k-dickinson

k-dickinson/evaltrust

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

1 stars 1 forks 1 watchers Python MIT License
View on GitHub
42 Open Issues Need Help Last updated: Jul 8, 2026

Open Issues Need Help

View All on GitHub
Adapter: HuggingFace lighteval 3 days ago
help wanted adapter priority: medium
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Adapter: SWE-bench results (convenience) 4 days ago
help wanted adapter priority: low
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Adapter: Weights & Biases / Weave eval export 4 days ago
help wanted adapter priority: medium
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Report a headline P(model A > model B) 9 days ago
enhancement help wanted statistics priority: medium
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Run-aware comparison: run-aware resampling estimator 13 days ago
enhancement help wanted statistics priority: high
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Adapter: Langfuse scores export 15 days ago
help wanted adapter priority: medium
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Adapter: OpenCompass 19 days ago
help wanted adapter priority: low
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Adapter: HELM per-instance results 19 days ago
help wanted adapter priority: medium
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Adapter: Arize Phoenix evals 19 days ago
help wanted adapter priority: medium
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Adapter: Braintrust experiment export 19 days ago
help wanted adapter priority: medium
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Adapter: MLflow evaluate 19 days ago
help wanted adapter priority: medium
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Support run-level (aggregate) score input + unpaired two-sample comparison 19 days ago
enhancement help wanted statistics priority: medium
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Run-aware comparison: expose repeated runs to the comparison path 19 days ago
enhancement help wanted priority: medium
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
All-pairs comparison for files with >2 models 20 days ago
enhancement help wanted priority: high
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Add a Makefile with common dev commands 24 days ago
documentation help wanted good first issue priority: low
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
List available model names when a --model-a/-b isn't found 24 days ago
enhancement help wanted good first issue priority: low
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Add __str__ to Status and VerdictLevel enums 24 days ago
enhancement help wanted good first issue priority: low
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Add unit tests for magnitude_label thresholds 24 days ago
help wanted good first issue priority: low
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Add a CITATION.cff so the project is citable 24 days ago
documentation help wanted good first issue priority: low
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Add CSV, JSONL, and multi-metric example files 24 days ago
documentation help wanted good first issue priority: low
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Add more column-name aliases for the generic adapter 24 days ago
enhancement help wanted good first issue priority: low
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Add more pass/fail score-word aliases 24 days ago
enhancement help wanted good first issue priority: low
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Parse percent-style scores like "85%" in coerce_score 24 days ago
enhancement help wanted good first issue priority: low
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Expose evaltrust.__version__ in Python 24 days ago
enhancement help wanted good first issue priority: low
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Adapter: EleutherAI lm-evaluation-harness 25 days ago
help wanted good first issue adapter priority: low
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Run judge reliability + calibration in single-model mode 25 days ago
enhancement help wanted priority: low
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Confidence interval on the effect size 25 days ago
enhancement help wanted statistics priority: medium
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Adapter: Ragas (RAG evaluation) 25 days ago
help wanted adapter priority: medium
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Support pairwise-preference judgments (win / lose / tie) 27 days ago
enhancement help wanted
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Adapter: Inspect (UK AISI) .eval logs 28 days ago
help wanted adapter
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Adapter: generic JSONL of per-example records 28 days ago
help wanted adapter
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Continuous-score judge calibration (correlation, not exact match) 28 days ago
enhancement help wanted statistics
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Single-model repeatability check 28 days ago
enhancement help wanted
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Holm-Bonferroni correction for multi-metric suites 28 days ago
enhancement help wanted statistics
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
BCa (bias-corrected accelerated) bootstrap intervals 28 days ago
enhancement help wanted statistics
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Weighted / gated metrics in a suite 28 days ago
enhancement help wanted
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Add a Markdown report output (`--md`) 28 days ago
enhancement good first issue
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Add an `evaltrust --version` flag 28 days ago
enhancement good first issue
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Add an OpenEvals format adapter 29 days ago
help wanted good first issue
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Add an optional HTML report (--html) 29 days ago
help wanted
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Add a LangSmith format adapter 29 days ago
help wanted good first issue
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python
Add a score-word alias (good first issue) 29 days ago
good first issue
k-dickinson/evaltrust
1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python