homayoun-safarpour

homayoun-safarpour/judge-reliability-kit

Diagnose whether a low-kappa LLM judge panel fails from item ambiguity or rubric underspecification.

0 stars 0 forks 0 watchers Python MIT License
ai-evaluation bias-detection cohen-kappa evaluation-metrics fleiss-kappa inter-rater-reliability llm-as-a-judge llm-evaluation python reliability
View on GitHub Website
1 Open Issue Need Help Last updated: Aug 5, 2026

Open Issues Need Help

View All on GitHub
Good first issue: add one ambiguous-item fixture for the kappa worked example about 13 hours ago
good first issue
homayoun-safarpour/judge-reliability-kit
0

Diagnose whether a low-kappa LLM judge panel fails from item ambiguity or rubric underspecification.

Python
#ai-evaluation#bias-detection#cohen-kappa#evaluation-metrics#fleiss-kappa#inter-rater-reliability#llm-as-a-judge#llm-evaluation#python#reliability