EleutherAI/lm-evaluation-harness

Add long context evaluation benchmarks such as LongBench and LEval. 24 days ago

help wanted feature request

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

[New Task] COLLIE about 1 month ago

help wanted feature request good first issue

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

New Task Request: LegalBench about 2 months ago

help wanted feature request good first issue

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

Add ZeroScrolls Benchmark about 2 months ago

help wanted feature request good first issue

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

Add Regression Testing about 2 months ago

help wanted feature request good first issue

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

New Task Request: InflectionAI's Physics GRE about 2 months ago

help wanted feature request good first issue

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

Hendrycks Math extraction rule seems too strict 3 months ago

good first issue validation

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

Implement TyDiQA 4 months ago

help wanted feature request good first issue

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

Better Arg Typechecking / Validation 5 months ago

help wanted feature request good first issue

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

reproduce llama 3 evals 6 months ago

good first issue validation

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

Wrong format of the few-shot examples in mgsm_direct tasks 6 months ago

good first issue validation

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

Add docstring for HFLM's many keyword args 8 months ago

documentation help wanted feature request good first issue

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

Plans to add new long-context benchmark like LongBench v2, Babilong, InfiniteBench datasets? 8 months ago

help wanted good first issue

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

Ruler QA tasks do not work for `max_seq_lengths` < 4096 9 months ago

bug good first issue

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

Regarding metric chrf's implementation 10 months ago

feature request good first issue

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

Evaluate Thinking models on MMLU tasks 12 months ago

help wanted validation

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

Open Issues Need Help