EleutherAI/lm-evaluation-harness

Add better test coverage for models 3 days ago

help wanted feature request good first issue

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

Implement XQuAD 28 days ago

help wanted feature request good first issue

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

Quac Dataset about 1 month ago

feature request good first issue

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

Add Regression Testing about 1 month ago

help wanted feature request good first issue

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

Add New Lambada Translations about 2 months ago

good first issue

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

Add TheoremQA about 2 months ago

help wanted feature request good first issue

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

CoQA's implementation only predicts the last answer of each text 2 months ago

bug good first issue

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

Add ZeroScrolls Benchmark 2 months ago

help wanted feature request good first issue

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

Implement TyDiQA 2 months ago

help wanted feature request good first issue

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

Add long context evaluation benchmarks such as LongBench and LEval. 4 months ago

help wanted feature request

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

[New Task] COLLIE 4 months ago

help wanted feature request good first issue

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

New Task Request: LegalBench 5 months ago

help wanted feature request good first issue

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

New Task Request: InflectionAI's Physics GRE 5 months ago

help wanted feature request good first issue

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

Hendrycks Math extraction rule seems too strict 6 months ago

good first issue validation

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

Better Arg Typechecking / Validation 8 months ago

help wanted feature request good first issue

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

reproduce llama 3 evals 9 months ago

good first issue validation

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

Wrong format of the few-shot examples in mgsm_direct tasks 9 months ago

good first issue validation

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

Add docstring for HFLM's many keyword args 11 months ago

documentation help wanted feature request good first issue

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

Plans to add new long-context benchmark like LongBench v2, Babilong, InfiniteBench datasets? 11 months ago

help wanted good first issue

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

Ruler QA tasks do not work for `max_seq_lengths` < 4096 about 1 year ago

bug good first issue

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

Regarding metric chrf's implementation about 1 year ago

feature request good first issue

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

Evaluate Thinking models on MMLU tasks about 1 year ago

help wanted validation

EleutherAI/lm-evaluation-harness

13.3K

A framework for few-shot evaluation of language models.

Python

#evaluation-framework#language-model#transformer

Open Issues Need Help