Open Issues Need Help
View All on GitHubThe robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
AI Summary: The user is encountering an error when running `euroeval` with a GPT-2 model, stating that the `generative` extra is missing, despite explicitly including `euroeval[generative]` in their `uv` script's dependencies. This suggests an issue where the `generative` extra is not being correctly recognized or installed, potentially due to a packaging problem within `euroeval` or a flaw in its internal dependency check.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
AI Summary: This GitHub issue is a request to add the 'Exam-et' dataset, an Estonian multiple-choice exam dataset available on Hugging Face, to be used as an Estonian knowledge benchmark. The request provides a direct link and a brief description of the dataset's content and purpose.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
AI Summary: This GitHub issue requests the addition of the `copa-lv` dataset, a Latvian translation and post-edited version of the English COPA common-sense reasoning dataset. The request specifies using machine-translated train and validation splits, and post-edited test splits, all available at the provided GitHub link.
The robust European language model benchmark.
The robust European language model benchmark.
AI Summary: This GitHub issue reports several grammatical flaws and awkward direct translations in Icelandic prompts used across various datasets, including 'Hotter and Colder', 'MÍM-GOLD-NER', and 'ScaLA-is'. The author provides specific suggestions for improving word choice, verb conjugation, and overall naturalness of the prompts to make them more idiomatic Icelandic.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
The robust European language model benchmark.
AI Summary: The task is to improve the EuroEval benchmark by replacing the current manual evaluation method for encoder models on multiple-choice tasks with the Hugging Face `AutoModelForMultipleChoice` class. This involves modifying the existing code to utilize this class and comparing the results to the previous method to assess the impact of the change.
The robust European language model benchmark.
AI Summary: Implement a new separator (#) for specifying generation arguments in the EuroEval benchmark, alongside the existing @ separator for LiteLLM models. The new separator should work for all model types, while maintaining backward compatibility with the @ separator (with a deprecation warning). The implementation must handle any order of @ and # separators.
The robust European language model benchmark.