OpenAgentHQ/openagent-eval

Add unit tests for ReportManager.reconstruct() edge cases about 10 hours ago

enhancement good first issue

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add common evaluation workflow examples to docs/examples.md about 10 hours ago

enhancement good first issue in-progress

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add CLI exit codes documentation to docs/cli.md about 10 hours ago

documentation good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add a docs/config-reference.md page documenting the full YAML schema 4 days ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Write a plugin development guide with step-by-step example 6 days ago

documentation good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Create end-to-end tutorial with free APIs and local embeddings 14 days ago

documentation enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add Professional Thumbnail Banner to README for Social Media Sharing 14 days ago

documentation enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add static/openagent-eval.png showing interactive CLI wizard for README 14 days ago

documentation enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

[DOCS] [FEATURE] Complete Colab Notebook Tutorial + Video Walkthrough 20 days ago

documentation enhancement good first issue help wanted

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add unit tests for the udit CLI command 21 days ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

[GOOD FIRST ISSUE] Create a Python SDK so users can evaluate in 3 lines of code 23 days ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Fix: Corpus auditor loads .jsonl files as single documents instead of one-per-line 23 days ago

bug good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add unit tests for the diagnose CLI command 23 days ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Improve error message when required config fields are missing during validation 23 days ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add --version flag to all CLI commands 23 days ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Fix: MetricTimeoutError drops imeout_seconds=0.0 due to falsy check 23 days ago

bug good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add a simple progress indicator to oaeval audit command 23 days ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Fix: Non-ASCII characters silently ignored in content gap analysis 23 days ago

bug good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add CorpusConfig to the main Config model for unified configuration 23 days ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Fix: Comparison report footer shows wrong winner when experiment score is lower 23 days ago

bug good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add input validation for dataset path — show helpful error for missing files 23 days ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Remove unused imports across the codebase 23 days ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add validation for retriever settings field to catch typos early 23 days ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add type hints to public API functions in the reports module 23 days ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Fix: ReportConfig.max_examples is ignored — each report generator hardcodes its limit 23 days ago

bug good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Standardize error messages across all LLM providers for consistency 23 days ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Document all supported environment variables in a single reference page 23 days ago

documentation good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add API key setup guide for all LLM providers to docs 23 days ago

documentation good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add docstrings to all CLI command functions 23 days ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add integration test for full evaluation pipeline with mock provider 23 days ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add unit tests for the synth CLI command 23 days ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Fix: commands/__init__.py missing imports for 6 command modules 23 days ago

bug good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Fix: --llm-provider mock flag silently creates OpenAI provider instead of mock 23 days ago

bug good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Fix: ctx variable shadowed from CLIContext to click.Context in main.py 23 days ago

bug good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Fix: Progress bar resets to 1/total_items on completion in run.py 23 days ago

bug good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Fix: Missing f-string prefix causes {config.timeout} to be printed literally in run.py 23 days ago

bug good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

docs: Add reports output formats documentation page 24 days ago

documentation good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Open Issues Need Help