aicostbudget

aicostbudget/llm-cost-per-task-benchmark

What one correct LLM answer really costs: 16 tasks, code-only validators, bring-your-own-key runner (OpenAI, Anthropic, Gemini, DeepSeek). First result: same accuracy, 9x the cost per success.

1 stars 0 forks 1 watchers Python MIT License
ai-cost anthropic benchmark cost-optimization deepseek finops gemini llm llm-benchmark llm-cost llm-evaluation llmops openai prompt-caching python reasoning-models text-to-sql tool-calling
View on GitHub Website
2 Open Issues Need Help Last updated: Oct 29, 2026

Open Issues Need Help

View All on GitHub
Add a model to models.json (e.g. GPT and Claude small models) 6 days ago
good first issue help wanted
aicostbudget/llm-cost-per-task-benchmark
1

What one correct LLM answer really costs: 16 tasks, code-only validators, bring-your-own-key runner (OpenAI, Anthropic, Gemini, DeepSeek). First result: same accuracy, 9x the cost per success.

Python
#ai-cost#anthropic#benchmark#cost-optimization#deepseek#finops#gemini#llm#llm-benchmark#llm-cost#llm-evaluation#llmops#openai#prompt-caching#python#reasoning-models#text-to-sql#tool-calling
Results wanted: run it on GPT, Claude and Gemini models 6 days ago
help wanted
aicostbudget/llm-cost-per-task-benchmark
1

What one correct LLM answer really costs: 16 tasks, code-only validators, bring-your-own-key runner (OpenAI, Anthropic, Gemini, DeepSeek). First result: same accuracy, 9x the cost per success.

Python
#ai-cost#anthropic#benchmark#cost-optimization#deepseek#finops#gemini#llm#llm-benchmark#llm-cost#llm-evaluation#llmops#openai#prompt-caching#python#reasoning-models#text-to-sql#tool-calling