dundysm/inference-doctor - 8 Help Wanted Issues

Open Issues Need Help

Format: read TensorRT-LLM trtllm-bench results about 23 hours ago

help wanted format

Performance regression gate for LLM inference: PASS, FAIL or INCONCLUSIVE from repeated benchmark runs, decided by a confidence interval. Reads vLLM, AIPerf, GuideLLM and SGLang results.

Python

#benchmark#continuous-integration#github-actions#llm#llm-inference#mlops#performance-regression#performance-testing#sglang#vllm

diagnose: re-verify QUEUE_PRESSURE on a real vLLM server 24 days ago

help wanted needs GPU

dundysm/inference-doctor

Performance regression gate for LLM inference: PASS, FAIL or INCONCLUSIVE from repeated benchmark runs, decided by a confidence interval. Reads vLLM, AIPerf, GuideLLM and SGLang results.

Python

#benchmark#continuous-integration#github-actions#llm#llm-inference#mlops#performance-regression#performance-testing#sglang#vllm

Statistics: a paired interval when runs are interleaved 24 days ago

enhancement help wanted

dundysm/inference-doctor

Performance regression gate for LLM inference: PASS, FAIL or INCONCLUSIVE from repeated benchmark runs, decided by a confidence interval. Reads vLLM, AIPerf, GuideLLM and SGLang results.

Python

#benchmark#continuous-integration#github-actions#llm#llm-inference#mlops#performance-regression#performance-testing#sglang#vllm

Example: an end-to-end workflow on a free CPU runner 24 days ago

documentation help wanted

dundysm/inference-doctor

Performance regression gate for LLM inference: PASS, FAIL or INCONCLUSIVE from repeated benchmark runs, decided by a confidence interval. Reads vLLM, AIPerf, GuideLLM and SGLang results.

Python

#benchmark#continuous-integration#github-actions#llm#llm-inference#mlops#performance-regression#performance-testing#sglang#vllm

Docs: walkthrough of gating a vLLM upgrade with vllm bench serve 24 days ago

documentation good first issue

dundysm/inference-doctor

Performance regression gate for LLM inference: PASS, FAIL or INCONCLUSIVE from repeated benchmark runs, decided by a confidence interval. Reads vLLM, AIPerf, GuideLLM and SGLang results.

Python

#benchmark#continuous-integration#github-actions#llm#llm-inference#mlops#performance-regression#performance-testing#sglang#vllm

Format: read llmperf summary files 24 days ago

good first issue format

dundysm/inference-doctor

Performance regression gate for LLM inference: PASS, FAIL or INCONCLUSIVE from repeated benchmark runs, decided by a confidence interval. Reads vLLM, AIPerf, GuideLLM and SGLang results.

Python

#benchmark#continuous-integration#github-actions#llm#llm-inference#mlops#performance-regression#performance-testing#sglang#vllm

Action: post the result as a pull request comment 24 days ago

enhancement help wanted

dundysm/inference-doctor

Performance regression gate for LLM inference: PASS, FAIL or INCONCLUSIVE from repeated benchmark runs, decided by a confidence interval. Reads vLLM, AIPerf, GuideLLM and SGLang results.

Python

#benchmark#continuous-integration#github-actions#llm#llm-inference#mlops#performance-regression#performance-testing#sglang#vllm

Report: --markdown output for pasting into a pull request 24 days ago

enhancement good first issue

dundysm/inference-doctor

Performance regression gate for LLM inference: PASS, FAIL or INCONCLUSIVE from repeated benchmark runs, decided by a confidence interval. Reads vLLM, AIPerf, GuideLLM and SGLang results.

Python

#benchmark#continuous-integration#github-actions#llm#llm-inference#mlops#performance-regression#performance-testing#sglang#vllm