nisaral/DIO

Export affinity LRU eviction counters and document the hit rate 9 days ago

enhancement good first issue

3

DIO: non-invasive LLM gateway that learns per-backend latency online (dual-timescale NLMS) and routes vLLM/Ollama/SGLang/TGI with SLO-aware admission.

Python

#admission-control#ai-infrastructure#inference-server#llm#llm-gateway#llm-inference#llm-serving#llmops#load-balancing#mcp-server#mlops#model-context-protocol#model-serving#ollama#openai-api#openai-compatible#python#self-hosted#sglang#vllm

Make model names consistent across /v1/models, /api/tags and routing 9 days ago

bug good first issue

nisaral/DIO

3

DIO: non-invasive LLM gateway that learns per-backend latency online (dual-timescale NLMS) and routes vLLM/Ollama/SGLang/TGI with SLO-aware admission.

Python

#admission-control#ai-infrastructure#inference-server#llm#llm-gateway#llm-inference#llm-serving#llmops#load-balancing#mcp-server#mlops#model-context-protocol#model-serving#ollama#openai-api#openai-compatible#python#self-hosted#sglang#vllm

Document a real production deployment end to end 9 days ago

documentation good first issue

nisaral/DIO

3

DIO: non-invasive LLM gateway that learns per-backend latency online (dual-timescale NLMS) and routes vLLM/Ollama/SGLang/TGI with SLO-aware admission.

Python

#admission-control#ai-infrastructure#inference-server#llm#llm-gateway#llm-inference#llm-serving#llmops#load-balancing#mcp-server#mlops#model-context-protocol#model-serving#ollama#openai-api#openai-compatible#python#self-hosted#sglang#vllm

Add mypy checking for scheduler.py and backends.py 9 days ago

enhancement good first issue

nisaral/DIO

3

DIO: non-invasive LLM gateway that learns per-backend latency online (dual-timescale NLMS) and routes vLLM/Ollama/SGLang/TGI with SLO-aware admission.

Python

#admission-control#ai-infrastructure#inference-server#llm#llm-gateway#llm-inference#llm-serving#llmops#load-balancing#mcp-server#mlops#model-context-protocol#model-serving#ollama#openai-api#openai-compatible#python#self-hosted#sglang#vllm

Publish dio-serve to PyPI with trusted publishing 9 days ago

enhancement good first issue

nisaral/DIO

3

DIO: non-invasive LLM gateway that learns per-backend latency online (dual-timescale NLMS) and routes vLLM/Ollama/SGLang/TGI with SLO-aware admission.

Python

#admission-control#ai-infrastructure#inference-server#llm#llm-gateway#llm-inference#llm-serving#llmops#load-balancing#mcp-server#mlops#model-context-protocol#model-serving#ollama#openai-api#openai-compatible#python#self-hosted#sglang#vllm

Expose Prometheus/OTel metrics on a real /metrics endpoint 9 days ago

enhancement good first issue

nisaral/DIO

3

DIO: non-invasive LLM gateway that learns per-backend latency online (dual-timescale NLMS) and routes vLLM/Ollama/SGLang/TGI with SLO-aware admission.

Python

#admission-control#ai-infrastructure#inference-server#llm#llm-gateway#llm-inference#llm-serving#llmops#load-balancing#mcp-server#mlops#model-context-protocol#model-serving#ollama#openai-api#openai-compatible#python#self-hosted#sglang#vllm

Add an Anthropic or Gemini api_style adapter 9 days ago

enhancement good first issue

nisaral/DIO

3

DIO: non-invasive LLM gateway that learns per-backend latency online (dual-timescale NLMS) and routes vLLM/Ollama/SGLang/TGI with SLO-aware admission.

Python

#admission-control#ai-infrastructure#inference-server#llm#llm-gateway#llm-inference#llm-serving#llmops#load-balancing#mcp-server#mlops#model-context-protocol#model-serving#ollama#openai-api#openai-compatible#python#self-hosted#sglang#vllm

Open Issues Need Help