pegainfer-project/pegainfer

perf(kimi-k2): compact local top-1 readback into one pinned transfer 3 days ago

enhancement help wanted kimi-k2 stale hw:multi-gpu

631

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Rust

#cuda#cuda-kernels#deepseek#gpu#inference#inference-engine#kimi#kimi-k2#kv-cache#llm#llm-inference#llm-serving#model-serving#moe#openai-api#paged-attention#qwen#qwen3#rust#vllm

kimi-k2: add an integration-test surface (tests/) 3 days ago

enhancement help wanted kimi-k2 hw:none stale

pegainfer-project/pegainfer

631

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Rust

#cuda#cuda-kernels#deepseek#gpu#inference#inference-engine#kimi#kimi-k2#kv-cache#llm#llm-inference#llm-serving#model-serving#moe#openai-api#paged-attention#qwen#qwen3#rust#vllm

kimi-k2: gate DeepEP decode graph-replay numerics (full-bucket graph-vs-eager) 3 days ago

enhancement help wanted kimi-k2 stale hw:multi-gpu

pegainfer-project/pegainfer

631

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Rust

#cuda#cuda-kernels#deepseek#gpu#inference#inference-engine#kimi#kimi-k2#kv-cache#llm#llm-inference#llm-serving#model-serving#moe#openai-api#paged-attention#qwen#qwen3#rust#vllm

dsv2lite: reduce mixed and long-prompt HTTP TTFT with prefill scheduling 3 days ago

enhancement help wanted dsv2lite stale hw:multi-gpu

pegainfer-project/pegainfer

631

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Rust

#cuda#cuda-kernels#deepseek#gpu#inference#inference-engine#kimi#kimi-k2#kv-cache#llm#llm-inference#llm-serving#model-serving#moe#openai-api#paged-attention#qwen#qwen3#rust#vllm

Gate the P2P/RDMA offload data plane behind a cargo feature 3 days ago

enhancement help wanted hw:none stale

pegainfer-project/pegainfer

631

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Rust

#cuda#cuda-kernels#deepseek#gpu#inference#inference-engine#kimi#kimi-k2#kv-cache#llm#llm-inference#llm-serving#model-serving#moe#openai-api#paged-attention#qwen#qwen3#rust#vllm

glm52: DSpark and prefix caching are mutually exclusive — draft context cold-start at the cache-hit boundary 3 days ago

enhancement help wanted stale glm52 hw:multi-gpu

pegainfer-project/pegainfer

631

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Rust

#cuda#cuda-kernels#deepseek#gpu#inference#inference-engine#kimi#kimi-k2#kv-cache#llm#llm-inference#llm-serving#model-serving#moe#openai-api#paged-attention#qwen#qwen3#rust#vllm

perf(dsv2lite): move decode attention and KV-cache access to CUDA 3 days ago

enhancement help wanted dsv2lite stale hw:multi-gpu

pegainfer-project/pegainfer

631

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Rust

#cuda#cuda-kernels#deepseek#gpu#inference#inference-engine#kimi#kimi-k2#kv-cache#llm#llm-inference#llm-serving#model-serving#moe#openai-api#paged-attention#qwen#qwen3#rust#vllm

perf(dsv2lite): keep NCCL route selection and route plans device-resident 3 days ago

enhancement help wanted dsv2lite stale hw:multi-gpu

pegainfer-project/pegainfer

631

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Rust

#cuda#cuda-kernels#deepseek#gpu#inference#inference-engine#kimi#kimi-k2#kv-cache#llm#llm-inference#llm-serving#model-serving#moe#openai-api#paged-attention#qwen#qwen3#rust#vllm

[Tracking] Dev Dockerfile / devcontainer 4 days ago

help wanted roadmap hw:none

pegainfer-project/pegainfer

631

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Rust

#cuda#cuda-kernels#deepseek#gpu#inference#inference-engine#kimi#kimi-k2#kv-cache#llm#llm-inference#llm-serving#model-serving#moe#openai-api#paged-attention#qwen#qwen3#rust#vllm

metrics: report real prefix-cache query/hit counters from the qwen3 scheduler 4 days ago

enhancement help wanted qwen3 observability hw:1-gpu

pegainfer-project/pegainfer

631

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Rust

#cuda#cuda-kernels#deepseek#gpu#inference#inference-engine#kimi#kimi-k2#kv-cache#llm#llm-inference#llm-serving#model-serving#moe#openai-api#paged-attention#qwen#qwen3#rust#vllm

Make explicitly selected Qwen model gates fail closed 4 days ago

bug help wanted qwen3 qwen35 hw:1-gpu

pegainfer-project/pegainfer

631

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Rust

#cuda#cuda-kernels#deepseek#gpu#inference#inference-engine#kimi#kimi-k2#kv-cache#llm#llm-inference#llm-serving#model-serving#moe#openai-api#paged-attention#qwen#qwen3#rust#vllm

qwen3: scheduler can park on blocking_recv while an async decode-overlap prefill is still in flight 4 days ago

bug good first issue qwen3 hw:none

pegainfer-project/pegainfer

631

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Rust

#cuda#cuda-kernels#deepseek#gpu#inference#inference-engine#kimi#kimi-k2#kv-cache#llm#llm-inference#llm-serving#model-serving#moe#openai-api#paged-attention#qwen#qwen3#rust#vllm

[Help wanted] Verify Qwen3.5-9B / Qwen3.5-27B on the qwen35-4b crate 5 days ago

enhancement help wanted qwen35 hw:1-gpu stale

pegainfer-project/pegainfer

631

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Rust

#cuda#cuda-kernels#deepseek#gpu#inference#inference-engine#kimi#kimi-k2#kv-cache#llm#llm-inference#llm-serving#model-serving#moe#openai-api#paged-attention#qwen#qwen3#rust#vllm

[Help wanted] Verify Qwen3-14B / Qwen3-32B on the existing qwen3 crate 5 days ago

enhancement help wanted qwen3 hw:1-gpu stale

pegainfer-project/pegainfer

631

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Rust

#cuda#cuda-kernels#deepseek#gpu#inference#inference-engine#kimi#kimi-k2#kv-cache#llm#llm-inference#llm-serving#model-serving#moe#openai-api#paged-attention#qwen#qwen3#rust#vllm

perf(qwen): reuse PrefillPagedPlan for uncompiled-GQA decode 5 days ago

enhancement help wanted qwen3 qwen35 hw:1-gpu stale

pegainfer-project/pegainfer

631

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Rust

#cuda#cuda-kernels#deepseek#gpu#inference#inference-engine#kimi#kimi-k2#kv-cache#llm#llm-inference#llm-serving#model-serving#moe#openai-api#paged-attention#qwen#qwen3#rust#vllm

qwen3: docs/dead-code cleanup — stale records, deleted-tool references, unused consts 6 days ago

documentation enhancement good first issue qwen3 hw:none stale

pegainfer-project/pegainfer

631

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Rust

#cuda#cuda-kernels#deepseek#gpu#inference#inference-engine#kimi#kimi-k2#kv-cache#llm#llm-inference#llm-serving#model-serving#moe#openai-api#paged-attention#qwen#qwen3#rust#vllm

frontend: preserve logprobs request semantics and return prompt logprobs instead of HTTP 500 18 days ago

bug help wanted hw:1-gpu

pegainfer-project/pegainfer

631

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Rust

#cuda#cuda-kernels#deepseek#gpu#inference#inference-engine#kimi#kimi-k2#kv-cache#llm#llm-inference#llm-serving#model-serving#moe#openai-api#paged-attention#qwen#qwen3#rust#vllm

Reduce duplicated CPU operator oracles in the HD256/HD512 QK-RoPE smokes 18 days ago

enhancement help wanted hw:1-gpu stale

pegainfer-project/pegainfer

631

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Rust

#cuda#cuda-kernels#deepseek#gpu#inference#inference-engine#kimi#kimi-k2#kv-cache#llm#llm-inference#llm-serving#model-serving#moe#openai-api#paged-attention#qwen#qwen3#rust#vllm

Structured / guided output (grammar-constrained decoding) for reliable tool calls 18 days ago

enhancement help wanted roadmap hw:1-gpu

pegainfer-project/pegainfer

631

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Rust

#cuda#cuda-kernels#deepseek#gpu#inference#inference-engine#kimi#kimi-k2#kv-cache#llm#llm-inference#llm-serving#model-serving#moe#openai-api#paged-attention#qwen#qwen3#rust#vllm

Migrate Gemma 4 onto the step contract 21 days ago

enhancement help wanted hw:1-gpu

pegainfer-project/pegainfer

631

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Rust

#cuda#cuda-kernels#deepseek#gpu#inference#inference-engine#kimi#kimi-k2#kv-cache#llm#llm-inference#llm-serving#model-serving#moe#openai-api#paged-attention#qwen#qwen3#rust#vllm

qwen35: migrate GDN prefill to FlashInfer and retire Triton AOT where supported 28 days ago

enhancement help wanted qwen35 hw:1-gpu stale

pegainfer-project/pegainfer

631

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Rust

#cuda#cuda-kernels#deepseek#gpu#inference#inference-engine#kimi#kimi-k2#kv-cache#llm#llm-inference#llm-serving#model-serving#moe#openai-api#paged-attention#qwen#qwen3#rust#vllm

qwen3: PerToken decode-graph memory grows linearly with batch bucket and is not budgeted about 1 month ago

help wanted qwen3 hw:1-gpu stale

pegainfer-project/pegainfer

631

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Rust

#cuda#cuda-kernels#deepseek#gpu#inference#inference-engine#kimi#kimi-k2#kv-cache#llm#llm-inference#llm-serving#model-serving#moe#openai-api#paged-attention#qwen#qwen3#rust#vllm

Keep Gemma 4 ignored gates compilable and define their maintainer runner about 1 month ago

enhancement help wanted hw:1-gpu

pegainfer-project/pegainfer

631

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Rust

#cuda#cuda-kernels#deepseek#gpu#inference#inference-engine#kimi#kimi-k2#kv-cache#llm#llm-inference#llm-serving#model-serving#moe#openai-api#paged-attention#qwen#qwen3#rust#vllm

Open Issues Need Help