bhargavchintam

bhargavchintam/llm-inference

LLM inference stack in NumPy - attention, KV cache, paged attention, continuous batching, speculative decoding, quantization, sampling, and a toy engine.

0 stars 1 forks 0 watchers Python MIT License
continuous-batching flash-attention kv-cache llm llm-inference ml-systems numpy paged-attention quantization speculative-decoding transformers
View on GitHub
3 Open Issues Need Help Last updated: Oct 27, 2026

Open Issues Need Help

View All on GitHub
Add multi-head latent attention (MLA) with a compressed KV cache 7 days ago
help wanted
bhargavchintam/llm-inference
0

LLM inference stack in NumPy - attention, KV cache, paged attention, continuous batching, speculative decoding, quantization, sampling, and a toy engine.

Python
#continuous-batching#flash-attention#kv-cache#llm#llm-inference#ml-systems#numpy#paged-attention#quantization#speculative-decoding#transformers
Add a chunked prefill check 7 days ago
help wanted
bhargavchintam/llm-inference
0

LLM inference stack in NumPy - attention, KV cache, paged attention, continuous batching, speculative decoding, quantization, sampling, and a toy engine.

Python
#continuous-batching#flash-attention#kv-cache#llm#llm-inference#ml-systems#numpy#paged-attention#quantization#speculative-decoding#transformers
Add a sliding-window attention check 8 days ago
good first issue
bhargavchintam/llm-inference
0

LLM inference stack in NumPy - attention, KV cache, paged attention, continuous batching, speculative decoding, quantization, sampling, and a toy engine.

Python
#continuous-batching#flash-attention#kv-cache#llm#llm-inference#ml-systems#numpy#paged-attention#quantization#speculative-decoding#transformers