novitalabs/pegaflow

feat(vllm): Add prefetch_threshold Parameter 20 days ago

good first issue

177

High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.

Rust

#inference#kv-cache#llm#vllm

Suppress harmless cudaErrorCudartUnloading warning during shutdown 4 months ago

good first issue

novitalabs/pegaflow

177

High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.

Rust

#inference#kv-cache#llm#vllm

Add custom io_uring benchmark (harness=false) and align with fio 4 months ago

good first issue

novitalabs/pegaflow

177

High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.

Rust

#inference#kv-cache#llm#vllm

feat(transfer): show PCIe link speed and width in startup topology log 4 months ago

enhancement good first issue

novitalabs/pegaflow

177

High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.

Rust

#inference#kv-cache#llm#vllm

fix(python): .pyi stubs missing req_id parameter for query/query_prefetch 4 months ago

good first issue

novitalabs/pegaflow

177

High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.

Rust

#inference#kv-cache#llm#vllm

fix(python): EngineRpcClient.call() loses RPC method name in error messages 4 months ago

good first issue

novitalabs/pegaflow

177

High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.

Rust

#inference#kv-cache#llm#vllm

build: pegaflow-metaserver-py binary missing from wheel 4 months ago

good first issue

novitalabs/pegaflow

177

High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.

Rust

#inference#kv-cache#llm#vllm

chore(python): clean up dead code in __init__.py 4 months ago

enhancement good first issue

novitalabs/pegaflow

177

High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.

Rust

#inference#kv-cache#llm#vllm

docs(python): remove outdated API examples from README 4 months ago

documentation good first issue

novitalabs/pegaflow

177

High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.

Rust

#inference#kv-cache#llm#vllm

KV-cache registration is O(layers) and gets expensive for large models 5 months ago

good first issue

novitalabs/pegaflow

177

High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.

Rust

#inference#kv-cache#llm#vllm

Fix inflight_bytes metric overcount on duplicate slot inserts 5 months ago

good first issue

novitalabs/pegaflow

177

High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.

Rust

#inference#kv-cache#llm#vllm

Restore gRPC client message size limits in Python bindings 5 months ago

good first issue

novitalabs/pegaflow

177

High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.

Rust

#inference#kv-cache#llm#vllm

Move insert worker from tokio::spawn to std::thread 5 months ago

good first issue

novitalabs/pegaflow

177

High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.

Rust

#inference#kv-cache#llm#vllm

Add Regular (cached) pinned memory allocation strategy 6 months ago

enhancement good first issue

novitalabs/pegaflow

177

High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.

Rust

#inference#kv-cache#llm#vllm

[Concurrency] Verify lock acquisition order for layer_name_to_id and layer_names to prevent deadlocks 6 months ago

good first issue

novitalabs/pegaflow

177

High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.

Rust

#inference#kv-cache#llm#vllm

parallel init workers on different device 6 months ago

AI Summary: The issue reports that workers are initializing sequentially rather than in parallel, despite the expectation for parallel initialization across different devices. The provided logs show significant time delays (several seconds) between the `PeagflowRadixCache` registration events for various workers (TP0-TP7), indicating a bottleneck or serialization in the initialization process.

Complexity: 3/5

good first issue

novitalabs/pegaflow

177

High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.

Rust

#inference#kv-cache#llm#vllm

End-to-End Accuracy Test for Pega KV Cache 7 months ago

good first issue

novitalabs/pegaflow

177

High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.

Rust

#inference#kv-cache#llm#vllm

[core] Support Prometheus-style pull metrics endpoint 7 months ago

good first issue

novitalabs/pegaflow

177

High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.

Rust

#inference#kv-cache#llm#vllm

chore(logs): Add block count to save logs 7 months ago

AI Summary: This chore issue requests adding the block count to the save logs, similar to how load logs already include both layer and block counts. The goal is to make the log outputs consistent, providing more complete information during save operations.

Complexity: 1/5

good first issue

novitalabs/pegaflow

177

High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.

Rust

#inference#kv-cache#llm#vllm

Open Issues Need Help