vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported59 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Bugfix] Llama4 pythonic parser: strip <|python_end|> even without <|python_start|>
- [ROCm][KV Offload] SimpleCPUOffload: batch copies on SDMA with hipMemcpyFlagExtPreferCE (HIP 7.16+)
- [Bugfix][KV Offload] SimpleCPUOffload: requeue a finished request's CPU prefix, free CPU blocks tail first
- [Bugfix] Keep KV-cache scale defaults under dummy loading
- [Bugfix][MoRIIO] Route decode requests by their own DP rank
- [Bug]: Qwen3.5-9B forced named tool choice is ~11x slower on A100 with v0.27.1 than v0.31.0
- [Bug]: Kimi K3 latent-MoE tail fusion fails at startup when NVLS multicast is unavailable (multi-GPU Confidential Computing), and cannot be disabled
- [Perf][Kernel] Fuse bias add + GELU + NVFP4 quant for BERT-style FFNs
- [Bugfix] Retain sliding window block for recomputed token on full remote KV load
- [Bugfix] Gracefully fallback to non-fused latent-MoE tail when NVLS multicast is unavailable
- Docs
- Python not yet supported