vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported61 Subscribers
Add a CodeTriage badge to vllm
Help out
- Issues
- [Frontend] Return flat logprobs as arrays on the generate endpoint
- [Core][Perf] Skip gaps when searching sliding-window cache hits
- [Model] Add opt-in PrimTS attention for Step5 on Blackwell
- [DSv4.1][BI] Bitwise batch invariance for DeepSeek-V4.1-Flash with CUDA graphs
- [Bug]: Qwen4Exp (Qwen3.8-Flash-Next) — single prefill crossing the 262,144 native boundary crashes the engine with CUDA illegal memory access (v0.32.0 and nightly, OK on v0.31.1)
- [Perf][Parser] Avoid O(N^2) arg conversion in Granite streaming parser
- [Bugfix] [Rust Frontend] Fix HTTP metrics for streaming and cancelled requests
- [Bug][HiSparse] Shutdown stalls ~90s: every TP rank unpins the full shared host pool and EngineCore blocks on dying workers
- [ROCm][Perf] Fuse QK-norm + MRoPE + KV-cache write for gated Qwen3.5 attention
- [Bugfix][ROCm] Restore MoE weight padding dropped by #46009
- Docs
- Python not yet supported