vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported61 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Attention][MLA] Add NVFP4 KV cache for dense MLA with FlashInfer Cake decode
- [ROCm][Perf][DSv4] GFX1250 A8W8 Blockscale GEMM preshuffled + shared_experts fused_clamp_act_mul
- [Perf][Kernel] Reduce CUDA MEAN pooling overhead for short sequences
- [Core] Fix 100% idle CPU usage
- [ROCm] Enable the custom all-reduce on MI200 (gfx90a)
- [Bugfix][Frontend] Accept web_search_call values in Responses include
- [Question]: Qwen4Exp PLE n-gram table is materialised dense in bf16 — can it stay compressed in pinned host memory?
- [ROCm][Perf][DSv4] Use AITER tuned GEMM for the Router GEMM on gfx1250
- [Bug]: DeepSeek-V4.1-Flash(-1017) returns U+FFFD for a 12-token prompt, clean one token longer (v0.31.0/v0.32.0, TP8+EP, 8xH20)
- [Bug]: CUDA builds without FA extensions (Jetson Thor sm_110) cannot start: vllm_flash_attn raises at import time and FA is still auto-selected
- Docs
- Python not yet supported