vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported59 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Bugfix][Elastic EP] Keep FlashInfer autotuning working on stateless groups
- [ROCm][MiniMax-M3] LM-path speedups: fused norm + FP8 quant, dense short MSA prefill, JIT warm-ups
- High-concurrency MTP on GLM-5.3 (H20 x8, batch 256): static K loses 10x; per-batch-size K table recovers it
- [Bugfix][CPU]-Support-grouped-INT8-WNA16-MoE
- [Bugfix][MRV2] Reserve MoE workspace before graph capture
- [Bugfix][CPU] Use supports_fused_gdn_convs() for every GDN conv path (fixes macOS)
- [Bug]: Requests queued during level-2 sleep run before `reload_weights` and return wrong output with HTTP 200
- [Bug]: shared streaming helper synthesizes delta_token_ids that serving never sends, changing engine behavior
- [model] Support FireRedFireRedASR2-AED
- [Kernel] Calibrate the NVFP4 W4A16 Marlin/CuTe-DSL choice per layer shape on sm_12x
- Docs
- Python not yet supported