vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported60 Subscribers
Add a CodeTriage badge to vllm
Help out
- Issues
- [ROCm][Perf] Muse Spark: MXFP4 MoE on native routes and column-sharded at TP prefill, via AITER's two-stage API
- [Perf][Multimodal] Add `mm_tensor_ipc="cuda_ipc_pool"`: TP-aware GPU transport
- [CPU] Warm up the remaining sampling kernels before serving
- [Performance]: Triton and other eager attention backends re-derive K/V cache views on every call (same class as #59534)
- [Bugfix] Rebuild npcache when source checkpoint shards are updated
- [Feature][Frontend] Accept forwarded multimodal features with reused prompt token ids
- [Rust Frontend] Let the server run over any `Llm`
- [ROCm][Bugfix] Fix AITER shared-expert fusion routing for gate-less MoE (DeepSeek-V2-Lite)
- [Quantization][MoE] Keep CSF-compressed NVFP4 expert scales resident on B12X
- [Model] Quantize Muse Spark BF16 routed experts with online nvfp4_per_token
- Docs
- Python not yet supported