vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported59 Subscribers
Add a CodeTriage badge to vllm
Help out
- Issues
- [RFC]: SM90 DeepGEMM MegaMoE for DeepSeek V4
- [Bugfix][Tool Parser] Stream all tool calls completed in one delta (ernie45, jamba, granite-20b-fc)
- [Bugfix][KV Offload] Preserve block extra keys across async request cleanup
- [Bugfix][MoE] Pad block-FP8 weight scales for the HPC MoE backend
- [Perf][MoE] Let DeepGEMM MoE fall back to Triton for small M on Hopper
- [Bug]: DeepSeek-V4.1-Flash: some DP ranks of a multi-node DP/EP decode engine produce degenerate output (rank-sticky, reproduces without P/D transfer)
- [Perf][MoE] Triton FP8 MoE: per-layer activation scale format and always-fused SiLU+quant
- [Core] Stop re-queuing waiting requests that max_loras blocks
- [Benchmark] Compare MXFP4 MoE backends at GLM-5.3 shapes
- [RFC]: Token-id parity with the Python frontend as a tested contract for the Rust frontend
- Docs
- Python not yet supported