vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported59 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [KV Connector][NIXL] Pull KV from pipeline-parallel prefill stages, with the hybrid KV cache manager
- [Bugfix][GDN] Size repeat_interleave from the device token total
- [Security] Use weights_only=True for EPLB all_gather_object collectives
- [Docs] Update SPEED-Bench preparation script documentation
- [Bugfix] Report reasoning_tokens in /inference/v1/generate derender usage
- [Bug]: GDN Triton/FLA prefill gives wrong, non-deterministic results on Hopper (H20) with triton 3.6: `chunk_scaled_dot_kkt_fwd` autotune configs with BK < K and num_warps >= 4 are broken
- [Bug]: TP workers wedge permanently in Triton kernel launch (GDN layer_norm_fwd) with idle GPUs; Qwen3.5 + MTP, v0.29.0
- [ROCm] A text-only model needs torchvision, and you find out 15 minutes into JIT compilation
- [Quantization] Add mxfp4_hadamard32_moe online quantization
- [Bug]: get_top_k_tokens applies scaling and soft capping in the wrong order
- Docs
- Python not yet supported