vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported59 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [RFC]: Zero-copy DeepEP low-latency combine, safe under DBO
- [MORI][DBO] Dual batch overlap with MoRI EPv2 (native and AsyncLL), and DBO fixes
- [Bug]: Sync KV load-failure recovery triggers num_output_placeholders assertion on v0.29.0 and v0.32.0
- [ROCm][Perf] Add fused MoE config for Gemma 4 26B-A4B on MI350P
- [Bug]: MessageQueue.shutdown() does not unblock writers blocked in acquire_write
- [Feature][NIXL] Add prefix cache reset to the toy proxy
- [Bug]: b12x attention — CUDA illegal memory access at the first MTP verify step (cold-JIT `PagedForwardKernel`) on v0.33.0; regression from v0.32.0 (SM120, NVFP4 MoE, fp8 KV, chunked prefill, prefix caching, MTP spec decode)
- [Bug]: Second `vllm serve --grpc` on the same port starts fine and steals traffic
- [Bugfix][Frontend] Don't let the gRPC server share a port that's already in use
- [Metrics][Quantization] Let quantization configs report weight bytes for MFU
- Docs
- Python not yet supported