vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported62 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Security] Fix ReDoS in error message path sanitizer
- [Bug]: TRITON_MLA is not batch invariant under chunked prefill with `VLLM_BATCH_INVARIANT=1` (DeepSeek-V2-Lite, DeepSeek-V3.1)
- [Bugfix][ROCm] Skip AITER Gluon MLA when Triton cannot compile it
- [ROCm][DSv4.1][Perf] Pass output_size to repeat_interleave in the Torch combine_topk_swa_indices
- [Docs][KV Connector] Explain effective HMA mismatches for NIXL peers
- [Metrics] Export scheduler-state gauges before the first request with --api-server-count > 1
- [Bugfix][Model] Fix LongCat MLA scale folding on partial loads
- [CPU] Collapse MRV2 staged writes and buffer pool on CPU
- [Bugfix][Pooling] Make MEAN pooling batch invariant when the flag asks for it
- [LoRA] Support saved multi-layer classification heads
- Docs
- Python not yet supported