vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported62 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Bugfix][Metrics] Skip latency stats for phases a request never reached
- [Core] Add fail-closed engine-level LLM.reload_weights with derived-cache invalidation
- [ROCm][Kimi-K3] Apply MLA RoPE with the cached AITER kernel
- [Model][Kernel] Add Granite Switch model
- [Security] Fix ReDoS in error message path sanitizer
- [Bug]: TRITON_MLA is not batch invariant under chunked prefill with `VLLM_BATCH_INVARIANT=1` (DeepSeek-V2-Lite, DeepSeek-V3.1)
- [Bugfix][ROCm] Skip AITER Gluon MLA when Triton cannot compile it
- [ROCm][DSv4.1][Perf] Pass output_size to repeat_interleave in the Torch combine_topk_swa_indices
- [Docs][KV Connector] Explain effective HMA mismatches for NIXL peers
- [Metrics] Export scheduler-state gauges before the first request with --api-server-count > 1
- Docs
- Python not yet supported