vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported60 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Spec Decode][MRV2] Add prefill_context_kv_only for P/D prefill instances
- [Attention] Sync FA with upstream (54b044c8)
- [Perf][KV Offload] Vectorize direct-layout descriptor fill across layers
- [ROCm][Disagg][CI] Give Wide-EP headless nodes the KV config
- [MORI][DBO] Dual batch overlap with MoRI EPv2 (native and AsyncLL), and DBO fixes
- [Bugfix][Frontend] Stop accepting gRPC connections when shutdown starts
- [Bugfix][Quantization] Index INT8 MoE per-channel weight scales independently of act quant - #58570
- [Bugfix][Frontend] Report the line of a run-batch request rejected by request validators
- [ROCm][Perf] Build AITER MLA full-graph decode metadata in one kernel
- [Bugfix] shm_broadcast: unblock writer in acquire_write when shutdown
- Docs
- Python not yet supported