vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported62 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Bugfix] Reduce threaded PT loader peak checkpoint-tensor memory by 52-59%
- [Bugfix] Fix startup plan over-allocating 2.83 GiB KV after multimodal changes
- [Misc] Log enabled fusion passes only under VLLM_COMPILE
- [Docs] Clarify Qwen3 implicit reasoning end for tool calls
- [Docs] Update batch invariance tested models (Blackwell validation)
- [Bug]: --disable-custom-all-reduce says it falls back to NCCL, but FlashInfer and symm-mem all-reduce stay enabled
- [Bug]: With --api-server-count > 1, gauges such as vllm:num_requests_running have no samples until the first request
- [Feature][Rust Frontend] Return token offsets from render endpoints
- [Feature]: Per-request prefix-cache miss attribution (prompt divergence vs. eviction)
- [MRV2][Spec Decode] Confidence stop for autoregressive drafting
- Docs
- Python not yet supported