vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported62 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Bug]: support_torch_compile's positional type check is off by one: each argument is checked against the previous parameter's annotation
- [MRV2][Spec Decode] Context Aware Sparse LM Head on quantized lm_heads
- [Bugfix] Reduce threaded PT loader peak checkpoint-tensor memory by 52-59%
- [Bugfix] Fix startup plan over-allocating 2.83 GiB KV after multimodal changes
- [Misc] Log enabled fusion passes only under VLLM_COMPILE
- [Docs] Clarify Qwen3 implicit reasoning end for tool calls
- [Docs] Update batch invariance tested models (Blackwell validation)
- [Bug]: --disable-custom-all-reduce says it falls back to NCCL, but FlashInfer and symm-mem all-reduce stay enabled
- [Bug]: With --api-server-count > 1, gauges such as vllm:num_requests_running have no samples until the first request
- [Feature][Rust Frontend] Return token offsets from render endpoints
- Docs
- Python not yet supported