vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported62 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Docs] Update batch invariance tested models (Blackwell validation)
- [Bug]: --disable-custom-all-reduce says it falls back to NCCL, but FlashInfer and symm-mem all-reduce stay enabled
- [Bug]: With --api-server-count > 1, gauges such as vllm:num_requests_running have no samples until the first request
- [Feature][Rust Frontend] Return token offsets from render endpoints
- [Feature]: Per-request prefix-cache miss attribution (prompt divergence vs. eviction)
- [MRV2][Spec Decode] Confidence stop for autoregressive drafting
- [Bugfix][Metrics] Skip latency stats for phases a request never reached
- [Core] Add fail-closed engine-level LLM.reload_weights with derived-cache invalidation
- [ROCm][Kimi-K3] Apply MLA RoPE with the cached AITER kernel
- [Model][Kernel] Add Granite Switch model
- Docs
- Python not yet supported