vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported62 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Docs][KV Connector] Explain effective HMA mismatches for NIXL peers
- [Metrics] Export scheduler-state gauges before the first request with --api-server-count > 1
- [Bugfix][Model] Fix LongCat MLA scale folding on partial loads
- [CPU] Collapse MRV2 staged writes and buffer pool on CPU
- [Bugfix][Pooling] Make MEAN pooling batch invariant when the flag asks for it
- [LoRA] Support saved multi-layer classification heads
- [Bug]: PEFT sequence-classification LoRA adapters fail to load on RoBERTa / XLM-RoBERTa
- [KV Offload] Add admission policy framework and backpressure adapter
- [Performance]: Hybrid Mamba prefix caching (`mamba_cache_mode="align"`) costs 11-16% throughput on Nemotron-3.5-Lightning with no cache hits: per-group eager block-table ops, per-step align kernels outside CUDA graphs, and block-boundary prefill splits
- [Bugfix][Benchmark] Preserve synthetic datasets in throughput validation
- Docs
- Python not yet supported