vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported62 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Doc] Fix placeholder count in prefix caching multi-modal example
- [Bugfix][Kernel] Mask the per-key gate load in fused recurrent KDA
- [Quantization] Allow online per-channel fp8 quantization of an unquantized lm_head
- ettin rereanker load additional modules
- [Bug]: Whisper long-audio transcription loses words at the internal 30 s cut points in json/text (split_audio cuts inside speech, no overlap or merge)
- [Feature]: Candidate-only scoring for DeepSeek-V4.1 indexer consumers on SM90 (H20)
- [Performance]: MoE-LoRA one-shot kernel: rank offsets defeat load vectorization (w2 at 14% DRAM, 92% excess sectors); ~2x with bit-identical output
- [Feature] Add optional GPU timing to iteration metrics
- [Feature] Add workload snapshots to iteration metrics
- [Perf][Model Loader] DeepSeek V4.1 Flash mixed RunAI loading
- Docs
- Python not yet supported