vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported49 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Bugfix][Frontend] Respect tokenizer padding side in renderer
- [Bugfix][Quantization] Index INT8 MoE per-channel weight scales independently of act quant
- [Bug]: --max-num-queued-reqs is checked before the render but only counted after it, so a burst renders requests it then refuses (503 after 4–9 s of multimodal preprocessing)
- [Perf][Kernel] Vectorized flat abs-max for dynamic per-tensor FP8 quantization
- [Performance]: 【求助】8卡H200跑DeepSeek-V4.1-Flash,并发才8就慢成狗,首字等好几分钟,有没有大佬遇到过?
- [Bug/Perf]: the dense default chosen for Mamba + EAGLE reverts to 0% prefix reuse under interleaved long conversations
- [Perf][MRV2] Allow FULL decode graphs for one-token prompt tails
- [Bug]: Inconsistent merge/resolution semantics for flat and scoped `mm_processor_kwargs`
- [Bugfix][Reasoning] Handle multi-token streaming deltas in the Hunyuan A13B parser
- [Bugfix] Stop allocator fragmentation from shrinking the KV cache during memory profiling
- Docs
- Python not yet supported