vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported62 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Bug]: gemma4 loader requires k_proj/v_proj/k_norm for KV-shared layers that transformers 5.5.4 never saves; any transformers-saved gemma4 checkpoint is refused
- [Bugfix][NIXL] Count pull completion notifications per transfer
- docs: add Gemma 4 Models tool-calling section
- [Kernel] Extend LiteTopK decode to BF16 V4.1 top512 and next_n 1-6
- [Bugfix][Frontend] Keep qwen3/MiMo unclosed last tool parameter only when the call is closed
- [Bug]: Priority scheduling can un-schedule a request it already scheduled in the same step (encoder cache miss, prefix hits on unwritten KV, priority inversion)
- [Bugfix] Fix ModernBERT attention with tensor parallelism
- Fix Xid 31 MMU fault in EP16 CUDA-graph replay during reduce_scatterv
- [Config] Support in-memory Hugging Face config dictionaries
- [Bugfix][Attention] Check draft model support for FlashInfer large pages
- Docs
- Python not yet supported