vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported60 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Perf][Spec Decode] Vocab-parallel DSpark Markov sampling and vocab-sharded verification over TP
- [Bugfix][Spec Decode][XPU] greedy_argmax: treat NaN as -inf on the torch fallback
- [KV Connector] Remove the in-tree LMCache integration package
- [Bugfix][Attention] Reject TORCH_SDPA for decoder attention
- [Bugfix][Structured Output] Preserve raw scores before grammar masking
- [Bugfix][Kernel] Plan b12x NVFP4 W4A16 linear token counts at load time
- [DeepSeek-V4] Warm up DeepGEMM fp8_einsum for the o_proj
- [Bugfix][Prefix] Skip EAGLE block drop for KV-shared drafters + widen periodic SWA tail
- [Rust Frontend] Support beam search (use_beam_search, length_penalty)
- [Feature][Core] Reset prefix cache without preempting running requests
- Docs
- Python not yet supported