vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported59 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Feature][Model Loader] Expose fastsafetensors transfer-mode selection
- [Doc][KV Offload] Document KV cache layout support for OffloadingConnector
- [Quantization][MoE] Keep MXFP4 expert scales compressed on B12X
- [Bugfix][Kernel] Add __syncwarp before the lane-0 stores in the fused KDA decode block reductions
- [Bug]: Hermes streaming tool parser leaks keys that follow "arguments" into the streamed arguments string; non-streaming is correct
- [ROCm][Perf][DSv4.1] Skip the redundant output copy in MoRI finalize
- [CI/Build] Rebuild lmcache against the image's torch
- [Bugfix][Frontend] Stop Hermes streaming tool parser leaking keys that follow "arguments"
- [ROCm][Perf][Attention] AITER gluon attention backend for Qwen3.8-27B
- [Performance]: extract_hidden_states loses ~38% throughput to a decode draft that is almost never accepted
- Docs
- Python not yet supported