vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported61 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Loading] Fix misleading logs when multiple models are loaded
- [ROCm][Bugfix] Skip AITER unquantized MoE for hidden sizes not divisible
- [ROCm] Update ROCm to 10.1 and MORI to v1.2.4 and amd-smi fixes + AITER v0.1.28.post1
- [Bugfix][Frontend] Refresh echoed text after prompt truncation
- [Frontend] Return flat logprobs as arrays on the generate endpoint
- [Core][Perf] Skip gaps when searching sliding-window cache hits
- [Model] Add opt-in PrimTS attention for Step5 on Blackwell
- [DSv4.1][BI] Bitwise batch invariance for DeepSeek-V4.1-Flash with CUDA graphs
- [Bug]: Qwen4Exp (Qwen3.8-Flash-Next) — single prefill crossing the 262,144 native boundary crashes the engine with CUDA illegal memory access (v0.32.0 and nightly, OK on v0.31.1)
- [Perf][Parser] Avoid O(N^2) arg conversion in Granite streaming parser
- Docs
- Python not yet supported