vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported60 Subscribers
Add a CodeTriage badge to vllm
Help out
- Issues
- [Bugfix][Tool Parser] Fix hunyuan_a13b streaming arguments with braces and deep nesting
- [Bugfix] Preserve cross-encoder pair boundaries during explicit truncation
- [Security] Keep prefill truncation state internal to each request
- [ROCm][Perf] Run fp8 DCP cprr MLA verify at 96 native heads when AITER supports it
- [Bug]: CudaPlatform.log_warnings() aborts startup on an unqueryable NVML device even when CUDA_VISIBLE_DEVICES excludes it
- [RFC]: Hybrid KV Backend: Eliminating Disk Restore Pipeline Bubbles Without CPU Memory Bloat
- [Core][Perf] Evict superseded Mamba align states first
- [Perf][MoE] Skip the token-count load on the low-latency fused-MoE path
- [Perf] Add SM90 skinny GEMM plans for dense Qwen3 and Llama-3.1 decode (TP=1)
- [Perf] W8A16 skinny GEMM for block-FP8 linear layers at 1-4 tokens on Hopper
- Docs
- Python not yet supported