vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported59 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Perf][Quantization] Batch the Marlin MoE scale and zero-point conversion over experts
- [Bug]: vLLM Kimi k3 KVV AMD GPU product quality issue (DeepSWE & BEAM)
- [Feature] Add POST /set_log_level to change logger levels at runtime
- [Bug][ROCm]: Dense attention output allocated with torch.empty — CUDA-graph padding rows poison the dynamic per-tensor FP8 activation scale (garbage under concurrency)
- [Bugfix][Step5] Fix FP32 residual truncation at pipeline boundaries
- [Bugfix][Mooncake] Reclaim unclaimed transfer placeholders
- [Bugfix] Fix Granite 4.1 Vision deepstack handoff between pipeline stages
- [ROCm][Perf] W4A16: serve 6-8 decode rows with the HIP skinny GEMM on gfx1151
- [Model] Gemma 4: native packed vision tower on MMEncoderAttention
- [Kernel] Marlin: deprecate VLLM_MARLIN_USE_ATOMIC_ADD and drop the bf16 sm90 build
- Docs
- Python not yet supported