vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported61 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Bug]: float32 models: Triton attention, prefill attention and fused-MoE kernels compute `tl.dot` in TF32 (~1e-2 error relative to output RMS instead of ~1e-5)
- [Usage]: cu129-nightly
- [Bugfix][Engine] Ignore clear_cache for pause mode=keep
- [Bug]: Misleading `--quantization gptq` suggestion in Marlin shape check (AWQ/GPTQ TP failures)
- [Bugfix][Kernel] Give float32 Triton attention and fused MoE float32 accuracy
- [CI] Preserve DockerHub cleanup pagination across filtered pages
- [Build][CPU] Publish indexed Triton CPU wheel dependencies
- [Bug]: A100 run Bolt (nv-community/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB) throw :B12X NVFP4 kernels require a Blackwell 12x device
- [Bugfix][Model Runner V2] Honor use_fp64_gumbel when FlashInfer sampling is available
- [Perf] 添加 RTX PRO 6000D 的可选 BF16 投影适配
- Docs
- Python not yet supported