vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported62 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [ROCm][Perf][GLM-5] Fused decode with the MonoKernel in vLLM
- [ROCm][Perf] Qwen3.8-2.4T specialized decode on MI355X (--monokernel)
- [Perf][Attention] Skip nonempty output rows during sparse MLA cleanup
- [Bugfix][Tool Parser] Recover when a stray <tool_call> opens a phantom call in the glm47 parser
- [Kernel] Add opt-in BF16 shared-gate and CUTLASS MoE decode fusions
- [Performance][NIXL] Optimize nixl descriptors for block-outermost kv cache layout
- [Bugfix][MoRIIO] Handle WRITE cancellation before submission
- [Bugfix] Name the cause when a compile cache dir cannot be created
- [Bug]: DotsToolParser drops tool calls when request.tools is None and crashes on FunctionTool
- [Usage]: How to reduce the number of Prometheus metrics exposed by vLLM?
- Docs
- Python not yet supported