vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported59 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Refactor][PCP] Separate physical topology from attention execution layout
- [Bug]: Streaming final usage chunk intermittently omits metrics.speculative_decoding while spec-decode Prometheus counters advance
- [ROCm][MiniMax-M3] Default to breakable CUDA graphs tests on ROCM
- [Bugfix][Spec Decode] Add packed_modules_mapping to DeepSeek-V4 DSpark models
- [Perf][Sampling] Use FlashInfer top-k selection for top-k/top-p masking
- [Frontend] Support repetition_detection in Responses and Messages to stop repetitive generation
- [Core][DP][WIP] Add a cross-DP prefill delayer scheduling policy
- [Perf][DSV4.1] Accelerate candidate prefill and decode with CuTe
- [Kernel][LoRA][MoE] Skip zero intermediate rank tiles in fused MoE LoRA expand
- [Bugfix][KV Offload] Size the lazy walk target for sliding-window allocations
- Docs
- Python not yet supported