vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported51 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- Treat max_tokens as output upper bound, not input-space reservation
- [Docs][Hardware][NVIDIA] Document GTX 16-series FP16 performance
- [Bugfix][Parser] DeepSeek V4: drop orphan DSML structural closers in content
- [Bug]: API server leaks request state when client cancels during AsyncLLM.add_request
- [CI] Move (H200 MIG 35GB / MI355 DPX) Multimodal Models (Standard) 4 core tests into standard/
- [MoE] Integrate FlashInfer CFT counted-write MoE all-to-all (issue #57069)
- [ROCm][Perf] FP8 o-projection for DeepSeek-V4.1-Flash MLA on AMD MI355X
- [Bugfix][Frontend] Handle FINISHED_ABORTED in streaming logic by emitting error event
- [Bugfix][Attention] Avoid NaN from empty seq_len in Triton MLA decode (CUDA graph pad)
- [Bugfix][Spec Decode] Separate DSpark from EAGLE prefix-cache block drop
- Docs
- Python not yet supported