vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported43 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Performance]: Inspired by nano-vllm, as vLLM-Omni is also complex, I tried building a nano-vLLM-Omni (~1k LOC)
- [Bug]: Wrongful detection of WSL
- [Feature]: Opt-in: stop caching KV blocks after thinking start tokens
- [Bug]: v1 Scheduler: preempted requests lose re-admission priority in PriorityRequestQueue, causing redundant recompute
- [Bug]: Mixed INT4/INT8 GPTQ MoE models crash on initialization (AssertionError in fused_marlin_moe)
- [Bug]: AsyncLLM silent-hangs on multimodal pooling requests when default max_num_batched_tokens too small (L4); sync LLM works on same hardware
- [Bug]: NIXL Disagg Does Not Support GDN Attention (Qwen3.5 Hybrid)
- [Bug]: Pipeline Parallelism Blocked on V1 Engine (Multi-Node PP)
- [EPLB] Simplify move_to_buffer() by decomposing into helper functions
- [Frontend][RFC]: Add unified Pod Snapshot API to support automatic cloud provider checkpoints
- Docs
- Python not yet supported