vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported59 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [MoE][RL] MegaMoE NVFP4: load each scale on its own, combine after loading
- [RFC]: Out-of-tree KV-export connectors that hand a vLLM prefill to MLX (EXO) decode on Apple Silicon
- [Quantization] Keep E8M0 scale bytes when loading into uint8 params
- [ROCm] Split-K bf16x3 router GEMV for fp32 router gates at 33-2047 tokens
- [RL][Model Loader] Make expert scale, Inkling and Mamba v2 sharded loaders copy_-only
- [Bugfix][KV Connector] Forward num_threads to non-UCX NIXL backends
- [Bugfix][Core] Fix Gemma-4 MTP prefix-cache hit loss when lookahead block hashes are off
- [Bug]: --max-num-queued-tokens is enforced per API server, not across API servers
- [Bugfix][Frontend] Share the max_num_queued_tokens count across API servers
- [Feature] Segment attention layout for one-pass multi-candidate scoring
- Docs
- Python not yet supported