vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported37 Subscribers
Add a CodeTriage badge to vllm
Help out
- Issues
- [Bug]: Mistral3 (HF format): default text-only LLM() init fails in multimodal profiling — "Failed to apply PixtralProcessor on data={'text': '[IMG]'}"
- [Bug] Speculative decoding with a large num_speculative_tokens fails with a bare negative max_num_scheduled_tokens instead of naming the flags that fix it
- [Bug]: NixlConnector rejects packed hybrid KV cache after HMA rewrites block size
- [KVConnector][NIXL] Skip block-size sync for packed KV caches
- [Usage]: Significant TTFT degradation and poor QPS scaling with Rust frontend on vLLM 0.25.1 (Qwen122B-A10B, avg input 44k tokens)
- [Bug][PD][Mooncake] Decode requests stuck in WAITING_FOR_REMOTE_KVS and fail with Timeout waiting for P side ready
- [Core] Support sparse checkpoint updates through native weight loaders
- [Bugfix][MoE] Fix fused block-scale orientation
- [Bugfix][Mamba] Fix overlapping state copy race
- [Feature]: Emit BlockInactive KV events when GPU block ref_cnt drops to zero
- Docs
- Python not yet supported