vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported37 Subscribers
Add a CodeTriage badge to vllm
Help out
- Issues
- Rust Frontend: Add supervisor-side control channel for Elastic EP scaling
- Update workflow comments and add first-interaction action.
- [Bug]: sm_120 + local CUDA toolkit < 12.9: FlashInfer JIT failures kill engine init in three default paths (sampler, fused-MoE, FP8 KV) instead of falling back
- [Bug]: Mistral3 (HF format): default text-only LLM() init fails in multimodal profiling — "Failed to apply PixtralProcessor on data={'text': '[IMG]'}"
- [Bug] Speculative decoding with a large num_speculative_tokens fails with a bare negative max_num_scheduled_tokens instead of naming the flags that fix it
- [Bug]: NixlConnector rejects packed hybrid KV cache after HMA rewrites block size
- [KVConnector][NIXL] Skip block-size sync for packed KV caches
- [Usage]: Significant TTFT degradation and poor QPS scaling with Rust frontend on vLLM 0.25.1 (Qwen122B-A10B, avg input 44k tokens)
- [Bug][PD][Mooncake] Decode requests stuck in WAITING_FOR_REMOTE_KVS and fail with Timeout waiting for P side ready
- [Core] Support sparse checkpoint updates through native weight loaders
- Docs
- Python not yet supported