vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported59 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [ROCm][Kimi-K3.1] Specialized decode: run DCP1 without M1 on vLLM's MLA ops (enables K3.1; K3 fallback)
- [Bug]: Pythonic tool parsers silently drop calls with Python reserved keywords as parameter names
- [Perf][Core] Look up single-group prefix-cache hits with one dict get per block
- [Bug]: Granite20bFCToolParser streaming misses tool calls when preceded by content
- [Bug]: xLAM non-streaming tool parser drops single tool call object in <tool_call> tags
- [Bug]: Step3ReasoningParser.extract_content_ids excludes think_end_token at last index
- [ROCm][MiniMax-M3] Select the whole-layer mono decode with --monokernel and remove mono/envs.py
- [Rust Frontend] Accept media_io_kwargs video fps over gRPC
- [Bug]: SM100 default prefill CUDA-graph grid (up to 8192 tokens, #63406) adds ~5 GiB/GPU graph memory and breaks max_model_len=256000 startup for Qwen3.8-2.4T-A95B-FP8 TP16 on GB200
- [Bug]: Extensible KV cache reserves the one-off FlashInfer autotune peak as a serving transient, cutting KV 2.4x under an explicit --gpu-memory-utilization (regression from #56492)
- Docs
- Python not yet supported