vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported62 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [ROCm][Perf] gfx950: use aiter.mqa_logits for the sparse indexer prefill
- [Bugfix] Disable CUDA graphs for --cpu-offload-gb without pinned memory
- [Perf][Attention] Optimize Triton unified attention for head_dim=512 and sliding-window decode
- [Perf][DSpark] Compute windowed draft context for trailing rows only
- [Bug]: Default -O2 leaves DeepSeek MLA RoPE and KV-cache fusion off
- [Bugfix] Serialize sweep list parameter values as JSON
- [Bugfix][Model] Accept list-typed compressed-tensors format in is_nvfp4_quantized
- [Bug] MiMoV2MTP silently drops FP8 KV cache scales and every draft layer falls back to scale 1.0
- [Frontend] Add `srt` and `vtt` response formats for audio transcription/translation
- [Bug]: Olmo3 reasoning parser drops the final streamed delta when it is a substring of `<think>`/`</think>` (e.g. "think")
- Docs
- Python not yet supported