vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported61 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Attention] Decode FlashInfer spec queries with fa2 on SM90 NVFP4
- [Tracking] Dory looped model: per-request loops, speculative decoding, GB10 performance
- [Feature]: Cohere grounding citations on the derender paths
- [Docs][KV Connector] NixlConnector: disable EFA shm for same-host P/D with LIBFABRIC
- [Bug]: GLM-5.3-Flash kpool prefill writer leaves the pool straddling an unaligned chunk start unwritten or pooled from another request's tokens
- [Attention] Allow an NVFP4 KV cache at head_size 64 on fa2
- [Bug]: Triton ReplaySSM (V1) drops un-flushed ring tokens when a running request skips a step
- [Bug]: Responses API: completed responses never carry `completed_at`
- [Bug]: Responses API rejects `include: ["web_search_call.results", "web_search_call.action.sources"]` with 400, although the `web_search` tool is accepted
- [Performance]: SM100 prefill indexer: BF16 logits are upcast to FP32 before top-k (as_fp32_logits); a BF16-native top_k_per_row_prefill is 20-25% faster end to end at 64k-192k
- Docs
- Python not yet supported