vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported59 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Tracking] Replace Marlin with Humming across NVIDIA architectures
- [Refactor][Worker] Expose platform executor and PP orchestration hooks
- [Bugfix][Spec Decode] Fall back from InstantTensor for PP-local draft models
- [Bugfix][KV Connector] Backoff last prompt token for sliding window in P/D (#65550)
- [ROCm][UMBP] Add waiting-request prefetch to UMBPStoreConnector
- [KV Events] Expose chunk size in BlockStored events
- [Bug]: gpt-oss-20b + nebius/EAGLE3-gpt-oss-20b gives corrupted greedy output at num_speculative_tokens >= 5 on MI355X (correct with --enforce-eager)
- [Bugfix][Multimodal] Molmo2: snap NTSC frame rates in fps sampling
- [Bug]: OffloadingConnector KV events drop the request's extra_keys (cache_salt), so a KV-event consumer can't place CPU-tier blocks in their salted namespace
- [KVConnector][NIXL] Add stream-ordered same-host CUDA IPC pulls
- Docs
- Python not yet supported