vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported59 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Bugfix][MRV2][Spec Decode] Count NaNs in logits per request, not per logits row
- [Bugfix] Warm up FlashInfer kernels on an autotune cache hit
- [Quantization][Fusion] Extend manual activation fusion to more NVFP4 linear backends
- [Bugfix][Rust Frontend] Preserve null DeepSeek tool descriptions
- [Bugfix][XPU] Make layer_norm batch invariant under VLLM_BATCH_INVARIANT
- [Bug]: Rust frontend: clients that connect during --shutdown-timeout hang instead of being refused
- [Bugfix][Frontend] Close the parent's listening sockets after launching the Rust frontend
- [Bugfix][Scheduler] Drain in-flight stale output on sync KV load-failure recovery (#65131)
- [Performance]: With EAGLE and the OffloadingConnector, a prefix-cache hit loads its dropped last block from the CPU tier
- [Bugfix][Frontend] Stop accepting gRPC connections when shutdown starts
- Docs
- Python not yet supported