vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported62 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [CPU] Add BF16 fallback for MXFP4 MoE without AMX
- [Transformers backend] add DistilBERT sequence classification via Transformers Backend
- [Frontend] Send ping keepalives while an Anthropic /v1/messages stream waits for its first token
- [FlashInfer] Add opt-in eager/graph autotune measurement
- [Bugfix][ROCm] Make Mooncake an optional connector dependency
- [Kernel][MoE] Add small-batch BF16 SiLU Triton expert path on Hopper
- [Docs][Spec Decode] Qualify Gumbel recovery watermark strength
- [Bugfix][Frontend] Keep special tokens when truncating from an explicit side
- [Bugfix][Spec Decode] Log EAGLE draft RoPE parameters at startup
- [ROCm][Misc] Write GDN prefill output in place for the Triton and FlyDSL backends
- Docs
- Python not yet supported