vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported59 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [KV Connector][Mooncake] Generalize cache registration and payload layouts
- [Bugfix] Broadcast scalar block scales in Triton fused MoE
- [Feature][Spec Decode] Add Ampere (SM80) Native Specialized Decode for DeepSeek-V4.1-Flash
- [Bugfix][KV Offload] Report ENOSPC instead of EIO when an FS-tier store fills the disk
- [ROCm][Perf][DSv4.1] Score adaptive-verification decode steps as varlen sequences in the MXFP4 indexer
- [Bug]: compressed-tensors loads checkpoint FP8 k/v scales into layers kept in BF16 by kv_cache_dtype_skip_layers, and FlashInfer applies them
- [Bugfix][KV Connector] Reconcile local hits when draft attention lags
- [ROCm][Perf] W4A16: fix the gfx1151 bf16 tile cliff at decode-sized M (up to +44% decode throughput)
- [Model][Quantization] Muse Spark: online FP8 routed experts (fp8_per_channel) on Hopper
- [Perf][Kernel] Parallelize mHC post for small SM100 batches
- Docs
- Python not yet supported