vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported59 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Bugfix][Core] Tell the model state when the extensible KV cache is rebound
- [Bug]: Rust vllm-bench random dataset sends prompts shorter than --random-input-len
- [Rust][Benchmark] Pad random text prompts to the target length
- [Bug][XPU]: Expert weights transposed twice with vllm_xpu_kernels 0.1.17
- [XPU] Bump vllm_xpu_kernels to 0.1.17.1, take expert weights as loaded
- [Bug][XPU]: UVA expert offload of an int4 MoE loses the device (UR_RESULT_ERROR_DEVICE_LOST) in the first decode step
- [XPU][Offload] Keep MoE weight scales on the device without vllm-xpu-kernels#605
- [Bug]: FlashAttention cache write fails with an explicit `"kv_cache_dtype": "bfloat16"` (e.g. a BF16 draft over an FP8 target)
- [Bugfix][Attention] reshape_and_cache_flash: accept an explicit plain KV dtype
- [Feature][XPU][Offload]: Spend `--cpu-offload-gb` per routed expert, by routing rank, instead of whole layers in module order
- Docs
- Python not yet supported