vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported43 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- Add release_kv_cache API
- [Bug]: TurboQuant decode attention: workspace allocation not sized during warmup → AssertionError on first decode after lock_workspace()
- [RFC]: Tensor descriptor (TD) adoption strategy for vLLM Triton kernels
- [Bug]: KV Block double free when using eager SimpleCPUOffloading + Sliding window attention
- [Bug]: Intermittent `vectorized_gather_kernel index out of bounds` in `triton_reshape_and_cache_flash` (KV-cache update) during Gemma 4 31B + MTP K=3 inference — both FP8-block and NVFP4
- [Bugfix] DeepSeek V4 MTP: fix AttributeError in compute_logits when calling hc_head_op
- [Bug]: Using the /generative_scoring may cause shape mismatches in the rejection sampler, causing vllm serve to crash
- [Bug]: --enable-return-routed-experts crashes with AttributeError on non-MoE models after full model load (missing architecture compatibility check)
- [Bug] custom_all_reduce IPC handle fails with expandable_segments when DP>1 AND TP>1
- [Bugfix] Validate stop_token_ids against vocab size to prevent OOB GPU access
- Docs
- Python not yet supported