vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported62 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Bug]: PEFT sequence-classification LoRA adapters fail to load on RoBERTa / XLM-RoBERTa
- [KV Offload] Add admission policy framework and backpressure adapter
- [Performance]: Hybrid Mamba prefix caching (`mamba_cache_mode="align"`) costs 11-16% throughput on Nemotron-3.5-Lightning with no cache hits: per-group eager block-table ops, per-step align kernels outside CUDA graphs, and block-boundary prefill splits
- [Bugfix][Benchmark] Preserve synthetic datasets in throughput validation
- [DRAFT] unconditionally zero kv-cache blocks when we get them to make attention kernel usage safer
- [Usage]: How to serve qwen3.8-flash-next-fp8 on 2 A100
- [Bug]: flashinfer_moe_ep_cutedsl fails at model load because vLLM CUDA images don't ship nvshmem4py (No module named 'nvshmem')
- [Test] Fix kernel tests on Blackwell GPUs
- [Bugfix] flashinfer_moe_ep_cutedsl: reject it without nvshmem4py and install nvshmem4py
- [Model][LoRA] Load Inkling PEFT adapters in the serving layout
- Docs
- Python not yet supported