vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported59 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Perf][KV Offload] Vectorize direct-layout descriptor fill across layers
- [Bugfix][NIXL] Reject P/D pairs whose KV cache groups scale differently
- [Bug]: Unbounded startup wait holds --data-parallel-rpc-port; crashed frontend leaves orphaned processes holding accelerator memory and the shared API listen socket
- [ROCm][Disagg][CI] Give Wide-EP headless nodes the KV config
- [Kernel] Exact top-p cutoffs and kept log-sum-exps for bf16 logits
- [Perf][Spec Decode] Apply top-p cutoffs as target logits load in verification
- [Perf][Spec Decode] Take the target log-sum-exp from the top-p cutoff kernel
- [CI/Build] Mirror release and nightly images to the vllm/vllm, vllm/vllm-rocm and vllm/vllm-xpu DockerHub aliases
- [Bugfix][LoRA] Fix GPT-OSS triton_kernels 3.8 MoE LoRA row order
- [Perf][HY4] Speed up the iHC pre reduction with M-blocked and split-K paths
- Docs
- Python not yet supported