vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported61 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [KV Connector] Warn when MultiConnector's first-hit load leaves tokens a later connector reported
- [Bug]: vllm/vllm-openai:flint can't serve the NVFP4 GGUF checkpoint without the gguf package and a speculators-probe workaround
- [Feature]: keep_tags for CuMemAllocator.sleep so selected allocations (e.g. kv_cache) stay mapped
- [Model] dots.ocr: support encoder CUDA graph
- [Bugfix][Muse Spark] Keep the FP32 RoPE tables in FP32
- [Core] Warn at startup when a quantized KV cache meets a large first-layer key bias
- [Bugfix][Core] Set TRITON_CACHE_DIR at worker startup to stop Triton kernels rebuilding on every start
- [Perf][Vision] Fuse Qwen vision QK packing and rotary embedding
- [Perf][GLM-5.x] DeepGEMM FP8 MegaMoE for DeepseekV2MoE (DSA models, PCP)
- [Bug]: `torchcodec` is unpinned and resolves to a CUDA-13 wheel in CUDA-12.9 builds, breaking `vllm serve` on `vllm/vllm-openai:v0.32.0-cu129`
- Docs
- Python not yet supported