vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported48 Subscribers
Add a CodeTriage badge to vllm
Help out
- Issues
- [MLA][DeepSeek] Add VLLM_MLA_FP8_PROJ to force FP8 for MLA q_b_proj layer
- [Performance]: Custom Helion Kernels
- [RFC]: All Ops should be determined during init and wrapped in a Layer Module to avoid envs.ENVIRON overhead
- [Perf] Async logits proc and sampling metadata state update
- [Performance]: Multi-Modal Benchmark on NVIDIA A100 – Qwen2.5-VL / MiniCPM-V-4 / InternVL3_5-4B / InternVL3_5-2B
- [Bugfix] Fix ShardedStateLoader support for DeepSeek models with MLA scaling parameters
- [cuda 13][aarch64][CI] Adding CI steps to build arm64 cuda13 nightly wheels and images
- [DRAFT] Add RoaringBitmap-based sparse attention mask implementation
- [AMD] Amd/deepseek aiter fusions
- Add pytest marks, update ci_config and test_area accordingly
- Docs
- Python not yet supported