vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported59 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Bugfix] Accept weight-only NVFP4 checkpoints in enable_weights_track
- [ROCm][MLA] Refuse the AITER fp8 decode config that silently returns wrong output
- [ROCm] Support DP in AiterW4A16ExpertsMonolithic
- [Bugfix][ROCm] Drop the dead min_seqlen_q parameter from AITER FA extend_forward
- [Bugfix] Resolve speculative draft model to local path in offline mode
- [Model] Add Dhir dense and MoE models
- [RL][Model Loader] Inkling: derive expert input scales from input_amax after loading
- [CPU] Support 2D (batched) weights in rms_norm, fixing DFlash acceptance on CPU
- [Feature]: Enable and validate pipeline parallelism for Domyn Edge
- [Perf][Reasoning] Skip the full reasoning-end check on deltas without the end token
- Docs
- Python not yet supported