vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported59 Subscribers
Add a CodeTriage badge to vllm
Help out
- Issues
- [Bugfix] Keep modules listed in FP8 modules_to_not_convert unquantized
- [Bug]: LFM2, LFM2-MoE, Jamba, Zamba2, Step3, Bailing MoE and PLaMo3 build `Attention` without `quant_config`, so checkpoints with static FP8 KV-cache scales fail to load
- [Bugfix] Pass quant_config to Attention in hybrid and MoE models
- [RL][Model Loader] Load FP8 indexer wk and MLA projections independently, as FP8 or BF16, in any scale layout
- [Rust Frontend] Support --log-level and --logging-config
- [Bug]: MiniMax-M3 sparse attention layers drop checkpoint FP8 KV-cache scales and run with scale 1.0
- feat: CPU WNA16 MoE INT8 weight-only support for mixed INT4/INT8 models (issue #65070)
- [Feature]: Integrate FlashTopK fused indexer top-k for DSA prefill on Hopper (SM90)
- [Bug]: `enable_weights_track` refuses correct weight-only NVFP4 checkpoints (input scales that no export writes and no kernel reads)
- [Perf][Kernel] CUTLASS FP8 scaled_mm: smaller tiles for narrow-N decode GEMMs on SM90 and SM100
- Docs
- Python not yet supported