vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported51 Subscribers
Add a CodeTriage badge to vllm
Help out
- Issues
- [Bug]: `VLLM_BATCH_INVARIANT=1` is not batch-invariant for fused-MoE experts with no warning
- [Bugfix] Bound request seeds to the int64 range
- [Bug][ROCm] AiterExperts + DeepEPHT/Mori PrepareAndFinalize: illegal memory access at world_size=2 (gfx942)
- 1M-context models on MRV2: `warmup_kernels()` dummy-execute leaves the engine dead at startup/capture on ROCm — skipping that call is the only thing that fixed it for us
- Treat max_tokens as output upper bound, not input-space reservation
- [Docs][Hardware][NVIDIA] Document GTX 16-series FP16 performance
- [Bugfix][Parser] DeepSeek V4: drop orphan DSML structural closers in content
- [Bug]: API server leaks request state when client cancels during AsyncLLM.add_request
- [CI] Move (H200 MIG 35GB / MI355 DPX) Multimodal Models (Standard) 4 core tests into standard/
- [MoE] Integrate FlashInfer CFT counted-write MoE all-to-all (issue #57069)
- Docs
- Python not yet supported