vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported43 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Bug]: mooncake-transfer-engine fails to import in CUDA 13 images: libcudart.so.12 missing
- [RFC]: External Elastic EP Scaling
- [Bug]: Gemma4 NVFP4 fails to start with pipeline parallel = 2, or TP = 2 without EP
- [Bug]: ngram_gpu speculative decoding can propose draft tokens past max_model_len budget
- [Bugfix] Abort request on ValueError/VLLMValidationError in generate() and encode()
- [Bug]: vLLM rejects requests when max_tokens exceeds available context instead of clamping
- [RFC] Production-boundary measurement on H100 + vLLM 0.19.1: throughput plateau at c=4→16, methodology critique invited
- [Bug]: Async double streaming_update with shared-prefix reuse can leave invalid -1 token ids in the worker input row
- [v1] Guard resumable streaming updates against max_model_len overflow
- [Performance] DFlash slower than baseline at concurrency > 8 on Qwen3.5-35B-A3B
- Docs
- Python not yet supported