vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported43 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Performance]: Inconsistent speculative decoding acceptance metrics between vLLM and SpecForge on Qwen3-32B baselines
- [Bug]: vllm serve crashes at KV cache initialization when using --block-size 1 or 8 despite being listed as valid choices
- [Bug]: mooncake-transfer-engine fails to import in CUDA 13 images: libcudart.so.12 missing
- [RFC]: External Elastic EP Scaling
- [Bug]: Gemma4 NVFP4 fails to start with pipeline parallel = 2, or TP = 2 without EP
- [Bug]: ngram_gpu speculative decoding can propose draft tokens past max_model_len budget
- [Bugfix] Abort request on ValueError/VLLMValidationError in generate() and encode()
- [Bug]: vLLM rejects requests when max_tokens exceeds available context instead of clamping
- [RFC] Production-boundary measurement on H100 + vLLM 0.19.1: throughput plateau at c=4→16, methodology critique invited
- [Bug]: Async double streaming_update with shared-prefix reuse can leave invalid -1 token ids in the worker input row
- Docs
- Python not yet supported