vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported37 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- docs(serving): note OpenAI client base_url for multi-model gateways
- [XPU] Route block-quantized FP8 weights to the W8A8 kernel
- [Draft] [Bugfix] Clone safetensors tensors before H2D copy on Ascend NPU to fix DMA hang
- Fix weight loading I/O contention and add FlashInfer autotune logging
- fix(config): apply presence_penalty/frequency_penalty from override-generation-config
- FlashInfer MLA decode workspace buffer overflow with decode-context-parallel
- [Feature][Gemma4] Configure video frame soft-token budget
- fix(model): ensure cpu tensor contiguity during Llama4 ModelOpt weight loading
- Fix OverflowError in prompt_logprobs when external KV pre-fills prompt
- [Docs] Document cross-rank aggregation semantics of NIXL KV transfer metrics
- Docs
- Python not yet supported