vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported37 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [MISC][Bench] refactor throughput and reuse serve's get samples
- [Doc] Troubleshooting: FlashInfer JIT failure on a GPU newer than the local CUDA toolkit
- Remove attention layer name from unified_kv_cache_update for torch.compile
- docs(serving): note OpenAI client base_url for multi-model gateways
- [XPU] Route block-quantized FP8 weights to the W8A8 kernel
- [Draft] [Bugfix] Clone safetensors tensors before H2D copy on Ascend NPU to fix DMA hang
- Fix weight loading I/O contention and add FlashInfer autotune logging
- fix(config): apply presence_penalty/frequency_penalty from override-generation-config
- FlashInfer MLA decode workspace buffer overflow with decode-context-parallel
- [Feature][Gemma4] Configure video frame soft-token budget
- Docs
- Python not yet supported