vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported60 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Feature] Add request-level prefill progress to gRPC streaming
- [Bugfix][Frontend] Memoize parser-engine drop info across requests
- [Bugfix][V1][Mamba] Keep recurrent-state bookkeeping when a running request is skipped for a step
- [Bug]: With multimodal pruning, the encoder compute budget is debited by the pruned placeholder count, so one step can send several times the profiled patches to the ViT (CUDA OOM)
- [Perf][FlashInfer] Opt-in trtllm-gen generation kernel for long-prefix chunked prefills
- [Perf][Spec Decode] Opt-in MTP draft prefill attention for the sampled rows only
- [ROCm] Add gfx1151 W8A8 blockscale skinny GEMM
- [Quantization] Prefer Humming before Machete for 4-bit weights on SM90
- [Feature][ROCm][DSv4.1] Mono decode: enable MI350P (gfx950, 128 CUs) via k-pass task decomposition
- [Bugfix][Tool Parser] Extract every InternLM2 tool-call block
- Docs
- Python not yet supported