vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported61 Subscribers
Add a CodeTriage badge to vllm
Help out
- Issues
- [Core][DBO] Dual batch overlap under pure tensor parallelism (split PR of M3.1)
- [Bugfix][Kernel][Benchmark] Fix benchmark_moe.py crash on TP-refined FP8 block shapes; add H100 E=128,N=192 fp8 [64,64] config
- [Perf][MoE] Size NIXL dispatch from synchronized execution slices
- [Bug]: DeepSeek-V4.1-Flash-1017 on GB300: long agent replies fall into repetition loops that another provider never shows on identical requests (persists with DSpark off, fp8 KV, FlashMLA, no decoder replay)
- [Misc] Remove dead chunk_size_in_s variable from speech-to-text serving
- [Perf][Spec Decode] DSpark: run the Markov walk in one kernel with int8 W2 on chip (Hopper)
- [Bugfix][Spec Decode] Keep FlashInfer CUDA graphs for drafters with their own head count
- [Bugfix][Spec Decode] Validate the dynamic SD schedule when building the config
- [Bug]: [b12x] extra pins b12x 1.5.1 — unguarded `fake_mode` in the paged-priming compile crashes engine-core startup on b12x paged attention + chunked prefill
- [Perf][Spec Decode] Enable torch.compile for GLM-4 MoE MTP draft model
- Docs
- Python not yet supported