vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported59 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Recipe] Add Qwen recipes with IAXL KVShrink support
- [Core][KV Connector] Add VLLM_KV_SCRAMBLE_BLOCKS to test and benchmark with a fragmented KV pool
- [Bugfix][Frontend] Report a tool call cut short by EOS as end_turn in Anthropic streaming
- Experiment OCTAVE KV on GFX1151
- [Build] Compile DeepGEMM's extension through the compiler launcher
- [Bugfix] Infer the per-layer attention module count from KV cache layer names
- [Kernel][Triton] Allow an fp8_e5m2 KV cache on SM80-SM88
- [Perf][KV Connector] Mooncake store: build and parse lookup candidates in bulk
- [Bugfix][GLM-5.3] Build the MTP eh_proj low-latency GEMM plan
- [Perf][GLM-5.3] Use FlashInfer's warp split-K BF16 GEMM for the SM107 low-latency table
- Docs
- Python not yet supported