vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported62 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Tracking] OffloadingConnector layout support
- [Bug]: IndexError in _create_chat_logprobs / _create_completion_logprobs (top_logprobs shorter than token_ids) causes 500s
- [Fast Start] Seed compiled caches through the D2D transport
- [Spec Decode] Support AdaFlash length heads with DFlash adaptive verification
- [AuxOutput] Validate SHM capacity and warn on host memory pressure
- [Quantization] Prefer Humming over Marlin for int4 g128 on SM100
- [Perf][Kernel] AWQ: vectorize the output store in dequantize_weights
- [Bug]: P2P KV offload tier does not fail fast on peer faults
- Match GPTQ dynamic patterns against all module-name forms
- [Bug]: P/D disaggregation (NixlConnector): decode output differs from a non-disaggregated run whenever repetition_penalty != 1.0
- Docs
- Python not yet supported