vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported61 Subscribers
Add a CodeTriage badge to vllm
Help out
- Issues
- [Frontend][Metrics] Add vllm:request_reasoning_effort counter
- [Bugfix][0.32] Keep Clef Omni media safe after schema refusal
- [Distributed] Opt-in FP16 partial sums for tensor-parallel row-parallel layers
- Fix GPTQ quantizations loading zero point parameters in symmetric configs
- [ROCm] torch.profiler stalls on ROCm 10 from a rocprofiler-sdk split lock
- [Bugfix][LoRA] Validate dtype at config time for Triton LoRA kernels
- feat(quantization): add AWQ-v2 64KB DMA page-aligned zero-repack loader and specification
- [Spec Decode] Keep the trailing prefix-cache block for hybrid models
- [Attention] Add opt-in FlashInfer fused DCP combine for MLA and GQA/MQA
- [Attention][SM86] Occupancy-adaptive Triton split-K segments for single-query decode
- Docs
- Python not yet supported