vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported59 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Bugfix][LoRA] Fix LLaVA-NeXT tower and connector buffer sizing
- [Security] Validate decoded scale-out Qwen image features
- [Bug][KV Offload] FS tier reports EIO instead of ENOSPC when the disk fills
- [Bugfix][Model] Zero Kimi-Audio's Whisper k_proj bias so the model loads again
- [Bug]: Phi-2 crashes with CUDA illegal memory access during compile/warmup at size 32
- [Bug]: Qwen2.5-0.5B crashes with CUDA illegal memory access during initialization warmup
- [XPU][CI][DEBUG] xccl TP=2 hang repro on intel_gpu_k8s
- [Bugfix] Validate compilation config backend
- [Bug]: phi-2 crashes with CUDA illegal memory access during compile/warmup at size 32
- [Bugfix][LoRA] Keep MoE-LoRA correct when DP gathers tokens
- Docs
- Python not yet supported