vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported59 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [ROCm][UMBP][Muse Spark] Support UMBP for Muse Spark Open, keeping the decode monokernel
- [Model Loader] Refill derived tensors with refill_derived (mHC broadcast, Muse Spark ROCm router planes, FlashInfer/b12x sinks)
- [Bugfix][SM120] Admit SM120 to the flattened device-query-lens path
- [Bug]: Precomputed image or video embeddings crash the engine when the encoder CUDA graph is on
- [Bugfix] Fix Cohere/Cohere2 compute_logits when LoRA is enabled
- [Core][MM] Make the multimodal encoder cache size configurable
- [Bugfix][Quantization] Normalize Quark exclude entries with parameter suffixes (#65034)
- [Bugfix] Reject empty/missing content on developer/system messages (400) instead of renderer crash (500)
- [RFC]: Scheduler-owned persistent state for cache-aware streaming RNN-T ASR
- [ROCm][DSv4.1] Score the sparse-indexer prefill on ShardQ token slices
- Docs
- Python not yet supported