vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported61 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Bugfix][Tokenizer] Fix Kimi-Audio special token encoding and skipping
- [ROCm][Perf] Muse Spark: column-shard the MoE pooler/unpooler over TP
- [Perf]Support Humming MXFP8 grouped output projection for DeepSeek V4/V4.1
- [Bugfix] Tell a stopped engine core apart from a dead one during AsyncLLM shutdown
- [CPU] Skip multimodal encoder profiling when the KV cache size is explicit
- [ROCm][Perf][DSv4.1] Mono decode: architecture-selected tiling + MI350P enablement path
- [Bugfix] Honor child engine IDs in MultiConnector
- [Spec Decode] Adaptive verification: wait for stale confidences only when ranking drafts
- [Bug]: Chat derender ignores `--response-role`, `add_generation_prompt: false` and `echo`
- [Bug]: Hermes tool parser: streaming and non-streaming disagree when a valid tool call is followed by a malformed <tool_call> region
- Docs
- Python not yet supported