vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported62 Subscribers
View all SubscribersAdd a CodeTriage badge to vllm
Help out
- Issues
- [Refactor][KV Connector][Mooncake] Move connector metadata and wire structs to their own modules
- [Docs] Add ROCm decode megakernel (MonoKernel) authoring guide
- [Tool Parser] Fix truncated/unclosed DSML parameters and add incremental DSML arg streaming
- [Bugfix][Tool Parser] Fix invalid streamed JSON for control characters in hy_v3 tool args
- [Bugfix][Structured Output][v0.31] Honor resolved backends and mask widths
- [Bugfix][Rust Frontend] Drop a DeepSeek DSML tool-call opener cut after its DSML token
- [Bugfix][Structured Output][v0.32] Honor resolved backends and mask widths
- [Bugfix][Tool Parser] Fix invalid streamed JSON for MiniCPM5 string args with leading or trailing spaces
- [Performance] Memoize waiting prefix cache hits across retries
- [Perf][DSv4.1] DSpark draft on the specialized decode
- Docs
- Python not yet supported