sglang
https://github.com/sgl-project/sglang
Python
SGLang is a high-performance serving framework for large language models and multimodal models.
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported1 Subscribers
Add a CodeTriage badge to sglang
Help out
- Issues
- [Fix] mem_cache: Pass v_head_dim to the hybrid linear full-attention pool
- Feature request: public single-iteration step API for an in-process `Scheduler`
- Feature request: in-process output sink instead of the detokenizer/tokenizer sockets
- Decouple per-request sampling seeds from `enable_deterministic_inference`'s kernel overrides; allow ignoring the generation config's EOS ids
- Feature request: extension points for externally defined models (registration of runtime classes, attention lookup, attention kernels, sampler logprobs)
- Support data-parallel attention/replicas inside one existing torch distributed world (in-process scheduler)
- [Feat] decision: Serve pplx-decider v1.2 checkpoints on /v1/systemone
- Feature request: per-request time stats in scheduler outputs without enable_metrics
- [Bug] [AMD] moe: GLM Gluon EP>1 32K prefill reads codes past the 2-GiB buffer-op window
- [Fix] parser: Keep DeepSeek recipe output when a length finish splits a UTF-8 character
- Docs
- Python not yet supported