deepspeed
https://github.com/microsoft/deepspeed
Python
DeepSpeed is a deep learning optimization library that makes distributed training easy, efficient, and effective.
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported21 Subscribers
Add a CodeTriage badge to deepspeed
Help out
- Issues
- Deprecate unused DeepSpeed features
- [RFC] Enable Correct and High-Performance Muon Optimizer under ZeRO CPU Offload (Stage 1/2/3)
- fix: track FusedAdam bias correction per parameter
- fix(inference): break circular import in ops.transformer.inference
- Support dynamic parameter offload for ZeRO-3 inference
- Rebuild the full argument list when partitioning activations
- [AutoSP] Aggregate the causal LM loss and gradients over the SP group
- AutoTP loads module weights by hardcoded class name, so 219 of 233 transformers norm classes are silently left on meta
- Muon under AutoTP orthogonalizes each shard, so the update depends on the TP degree
- Default gradient_clipping divides every Muon update by its own norm, shrinking the step by a model-sized factor
- Docs
- Python not yet supported