deepspeed
https://github.com/microsoft/deepspeed
Python
DeepSpeed is a deep learning optimization library that makes distributed training easy, efficient, and effective.
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported21 Subscribers
Add a CodeTriage badge to deepspeed
Help out
- Issues
- Overlap AutoEP's expert weight gradients with the DeepEP backward combine
- Fix: [BUG] In step3, a runtime error will be thrown when inference_tp_size>
- Average qgZ gradients over every rank, not just the nodes
- Reshuffle the training data every epoch when DeepSpeed builds the sampler
- [BUG] torch_autocast fp16 at ZeRO stage 0: skipped (overflow) steps are counted as applied and still step the LR scheduler
- Keep the sequence-parallel data-parallel size an integer in DeepSpeedConfig
- Bias-correct LAMB's moments instead of its step size
- Count GradScaler-skipped steps as skipped with torch_autocast fp16 at ZeRO-0
- Coalesce sparse gradients in DeepSpeedCPUAdagrad
- Restore ROCm architecture environment after JIT loading
- Docs
- Python not yet supported