deepspeed
https://github.com/microsoft/deepspeed
Python
DeepSpeed is a deep learning optimization library that makes distributed training easy, efficient, and effective.
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported21 Subscribers
Add a CodeTriage badge to deepspeed
Help out
- Issues
- ZeRO-1/2: record reduction stream on oversized gradients under overlap_comm
- [RFC] Two-stage relay for cross-node weight sync, priced by a calibrated cost model
- [checkpoint] Expose the affine transfer schedule as schedule_transfers
- Count replicated GPTBigCode K/V rows once in clip_grad_norm_
- SuperOffload - PCIe Optimization
- AutoTP rollout engine: generate with a tensor-parallel copy of a ZeRO training model
- [BUG] AutoTP counts replicated GPTBigCode key/value rows once per tensor-parallel rank in the global gradient norm
- CPU: test_finite_step_after_overflow fp16 cases fail with 'Type fp16 is not supported on your device'
- feat(rollout): add AutoTP rollout engine with in-memory weight sync
- Reuse continuous rollout graphs and release captured resources on shutdown
- Docs
- Python not yet supported