deepspeed
https://github.com/microsoft/deepspeed
Python
DeepSpeed is a deep learning optimization library that makes distributed training easy, efficient, and effective.
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported21 Subscribers
Add a CodeTriage badge to deepspeed
Help out
- Issues
- [Fix] Reject duplicate block ids in BlockedAllocator.free to prevent KV-cache double allocation
- Emit intra_op_parallelism and JSON booleans from ds_nvme_tune
- [BUG] Low-precision optimizer initialization advances Adam and applies weight decay before training
- [BUG] ZeRO stage 1/2 marks optimizer-state fragments linked before later groups initialize
- [BUG] get_global_norm_of_tensors crashes on ranks with an empty norm contribution
- [Precision] BF16 AutoEP folding rounds FP32 TP reduction before FP32 gradient accumulation
- [RFC] Deterministic training and versioned Megatron bitwise compatibility for production-common configurations
- Fix the default data parallel checkpoint writer crash on dp_rank
- Preserve FP32 folded gradients through FP32 accumulation
- Fix ZeRO-3 NVMe module-only checkpoint weight refresh
- Docs
- Python not yet supported