accelerate
https://github.com/huggingface/accelerate
Python
🚀 A simple way to train and use PyTorch models with multi-GPU, TPU, mixed-precision
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported1 Subscribers
Add a CodeTriage badge to accelerate
Help out
- Issues
- enable_model_cpu_offload(): first post-load generation thrashes the Windows pagefile (~6x step time), silently recovers
- Fix dead links to removed transformers examples in the docs
- restore dataloader epoch when resuming a stateful dataloader
- Fix stateful dataloader replaying the finished epoch after a boundary checkpoint
- Add RunyardTracker: log_with="runyard"
- Stateful DataLoader resume at an epoch boundary replays the completed epoch (shard iteration counter is not checkpointed)
- Fix typos in docs, examples and comments
- FSDP2 + cpu_ram_efficient_loading: ignored_modules are sharded anyway, and models with persistent buffers crash
- FSDP2 cpu_ram_efficient_loading: honor ignored modules and load buffers
- Sequence parallelism: allow an opt-out of the linear-attention refusal, and make disable_in_eval configurable
- Docs
- Python not yet supported