accelerate
https://github.com/huggingface/accelerate
Python
🚀 A simple way to train and use PyTorch models with multi-GPU, TPU, mixed-precision
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported1 Subscribers
Add a CodeTriage badge to accelerate
Help out
- Issues
- FSDP2 QLoRA: with `bnb_4bit_quant_storage=bf16` (the storage the #3987 warning recommends), the all-gather alters packed 4-bit weights at world size ≥ 2
- FSDP2: prepare() keeps the whole unsharded model alive during sharding when cpu_ram_efficient_loading=False
- Continue automatic checkpoint numbering after load_state()
- Checkpoints saved during gradient accumulation lose partial gradients on resume
- BatchSamplerShard padding mutates retained sampler batches and changes original loader samples
- Avoid mutating sampler-owned batches during distributed padding
- Stop BatchSamplerShard padding from mutating sampler-owned batches
- Allow variadic arguments in find_executable_batch_size
- Persist partial gradients in save_state for mid-accumulation checkpoints
- Fix metric gathering with sized samplers
- Docs
- Python not yet supported