pytorch-fsdp

Fully sharded data parallel training shards model parameters and gradients across data-parallel workers, allowing you to train models that exceed single GPU memory limits. It uses a device mesh to manage distributed resources efficiently.

The best way to train large transformer models across multi-GPU clusters is using per-parameter sharding with FSDP2. By applying fully_shard to your model and device mesh, you achieve memory-efficient, scalable training with optional mixed precision and CPU offloading.

To apply FSDP2, run fully_shard(model, mesh) across your configured 1D or 2D device mesh. This initiates per-parameter sharding for scalable training, requiring PyTorch >= 2.x and access to torch.distributed.

Yes, PyTorch FSDP supports mixed-precision training and CPU offloading to optimize memory usage and throughput. These features help maximize resource efficiency when training large models across multi-GPU and multi-node clusters.

Yes, you can migrate from FSDP1 to FSDP2 as the platform is migration-friendly. FSDP2 introduces per-parameter sharding and supports both 1D and 2D meshes, providing a scalable upgrade path for your distributed training workflows.