What problem does it solve? Training large models that exceed single-GPU memory requires correct parameter sharding, and misusing PyTorch FSDP2 (wrong sharding order, optimizer timing, or naive checkpointing) causes memory blowups, silent hook failures, and broken checkpoints. ## Core Features & Use Cases - Correct FSDP2 Integration: Applies fully_shard() bottom-up with proper process group init, DeviceMesh setup, and meta-device model initialization. - Memory & Performance Tuning: Configures reshard_after_forward, MixedPrecisionPolicy, and CPUOffloadPolicy for memory/throughput trade-offs. - Distributed Checkpointing: Implements DCP save/load and distributed state-dict helpers with load-time resharding support. - Use Case: You have a transformer training script that OOMs on one GPU. Use this Skill to retrofit FSDP2 sharding, BF16 mixed precision, and DCP checkpointing so it trains across multiple GPUs. ## Quick Start Add PyTorch FSDP2 fully_shard sharding with distributed checkpointing to my training script so the model fits across multiple GPUs.
