pytorch-fsdp2

What problem does it solve?

This Skill enables coding agents to correctly integrate PyTorch's Fully Sharded Data Parallel (FSDP2) into training scripts, overcoming challenges with model memory limits and distributed training complexities.

Core Features & Use Cases

• FSDP2 Integration: Adds PyTorch FSDP2 (fully_shard) with proper initialization, sharding, mixed precision, and distributed checkpointing.
• Memory Optimization: Essential for training models that exceed single-GPU memory capacity.
• DTensor-based Sharding: Leverages DTensor for inspectable, per-parameter sharding, composable with DeviceMesh.
• Use Case: When training a large language model that requires more VRAM than available on a single GPU, this Skill ensures FSDP2 is applied correctly to shard parameters, gradients, and optimizer states across multiple GPUs or nodes.

Quick Start

Use the pytorch-fsdp2 skill to add PyTorch FSDP2 to your existing training script, ensuring correct initialization, sharding, and checkpointing.