deepspeed

Create a ds_config JSON with zero_optimization stage set to 3, optionally adding offload_param and offload_optimizer entries targeting cpu or nvme devices. Launch training with the deepspeed CLI command specifying your config file and number of GPUs.

ZeRO stage 1 partitions optimizer states, stage 2 additionally partitions gradients, and stage 3 also partitions model parameters across GPUs. Higher stages save more memory but increase communication overhead between devices.

Yes, ZeRO-Offload moves optimizer states and gradients to CPU memory, while ZeRO-Infinity extends offloading to NVMe storage. This enables training models with trillions of parameters on limited GPU hardware.

Add an fp16 or bf16 block with enabled set to true in the DeepSpeed config JSON. DeepSpeed handles automatic loss scaling for FP16 and supports FP8 on compatible hardware.

Out-of-memory errors typically occur when the ZeRO stage is too low for the model size or activation checkpointing is disabled. Increase the ZeRO stage, enable gradient checkpointing, or offload parameters to CPU or NVMe.

Yes, DeepSpeed supports MoE training with expert parallelism via num-experts and moe-expert-parallel-size parameters. It also supports PR-MoE and Mixture-of-Students distillation to reduce model size by up to 3.7x.