huggingface-accelerate

Add four lines with HuggingFace Accelerate: create an Accelerator, pass your model, optimizer, and dataloader to accelerator.prepare(), and replace loss.backward() with accelerator.backward(loss). Then run with accelerate launch --multi_gpu --num_processes 8 train.py.

Accelerate offers the lightest abstraction with a unified API over DDP, DeepSpeed, FSDP, and Megatron. Lightning suits users wanting callbacks and high-level structure, while raw DeepSpeed gives direct control over advanced ZeRO features.

Yes. Pass a DeepSpeedPlugin with zero_stage=3 or a FullyShardedDataParallelPlugin to the Accelerator, or select them during accelerate config. The training loop code stays identical across backends.

Set mixed_precision='bf16' or 'fp8' when creating the Accelerator. BF16 needs no gradient scaler and is more stable, FP16 uses GradScalerKwargs for loss scaling, and FP8 requires H100 GPUs with TransformerEngine.

Gradient accumulation requires wrapping the step in the accelerator.accumulate(model) context manager and setting gradient_accumulation_steps on the Accelerator. Without the context manager, gradients synchronize every step instead of accumulating.

Use accelerator.save_state('checkpoint/') after accelerator.wait_for_everyone(), and guard main-process-only logic with accelerator.is_main_process. Load with accelerator.load_state() so every process restores its correct shard.