skills.rest
azaanaliraza
azaanalirazaCOMMUNITY
Explore all skills by this author →
GitHub

huggingface-accelerate

Adds distributed training support to PyTorch scripts with a unified API for DDP, DeepSpeed, FSDP, and Megatron.

Updated Apr 18, 2026
Search Tags:#pytorch#distributed training#fsdp#mixed precision#deepspeed#accelerate#multi-gpu
One-click install
npx skills add https://github.com/azaanaliraza/operarius --skill huggingface-accelerate-azaanaliraza
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: huggingface-accelerate
Source: https://github.com/azaanaliraza/operarius/tree/main/src-tauri/bin/hermes/optional-skills/mlops/accelerate
Command: npx skills add https://github.com/azaanaliraza/operarius --skill huggingface-accelerate-azaanaliraza

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires accelerate, torch, transformers, and includes references (resource) components.

What problem does it solve? Scaling PyTorch training from a single GPU to multi-GPU or multi-node clusters normally requires rewriting training loops for each distributed backend. This Skill shows how to add distributed support with four lines of code using HuggingFace Accelerate, covering device placement, mixed precision, and sharding automatically. ## Core Features & Use Cases - Unified Distributed API: One code path works across DDP, DeepSpeed ZeRO, FSDP, and Megatron-LM without rewriting the training loop. - Mixed Precision Training: Enable FP16, BF16, or FP8 with automatic gradient scaling and device placement. - Interactive Configuration: Use accelerate config and a single accelerate launch command for single-GPU, multi-GPU, or multi-node runs. - Use Case: You have a single-GPU training script that now needs to run on 8 GPUs with BF16. Add the Accelerator, call prepare() on your model, optimizer, and dataloader, replace loss.backward() with accelerator.backward(loss), then launch with accelerate launch --multi_gpu --num_processes 8 train.py. ## Quick Start Convert my PyTorch training script to run on multiple GPUs using HuggingFace Accelerate and show me the launch command.

Frequently Asked Questions about huggingface-accelerate

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I convert a PyTorch script to multi-GPU training?▼

Add four lines with HuggingFace Accelerate: create an Accelerator, pass your model, optimizer, and dataloader through accelerator.prepare(), and replace loss.backward() with accelerator.backward(loss). Then run it with accelerate launch --multi_gpu --num_processes N train.py.

Accelerate vs PyTorch Lightning vs DeepSpeed for distributed training?▼

Accelerate offers the lightest abstraction with a unified API over DDP, DeepSpeed, FSDP, and Megatron. PyTorch Lightning suits users wanting callbacks and high-level structure, while raw DeepSpeed gives direct control over advanced features.

Does Accelerate support DeepSpeed ZeRO-3 and FSDP?▼

Yes. Pass a DeepSpeedPlugin with zero_stage=3 or a FullyShardedDataParallelPlugin to the Accelerator, or select them during accelerate config. The training loop code stays identical across backends.

How do I enable mixed precision training with Accelerate?▼

Set mixed_precision='fp16', 'bf16', or 'fp8' when creating the Accelerator. BF16 needs no gradient scaling and is more stable, FP16 uses automatic gradient scaling, and FP8 requires H100-class hardware.

Why is gradient accumulation not working in my training loop?▼

Gradient accumulation requires wrapping the step in the accelerator.accumulate(model) context manager. Without it, gradients synchronize every step and the effective batch size stays unchanged.

Can Accelerate train large models with Megatron tensor parallelism?▼

Yes, via MegatronLMPlugin with configurable tensor, pipeline, and sequence parallelism degrees. Tensor parallelism requires fast NVLink interconnects within a node, while pipeline parallelism works across nodes.

Related Skills

Explore More Software Engineering Skills→
xiaoquqixiaoquqi

pytorch-fsdp

Apply PyTorch FSDP memory sharding across data-parallel workers for distributed training.

Community
Advanced
rocklambrosrocklambros
2

auditing-rlhf-reward-hacking

Audit RLHF, DPO, and RLAIF checkpoints for reward hacking by comparing reward win-rates with preference win-rates.

Community
Advanced
WangJie-cnWangJie-cn

experiment-manager

Track ML experiment configurations, metrics, and GPU usage.

Community
Advanced
Orchestra-ResearchOrchestra-Research
13.0k

ray-train

Orchestrates distributed PyTorch, TensorFlow, and HuggingFace training across multi-node GPU clusters.

Official
Advanced