skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

huggingface-accelerate

Adds distributed training support to PyTorch scripts with four lines of code.

1|Updated Mar 5, 2026
Search Tags:#pytorch#distributed training#huggingface#fsdp#mixed precision#deepspeed#accelerate
One-click install
npx skills add https://github.com/Clay-HHK/claude-skills --skill huggingface-accelerate-clay-hhk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: huggingface-accelerate
Source: https://github.com/Clay-HHK/claude-skills/tree/main/accelerate
Command: npx skills add https://github.com/Clay-HHK/claude-skills --skill huggingface-accelerate-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires accelerate, torch, transformers, and includes references (resource) components.

What problem does it solve? Scaling PyTorch training from a single GPU to multi-GPU or multi-node clusters normally requires rewriting code for DDP, DeepSpeed, FSDP, or Megatron. This Skill provides a unified API that handles device placement, mixed precision, and sharding automatically. ## Core Features & Use Cases - Unified Distributed API: Convert any PyTorch script to run on DDP, DeepSpeed ZeRO, FSDP, or Megatron with the same code and a single launch command. - Mixed Precision Training: Enable FP16, BF16, or FP8 training with automatic gradient scaling and autocast handling. - Interactive Configuration: Generate launch configs via accelerate config instead of writing manual launcher scripts. - Use Case: You have a single-GPU training script that OOMs on a 70B model. Use this Skill to configure FSDP with full sharding and BF16, then launch across 8 GPUs without changing your training loop. ## Quick Start Convert my PyTorch training script to run on multiple GPUs using HuggingFace Accelerate and show me the launch command.

Frequently Asked Questions about huggingface-accelerate

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I convert a PyTorch script to multi-GPU training?▼

Add four lines with HuggingFace Accelerate: create an Accelerator, pass your model, optimizer, and dataloader to accelerator.prepare(), and replace loss.backward() with accelerator.backward(loss). Then run with accelerate launch train.py.

Accelerate vs PyTorch Lightning vs DeepSpeed for distributed training?▼

Accelerate offers the lightest abstraction with a unified API over DDP, DeepSpeed, FSDP, and Megatron. PyTorch Lightning provides high-level callbacks and structure, while raw DeepSpeed gives direct control over advanced features.

Does Accelerate support DeepSpeed ZeRO-3 and FSDP?▼

Yes. Pass a DeepSpeedPlugin or FullyShardedDataParallelPlugin to the Accelerator, or select them during accelerate config. Both support CPU offloading and BF16 mixed precision for training large models.

Why is gradient accumulation not working with Accelerate?▼

You must wrap the training step in the accelerator.accumulate(model) context manager and set gradient_accumulation_steps when creating the Accelerator. Calling optimizer.step() outside this context skips the accumulation logic.

Can Accelerate train on TPUs or Apple Silicon?▼

Yes. Accelerate supports TPUs and Apple MPS in addition to CPU, single GPU, and multi-GPU setups. The same script and launch command work across all supported hardware.

Related Skills

Explore More Software Engineering Skills→
thinking-machines-labthinking-machines-lab
4.0k

datasets

Construct and normalize ML datasets across JSONL, HuggingFace, and conversation formats.

Official
Intermediate
RachasumanthRachasumanth

inference-deploy

Deploy trained machine learning models for inference using vLLM, TGI, Ollama, and llama.cpp.

Community
Advanced
nishide-devnishide-dev

ml-profile

Profile PyTorch Lightning training to diagnose data loading, computation, and GPU memory bottlenecks.

Community
Intermediate
kaminocorpkaminocorp
1

pytorch-lightning

Organizes PyTorch training code with automatic distributed training, callbacks, and checkpointing.

Official
Advanced