skills.rest
80portisfound
80portisfoundCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

huggingface-accelerate

Adds distributed training support to PyTorch scripts with four lines of code.

Updated May 3, 2026
Search Tags:#pytorch#distributed training#huggingface#fsdp#mixed precision#deepspeed#multi-gpu
One-click install
npx skills add https://github.com/80portisfound/vibe-learning --skill huggingface-accelerate-80portisfound
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: huggingface-accelerate
Source: https://github.com/80portisfound/vibe-learning/tree/main/packages/hermes/optional-skills/mlops/accelerate
Command: npx skills add https://github.com/80portisfound/vibe-learning --skill huggingface-accelerate-80portisfound

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires accelerate, torch, transformers, and includes references (resource) components.

What problem does it solve? Scaling PyTorch training from a single GPU to multi-GPU or multi-node clusters normally requires rewriting code for DDP, DeepSpeed, FSDP, or Megatron, each with different APIs and launcher setups. This Skill provides a unified approach where the same training script runs on any hardware configuration with minimal changes. ## Core Features & Use Cases - Unified Distributed API: Convert any PyTorch script to distributed training by adding an Accelerator, calling prepare(), and replacing loss.backward() with accelerator.backward(). - Backend Flexibility: Switch between DDP, DeepSpeed ZeRO, FSDP, and Megatron-LM through configuration rather than code rewrites, with support for FP16, BF16, and FP8 mixed precision. - Use Case: You have a single-GPU training script that now needs to run on an 8-GPU node with BF16 mixed precision. Add the four Accelerate lines, run accelerate config interactively, and launch with accelerate launch train.py. ## Quick Start Convert my PyTorch training script to run on multiple GPUs using HuggingFace Accelerate with BF16 mixed precision.

Frequently Asked Questions about huggingface-accelerate

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I convert a PyTorch script to multi-GPU training?▼

Add four lines with HuggingFace Accelerate: import Accelerator, instantiate it, pass model/optimizer/dataloader to accelerator.prepare(), and replace loss.backward() with accelerator.backward(loss). Then run accelerate config and accelerate launch train.py.

Accelerate vs PyTorch Lightning vs DeepSpeed for distributed training?▼

Accelerate offers a minimal unified API over DDP, DeepSpeed, FSDP, and Megatron with only four lines of code changes. PyTorch Lightning provides higher-level abstractions with callbacks, while raw DeepSpeed gives direct control over advanced features.

Does Accelerate support DeepSpeed ZeRO-3 and FSDP?▼

Yes, Accelerate supports DeepSpeed ZeRO stages 2 and 3 via DeepSpeedPlugin or a JSON config, and FSDP via FullyShardedDataParallelPlugin with configurable sharding strategies and auto-wrap policies. Both work with BF16 mixed precision.

Why is gradient accumulation not working with Accelerate?▼

Gradient accumulation requires wrapping the training step in the accelerator.accumulate(model) context manager and setting gradient_accumulation_steps in the Accelerator constructor. Without the context manager, gradients synchronize every step.

Can Accelerate train large models with Megatron tensor parallelism?▼

Yes, Accelerate integrates Megatron-LM through MegatronLMPlugin, supporting tensor parallelism, pipeline parallelism, sequence parallelism, and activation checkpointing. It requires installing Megatron-LM and NVIDIA Apex, and TP works best with NVLink-connected GPUs.

Why does my Accelerate training run out of memory?▼

Reduce memory by enabling gradient checkpointing, increasing gradient accumulation steps, using BF16 mixed precision, or switching to FSDP or DeepSpeed ZeRO-3 with CPU offloading. Flash Attention 2 also cuts attention memory roughly in half on A100/H100 GPUs.

Related Skills

Explore More Software Engineering Skills→
KaiserWhoLearnsKaiserWhoLearns

nanogpt-training

Train GPT-2-scale models on a single GPU with mixed precision.

Community
Advanced
rocklambrosrocklambros
2

auditing-rlhf-reward-hacking

Audit RLHF, DPO, and RLAIF checkpoints for reward hacking by comparing reward win-rates with preference win-rates.

Community
Advanced
WangJie-cnWangJie-cn

experiment-manager

Track ML experiment configurations, metrics, and GPU usage.

Community
Advanced
Orchestra-ResearchOrchestra-Research
13.0k

ray-train

Orchestrates distributed PyTorch, TensorFlow, and HuggingFace training across multi-node GPU clusters.

Official
Advanced