skills.rest
devMoez
devMoezCOMMUNITY
Explore all skills by this author →
GitHub

distributed-llm-pretraining-torchtitan

Configure torchtitan 4D parallelism for distributed LLM pretraining.

1|Updated May 16, 2026
Search Tags:#pytorch#distributed training#pipeline parallelism#float8#fsdp2#llm pretraining#torchao
One-click install
npx skills add https://github.com/devMoez/titan --skill distributed-llm-pretraining-torchtitan-devmoez
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: distributed-llm-pretraining-torchtitan
Source: https://github.com/devMoez/titan/tree/main/optional-skills/mlops/torchtitan
Command: npx skills add https://github.com/devMoez/titan --skill distributed-llm-pretraining-torchtitan-devmoez

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch>=2.6.0, torchtitan>=0.2.0, torchao>=0.5.0, and includes references (resource) components.

What problem does it solve?

This Skill helps you pretrain large language models efficiently across many GPUs by providing a PyTorch-native torchtitan setup with composable 4D parallelism (FSDP2, tensor, pipeline, and context parallel).

Core Features & Use Cases

  • 4D parallel pretraining recipes: Run single-node and multi-node training with configurable FSDP2, TP, PP, and CP degrees to match your model size and hardware budget.
  • Deterministic distributed training primitives: Use torchrun/SLURM launch patterns, seed checkpoint creation for pipeline parallelism, and distributed checkpointing flows.
  • Performance-focused techniques: Enable Float8 training via torchao and optionally combine it with torch.compile to improve throughput on H100-class hardware.

Use case example: Pretrain a Llama 3.1-class model from scratch on 8–512+ GPUs, selecting parallelism degrees and checkpointing settings so training can resume reliably and scale from a workstation to a cluster.

Quick Start

Run distributed pretraining for an 8B-class config by executing one training launch with the provided torchtitan train config on your target GPU count using the torchtitan run_train.sh or torchrun command.

Frequently Asked Questions about distributed-llm-pretraining-torchtitan

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I pretrain LLMs from scratch using distributed PyTorch across multiple GPUs?▼

To configure 4D parallelism for LLM pretraining, you set the degrees for FSDP2, tensor, pipeline, and context parallelism. This composable approach matches your model size and hardware budget across multi-node setups.

Does torchtitan support Float8 training with torchao on H100 GPUs?▼

Yes, torchtitan supports Float8 training via torchao to improve throughput on H100-class hardware. You can optionally combine this with torch.compile integration for further performance gains during large-scale LLM pretraining.

What is the best way to resume distributed training reliably after a cluster failure?▼

The best way to resume distributed training after a cluster failure is using distributed checkpointing and seed-checkpoint workflows. Torchtitan provides these fault-tolerant primitives to ensure reliable resumption across multi-node setups.

Can I use pipeline parallelism with FSDP2 for large-sequence LLM pretraining?▼

Yes, you can use pipeline parallelism with FSDP2 for large-sequence LLM pretraining. Torchtitan enables composable 4D parallelism, allowing you to configure FSDP2 alongside tensor, pipeline, and context parallel degrees simultaneously.

Do I need specific PyTorch versions to run torchtitan distributed training?▼

Yes, you need PyTorch version 2.6.0 or higher to run torchtitan distributed training. You also need torchtitan 0.2.0 and torchao 0.5.0 to properly enable the 4D parallelism and Float8 performance features.

Related Skills

Explore More Software Engineering Skills→
Z43LZ43L

distributed-llm-pretraining-torchtitan

Configure torchtitan TOML files for distributed LLM pretraining with 4D parallelism.

Community
Advanced
Jonah-1Jonah-1

eval

Run inference on neural network checkpoints and generate PSNR, SSIM, and LPIPS metrics.

Community
Intermediate
Orchestra-ResearchOrchestra-Research
13.0k

distributed-llm-pretraining-torchtitan

Pretrains large language models at scale using PyTorch torchtitan with 4D parallelism.

Official
Advanced
zangjeicyzangjeicy

slime-rl-training

Train LLMs with GRPO and PPO using Megatron-LM and SGLang.

Community
Advanced