skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

distributed-llm-pretraining-torchtitan

Pretrain large language models at scale using PyTorch-native 4D parallelism with TorchTitan.

1|Updated Mar 5, 2026
Search Tags:#pytorch#distributed training#llama#pretraining#torchtitan#float8#fsdp2
One-click install
npx skills add https://github.com/Clay-HHK/claude-skills --skill distributed-llm-pretraining-torchtitan-clay-hhk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: distributed-llm-pretraining-torchtitan
Source: https://github.com/Clay-HHK/claude-skills/tree/main/torchtitan
Command: npx skills add https://github.com/Clay-HHK/claude-skills --skill distributed-llm-pretraining-torchtitan-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch>=2.6.0, torchtitan>=0.2.0, torchao>=0.5.0, and includes references (resource) components.

What problem does it solve? Pretraining large language models from scratch requires coordinating complex distributed training strategies across hundreds of GPUs, and configuring FSDP, tensor parallelism, pipeline parallelism, and Float8 quantization correctly is error-prone and time-consuming. ## Core Features & Use Cases - 4D Parallelism Configuration: Compose FSDP2, tensor parallelism, pipeline parallelism, and context parallelism for models from 8B to 405B+ parameters across 8 to 512+ GPUs. - Float8 and torch.compile Optimization: Enable Float8 training with tensorwise or rowwise scaling on H100 GPUs for up to 48% throughput gains. - Checkpoint Management: Use distributed checkpointing with async saves, HuggingFace conversion, seed checkpoints for pipeline parallelism, and resharding across parallelism configurations. - Use Case: Pretrain Llama 3.1 70B on 256 GPUs by generating a TOML config with FSDP plus tensor parallelism, launching via SLURM, and monitoring training through TensorBoard. ## Quick Start Ask the assistant to configure and launch a TorchTitan pretraining run for Llama 3.1 8B on 8 GPUs with a TOML config and checkpointing enabled.

Frequently Asked Questions about distributed-llm-pretraining-torchtitan

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I pretrain Llama 3.1 8B on multiple GPUs with TorchTitan?▼

Download the tokenizer with the download_hf_assets.py script, create a TOML config specifying the llama3 model and 8B flavor, then launch with torchrun using 8 processes per node. Training logs are written to TensorBoard under the outputs directory.

TorchTitan vs Megatron-LM for large-scale pretraining?▼

TorchTitan is PyTorch-native with composable 4D parallelism and no third-party dependencies, making it easier to modify. Megatron-LM targets maximum performance on NVIDIA-only deployments but involves a heavier framework.

Does TorchTitan support Float8 training on H100 GPUs?▼

Yes, TorchTitan supports Float8 training on H100 GPUs via torchao with tensorwise or rowwise scaling. Enable it with the quantize.linear.float8 converter plus torch.compile, which yields up to 48% throughput improvement over FSDP baselines.

How do I convert TorchTitan checkpoints to HuggingFace format?▼

Use the convert_to_hf.py script in scripts/checkpoint_conversion with the model name and flavor, or set last_save_in_hf = true in the checkpoint config to save directly in HuggingFace format during training.

Why does TorchTitan run out of memory on large models?▼

Out-of-memory errors occur when activation memory exceeds GPU capacity. Enable full activation checkpointing, reduce local_batch_size, or use gradient accumulation with a larger global_batch_size to fit the model.

When should I not use TorchTitan for training?▼

Avoid TorchTitan for fine-tuning tasks, where Axolotl or TRL are better suited, and for small-scale educational experiments where LitGPT is simpler. It is designed specifically for large-scale pretraining from scratch.

Related Skills

Explore More Education & Research Skills→
afel6afel6

distributed-llm-pretraining-torchtitan

Enable PyTorch-native distributed LLM pretraining across multi-GPU clusters.

Community
Advanced
itallstartedwithaideaitallstartedwithaidea
31

ml-model-integration

Select and deploy HuggingFace Hub models with task-first filtering and evaluation.

Community
Advanced
yo-stevenyo-steven

mlm-bridge-training

Correlate Megatron-LM and Megatron Bridge training runs by matching configs and diagnosing loss divergence.

Community
Advanced
arsityarsity

huggingface-accelerate

Run distributed PyTorch training across DDP, DeepSpeed, FSDP, and Megatron backends.

Community
Advanced