skills.rest
twjarviszyra-web
twjarviszyra-webCOMMUNITY
Explore all skills by this author →
GitHub

distributed-llm-pretraining-torchtitan

Configure distributed LLM pretraining workflows with TorchTitan and parallelism strategies.

3|Updated Apr 4, 2026
Search Tags:#checkpointing#distributed-training#slurm#torchtitan#float8#fsdp2#llm-pretraining
One-click install
npx skills add https://github.com/twjarviszyra-web/hermes-unbound --skill distributed-llm-pretraining-torchtitan-twjarviszyra-web
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: distributed-llm-pretraining-torchtitan
Source: https://github.com/twjarviszyra-web/hermes-unbound/tree/main/optional-skills/mlops/torchtitan
Command: npx skills add https://github.com/twjarviszyra-web/hermes-unbound --skill distributed-llm-pretraining-torchtitan-twjarviszyra-web

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

TorchTitan simplifies large-scale LLM pretraining by giving teams a PyTorch-native way to run distributed training without assembling separate sharding, checkpointing, and orchestration systems.

Core Features & Use Cases

  • 4D Parallelism: Combine FSDP2, tensor parallelism, pipeline parallelism, and context parallelism for efficient scaling.
  • Performance Optimization: Enable torch.compile and Float8 training to improve throughput on H100 and newer hardware.
  • Checkpointing and Portability: Use distributed checkpointing, seed checkpoints, and Hugging Face conversion for recovery and interoperability.
  • Use Cases: Pretrain Llama 3.1 at single-node or multi-node scale, run SLURM-based 70B and 405B jobs, or add a custom model with TrainSpec and state-dict adapters.

Quick Start

Ask for a TorchTitan training setup for your model, GPU count, parallelism plan, and checkpointing strategy.

Frequently Asked Questions about distributed-llm-pretraining-torchtitan

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I configure distributed LLM pretraining with 4D parallelism using PyTorch?▼

Distributed LLM pretraining with 4D parallelism combines FSDP2, tensor parallelism, pipeline parallelism, and context parallelism. TorchTitan provides a PyTorch-native way to configure these dimensions together for efficient scaling across single-node and multi-node setups.

Does TorchTitan support multi-node LLM pretraining on SLURM for large models like Llama 405B?▼

Yes, TorchTitan supports multi-node LLM pretraining orchestrated via SLURM for 70B and 405B parameter models. It integrates distributed checkpointing and Float8 training for high-throughput execution on H100 and newer hardware.

What is the best way to enable Float8 training and torch.compile for LLM pretraining?▼

Float8 training and torch.compile are enabled through TorchTitan's TOML job configuration to optimize pretraining throughput. These performance optimizations target H100 hardware to accelerate distributed LLM training workflows.

How do I add a custom model to TorchTitan for distributed pretraining?▼

Custom models are added to TorchTitan by defining a TrainSpec and implementing state-dict adapters. This allows new architectures to integrate seamlessly with existing FSDP2 and tensor parallelism configurations.

Can I convert distributed checkpoints to Hugging Face format after pretraining?▼

Yes, TorchTitan includes Hugging Face conversion capabilities for distributed checkpoints. It supports both seed checkpoints and async checkpointing, ensuring model portability and recovery for downstream interoperability.

Related Skills

Explore More Software Engineering Skills→
cxnaivecxnaive

distributed-llm-pretraining-torchtitan

Configure and launch distributed LLM pretraining with PyTorch 4D parallelism.

Community
Advanced
ovachieverovachiever
52

transformers

Load, infer, and fine-tune pre-trained transformer models across modalities.

Community
Advanced
Orchestra-ResearchOrchestra-Research
13.0k

distributed-llm-pretraining-torchtitan

Pretrains large language models at scale using PyTorch torchtitan with 4D parallelism.

Official
Advanced
ovachieverovachiever
52

training-llms-megatron

Train large language models with Megatron-Core using configurable 3D and 4D parallelism.

Community
Advanced