distributed-llm-pretraining-torchtitan

Pretrain large language models using 4D parallelism by combining FSDP2, Tensor Parallel, Pipeline Parallel, and Context Parallel. This approach enables scalable distributed training across up to 512 GPUs using the torchtitan library.

Distributed LLM pretraining with torchtitan supports Float8 through the torchao library, delivering a 30-50% speedup on H100 GPUs. This feature accelerates training while maintaining efficiency for large models.

The best way to scale LLM pretraining across 512 GPUs is utilizing torchtitan with 4D parallelism. It combines FSDP2, Tensor, Pipeline, and Context Parallelism to efficiently distribute workloads for models like Llama and DeepSeek.

You can use distributed checkpointing during LLM pretraining to ensure fault tolerance. This mechanism saves model states efficiently across multiple GPUs, allowing training to resume without losing progress.

You need PyTorch version 2.6.0 or higher, torchtitan 0.2.0 or higher, and torchao 0.5.0 or higher to run distributed LLM pretraining. These dependencies provide the required 4D parallelism and Float8 functionalities.

Start distributed training for a Llama 3.1 model by executing torchrun with the nproc_per_node flag set to 8. You must also specify your job configuration file, such as a TOML file, to launch the pretraining process.