distributed-llm-pretraining-torchtitan

Download the tokenizer with the download_hf_assets script, create a TOML config specifying the llama3 model and 8B flavor, then launch with torchrun using 8 processes per node. Training logs to TensorBoard and checkpoints save at configured intervals.

TorchTitan is PyTorch-native with composable 4D parallelism and no third-party dependencies, making it suitable for research and HuggingFace-interoperable checkpoints. Megatron-LM targets maximum performance for NVIDIA-only production deployments.

Yes, torchtitan supports Float8 training on H100 or newer GPUs via torchao, with tensorwise and rowwise dynamic scaling recipes. Combined with torch.compile, benchmarks show up to 48% throughput improvement over FSDP baselines on Llama 8B.

Use the convert_to_hf.py script with the input directory, output directory, model name, and flavor, or enable last_save_in_hf in the checkpoint config to save directly in HuggingFace format during training. Reverse conversion uses convert_from_hf.py.

Out of memory occurs when activation memory exceeds GPU capacity. Enable full activation checkpointing instead of selective, reduce local batch size, or use gradient accumulation with a larger global batch size to lower peak memory.

Avoid torchtitan for fine-tuning tasks, where Axolotl, TRL, or torchtune are better suited, and for inference workloads where DeepSpeed or serving frameworks apply. It is designed specifically for large-scale pretraining from scratch.