distributed-llm-pretraining-torchtitan

Use TorchTitan with a TOML config specifying the model flavor, dataset, and parallelism degrees, then launch with torchrun or the run_train.sh script. FSDP2 shards the model across all GPUs by default when data_parallel_shard_degree is set to -1.

TorchTitan is PyTorch-native with composable 4D parallelism and no third-party dependencies, making it easier to modify and integrate with torchtune and HuggingFace. Megatron-LM may offer maximum performance for NVIDIA-only deployments but with a heavier framework.

Yes, TorchTitan supports Float8 training on H100 GPUs via torchao with tensorwise or rowwise scaling. Benchmarks show up to 48% throughput improvement when combined with torch.compile, though small layers should be filtered out.

Training auto-resumes from the latest checkpoint in the configured folder when checkpointing is enabled. To resume from a specific step, set load_step in the checkpoint config section of your TOML file.

Out-of-memory errors occur when activation memory exceeds GPU capacity. Enable full activation checkpointing, reduce local_batch_size, use gradient accumulation, or increase parallelism degrees to shard the model across more GPUs.

Yes, TorchTitan supports saving directly in HuggingFace format during training or offline conversion using the convert_to_hf.py and convert_from_hf.py scripts. Checkpoints also interoperate with torchtune for fine-tuning.