distributed-llm-pretraining-torchtitan

What problem does it solve?

This Skill addresses the challenge of efficiently pretraining large language models (LLMs) at scale, enabling users to train models like Llama 3.1 from 8 to 512+ GPUs.

Core Features & Use Cases

• 4D Parallelism: Supports FSDP2, Tensor Parallelism (TP), Pipeline Parallelism (PP), and Context Parallelism (CP) for optimal resource utilization.
• Advanced Training Techniques: Integrates Float8 precision and torch.compile for significant speedups on H100 GPUs.
• Use Case: Pretraining a custom LLM from scratch on a large dataset, requiring distributed training across multiple nodes and GPUs, leveraging advanced optimization techniques for faster convergence and reduced memory footprint.

Quick Start

Launch distributed LLM pretraining for Llama 3.1 8B on 8 GPUs using a custom configuration file.