training-llms-megatron

What problem does it solve?

This Skill addresses the immense computational and memory challenges of training large language models (LLMs) by providing a robust framework for distributed training with advanced parallelism strategies.

Core Features & Use Cases

• Massive Model Training: Train models from 2 billion to over 400 billion parameters.
• GPU Efficiency: Achieves high Model FLOP Utilization (MFU), up to 47% on H100 GPUs, maximizing hardware investment.
• Advanced Parallelism: Implements Tensor Parallelism (TP), Pipeline Parallelism (PP), Sequence Parallelism (SP), Context Parallelism (CP), and Expert Parallelism (EP) for optimal scaling.
• Use Case: Train a 70B parameter LLaMA-style model on a cluster of 64 H100 GPUs using a combination of TP, PP, and DP to achieve maximum training throughput and efficiency.

Quick Start

Use the training-llms-megatron skill to train a LLaMA-style model with 3D parallelism on 64 GPUs.