skills.rest
Orchestra-Research
Orchestra-ResearchOFFICIAL
Explore all skills by this author →
GitHub

training-llms-megatron

Trains large language models using NVIDIA Megatron-Core with tensor, pipeline, and expert parallelism.

13.0k|930|Updated Nov 3, 2025
Search Tags:#distributed training#pipeline parallelism#megatron-core#tensor parallelism#h100#mixture of experts#llm pretraining
One-click install
npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill training-llms-megatron-orchestra-research
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: training-llms-megatron
Source: https://github.com/Orchestra-Research/AI-research-SKILLs/tree/main/08-distributed-training/megatron-core
Command: npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill training-llms-megatron-orchestra-research

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires megatron-core, torch, apex, transformer-engine, and includes references (resource) components.

What problem does it solve? Training LLMs from 2B to 462B parameters requires coordinating complex parallelism strategies across hundreds of GPUs, and misconfiguration leads to out-of-memory errors, low GPU utilization, or diverging loss. ## Core Features & Use Cases - 3D/4D Parallelism Configuration: Provides tested tensor, pipeline, context, sequence, and expert parallelism setups with GPU-count decision tables for models from 7B to 405B. - MoE Training Support: Configures expert parallelism and load balancing for Mixtral-style sparse models, reducing per-GPU memory by up to 75%. - Performance Optimization: Covers FP8 training on H100, Flash Attention, micro-batch tuning, and interleaved pipeline schedules targeting 40-47% MFU. - Use Case: A team training a LLaMA-style 70B model on 64 H100 GPUs uses the provided TP=4/PP=4 configuration and launch scripts to start distributed training with monitored throughput metrics. ## Quick Start Train a LLaMA 70B model on my 64-GPU H100 cluster using Megatron-Core with the recommended tensor and pipeline parallelism configuration.

Frequently Asked Questions about training-llms-megatron

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a 70B LLM across multiple GPUs with Megatron-Core?▼

Use tensor parallelism within nodes and pipeline parallelism across nodes, such as TP=4 and PP=4 on 64 GPUs for a 70B model. Launch with torchrun specifying --tensor-model-parallel-size, --pipeline-model-parallel-size, and --sequence-parallel flags.

Megatron-Core vs DeepSpeed vs PyTorch FSDP for large model training?▼

Megatron-Core targets models above 10B parameters needing maximum efficiency and fine-grained parallelism control on NVIDIA GPUs. FSDP suits models under 70B with a simpler PyTorch-native API, while DeepSpeed offers easier setup for models under 100B.

Does Megatron-Core support Mixture of Experts training?▼

Yes, Megatron-Core supports MoE through expert parallelism via --expert-model-parallel-size, --num-experts, and --moe-router-topk flags. Expert parallelism distributes experts across GPUs, cutting per-GPU memory by up to 75% for models like Mixtral 8x7B.

Why is my Megatron training MFU below 30 percent?▼

Low MFU usually comes from a micro-batch size that is too small, missing Flash Attention, or excessive tensor parallelism across nodes. Increase micro-batch size, enable --sequence-parallel and Flash Attention, and keep TP at 8 or below within a single node.

Can I use FP8 training with Megatron-Core?▼

FP8 training requires Hopper, Ada, or Blackwell GPUs such as the H100 and is enabled with --fp8-hybrid using Transformer Engine. It delivers 1.5-2x speedup over BF16 with minimal quality degradation when properly scaled.

How do I fix out-of-memory errors during Megatron training?▼

Increase tensor or pipeline parallelism to split the model further, enable gradient checkpointing with --recompute-granularity full, or reduce micro-batch size. CPU optimizer offloading via --cpu-optimizer is also available as a fallback.

Related Skills

Explore More Software Engineering Skills→
travelinman1013travelinman1013

nemo-curator

Curate multimodal LLM training data with GPU-accelerated deduplication and filtering.

Community
Advanced
NVIDIANVIDIA
3.7k

deployment

Serve quantized or unquantized LLM checkpoints as OpenAI-compatible endpoints using vLLM, SGLang, or TRT-LLM.

Official
Advanced
wanshuiyinwanshuiyin
15.6k

training-check

Monitors WandB training metrics periodically to detect NaN, divergence, and idle GPUs.

Community
Intermediate
kngender5kngender5

colab-distributed

Distribute machine learning training across GPUs in Google Colab using DDP, DeepSpeed ZeRO, and FSDP.

Community
Advanced