skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

training-llms-megatron

Trains large language models using NVIDIA Megatron-Core with tensor, pipeline, and expert parallelism.

1|Updated Mar 5, 2026
Search Tags:#distributed training#pipeline parallelism#megatron-core#tensor parallelism#mixture of experts#llm pretraining#nvidia h100
One-click install
npx skills add https://github.com/Clay-HHK/claude-skills --skill training-llms-megatron-clay-hhk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: training-llms-megatron
Source: https://github.com/Clay-HHK/claude-skills/tree/main/megatron-core
Command: npx skills add https://github.com/Clay-HHK/claude-skills --skill training-llms-megatron-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires megatron-core, torch, apex, transformer-engine, and includes references (resource) components.

What problem does it solve? Training LLMs from 2B to 462B parameters requires coordinating complex parallelism strategies across hundreds of GPUs, and misconfiguration leads to out-of-memory errors, low GPU utilization, or diverging loss. This Skill provides production-tested configurations, parallelism decision guides, and troubleshooting workflows for NVIDIA Megatron-Core. ## Core Features & Use Cases - Parallelism Configuration: Step-by-step guidance for combining tensor, pipeline, context, sequence, and expert parallelism based on model size and GPU topology. - Performance Optimization: Instructions for reaching up to 47% MFU on H100 using Flash Attention, FP8 precision, and tuned micro-batch sizes. - Production Recipes: Complete hyperparameter configurations for LLaMA 3 (8B-405B), Mixtral MoE, GPT-3, and DeepSeek-V3, plus real deployment examples from Meta, NVIDIA, and BigScience. - Use Case: You need to pretrain a 70B LLaMA-style model on 64 H100 GPUs. The Skill provides the exact TP=4/PP=4 configuration, launch script, memory estimates, and monitoring metrics to run it. ## Quick Start Ask the AI to generate a Megatron-Core training configuration for a 70B LLaMA model on 64 H100 GPUs with tensor and pipeline parallelism.

Frequently Asked Questions about training-llms-megatron

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a 70B LLaMA model with Megatron-Core?▼

Use tensor parallelism 4 and pipeline parallelism 4 across 64 GPUs with micro-batch size 1 and global batch 1024. Launch with torchrun passing --tensor-model-parallel-size 4, --pipeline-model-parallel-size 4, --sequence-parallel, and --bf16 flags.

Megatron-Core vs DeepSpeed vs PyTorch FSDP for large model training?▼

Megatron-Core targets models above 10B parameters needing maximum efficiency (over 40% MFU) with fine-grained parallelism control. FSDP suits models under 70B with a simpler PyTorch-native API, while DeepSpeed offers easier setup for models under 100B.

How do I fix out of memory errors during Megatron training?▼

Increase tensor or pipeline parallelism to split the model across more GPUs, enable gradient checkpointing with --recompute-granularity full and --recompute-method block, or reduce micro-batch size. CPU optimizer offloading via --cpu-optimizer is also supported.

Why is my MFU below 30% on H100 GPUs?▼

Low MFU usually comes from a micro-batch size that is too small, missing Flash Attention, or excessive tensor parallelism across nodes. Increase micro-batch size, enable --sequence-parallel, and keep tensor parallelism at 8 or below within a single NVLink node.

Does Megatron-Core support Mixture of Experts training?▼

Yes, Megatron-Core supports MoE through expert parallelism via --expert-model-parallel-size, --num-experts, and --moe-router-topk flags. Expert parallelism distributes experts across GPUs, cutting per-GPU memory by up to 75% for models like Mixtral 8x7B.

What hardware is required for Megatron-Core training?▼

Megatron-Core requires NVIDIA Ampere or newer GPUs (A100, H100, B200), with FP8 training limited to Hopper and newer. Multi-node training needs InfiniBand or 400Gb+ Ethernet, and 70B+ models need fast NVMe storage for checkpoints.

Related Skills

Explore More Software Engineering Skills→
Supporter09Supporter09

sentencepiece

Train BPE or Unigram tokenizers on raw Unicode text.

Community
Intermediate
RachasumanthRachasumanth

inference-deploy

Deploy trained machine learning models for inference using vLLM, TGI, Ollama, and llama.cpp.

Community
Advanced
TabithaFannyTabithaFanny

training-check

Monitor WandB training metrics for NaN values, loss divergence, and idle GPUs.

Community
Advanced
Hung-3008Hung-3008
1

pytorch-lightning

Automate PyTorch Lightning training workflows with distributed strategies and checkpointing.

Community
Advanced