skills.rest
Nzettodess
NzettodessCOMMUNITY
Explore all skills by this author →
GitHub

training-llms-megatron

Trains large language models using NVIDIA Megatron-Core with tensor, pipeline, and expert parallelism.

2|Updated Jan 26, 2026
Search Tags:#llm#distributed training#megatron-core#tensor parallelism#h100#mixture of experts#gpu cluster
One-click install
npx skills add https://github.com/Nzettodess/Awesome-Agent-Skills --skill training-llms-megatron-nzettodess
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: training-llms-megatron
Source: https://github.com/Nzettodess/Awesome-Agent-Skills/tree/main/Skills/Agentic%20AI%20Development/AI-research-SKILLs-1.1.0/08-distributed-training/megatron-core
Command: npx skills add https://github.com/Nzettodess/Awesome-Agent-Skills --skill training-llms-megatron-nzettodess

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires megatron-core, torch, apex, transformer-engine, and includes references (resource) components.

What problem does it solve? Training LLMs from 2B to 462B parameters requires coordinating parallelism across hundreds of GPUs, and misconfigured tensor, pipeline, or expert parallelism leads to out-of-memory errors, low GPU utilization, and wasted compute budget. ## Core Features & Use Cases - Parallelism Configuration: Provides decision tables and commands for combining tensor, pipeline, data, context, sequence, and expert parallelism based on model size and GPU topology. - Production Training Recipes: Ships complete hyperparameter configurations for GPT-3, LLaMA-3 (8B/70B/405B), Mixtral MoE, and DeepSeek-V3 architectures. - Performance Optimization: Documents how to reach 40-47% MFU on H100 using Flash Attention, FP8 precision, interleaved pipeline schedules, and gradient checkpointing. - Use Case: A team training a 70B LLaMA-style model on 64 H100 GPUs uses the provided TP=4/PP=4 configuration and launch script to start distributed training without OOM errors. ## Quick Start Ask the agent to generate a Megatron-Core training configuration and launch script for a 70B LLaMA model on 64 H100 GPUs.

Frequently Asked Questions about training-llms-megatron

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a 70B LLaMA model on multiple GPUs with Megatron-Core?▼

Use tensor parallelism of 4 and pipeline parallelism of 4 across 64 GPUs, with micro-batch size 1 and global batch size 1024. Launch with torchrun passing --tensor-model-parallel-size 4, --pipeline-model-parallel-size 4, --sequence-parallel, and --use-mcore-models flags.

Megatron-Core vs DeepSpeed vs PyTorch FSDP for large model training?▼

Megatron-Core suits models above 10B parameters needing maximum efficiency and fine-grained parallelism control on NVIDIA GPUs. FSDP fits models under 70B with a simpler PyTorch-native API, while DeepSpeed offers easier setup for models under 100B.

How do I configure expert parallelism for Mixture of Experts training?▼

Set --expert-model-parallel-size to split experts across GPUs, for example EP=4 for Mixtral 8x7B reduces per-GPU memory from 56GB to 14GB. Combine with --num-experts 8, --moe-router-topk 2, and --moe-aux-loss-coeff 0.01 for load balancing.

Why is my Megatron training MFU below 30 percent?▼

Low MFU usually comes from micro-batch size being too small, missing Flash Attention, or excessive tensor parallelism across nodes. Increase micro-batch size, enable --sequence-parallel and transformer_engine, and keep TP at 8 or below within a single NVLink node.

How do I fix out of memory errors during Megatron training?▼

Increase tensor or pipeline parallelism to split the model further, or enable gradient checkpointing with --recompute-granularity full and --recompute-method block. You can also reduce micro-batch size or offload the optimizer to CPU with --cpu-optimizer.

Does Megatron-Core support FP8 training on H100 GPUs?▼

Yes, FP8 hybrid training is supported on Hopper, Ada, and Blackwell GPUs through Transformer Engine. Enable it with --fp8-hybrid, which typically delivers 1.5-2x speedup over BF16 with minimal quality degradation.

Related Skills

Explore More Software Engineering Skills→
qcmuuqcmuu

nanogpt

Train minimalist GPT-2-style language models with PyTorch and configurable hyperparameters.

Community
Intermediate
RachasumanthRachasumanth

inference-deploy

Deploy trained machine learning models for inference using vLLM, TGI, Ollama, and llama.cpp.

Community
Advanced
TabithaFannyTabithaFanny

training-check

Monitor WandB training metrics for NaN values, loss divergence, and idle GPUs.

Community
Advanced
adm-humanerdadm-humanerd

huggingface-accelerate

Unify distributed training workflows for PyTorch models with HuggingFace Accelerate.

Community
Advanced