skills.rest
sayalinvidia
sayalinvidiaCOMMUNITY
Explore all skills by this author →
GitHub

nemo-mbridge-perf-parallelism-strategies

Select and tune Megatron Bridge parallelism configurations for model and hardware.

Updated Apr 16, 2026
Search Tags:#scaling#parallelism#dp#moe#pp#tp#megatron-bridge
One-click install
npx skills add https://github.com/sayalinvidia/sayali-skills-test --skill nemo-mbridge-perf-parallelism-strategies
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-mbridge-perf-parallelism-strategies
Source: https://github.com/sayalinvidia/sayali-skills-test/tree/main/skills/nemo-mbridge-perf-parallelism-strategies
Command: npx skills add https://github.com/sayalinvidia/sayali-skills-test --skill nemo-mbridge-perf-parallelism-strategies

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This skill helps engineers select and size parallelism strategies for Megatron Bridge, enabling efficient training across diverse model sizes and hardware topologies.

Core Features & Use Cases

  • Decision guidance: choose appropriate data and tensor parallelism (DP, TP) and pipeline/sequence controls (PP, SP, CP) for target models.
  • MoE support: provide guidance for expert_parallel and EP/ETP configurations on mixture-of-experts models.
  • Use Case Example: when configuring training for a 236B MoE model across 16 GPUs, the skill helps derive a robust parallelism recipe that respects topology constraints.

Quick Start

Try asking for a recommended parallelism setup given a model size, hardware topology, and target training scale.

Frequently Asked Questions about nemo-mbridge-perf-parallelism-strategies

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I configure Megatron Bridge parallelism for a large MoE model?▼

To configure Megatron Bridge parallelism for MoE models, select expert parallel (EP) and expert tensor parallel (ETP) sizes, compute minimum GPUs using PP * max(TP * CP, EP * ETP), and calculate expert data parallelism based on your hardware topology.

What is the best way to choose tensor and pipeline parallel sizes for multi-node training?▼

Choosing tensor and pipeline parallel sizes requires keeping tensor parallelism within a single NVLink domain and using pipeline parallelism to scale across multi-node InfiniBand, computing dense data parallelism as world_size divided by the product of TP, PP, and CP.

Does sequence parallelism work when tensor parallelism is set to 1?▼

Sequence parallelism does not work with tensor parallelism set to 1, because the parallelism configuration enforces a constraint that SP requires TP to be strictly greater than 1 for valid execution.

How do I calculate context parallelism constraints for sequence length?▼

To calculate context parallelism constraints, ensure the model's sequence length is divisible by 2 multiplied by the context_parallel_size, which enforces the topology and sequence-length constraints required by the configuration.

Can I use this parallelism tuner for sub-1B parameter models on a single node?▼

You can use this parallelism tuner for sub-1B parameter models on single-node NVLink hardware, as it applies to model sizes ranging from sub-1B to multi-hundred-B parameters across diverse hardware topologies.

Related Skills

Explore More Software Engineering Skills→
qcmuuqcmuu

training-llms-megatron

Configure Megatron-Core distributed training for large language models.

Community
Advanced
yo-stevenyo-steven

perf-parallelism-strategies

Map model size and hardware topology to Megatron Bridge TP/DP/PP/SP/CP/EP degrees.

Community
Advanced
ovachieverovachiever
52

training-llms-megatron

Train large language models with Megatron-Core using configurable 3D and 4D parallelism.

Community
Advanced
yo-stevenyo-steven

perf-tp-dp-comm-overlap

Configure Megatron-Bridge communication overlap for tensor, data, and pipeline parallelism.

Community
Advanced