skills.rest
sayalinvidia
sayalinvidiaCOMMUNITY
Explore all skills by this author →
GitHub

nemo-mbridge-perf-cuda-graphs

Implements CUDA graph capture for Megatron Bridge with Transformer Engine and RNG tracking.

Updated Apr 16, 2026
Search Tags:#gpu-training#cuda-graphs#megatron-bridge#te-scoped-graphs#local-full-iteration#performance-harness#rng-tracker
One-click install
npx skills add https://github.com/sayalinvidia/sayali-skills-test --skill nemo-mbridge-perf-cuda-graphs
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-mbridge-perf-cuda-graphs
Source: https://github.com/sayalinvidia/sayali-skills-test/tree/main/skills/nemo-mbridge-perf-cuda-graphs
Command: npx skills add https://github.com/sayalinvidia/sayali-skills-test --skill nemo-mbridge-perf-cuda-graphs

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

CUDA graph capture reduces host-driver overhead in Megatron Bridge training by freezing and replaying GPU workloads to optimize throughput and stability.

Core Features & Use Cases

  • Local full-iteration graphs using MCore wrappers to capture forward and backward passes.
  • Transformer Engine scoped graphs per layer for attn, mlp, moe, moe_router, moe_preprocess, and mamba.
  • Clear enablement prerequisites, including static tensor shapes and RNG trackers, to ensure reliable replay.

Quick Start

Launch the TE-scoped or local full-iteration CUDA graphs path with static shapes and enable the RNG tracker to run the performance harness and validate replay.

Frequently Asked Questions about nemo-mbridge-perf-cuda-graphs

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I reduce host-driver overhead during Megatron Bridge GPU training?▼

You can reduce host-driver overhead by implementing CUDA graph capture to freeze and replay GPU workloads, optimizing throughput and stability during Megatron Bridge training.

What is the difference between TE-scoped and local full-iteration CUDA graphs?▼

TE-scoped graphs capture per-layer operations for attention, MLP, and MoE modules, while local full-iteration graphs use MCore wrappers to capture entire forward and backward passes.

Do I need static tensor shapes to enable CUDA graph capture?▼

Yes, static tensor shapes are a mandatory prerequisite. You must also configure an RNG tracker to ensure correct capture and replay behavior.

How do I validate CUDA graph replay behavior?▼

You can validate replay behavior by running the performance harness with static shapes and the RNG tracker enabled on either the TE-scoped or local full-iteration path.

Which Transformer Engine modules support scoped CUDA graph capture?▼

Scoped CUDA graph capture supports attention, MLP, MoE, MoE router, MoE preprocess, and Mamba modules within the Transformer Engine layers.

Related Skills

Explore More Software Engineering Skills→
NVIDIANVIDIA
2.8k

cuda-graphs

Capture and replay GPU operations to reduce host overhead in Megatron Bridge training.

Official
Advanced
tianhao909tianhao909
1

miles-rl-training

Train large-scale Mixture-of-Experts models with low-precision RL and Rollout Routing Replay.

Community
Advanced
yo-stevenyo-steven

perf-torch-cuda-graphs

Captures and replays PyTorch GPU workloads using CUDA Graphs to reduce launch overhead.

Community
Advanced
NVIDIA-NeMoNVIDIA-NeMo
852

perf-cuda-graphs

Capture CUDA graphs to reduce host overhead during GPU model training.

Official
Advanced