perf-torch-cuda-graphs

Reduce PyTorch GPU kernel launch overhead by capturing and replaying execution with CUDA Graphs, which records the workload once to eliminate CPU-bound iteration overhead and improve low GPU utilization.

The best way to capture CUDA Graphs depends on your stack: use torch.compile(mode="reduce-overhead") for native PyTorch, Megatron-LM CudaGraphManager for large models, or manual torch.cuda.graph() for custom loops.

Yes, CUDA Graph capture works with Transformer Engine FP8 training by using TE's make_graphed_callables, ensuring proper warmup steps and deterministic replay order to handle FP8 scaling factors correctly.

CUDA Graph replay fails or shows no speedup with DDP and pipeline parallelism due to sync points, wrong replay order, or memory fragmentation, requiring targeted troubleshooting to remove sync-free constraint violations.

Avoid using CUDA Graphs for PyTorch inference when workloads are dynamic, require frequent synchronization, or have variable tensor shapes, as graph capture demands static constraints and sync-free execution for deterministic replay.