nvidia-resiliency-ext

What problem does it solve?

This Skill provides tools and utilities to enhance the resiliency of NVIDIA applications, particularly in distributed computing environments, by offering fault tolerance mechanisms and diagnostic capabilities.

Core Features & Use Cases

• Fault Injection: Simulate various failure scenarios (e.g., rank failures, hangs) to test application robustness.
• Checkpointing Utilities: Asynchronous and robust checkpointing mechanisms for saving and resuming training states.
• Monitoring and Diagnostics: Tools for monitoring process health, capturing logs, and analyzing execution traces.
• Use Case: During a large-scale distributed training job, simulate a GPU failure to ensure the training process can gracefully recover and resume from the last checkpoint without significant data loss.

Quick Start

Use the nvidia-resiliency-ext skill to inject a simulated GPU error after a 60-second delay.