troubleshoot-nvidia-gpu

Triage Nvidia GPU failures by mapping Netdata MCP metrics to signal-domain rule files.

1|Updated Apr 17, 2026
One-click install
npx skills add https://github.com/netdata/skills --skill troubleshoot-nvidia-gpu
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: troubleshoot-nvidia-gpu
Source: https://github.com/netdata/skills/tree/main/skills/troubleshoot-nvidia-gpu
Command: npx skills add https://github.com/netdata/skills --skill troubleshoot-nvidia-gpu

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

It helps you triage and remediate Nvidia GPU issues by identifying the dominant failure archetype (OOM, thermal-throttle cascade, silent degradation, interconnect straggler, or GPU hang) using Netdata signals accessed through an MCP query flow.

Core Features & Use Cases

  • Structured Nvidia Gpu triage tree: Routes an agent through domain-specific rule files aligned to the Netdata operator playbook’s signal domains (availability, thermal/power, memory, compute/utilization, interconnect, errors/Xid, plus support domains).
  • MCP-driven verification: Uses MCP calls to discover nodes, pull the last-window metrics, rank anomalies, correlate signals, and confirm whether remediation actually returns signals to expected ranges.
  • Failure-mode coverage: Targets common real-world incidents like memory exhaustion (OOM), thermal runaway with throttling, ECC-related silent corruption, PCIe/NVLink degradation patterns, and catastrophic “fallen off bus” behaviors (including XID 79).
  • Operational on-call workflow: Designed for situations where an on-call engineer needs a paging-ready, evidence-based triage path tied to Netdata alerts on a specific GPU host.

Quick Start

Use the troubleshoot-nvidia-gpu skill to diagnose a failing Nvidia GPU by querying the Netdata instance through MCP for the last 15 to 30 minutes of nvidia_smi signals, matching anomalies to the appropriate rule-file domain, and running the verification queries to confirm remediation effectiveness.

Frequently Asked Questions about troubleshoot-nvidia-gpu

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I troubleshoot an Nvidia GPU that has fallen off the bus or hung?▼

To troubleshoot an Nvidia GPU hang or fallen-off-bus scenario, map observed Xid errors to the Netdata operator playbook’s signal-domain triage tree using MCP queries for nvidia_smi metrics to verify remediation effectiveness.

What is the best way to diagnose silent ECC memory degradation on Nvidia GPUs?▼

Diagnosing silent ECC memory degradation requires querying Netdata through MCP for recent nvidia_smi signals, ranking anomalies, and cross-correlating memory errors against the operator playbook's triage tree.

How can I identify thermal runaway and throttling cascades on an Nvidia GPU?▼

Identify thermal runaway and throttling cascades by applying the Netdata operator playbook’s thermal/power signal domain rules to nvidia_smi metrics retrieved via MCP, correlating temperature spikes with throttling behaviors.

Does this Nvidia GPU triage workflow support NVLink and PCIe interconnect stragglers?▼

Yes, the Nvidia GPU triage workflow supports NVLink and PCIe interconnect stragglers by mapping interconnect degradation patterns to the Netdata operator playbook and verifying signal normalization through MCP-based metric retrieval.

How do I verify if an OOM remediation resolved an Nvidia GPU memory exhaustion issue?▼

Verify OOM remediation by running post-remediation MCP verification queries against the Netdata instance to confirm that nvidia_smi memory metrics and anomalies return to expected load-bearing ranges.