skills.rest
aminojagh
aminojaghCOMMUNITY
Explore all skills by this author →
GitHub

debug-pipeline

Diagnose dlt pipeline failures by inspecting traces, load packages, schema, and job errors.

Updated Jun 15, 2026
Search Tags:#troubleshooting#data ingestion#etl#rest api#dlt#pipeline debugging#dlthub cli
One-click install
npx skills add https://github.com/aminojagh/LLMZC --skill debug-pipeline-aminojagh
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: debug-pipeline
Source: https://github.com/aminojagh/LLMZC/tree/main/05_02_dlt_workshop/.claude/skills/debug-pipeline
Command: npx skills add https://github.com/aminojagh/LLMZC --skill debug-pipeline-aminojagh

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? When a dlt data pipeline fails, hangs, or silently loads zero rows, finding the root cause requires digging through traces, load packages, and destination error messages. This Skill provides a structured debugging workflow that increases logging verbosity, runs the pipeline, and inspects every artifact to pinpoint issues like missing credentials, paginator loops, rate limits, or wrong data selectors. ## Core Features & Use Cases - Verbosity Setup: Configures log levels, HTTP error body visibility, and progress logging in .dlt/config.toml and pipeline code before debugging, then reverts changes afterward. - Failure Diagnosis: Interprets common exceptions like ConfigFieldMissingException and PipelineFailedException, and detects infinite paginator loops, HTTP 429 retry stalls, and zero-row loads. - Post-Mortem Inspection: Uses the dlthub CLI to examine pipeline traces, load packages, failed jobs, and raw load files (INSERT VALUES, JSONL, Parquet, CSV) per destination. - Use Case: Your REST API pipeline runs for 20 minutes without finishing. The Skill identifies a JSONResponseCursorPaginator with a wrong cursor_path causing an infinite loop and shows how to fix it. ## Quick Start Ask the AI to debug your dlt pipeline by name, for example: debug my github_pipeline that keeps failing with a 401 error.

Frequently Asked Questions about debug-pipeline

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I debug a failing dlt pipeline?▼

First set log_level to INFO and enable http_show_error_body in .dlt/config.toml, then rerun the pipeline. Afterward use the dlthub CLI commands pipeline trace, load-package, and failed-jobs to inspect errors, timing, and destination messages.

Why does my dlt pipeline run forever without finishing?▼

A common cause is a paginator loop where the auto-detected paginator guesses wrong or a cursor_path never advances. Set an explicit paginator in the resource config, or add stop conditions like total_path or maximum_offset for offset paginators.

Why does my dlt pipeline succeed but load zero rows?▼

This usually means a wrong or missing data_selector, since dlt can fail silently when auto-detecting the data array in nested responses. Fix it by explicitly setting data_selector as a JSONPath to the array, such as data or results.items.

How does dlt handle HTTP 429 rate limit errors?▼

dlt automatically retries HTTP 429 responses and respects Retry-After headers, with 5 attempts and exponential backoff by default. You can tune request_max_attempts and request_backoff_factor in .dlt/config.toml for stricter APIs.

How do I check why incremental loading stopped picking up new data?▼

Run dlthub local pipeline info with the -v flag to inspect the stored last_value cursor in the resource state and verify it updates between runs. Also check logs for the Bind incremental message confirming the incremental parameter was bound.

Related Skills

Explore More Data & Analytics Skills→
dlt-hubdlt-hub
53

create-rest-api-pipeline

Generate a dlt pipeline for ingesting data from a REST API source.

Official
Intermediate
Datus-aiDatus-ai
1.5k

bi-validation

Validate BI dashboard configurations and data accuracy for Superset and Grafana.

Official
Advanced
hmzainjamilhmzainjamil
1

data-pipeline

Build automated ETL pipelines extracting, transforming, and loading data across systems.

Community
Advanced
dlt-hubdlt-hub
53

find-source

Identify dlt source connectors for APIs and data providers.

Official
Intermediate