pipeline-review

Audits data pipelines for idempotency, reliability, observability, and cost issues.

15|3|Updated Jul 9, 2026
One-click install
npx skills add https://github.com/kiurakku/cursor-kit-for-ai --skill pipeline-review-kiurakku
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: pipeline-review
Source: https://github.com/kiurakku/cursor-kit-for-ai/tree/main/plugins/data/skills/pipeline-review
Command: npx skills add https://github.com/kiurakku/cursor-kit-for-ai --skill pipeline-review-kiurakku

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Data pipelines fail in three ways: duplicate data, missing data, and silently wrong data. This Skill audits Airflow DAGs, dbt models, cron ETL jobs, and streaming consumers against these failure modes so you catch risks before they corrupt production data. ## Core Features & Use Cases - Idempotency Analysis: Applies the double-run test to every step, checking upserts, partition delete-and-replace patterns, and deduplication for at-least-once Kafka consumers. - Structured Review Checklist: Covers reliability (retries, dead-letter queues, backfills), correctness (schema validation, timezone discipline, join fan-out), observability (freshness alerts, anomaly guards), and cost (partition pruning, incremental vs full-refresh). - Orchestrator-Specific Guidance: Provides targeted checks for Airflow, dbt, and cron-based ETL, plus a standardized Markdown verdict report. - Use Case: When debugging duplicate rows in a warehouse table, run this review to trace which step lacks an upsert or partition-scoped delete and get a concrete fix. ## Quick Start Review this Airflow DAG for idempotency and data-loss risks and give me the verdict report.

Frequently Asked Questions about pipeline-review

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I review a data pipeline for reliability?▼

Check that every step is idempotent by mentally running it twice, verify retries use bounded exponential backoff with dead-letter handling for poison records, and confirm backfills are parameterized by logical date rather than now(). This Skill walks through the full checklist.

How to make an ETL pipeline idempotent?▼

Use upserts (INSERT ... ON CONFLICT DO UPDATE) instead of bare inserts, delete-and-replace partitions for the run's logical date, recompute aggregates from source instead of incrementing, and deduplicate stream events by event id.

What should I check when reviewing Airflow DAGs?▼

Verify tasks are atomic, no heavy compute runs in the scheduler, catchup and max_active_runs are set consciously, and no top-level DAG code performs I/O since files are parsed every 30 seconds.

Why does my pipeline produce duplicate data after retries?▼

Retries re-execute steps that already wrote partial output, so non-idempotent writes like bare INSERTs create duplicates. Fix by using upserts, partition-scoped delete-and-replace, or idempotency keys derived from run_id and record_id.

How do I detect silent data corruption in pipelines?▼

Add schema validation at the ingestion boundary, row-count assertions before and after joins to catch fan-out or fan-in, and an anomaly guard that warns when daily volume deviates over 50% from the 7-day average.