What problem does it solve? Data pipelines often break silently under retries, late-arriving data, and schema drift, corrupting downstream consumers without anyone noticing. This Skill applies senior data-engineering discipline to build pipelines and warehouse structures that preserve correctness, lineage, and data contracts under real operational conditions. ## Core Features & Use Cases - Pipeline and Orchestration Design: Maps source-to-sink flows, designs dependency graphs, retry policies, and backfill mechanics for unattended operation. - Idempotency and Data Quality: Ensures reruns do not duplicate or drop records, and builds completeness, uniqueness, and anomaly checks directly into the pipeline. - Warehouse Modeling: Designs dimensional models, dbt transformation layers, and schema evolution strategies that protect downstream consumers. - Use Case: When a nightly ETL job occasionally double-counts records after a retry, use this Skill to trace the failure mode, implement an idempotent load pattern, and add a dead-letter path for malformed records. ## Quick Start Ask the agent to review this ETL pipeline for idempotency and data-quality gaps and propose the smallest safe fix.
