What problem does it solve? Data pipelines often fail silently, lose records, or produce metrics nobody can trust because contracts, quality checks, and operational behavior were never defined. This Skill guides the design, implementation, and review of data systems whose correctness, lineage, and reliability remain inspectable over time. ## Core Features & Use Cases - Data Contract Definition: Establish producers, consumers, grain, keys, event time, update semantics, retention, privacy classification, and freshness needs before choosing technologies. - Pipeline & Modeling Guidance: Apply idempotent ingestion, watermarking, backfill safety, partitioning, and explicit grain definitions for dimensions, facts, and derived metrics. - Data Quality & Operations: Define actionable checks for schema conformance, uniqueness, completeness, referential integrity, and freshness, plus instrumentation for throughput, latency, cost, and lineage. - Use Case: When building a new streaming ingestion pipeline into a warehouse, use this Skill to define the data contract, design duplicate handling and late-arrival behavior, assign quality check owners, and plan schema evolution and rollback. ## Quick Start Ask the AI to review your pipeline design or plan a new data pipeline using the senior data engineer skill, describing your sources, consumers, and latency requirements.
