What problem does it solve? Building reliable data pipelines often fails silently: upstream schema changes corrupt tables, bad records poison downstream analytics, and retries create duplicates. This Skill enforces data contracts, isolated audit branches, and deterministic upserts so pipelines stay consistent and replayable. ## Core Features & Use Cases - Contract-First Ingestion: Validates payloads against ODCS v3.1.0 schemas at producer boundaries before any data reaches storage. - Write-Audit-Publish (WAP): Stages writes on isolated Iceberg snapshot branches, audits them in-process with DuckDB over Arrow, then atomically publishes to main. - DLQ Quarantine with Circuit Breaker: Routes malformed records to a structured dead-letter queue and halts the pipeline when the quarantine ratio exceeds 2.0%. - Use Case: You ingest high-volume order events into an Iceberg lakehouse. This Skill stages each batch on a wap_audit_<run_id> branch, quarantines contract violations, runs DuckDB audit queries, and only fast-forwards main after all checks pass. ## Quick Start Use the build-data-pipeline skill to design an Iceberg ingestion pipeline with ODCS contract validation, WAP branching, and DLQ quarantine for my orders event stream.
