agency-data-engineer

Use Delta Lake merge operations keyed on primary keys instead of blind appends, so rerunning a pipeline updates existing rows rather than duplicating them. Combine this with append-only Bronze ingestion and deterministic deduplication via window functions in the Silver layer.

Medallion Architecture organizes data into Bronze (raw, append-only), Silver (cleansed, deduplicated, conformed), and Gold (business-ready aggregates) layers. Each layer has explicit contracts, and consumers should only read from Gold, never directly from Bronze or Silver.

Define dbt model contracts with enforced column types, not-null and unique constraints, and tests like dbt_expectations range checks. Add Great Expectations validation in the pipeline that raises an exception and blocks promotion when critical checks fail.

Delta Lake integrates tightly with Spark and Databricks features like liquid clustering, while Iceberg offers broader cross-engine compatibility across query engines. Choose based on your engine ecosystem; both support time travel, schema evolution, and ACID merges.

Yes. Use spark.readStream with the Kafka source, parse JSON payloads against an explicit schema, and write to Delta with a checkpoint location and a processing-time trigger. This gives micro-batch ingestion with exactly-once semantics via checkpointing.

When source schemas change without enforced contracts, new or renamed columns propagate silently and break joins or aggregations downstream. Enforcing dbt contracts and alerting on mergeSchema changes surfaces drift before it reaches Gold-layer consumers.