amee-joshi-data-engineering-portfolio

Medallion architecture uses three Delta Lake layers: Bronze for raw ingestion with audit columns, Silver for cleansing and deduplication, and Gold for analytics-ready dimensional models. The portfolio provides PySpark code for each layer including schema enforcement and merge patterns.

Use DeltaTable.merge with whenMatchedUpdate to expire current records by setting is_current to false and end_date, then whenNotMatchedInsertAll to add new versions. The source data needs effective_date, end_date, and is_current columns prepared before the merge.

Store the last processed watermark value in a control table, filter source data where the watermark column exceeds that value, merge results into the target Delta table, then update the watermark. This avoids full reloads on each pipeline run.

Yes, ADF supports parameterized pipelines where table names, schemas, and target paths come from a JSON configuration. The portfolio shows how to generate copy activities dynamically from a pipeline_config containing source system, watermark column, and table definitions.

Slow merges usually result from small file fragmentation. Run optimize with executeCompaction before merging, enable auto-optimize and auto-compact table properties, and use Z-ordering on frequently filtered columns like date_key and customer_key.

Standard checks include null validation on key columns, duplicate detection on business keys, referential integrity against dimension tables, and value range validation. The portfolio's DataQualityChecker class implements all four checks and returns pass/fail results per rule.