02-merge-patterns

What problem does it solve?

This Skill provides robust and production-grade patterns for performing MERGE operations when transitioning data from the Silver to the Gold layer in a data lakehouse, ensuring data integrity and consistency.

Core Features & Use Cases

• SCD Type 1 & 2: Implements patterns for handling slowly changing dimensions, allowing for either overwriting historical data or tracking changes over time.
• Fact Table Aggregation: Offers templates for aggregating transactional data from Silver into summarized fact tables in the Gold layer.
• Schema Evolution: Provides guidance on handling data type changes and adding derived columns during the merge process.
• Use Case: When migrating customer data from a Silver table to a Gold dimension table, use this Skill to correctly apply SCD Type 2 logic, ensuring that historical changes in customer addresses or status are preserved and queryable.

Quick Start

Use the 02-merge-patterns skill to implement an SCD Type 1 merge for the dim_product table.