04-grain-validation

What problem does it solve?

This Skill prevents costly data errors and table rewrites in the Gold layer by validating that the grain of DataFrames matches the DDL PRIMARY KEY before executing a MERGE operation.

Core Features & Use Cases

• Pre-Merge Grain Validation: Catches mismatches between DataFrame grain and DDL PRIMARY KEY before data is merged.
• Grain Inference: Analyzes DDL to understand the expected grain type (transaction, aggregated, snapshot).
• Common Mistake Prevention: Provides patterns to fix issues like aggregated DDL with transaction scripts, or vice-versa.
• Use Case: When merging a daily aggregated sales fact table, this skill ensures the incoming DataFrame is correctly grouped by date_key, store_key, and product_key as defined in the table's PRIMARY KEY, preventing duplicate or incorrectly aggregated rows.

Quick Start

Use the 04-grain-validation skill to validate the grain of the DataFrame against the 'fact_sales_daily' table's PRIMARY KEY before performing a MERGE operation.