dlt-expectations-patterns

What problem does it solve?

This Skill addresses the challenge of maintaining robust data quality in Databricks DLT pipelines by providing a standardized, dynamic, and manageable approach to defining and enforcing data quality expectations.

Core Features & Use Cases

• Dynamic Expectations: Load data quality rules from a Unity Catalog Delta table, allowing updates without redeploying pipeline code.
• Severity Levels: Differentiate between critical rules (which can cause data to be dropped or quarantined) and warning rules (which are logged but allow data to pass).
• Quarantine Pattern: Automatically capture records that fail critical checks into a dedicated quarantine table for investigation.
• Use Case: Implement a Silver layer DLT pipeline where critical fields like transaction_id and store_number must be present, and reasonable ranges are enforced for quantity_sold. If a transaction is missing an ID, it's quarantined with a specific reason.

Quick Start

Use the dlt-expectations-patterns skill to implement data quality checks for your Silver layer DLT pipeline by defining rules in a Delta table and loading them dynamically.