databricks-dq-professionally

What problem does it solve?

Professional-grade data quality governance on Databricks that standardizes checks, centralizes rule definitions, and outputs a single DQ_RESULT per row to support dashboards, audits, and remediation workflows.

Core Features & Use Cases

• Generic Rules Registry: A centralized catalogue of reusable DQ checks (e.g., NOT NULL, ISO country codes, numeric bounds) that can be mapped to any table field.
• Rule Mappings: Per-field mappings that bind generic rules to specific catalog.schema.table.field combinations with optional category overrides for contextual accuracy.
• Dynamic Rule Application: Deterministic evaluation of active mappings to produce a consolidated DQ_RESULT string per row (e.g., "RULE_101: 1 | RULE_102: 0 | RULE_103: NULL").
• Schema Setup & Enrichment: Automatic workspace schema creation, source table copying into a dedicated DQ schema, and output enrichment into bikes_dq and dq_results for traceability.
• Observability & Governance: Auditable outputs and dashboards built around dq_results and DQ_OUTPUTs to support regulatory and internal quality standards.

Quick Start

Set up the DQ schema, copy the source table into the dq_professional workspace, and run the enrichment notebook to generate the bikes_dq table with a DQ_RESULT column.