What problem does it solve? Moving or ingesting data from BigQuery or Google Cloud Storage without profiling leads to hidden data quality issues like garbage values, inconsistent formats, and unexpected NULLs. This Skill enforces a Dataplex-driven profiling and verification workflow so every transformation is evidence-based and validated before completion. ## Core Features & Use Cases - Dataplex-Based Profiling: Automatically scans source tables (with dynamic sampling for tables over 1M rows) to detect null rates, distinct values, and anomalies before writing any transformation. - SQL Transformation Generation: Produces cleaning SQL covering schema alignment, type conversion with SAFE.PARSE functions, JSON parsing, array handling, and STRUCT field mapping. - Quality Review Loop: Re-profiles a temporary output table after transformation and compares profiles column-by-column to catch regressions like unexpected NULL increases. - Use Case: When asked to copy a GCS dataset into a BigQuery destination table, the Skill profiles the source, generates cleansing SQL matched to the destination schema, and verifies the result with a post-transformation Dataplex scan. ## Quick Start Ask the agent to clean and load data from a BigQuery table or GCS path into a destination table, and it will profile, transform, and verify the data automatically.
