What problem does it solve? Data pipelines built on BigQuery or Google Cloud Storage often contain hidden quality issues such as garbage values, inconsistent date formats, malformed JSON, and schema mismatches. This Skill automates profiling, cleansing, and verification so every data movement or ingestion task is grounded in real profile evidence rather than guesswork. ## Core Features & Use Cases - Dataplex-Based Profiling: Runs automated data profile scans (with dynamic sampling for tables over 1M rows) before and after transformations to detect nulls, anomalies, and distribution shifts. - SQL Transformation Generation: Produces cleaning SQL covering schema alignment, type conversion with SAFE.PARSE functions, JSON parsing, array handling, and STRUCT field mapping. - Quality Review Protocol: Validates transformations by profiling a temporary sample output table and comparing results against the source profile, iterating until anomalies are resolved. - Use Case: When copying a raw GCS export into a curated BigQuery table, the Skill profiles the source, generates cleansing SQL for malformed dates and JSON fields, then verifies the output profile shows no unexpected NULL increases. ## Quick Start Ask the agent to clean and load data from a BigQuery table or GCS path into a destination table, and it will profile the source with Dataplex, generate the cleansing SQL, and verify the results.
