skills.rest
Lathika-laa
Lathika-laaCOMMUNITY
Explore all skills by this author →
GitHub

data-autocleaning

Profiles and cleans BigQuery and GCS data using Dataplex scans and SQL transformations.

Updated Aug 26, 2026
Search Tags:#data quality#data cleaning#bigquery#gcs#data profiling#sql transformation#dataplex
One-click install
npx skills add https://github.com/Lathika-laa/Recipe_Box --skill data-autocleaning-lathika-laa
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-autocleaning
Source: https://github.com/Lathika-laa/Recipe_Box/tree/main/.github/.gemini/skills/data-autocleaning
Command: npx skills add https://github.com/Lathika-laa/Recipe_Box --skill data-autocleaning-lathika-laa

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve? Moving or ingesting data from BigQuery or Google Cloud Storage often introduces hidden data quality issues like malformed dates, garbage values, and inconsistent units. This Skill automates profiling, cleansing, and verification so pipelines built on Dataform, dbt, or BigQuery start from trustworthy data. ## Core Features & Use Cases - Dataplex-Based Profiling: Runs automated data profile scans on source tables before and after transformation, with dynamic sampling for large tables and concurrent multi-table scans. - SQL Transformation Generation: Produces cleansing SQL covering schema alignment, type conversion with SAFE.PARSE functions, JSON parsing, array handling, and STRUCT field mapping. - Quality Review Protocol: Validates transformations by profiling a temporary sample output table and comparing results against the source profile to catch NULL increases or value shifts. - Use Case: When asked to copy a GCS dataset into a BigQuery destination table, the Skill profiles the source, generates cleaning SQL matched to the destination schema, and verifies the output with a post-transformation Dataplex scan. ## Quick Start Ask the assistant to clean and load a BigQuery or GCS source into a destination table, for example: profile and clean the data in project.dataset.source_table and load it into project.dataset.destination_table.

Frequently Asked Questions about data-autocleaning

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I clean data before loading it into BigQuery?▼

Profile the source table with a Dataplex data scan, then generate SQL transformations that handle garbage values, type conversions with SAFE.PARSE functions, and schema alignment. Verify the result by profiling a temporary sample output table and comparing it to the source profile.

How to profile BigQuery tables with Dataplex?▼

Run the dataplex_scanner.py script with table IDs in project.dataset.table format, a specific region like us-central1, and an output directory. It creates one-time data profile scans, applies sampling for tables over 1M rows, and saves results as JSON.

Does Dataplex profiling work with GCS data sources?▼

Yes, but GCS sources require creating an external table first before running the Dataplex scan. The location must also be a specific Google Cloud region, since multi-regions like us are not supported for Dataplex scans.

Can I use this with BigLake Iceberg tables?▼

Yes, the scanner script accepts BigLake Iceberg table IDs in the format project.catalog.namespace.table alongside standard BigQuery project.dataset.table identifiers. Multiple tables can be scanned concurrently in a single run.

Why does my Dataplex scan fail in a multi-region location?▼

Dataplex data profile scans do not support multi-region locations such as us or eu. Specify a single Google Cloud region like us-central1 in the location argument to create the scan successfully.

When should JSON fields be flattened in BigQuery?▼

Flatten or extract JSON fields only when a destination schema explicitly requires it. Use SAFE.PARSE_JSON to cast strings to JSON type and JSON_VALUE or JSON_QUERY accessors, keeping the original string if parsing returns NULL.

Related Skills

Explore More Data & Analytics Skills→
danielpratsdanielprats

uploadCarto

Upload GeoParquet files from a local directory to Google Cloud Storage and import them into CARTO Data Warehouse.

Community
Intermediate
teradata-labsteradata-labs
7

td-column-transformer

Generate SQL workflows for Teradata column transformations and feature engineering.

Official
Advanced
tonone-aitonone-ai
69

flux-health

Detect data quality issues and pipeline health across databases and warehouses.

Official
Intermediate
wangx7wangx7

bigquery-basics

Manage BigQuery datasets, tables, and jobs via CLI, client libraries, and MCP servers.

Community
Intermediate