skills.rest
tottenjordan
tottenjordanCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

data-autocleaning

Profiles and cleans BigQuery and GCS data using Dataplex scans and SQL transformations.

1|Updated Mar 19, 2026
Search Tags:#data quality#data cleaning#bigquery#gcs#data profiling#sql transformation#dataplex
One-click install
npx skills add https://github.com/tottenjordan/me-skittles --skill data-autocleaning-tottenjordan
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-autocleaning
Source: https://github.com/tottenjordan/me-skittles/tree/main/gemini/data-autocleaning
Command: npx skills add https://github.com/tottenjordan/me-skittles --skill data-autocleaning-tottenjordan

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve? Moving or ingesting data from BigQuery or Google Cloud Storage without profiling leads to hidden data quality issues like garbage values, inconsistent formats, and unexpected NULLs propagating into Dataform, dbt, or BigQuery pipelines. ## Core Features & Use Cases - Dataplex-Based Profiling: Automatically profiles source tables with Dataplex data scans, including dynamic sampling for tables over 1M rows, before any transformation is written. - SQL Transformation Generation: Produces cleansing SQL covering schema alignment, type conversion with SAFE.PARSE functions, JSON parsing, array handling, and STRUCT field mapping. - Post-Transformation Verification: Re-profiles a temporary sample output table and compares profiles column-by-column to catch NULL increases or value range shifts. - Use Case: When asked to copy a GCS file into a BigQuery table, the agent creates an external table, runs a Dataplex profile, generates cleaning SQL matched to the destination schema, and verifies the result with a second profile before finishing. ## Quick Start Ask the agent to move or copy data from a BigQuery table or GCS path into a destination table and it will profile, clean, and verify the data automatically.

Frequently Asked Questions about data-autocleaning

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I profile a BigQuery table with Dataplex before cleaning it?▼

Run the dataplex_scanner.py script with the table in project.dataset.table format, a specific region like us-central1, and an output directory. It creates a one-time Dataplex data profile scan, polls until results are ready, and saves the profile as JSON.

How do I clean data when moving from GCS to BigQuery?▼

Create an external table over the GCS source first, then run a Dataplex profile scan on it. Use the profile findings to generate cleansing SQL with SAFE.PARSE functions and schema alignment before loading into the destination table.

Does Dataplex profiling work with multi-region locations like us?▼

No, Dataplex data profile scans require a specific Google Cloud region such as us-central1. Multi-region locations like us are not supported, so you must specify a single region when running the scan.

How are large BigQuery tables handled during profiling?▼

Tables with more than 1 million rows are automatically sampled. The scanner calculates a sampling percentage targeting roughly 1 million rows, clamps it between 0.01 and 100 percent, and passes it to the Dataplex scan to control cost.

What happens if the user denies the Dataplex scan approval?▼

The denial is documented in the implementation plan instead of a job ID, and verification falls back to manual bq sample queries. The scanner script is not run again and the user is not re-asked for approval.

Related Skills

Explore More Data & Analytics Skills→
2nth-ai2nth-ai
1

tech/google/cloud/data

Query BigQuery, publish to Pub/Sub, and run Dataflow pipelines.

Official
Advanced
vnatarajan4061vnatarajan4061

document-dbt

Profile BigQuery tables and generate dbt source and model YAML documentation.

Community
Advanced
interflownexinterflownex
2

data-autocleaning

Profile BigQuery and Cloud Storage data with Dataplex scans and generate SQL transformations.

Community
Advanced
trungenglishtrungenglish
1

data-autocleaning

Profiles and cleans BigQuery and GCS data using Dataplex scans and SQL transformations.

Community
Advanced