skills.rest
trungenglish
trungenglishCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

data-autocleaning

Profiles and cleans BigQuery and GCS data using Dataplex scans and SQL transformations.

1|Updated Jul 4, 2026
Search Tags:#data quality#data cleaning#bigquery#gcs#data profiling#sql transformation#dataplex
One-click install
npx skills add https://github.com/trungenglish/SHOPWISE --skill data-autocleaning-trungenglish
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-autocleaning
Source: https://github.com/trungenglish/SHOPWISE/tree/main/.agents/skills/data-autocleaning
Command: npx skills add https://github.com/trungenglish/SHOPWISE --skill data-autocleaning-trungenglish

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve? Moving or ingesting data from BigQuery or Google Cloud Storage often introduces hidden quality issues like malformed dates, garbage values, inconsistent units, and schema mismatches. This Skill automates profiling, cleansing, and verification so pipelines built on Dataform, dbt, or BigQuery start from trustworthy data. ## Core Features & Use Cases - Dataplex-Based Profiling: Runs automated data profile scans (with dynamic sampling for tables over 1M rows) before and after transformation to drive data-driven cleaning decisions. - SQL Transformation Generation: Produces cleansing SQL covering schema alignment, type conversion with SAFE.PARSE functions, JSON parsing, array handling, and STRUCT field mapping. - Quality Review Protocol: Validates transformations by profiling a temporary sample output table and comparing results against the source profile to catch NULL increases or value range shifts. - Use Case: When copying a raw GCS export into a curated BigQuery table, the Skill profiles the source, generates cleaning SQL for unparseable timestamps and invalid JSON, then verifies the output resolves every detected anomaly. ## Quick Start Ask the AI to clean and load the BigQuery table my-project.raw_dataset.orders into the curated dataset using the data-autocleaning skill.

Frequently Asked Questions about data-autocleaning

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I clean data when moving tables in BigQuery?▼

Profile the source table with a Dataplex data profile scan, then generate SQL transformations addressing detected anomalies such as unparseable dates, garbage values, and unit inconsistencies. Verify the result by profiling a temporary sample output table and comparing it against the source profile.

How to profile BigQuery tables with Dataplex before transformation?▼

Run the dataplex_scanner.py script with the table IDs, a specific region like us-central1, and an output directory. It handles concurrent scans, applies dynamic sampling for tables over 1M rows, skips empty tables, and saves profile results as JSON files.

Does this work with Google Cloud Storage data sources?▼

Yes, GCS sources are supported, but you must create an external table over the GCS data before running the Dataplex scan. After profiling, the same cleansing and verification workflow applies as with native BigQuery tables.

Why does the Dataplex scan fail with a multi-region location?▼

Dataplex data profile scans do not support multi-region locations like 'us'. You must specify a specific Google Cloud region such as us-central1 when running the scanner script.

What are the limitations of automated data cleaning in BigQuery?▼

Cleaning rules avoid case conversions and only normalize units when a dominant unit exists; highly varied units are left as-is. Column splits and merges are skipped when no destination schema is provided, and invalid JSON strings are preserved rather than dropped.

Related Skills

Explore More Data & Analytics Skills→
starlake-aistarlake-ai
209

extract-bq-schema

Extract BigQuery schemas from Google Cloud projects using CLI options.

Official
Intermediate
borgheiborghei
467

analytics-engineer

Model data and build dbt pipelines from staging to marts.

Community
Advanced
databricks-solutionsdatabricks-solutions
5

anomaly-detection

Detect schema-level anomalies for table freshness and completeness in Databricks Unity Catalog.

Official
Intermediate
TriNgo0108TriNgo0108

data-quality-frameworks

Implement data quality validation with Great Expectations, dbt tests, and data contracts.

Community
Advanced