skills.rest
1919114514yasenpai-maker
1919114514yasenpai-makerCOMMUNITY
Explore all skills by this author →
GitHub

data-autocleaning

Profiles and cleans BigQuery and GCS data using Dataplex scans and SQL transformations.

Updated Aug 16, 2026
Search Tags:#data quality#data cleaning#bigquery#gcs#data profiling#sql transformation#dataplex
One-click install
npx skills add https://github.com/1919114514yasenpai-maker/focus-quest --skill data-autocleaning-1919114514yasenpai-maker
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-autocleaning
Source: https://github.com/1919114514yasenpai-maker/focus-quest/tree/main/.gemini/skills/data-autocleaning
Command: npx skills add https://github.com/1919114514yasenpai-maker/focus-quest --skill data-autocleaning-1919114514yasenpai-maker

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve? Moving or ingesting data from BigQuery or Google Cloud Storage often introduces hidden data quality issues like malformed dates, garbage values, inconsistent units, and schema mismatches. This Skill automates profiling, cleansing, and verification so pipelines built on Dataform, dbt, or BigQuery start from trustworthy data. ## Core Features & Use Cases - Dataplex-Based Profiling: Runs automated data profile scans on source tables (with dynamic sampling for large tables) before and after transformation to drive and validate cleaning decisions. - SQL Transformation Generation: Produces cleansing SQL covering schema alignment, type conversion with SAFE.PARSE functions, JSON parsing, array handling, and STRUCT field mapping. - Quality Review Protocol: Creates a temporary sample output table, re-profiles it, and compares profiles column-by-column to catch NULL increases or value range shifts. - Use Case: When copying a raw GCS export into a curated BigQuery dataset, the Skill profiles the source, generates cleaning SQL for unparseable timestamps and invalid JSON, then verifies the output profile shows no regressions. ## Quick Start Ask the assistant to clean and copy data from a BigQuery table or GCS path into a destination table using the data-autocleaning skill.

Frequently Asked Questions about data-autocleaning

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I clean data when copying tables in BigQuery?▼

Profile the source table with a Dataplex data scan, then generate cleansing SQL that handles malformed values, type conversions with SAFE.PARSE functions, and schema alignment. Verify the result by profiling a temporary output table and comparing it against the source profile.

How to profile BigQuery tables with Dataplex?▼

Run the dataplex_scanner.py script with table IDs in project.dataset.table format, a specific region like us-central1, and an output directory. It creates one-time Dataplex data profile scans, applies sampling for tables over 1M rows, and saves results as JSON.

Does Dataplex profiling work on GCS data?▼

Yes, but you must create an external BigQuery table over the GCS files first before running the Dataplex scan. Multi-region locations like 'us' are not supported; use a specific region such as us-central1.

How do I parse JSON columns in BigQuery safely?▼

Use SAFE.PARSE_JSON to cast JSON strings to the JSON type, which returns NULL for invalid input instead of failing. Use JSON_VALUE and JSON_QUERY for extraction, and avoid the deprecated JSON_EXTRACT functions.

Why does my BigQuery array transformation fail with NULL elements?▼

BigQuery arrays cannot contain NULL values, so casts that produce NULLs cause errors. Filter them out after SAFE_CAST using ARRAY_FILTER with a condition like e IS NOT NULL before aggregating.

Related Skills

Explore More Data & Analytics Skills→
danielpratsdanielprats

uploadCarto

Upload GeoParquet files from a local directory to Google Cloud Storage and import them into CARTO Data Warehouse.

Community
Intermediate
ihistandihistand
1

acuantia-dataform

Enforce Acuantia Dataform conventions for BigQuery ODS and Looker tables.

Community
Intermediate
databricks-solutionsdatabricks-solutions
5

anomaly-detection

Detect schema-level anomalies for table freshness and completeness in Databricks Unity Catalog.

Official
Intermediate
gemini-cli-extensionsgemini-cli-extensions
161

data-autocleaning

Automate data quality checks and transformations for BigQuery and GCS pipelines.

Official
Advanced