skills.rest
FeexSystems
FeexSystemsCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

dataform-bigquery

Generates and validates Dataform SQLX pipeline code for BigQuery ELT workflows.

1|Updated Aug 30, 2026
Search Tags:#gcp#data pipeline#bigquery#sqlx#elt#dataform#gcs ingestion
One-click install
npx skills add https://github.com/FeexSystems/3WM-SONIK-LABS --skill dataform-bigquery-feexsystems
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: dataform-bigquery
Source: https://github.com/FeexSystems/3WM-SONIK-LABS/tree/main/.gemini/skills/dataform-bigquery
Command: npx skills add https://github.com/FeexSystems/3WM-SONIK-LABS --skill dataform-bigquery-feexsystems

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Building Dataform pipelines for BigQuery requires correct SQLX syntax, proper source declarations, incremental table configuration, and validated compilation — mistakes cause broken DAGs and failed runs. This Skill guides the creation, modification, and validation of Dataform projects with enforced best practices. ## Core Features & Use Cases - Pipeline Generation: Creates SQLX actions, source declarations, and incremental tables following Dataform conventions, including GCS-to-BigQuery ingestion via external tables. - Validation Workflow: Compiles projects with dataform compile and validates with dataform run --dry-run or bq query --dry_run, never executing real runs without confirmation. - Automatic Optimization: Applies mandatory data cleaning and BigQuery SQL optimization protocols, with summary sections in every response. - Use Case: A data engineer needs to ingest CSV files from GCS into an existing BigQuery table. The Skill initializes the Dataform repo, declares sources, creates an incremental SQLX action with schema-aligned columns, and validates the pipeline via dry run. ## Quick Start Ask the assistant to create a Dataform pipeline that loads data from a GCS bucket into a BigQuery table using incremental SQLX actions.

Frequently Asked Questions about dataform-bigquery

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create a Dataform pipeline for BigQuery?▼

Initialize a repository with `dataform init <PROJECT_DIR> <PROJECT_ID> <LOCATION>`, then define SQLX actions with config blocks specifying type (table, incremental, or declaration). Validate with `dataform compile` and `dataform run --dry-run` before any real execution.

How to append data to an existing BigQuery table with Dataform?▼

Use `type: "incremental"` with the existing table name in the config, and write only a SELECT statement in the body — Dataform auto-generates the INSERT. Never use `type: "operations"` for append operations, and ensure SELECT columns match the target schema.

Does Dataform support BigLake Iceberg tables with 4-part naming?▼

Dataform does not natively support 4-part Project.Catalog.Dataset.Table names. Work around this by concatenating catalog and namespace into the declaration's schema field, e.g. schema: "my_catalog.my_namespace".

Why does dataform run --dry-run fail with table not found errors?▼

Dry-run fails with 'Table not found' when upstream dependencies defined in the project haven't been materialized in BigQuery yet. These specific errors can be ignored; rely on `dataform compile` and `bq query --dry_run` for validation instead.

How do I ingest CSV files from GCS into BigQuery using Dataform?▼

Create an external table in a SQLX operations file pointing at the GCS URI. Use STRING for all CSV columns and set allow_jagged_rows, allow_quoted_newlines, and ignore_unknown_values to true for resilient parsing.

Related Skills

Explore More Data & Analytics Skills→
pbc-ospbc-os
10

semantic-layer-audit

Discover and document BigQuery datasets, tables, views, APIs, and Secret Manager credentials.

Community
Intermediate
trungenglishtrungenglish
1

dataform-bigquery

Generates and validates Dataform SQLX pipelines for BigQuery ELT workflows.

Community
Advanced
Adelie-SquadAdelie-Squad
17

data-engineer

Clean and transform raw datasets into analysis-ready tables with pandas pipelines.

Official
Advanced
RxFitRxFit

gcp-pipeline-orchestration

Generate and deploy Cloud Composer orchestration pipeline definitions for GCP workflows.

Community
Advanced