skills.rest
Yashyasik
YashyasikCOMMUNITY
Explore all skills by this author →
GitHub

gcp-data-pipelines

Routes Google Cloud data pipeline requests to the appropriate dbt, Dataflow, Dataform, Spark, or Composer skill.

Updated Aug 11, 2026
Search Tags:#gcp#data pipeline#dbt#spark#dataflow#dataform#cloud composer
One-click install
npx skills add https://github.com/Yashyasik/zexca-api --skill gcp-data-pipelines-yashyasik
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-data-pipelines
Source: https://github.com/Yashyasik/zexca-api/tree/main/.gemini/skills/gcp-data-pipelines
Command: npx skills add https://github.com/Yashyasik/zexca-api --skill gcp-data-pipelines-yashyasik

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Choosing the right Google Cloud tool for a data pipeline is confusing when a workspace may already contain dbt, Dataflow, Dataform, Spark, or Airflow artifacts. This Skill detects existing pipelines, presents the correct GCP tool options, and routes the request to the matching specialized skill. ## Core Features & Use Cases - Workspace Detection: Scans for indicator files such as dbt_project.yml, workflow_settings.yaml, dataform.json, deployment.yaml, and Apache Beam or PySpark imports to identify existing pipelines. - Tool Selection Guidance: Presents a comparison of BigQuery DTS, dbt, Dataflow, Dataform, Spark on Dataproc Serverless, Cloud Composer orchestration, and declarative resource provisioning. - Skill Routing: Activates the corresponding downstream skill (dbt-bigquery, gcp-dataflow, dataform-bigquery, gcp-spark, gcp-pipeline-orchestration, and others) once the user confirms. - Use Case: A user asks to "run the pipeline" in a repo containing both dbt and Spark code. The Skill detects both, asks which component to run, clarifies whether to run directly or orchestrate via Cloud Composer, then hands off to the right skill. ## Quick Start Help me build a data pipeline on Google Cloud and recommend the right tool for my project.

Frequently Asked Questions about gcp-data-pipelines

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I choose between dbt, Dataflow, Dataform, and Spark on Google Cloud?▼

Choose dbt for SQL-first transformations inside BigQuery, Dataflow for streaming and unified batch processing with Apache Beam, Dataform for Google-native ELT with SQLX, and Spark on Dataproc Serverless for large-scale PySpark or Java workloads. The Skill presents these options and highlights the most likely fit based on your request.

How does the skill detect existing data pipelines in my workspace?▼

It scans for indicator files: dbt_project.yml for dbt, workflow_settings.yaml or dataform.json for Dataform, Apache Beam imports for Dataflow, PySpark imports for Spark, and deployment.yaml or *-pipeline.yaml for orchestration. Unambiguous indicators route directly to the matching skill.

When should I use Cloud Composer for pipeline orchestration?▼

Use Cloud Composer when you need to schedule, deploy, or coordinate multiple existing pipelines such as dbt and Spark as a unified workflow. Requests mentioning scheduling, cron, or automation are routed to the gcp-pipeline-orchestration skill.

What happens if my repo contains both dbt and Spark pipelines?▼

When multiple pipelines are detected and the request is ambiguous, the Skill asks which component you want to target. For run requests, it also clarifies whether to execute directly or deploy through Cloud Composer orchestration.

Does a Python file in my workspace mean it is a Spark pipeline?▼

No. A .py file could be Airflow, Spark, or another pipeline type, so the Skill asks you to confirm which pipeline you are working with. Only explicit indicators like PySpark imports or dbt_project.yml trigger automatic routing.

Related Skills

Explore More Data & Analytics Skills→
Lathika-laaLathika-laa

data-autocleaning

Profiles and cleans BigQuery and GCS data using Dataplex scans and SQL transformations.

Community
Advanced
tottenjordantottenjordan
1

gcp-data-pipelines

Routes data pipeline requests to the appropriate Google Cloud tool and skill.

Community
Intermediate
tottenjordantottenjordan
1

gcp-pipeline-resource-provisioning

Provisions BigQuery, Dataform, and Dataproc resources declaratively via deployment.yaml files.

Community
Intermediate
trungenglishtrungenglish
1

bigquery-data-transfer-service

Discovers and inspects BigQuery Data Transfer Service configurations for data ingestion pipelines.

Community
Advanced