skills.rest
DingJun1028
DingJun1028COMMUNITY
Explore all skills by this author →
GitHub

gcp-data-pipelines

Routes data pipeline requests to the appropriate Google Cloud tool and skill.

Updated Aug 17, 2026
Search Tags:#gcp#data pipeline#orchestration#dbt#dataflow#dataform#cloud composer
One-click install
npx skills add https://github.com/DingJun1028/esggo-kv --skill gcp-data-pipelines-dingjun1028
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-data-pipelines
Source: https://github.com/DingJun1028/esggo-kv/tree/main/.agents/skills/gcp-data-pipelines
Command: npx skills add https://github.com/DingJun1028/esggo-kv --skill gcp-data-pipelines-dingjun1028

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Choosing the right Google Cloud tool for a data pipeline is confusing when options like dbt, Dataflow, Dataform, Spark, BigQuery DTS, and Cloud Composer all overlap. This Skill removes the guesswork by detecting existing pipelines in your workspace and guiding you to the correct specialized skill. ## Core Features & Use Cases - Workspace Detection: Scans for indicator files such as dbt_project.yml, workflow_settings.yaml, dataform.json, Apache Beam imports, and PySpark notebooks to identify existing pipelines before recommending anything. - Tool Selection Guidance: Presents a structured comparison of BigQuery DTS, dbt, Dataflow, Dataform, Dataproc Serverless Spark, and Cloud Composer orchestration so you can pick the right fit. - Run Request Clarification: Distinguishes between running a pipeline directly (e.g., dbt run, gcloud dataproc jobs submit) and deploying it to Cloud Composer for orchestrated scheduling. - Use Case: You open a repository containing a dbt_project.yml and ask to build a transformation pipeline. The Skill detects dbt automatically and routes you to the dbt-bigquery skill without redundant questions. ## Quick Start Ask the assistant to help you build or run a data pipeline on Google Cloud and let it detect your project setup or present the available tool options.

Frequently Asked Questions about gcp-data-pipelines

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I choose between dbt, Dataflow, and Dataform on Google Cloud?▼

Choose dbt for SQL-first teams wanting modular models with built-in tests running inside BigQuery, Dataflow for streaming and unified batch processing with Apache Beam, and Dataform for Google-native ELT with SQLX and GCP Console integration. The Skill presents these options and highlights the likely fit based on your requirements.

How does the skill detect existing data pipelines in my workspace?▼

It scans for indicator files: dbt_project.yml for dbt, workflow_settings.yaml or dataform.json for Dataform, Apache Beam imports for Dataflow, PySpark imports for Spark, and deployment.yaml or *-pipeline.yaml for orchestration. Unambiguous indicators trigger direct routing without re-asking.

When should I use Cloud Composer for data pipeline orchestration?▼

Use Cloud Composer when you need to schedule, deploy, or coordinate multiple existing pipelines such as dbt and Spark as a unified workflow. If your request mentions scheduling, cron, or automation, the Skill highlights orchestration as the most likely fit.

What is the difference between running a pipeline directly and orchestrating it?▼

Running directly executes the pipeline in your development environment using commands like dbt run or gcloud dataproc jobs submit. Orchestrating deploys the pipeline to a managed service like Cloud Composer and triggers it as part of a larger scheduled workflow.

Does a Python file in my repo mean I have a Spark pipeline?▼

Not necessarily. Python files could be Airflow DAGs, Spark jobs, or generic scripts. The Skill confirms with you which pipeline type you are working with rather than assuming Spark based on .py files alone.

Related Skills

Explore More Data & Analytics Skills→
SmileAfterBurnSmileAfterBurn

data-autocleaning

Profiles and cleans BigQuery and GCS data using Dataplex scans and SQL transformations.

Community
Advanced
tottenjordantottenjordan
1

gcp-data-pipelines

Routes data pipeline requests to the appropriate Google Cloud tool and skill.

Community
Intermediate
rose4320rose4320

gcp-pipeline-resource-provisioning

Automate GCP data pipeline resource provisioning via deployment.yaml and gcloud commands.

Community
Intermediate
RxFitRxFit

bigquery-data-transfer-service

Discover BigQuery Data Transfer Service configurations and extract pipeline metadata via Python scripts.

Community
Advanced