skills.rest
trungenglish
trungenglishCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

gcp-data-pipelines

Routes Google Cloud data pipeline requests to the appropriate dbt, Dataflow, Dataform, Spark, or Composer skill.

1|Updated Jul 4, 2026
Search Tags:#gcp#data pipeline#orchestration#dbt#dataflow#dataform#cloud composer
One-click install
npx skills add https://github.com/trungenglish/SHOPWISE --skill gcp-data-pipelines-trungenglish
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-data-pipelines
Source: https://github.com/trungenglish/SHOPWISE/tree/main/.agents/skills/gcp-data-pipelines
Command: npx skills add https://github.com/trungenglish/SHOPWISE --skill gcp-data-pipelines-trungenglish

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Choosing the right Google Cloud tool for a data pipeline is confusing, and picking the wrong one wastes effort. This Skill acts as a router: it detects existing pipelines in your workspace, clarifies ambiguous requests, and directs you to the correct specialized skill for dbt, Dataflow, Dataform, Spark, BigQuery DTS, or Cloud Composer orchestration. ## Core Features & Use Cases - Workspace Detection: Scans for indicator files like dbt_project.yml, workflow_settings.yaml, dataform.json, Apache Beam imports, and PySpark notebooks to identify existing pipelines before recommending anything. - Guided Tool Selection: Presents a comparison of GCP pipeline options (BigQuery DTS, dbt, Dataflow, Dataform, Dataproc Serverless Spark) plus provisioning and Cloud Composer orchestration, then waits for user confirmation before acting. - Run Request Clarification: Distinguishes between running a pipeline directly (e.g., dbt run, gcloud dataproc jobs submit) and deploying it to Cloud Composer for scheduled orchestration. - Use Case: You open a repository containing a dbt_project.yml and ask to "run the pipeline". The Skill detects dbt, asks whether to run locally or orchestrate via Cloud Composer, and activates the dbt-bigquery or gcp-pipeline-orchestration skill accordingly. ## Quick Start Ask the assistant to help you build or run a data pipeline on Google Cloud and let it detect your project setup and recommend the right tool.

Frequently Asked Questions about gcp-data-pipelines

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I choose between dbt, Dataflow, and Dataform on Google Cloud?▼

Choose dbt for SQL-first transformations inside BigQuery with built-in tests and docs, Dataflow for streaming and unified batch processing with Apache Beam, and Dataform for Google-native ELT with SQLX and GCP Console integration. The Skill presents these options and waits for your confirmation.

How does the skill detect existing data pipelines in my workspace?▼

It scans for indicator files: dbt_project.yml for dbt, workflow_settings.yaml or dataform.json for Dataform, Apache Beam imports for Dataflow, PySpark imports for Spark, and deployment.yaml or *-pipeline.yaml for orchestration. Unambiguous indicators route directly to the matching skill.

Should I run my pipeline directly or use Cloud Composer?▼

Run directly with commands like dbt run or gcloud dataproc jobs submit for local development and testing. Use Cloud Composer when you need scheduling, automation, or coordinating multiple pipelines such as dbt plus Spark as a unified workflow.

What happens if my repo has both dbt and Spark pipelines?▼

When multiple pipelines are detected and the request is ambiguous, the Skill asks which component you want to target before proceeding. It will not assume which pipeline to run or modify without your confirmation.

Can a plain Python file be treated as a Spark pipeline?▼

No. A .py file alone is ambiguous since it could be Airflow, Spark, or something else. The Skill requires explicit indicators like a pyspark import and will ask you to confirm the pipeline type before acting.

Related Skills

Explore More Data & Analytics Skills→
1919114514yasenpai-maker1919114514yasenpai-maker

gcp-pipeline-resource-provisioning

Automates declarative provisioning of GCP data pipeline resources via deployment.yaml configurations.

Community
Intermediate
mahg-esmahg-es
3

etl-orchestration

Automate ETL orchestration by defining DAGs and workflows across Airflow, Dagster, or Prefect.

Official
Advanced
1919114514yasenpai-maker1919114514yasenpai-maker

gcp-pipeline-orchestration

Generate, validate, deploy, and trigger Google Cloud Composer orchestration pipeline definitions.

Community
Advanced
AubreyHanAubreyHan

bigquery-data-transfer-service

Discover BigQuery Data Transfer Service configurations and extract transfer metadata.

Community
Advanced