gcp-data-pipelines

Routes data pipeline requests to the appropriate Google Cloud tool and skill.

1|Updated Mar 19, 2026
One-click install
npx skills add https://github.com/tottenjordan/me-skittles --skill gcp-data-pipelines-tottenjordan
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-data-pipelines
Source: https://github.com/tottenjordan/me-skittles/tree/main/gemini/gcp-data-pipelines
Command: npx skills add https://github.com/tottenjordan/me-skittles --skill gcp-data-pipelines-tottenjordan

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Choosing the right Google Cloud tool for a data pipeline is confusing when options include dbt, Dataflow, Dataform, Spark, BigQuery DTS, and Cloud Composer. This Skill acts as a decision router that detects existing pipelines in your workspace and guides you to the correct specialized skill. ## Core Features & Use Cases - Workspace Detection: Scans for indicator files like dbt_project.yml, workflow_settings.yaml, dataform.json, and Apache Beam imports to identify existing pipelines automatically. - Tool Selection Guidance: Presents a comparison of GCP pipeline tools (dbt, Dataflow, Dataform, Spark on Dataproc Serverless, BigQuery DTS) with best-fit scenarios for each. - Orchestration Routing: Directs scheduling and deployment requests to Cloud Composer orchestration or declarative resource provisioning skills. - Use Case: You ask to "run the pipeline" in a repo containing both dbt and Spark jobs. The Skill detects both, asks which component to run, clarifies whether to run locally or deploy via Cloud Composer, then activates the matching skill. ## Quick Start Ask the agent to help you build or run a data pipeline on Google Cloud and let it detect your workspace setup before recommending a tool.

Frequently Asked Questions about gcp-data-pipelines

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I choose between dbt, Dataflow, and Dataform on Google Cloud?▼

Choose dbt for SQL-first teams wanting modular models with built-in tests running inside BigQuery, Dataflow for streaming and unified batch processing with Apache Beam, and Dataform for Google-native ELT with SQLX and GCP Console integration.

How do I orchestrate multiple data pipelines on GCP?▼

Use Cloud Composer to deploy and schedule existing pipelines such as dbt and Spark jobs as a unified workflow. The gcp-pipeline-orchestration skill handles deployment and triggering of these coordinated runs.

When should I use BigQuery Data Transfer Service instead of building a pipeline?▼

Use BigQuery DTS for managed ingestion from supported datasources when you need scheduled data loading without writing transformation code. Custom transformations require dbt, Dataform, Dataflow, or Spark instead.

Does this skill detect existing pipelines in my workspace?▼

Yes, it scans for indicator files like dbt_project.yml for dbt, workflow_settings.yaml or dataform.json for Dataform, Apache Beam imports for Dataflow, and PySpark imports for Spark before recommending a tool.

What is the difference between running a pipeline directly and orchestrating it?▼

Running directly executes the pipeline in your development environment using commands like dbt run or gcloud dataproc jobs submit. Orchestrating deploys the pipeline to Cloud Composer for scheduled, coordinated execution as part of a larger workflow.