skills.rest
Lathika-laa
Lathika-laaCOMMUNITY
Explore all skills by this author →
GitHub

gcp-data-pipelines

Routes data pipeline requests to the appropriate Google Cloud tool skill.

Updated Aug 26, 2026
Search Tags:#gcp#data pipeline#orchestration#dbt#bigquery#dataflow#cloud composer
One-click install
npx skills add https://github.com/Lathika-laa/Recipe_Box --skill gcp-data-pipelines-lathika-laa
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-data-pipelines
Source: https://github.com/Lathika-laa/Recipe_Box/tree/main/.github/.gemini/skills/gcp-data-pipelines
Command: npx skills add https://github.com/Lathika-laa/Recipe_Box --skill gcp-data-pipelines-lathika-laa

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Choosing the right Google Cloud tool for a data pipeline is confusing when options include dbt, Dataflow, Dataform, Spark, BigQuery DTS, and Cloud Composer. This Skill acts as a decision router that detects existing pipelines in your workspace and guides you to the correct specialized skill. ## Core Features & Use Cases - Workspace Detection: Scans for indicator files like dbt_project.yml, workflow_settings.yaml, and Apache Beam imports to identify existing pipelines automatically. - Tool Selection Guidance: Presents a comparison of GCP pipeline tools (dbt, Dataflow, Dataform, Spark, BigQuery DTS) with best-fit scenarios before any implementation begins. - Orchestration Routing: Directs scheduling and deployment requests to Cloud Composer orchestration or resource provisioning skills. - Use Case: You ask to "run the pipeline" in a repo containing both dbt and Spark jobs. The Skill detects both, asks which component to run, clarifies whether to run directly or deploy via Cloud Composer, then activates the matching skill. ## Quick Start Ask the assistant to help you build or run a data pipeline on Google Cloud and let it detect your workspace setup.

Frequently Asked Questions about gcp-data-pipelines

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I choose between dbt, Dataflow, and Dataform on Google Cloud?▼

Choose dbt for SQL-first teams wanting modular models with built-in tests running inside BigQuery, Dataflow for streaming and unified batch processing with Apache Beam, and Dataform for Google-native ELT with SQLX and GCP Console integration. The Skill presents these options and asks you to confirm before proceeding.

How do I run an existing data pipeline on GCP?▼

First clarify which component to run if multiple pipelines exist, then choose between running directly (dbt run, gcloud dataproc jobs submit, dataform run) or deploying to Cloud Composer for orchestrated execution. The Skill routes you to the matching specialized skill.

What is the difference between Dataflow and Spark on Dataproc Serverless?▼

Dataflow uses Apache Beam for unified stream and batch processing with Pub/Sub integration and serverless execution. Spark on Dataproc Serverless suits large-scale PySpark, Java, or Scala workloads including ML preprocessing and Iceberg/BigLake tables.

Can Cloud Composer orchestrate dbt and Spark jobs together?▼

Yes, Cloud Composer orchestration deploys and schedules existing pipelines such as dbt and Spark as a unified workflow. Requests mentioning scheduling, cron, or coordinating scripts are routed to the gcp-pipeline-orchestration skill.

When should I use BigQuery Data Transfer Service instead of building a pipeline?▼

Use BigQuery DTS for managed ingestion from supported datasources when you do not need custom transformation logic. For transformations inside BigQuery, dbt or Dataform are better fits.

Related Skills

Explore More Data & Analytics Skills→
tottenjordantottenjordan
1

gcp-pipeline-resource-provisioning

Provisions BigQuery, Dataform, and Dataproc resources declaratively via deployment.yaml files.

Community
Intermediate
RxFitRxFit

gcp-pipeline-orchestration

Generate and deploy Cloud Composer orchestration pipeline definitions for GCP workflows.

Community
Advanced
trungenglishtrungenglish
1

gcp-data-pipelines

Routes Google Cloud data pipeline requests to the appropriate dbt, Dataflow, Dataform, Spark, or Composer skill.

Community
Intermediate
2nth-ai2nth-ai
1

tech/google/cloud/compute

Deploy and manage Google Cloud resources using gcloud CLI.

Official
Advanced