skills.rest
SmileAfterBurn
SmileAfterBurnCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

gcp-data-pipelines

Routes data pipeline requests to the appropriate Google Cloud tool skill.

Updated Aug 26, 2026
Search Tags:#gcp#data pipeline#dbt#spark#dataflow#dataform#cloud composer
One-click install
npx skills add https://github.com/SmileAfterBurn/pani-dumka-ai --skill gcp-data-pipelines-smileafterburn
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-data-pipelines
Source: https://github.com/SmileAfterBurn/pani-dumka-ai/tree/main/.gemini/skills/gcp-data-pipelines
Command: npx skills add https://github.com/SmileAfterBurn/pani-dumka-ai --skill gcp-data-pipelines-smileafterburn

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Choosing the right Google Cloud tool for a data pipeline is confusing when options like dbt, Dataflow, Dataform, Spark, and BigQuery DTS overlap. This Skill removes the guesswork by detecting existing pipelines in your workspace and guiding you to the correct specialized skill. ## Core Features & Use Cases - Workspace Detection: Scans for indicator files like dbt_project.yml, workflow_settings.yaml, or import apache_beam to identify existing pipelines automatically. - Guided Tool Selection: Presents a comparison of GCP pipeline tools (dbt, Dataflow, Dataform, Spark, BigQuery DTS, Cloud Composer) matched to your use case. - Run Request Clarification: Distinguishes between running a pipeline directly and deploying it via Cloud Composer orchestration. - Use Case: You open a repository containing a dbt_project.yml and ask to update a model. The Skill detects dbt and routes you directly to the dbt-bigquery skill without redundant questions. ## Quick Start Ask the assistant to help you build or run a data pipeline on Google Cloud and let it detect your project setup.

Frequently Asked Questions about gcp-data-pipelines

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I choose between dbt, Dataflow, and Dataform on Google Cloud?▼

Choose dbt for SQL-first teams wanting modular models with built-in tests running inside BigQuery. Choose Dataflow for streaming and unified batch processing with Apache Beam. Choose Dataform for Google-native ELT with SQLX and dependency management in the GCP Console.

How do I run a data pipeline on Google Cloud?▼

You can run a pipeline directly in the development environment using commands like dbt run or gcloud dataproc jobs submit, or deploy it to Cloud Composer for scheduled orchestration. The Skill clarifies which method fits before proceeding.

What is the difference between Dataflow and Spark on Dataproc Serverless?▼

Dataflow uses Apache Beam for unified stream and batch processing with serverless autoscaling and Pub/Sub integration. Dataproc Serverless runs PySpark, Java, or Scala jobs suited for large-scale data, ML preprocessing, and Iceberg or BigLake tables.

When should I use Cloud Composer for pipeline orchestration?▼

Use Cloud Composer when you need to schedule, automate, or coordinate multiple existing pipelines such as dbt and Spark as a unified workflow. It is not required for running a single pipeline directly during development.

Can the Skill detect which pipeline tool my project already uses?▼

Yes, it scans the workspace for indicator files such as dbt_project.yml for dbt, workflow_settings.yaml for Dataform, and apache_beam imports for Dataflow. Ambiguous cases like plain Python scripts require user confirmation.

Related Skills

Explore More Data & Analytics Skills→
tottenjordantottenjordan
1

data-autocleaning

Profiles and cleans BigQuery and GCS data using Dataplex scans and SQL transformations.

Community
Advanced
RxFitRxFit

gcp-pipeline-orchestration

Generate and deploy Cloud Composer orchestration pipeline definitions for GCP workflows.

Community
Advanced
tottenjordantottenjordan
1

gcp-pipeline-resource-provisioning

Provisions BigQuery, Dataform, and Dataproc resources declaratively via deployment.yaml files.

Community
Intermediate
FeexSystemsFeexSystems

bigquery-data-transfer-service

Discovers and inspects BigQuery Data Transfer Service configurations for data ingestion pipelines.

Community
Advanced