skills.rest
FeexSystems
FeexSystemsCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

gcp-data-pipelines

Routes data pipeline requests to the appropriate Google Cloud tool and skill.

1|Updated Aug 30, 2026
Search Tags:#gcp#data pipeline#orchestration#dbt#bigquery#dataflow#cloud composer
One-click install
npx skills add https://github.com/FeexSystems/3WM-SONIK-LABS --skill gcp-data-pipelines-feexsystems
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-data-pipelines
Source: https://github.com/FeexSystems/3WM-SONIK-LABS/tree/main/.gemini/skills/gcp-data-pipelines
Command: npx skills add https://github.com/FeexSystems/3WM-SONIK-LABS --skill gcp-data-pipelines-feexsystems

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Choosing the right Google Cloud tool for a data pipeline is confusing when options like dbt, Dataflow, Dataform, Spark, BigQuery DTS, and Cloud Composer overlap. This Skill removes the guesswork by detecting existing pipelines in your workspace and guiding you to the correct tool before any implementation begins. ## Core Features & Use Cases - Workspace Detection: Scans for indicator files such as dbt_project.yml, workflow_settings.yaml, dataform.json, and Apache Beam or PySpark imports to identify existing pipelines. - Guided Tool Selection: Presents a comparison of GCP pipeline options (dbt, Dataflow, Dataform, Spark on Dataproc Serverless, BigQuery DTS) plus orchestration and provisioning choices, then waits for explicit user confirmation. - Run Request Clarification: Distinguishes between running a pipeline directly (dbt run, gcloud dataproc jobs submit) and deploying it through Cloud Composer orchestration. - Use Case: A data engineer opens a repository containing a dbt project and asks to schedule nightly runs; the Skill detects dbt_project.yml, routes to orchestration guidance, and deploys the workflow via Cloud Composer. ## Quick Start Ask the assistant to help you build or run a data pipeline on Google Cloud and let it detect your project setup.

Frequently Asked Questions about gcp-data-pipelines

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I choose between dbt, Dataflow, and Dataform on Google Cloud?▼

Choose dbt for SQL-first transformations with built-in tests and docs inside BigQuery, Dataflow for streaming and unified batch processing with Apache Beam, and Dataform for Google-native ELT with SQLX and dependency management. The Skill presents these options and waits for your confirmation.

How do I run an existing data pipeline on GCP?▼

First clarify which component to run if multiple pipelines exist, then choose between running directly (dbt run, gcloud dataproc jobs submit, dataform run) or deploying through Cloud Composer orchestration. The Skill routes orchestration requests to the gcp-pipeline-orchestration skill.

What is the best GCP tool for streaming data pipelines?▼

Dataflow is the recommended option for streaming pipelines, built on Apache Beam with unified stream and batch processing, Pub/Sub integration, and serverless execution. It also supports ML preprocessing and inference at scale.

Does a Python file in my repo mean it is a Spark pipeline?▼

No. A .py file could be Airflow, Spark, or another framework, so the Skill requires confirming with the user. Only files importing pyspark or notebooks indicate Spark, while apache_beam imports indicate Dataflow.

When should I use Cloud Composer for data pipelines?▼

Use Cloud Composer when you need to schedule, automate, or coordinate multiple existing pipelines such as dbt and Spark as a unified workflow. Requests mentioning cron, scheduling, or deployment are routed to the orchestration skill.

Related Skills

Explore More Data & Analytics Skills→
rose4320rose4320

gcp-pipeline-resource-provisioning

Automate GCP data pipeline resource provisioning via deployment.yaml and gcloud commands.

Community
Intermediate
trungenglishtrungenglish
1

gcp-pipeline-orchestration

Generate, validate, deploy, and trigger Google Cloud Composer orchestration pipeline YAML definitions.

Community
Advanced
camilooscargbaptistacamilooscargbaptista
7

data-engineering

Assess data pipelines, ETL/ELT processes, and governance across batch and streaming systems.

Community
Advanced
IndustrialIndustrial

data-engineer

Design scalable data pipelines and warehouses with Apache Spark, dbt, and Airflow.

Community
Advanced