skills.rest
Jatinkrmahato993203
Jatinkrmahato993203COMMUNITY
Explore all skills by this author →
GitHub

gcp-data-pipelines

Routes data pipeline requests to the appropriate Google Cloud tool and skill.

Updated Jul 20, 2026
Search Tags:#gcp#data pipeline#orchestration#dbt#bigquery#dataflow#cloud composer
One-click install
npx skills add https://github.com/Jatinkrmahato993203/crime --skill gcp-data-pipelines-jatinkrmahato993203
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-data-pipelines
Source: https://github.com/Jatinkrmahato993203/crime/tree/main/skills/gcp-data-pipelines
Command: npx skills add https://github.com/Jatinkrmahato993203/crime --skill gcp-data-pipelines-jatinkrmahato993203

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Choosing the right Google Cloud tool for a data pipeline is confusing when options include dbt, Dataflow, Dataform, Spark, BigQuery DTS, and Cloud Composer. This Skill removes that ambiguity by detecting existing pipelines in your workspace and guiding you to the correct tool before any implementation begins. ## Core Features & Use Cases - Workspace Detection: Scans for indicator files like dbt_project.yml, workflow_settings.yaml, dataform.json, and Apache Beam imports to identify existing pipelines automatically. - Guided Tool Selection: Presents a comparison of GCP pipeline options (dbt, Dataflow, Dataform, Spark, BigQuery DTS) with best-fit scenarios so you can confirm the right choice. - Orchestration Routing: Directs scheduling, deployment, and coordination requests to Cloud Composer orchestration or resource provisioning skills. - Use Case: You open a repository containing a dbt_project.yml and ask to run the pipeline. The Skill detects dbt, confirms whether you want a direct run or Composer deployment, and activates the dbt-bigquery skill. ## Quick Start Ask the assistant to help you build or run a data pipeline on Google Cloud and confirm which tool you want to use.

Frequently Asked Questions about gcp-data-pipelines

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I choose between dbt, Dataflow, and Dataform on Google Cloud?▼

Choose dbt for SQL-first teams wanting modular models with built-in tests running inside BigQuery. Choose Dataflow for streaming and unified batch processing with Apache Beam. Choose Dataform for Google-native ELT with SQLX and GCP Console integration.

How do I run an existing data pipeline on GCP?▼

First clarify which component to run if multiple pipelines exist, then choose a method. You can run directly with commands like dbt run or gcloud dataproc jobs submit, or deploy to Cloud Composer for orchestrated execution.

When should I use Cloud Composer for data pipelines?▼

Use Cloud Composer when you need to schedule, automate, or coordinate multiple existing pipelines such as dbt and Spark as a unified workflow. It handles deployment and triggering rather than direct local execution.

Does a Python file in my workspace mean I have a Spark pipeline?▼

Not necessarily. A .py file could be Airflow, Spark, or another framework. The Skill confirms with you which pipeline type you are working with unless it finds an unambiguous indicator like a pyspark import.

What is the difference between BigQuery DTS and Dataflow for data ingestion?▼

BigQuery Data Transfer Service provides managed ingestion from supported datasources with minimal configuration. Dataflow is a serverless Apache Beam service suited for custom streaming, high-throughput Pub/Sub integration, and ML preprocessing at scale.

Related Skills

Explore More Data & Analytics Skills→
DingJun1028DingJun1028

data-autocleaning

Profiles and cleans BigQuery and GCS data using Dataplex scans and SQL transformations.

Community
Advanced
ironkid90ironkid90

gcp-data-pipelines

Guide building, managing, and orchestrating data pipelines on Google Cloud.

Community
Advanced
AubreyHanAubreyHan

gcp-pipeline-resource-provisioning

Provision GCP pipeline resources from a deployment.yaml file.

Community
Advanced
ironkid90ironkid90

federate-lakehouse-catalog

Connect remote Iceberg REST Catalogs to GCP engines for cross-cloud querying.

Community
Advanced