gcp-data-pipelines

Choose dbt for SQL-first teams with modular models and built-in tests running inside BigQuery, Dataflow for streaming and unified batch processing with Apache Beam, and Dataform for Google-native ELT with SQLX and GCP Console integration. The Skill presents these options and asks you to confirm before proceeding.

It scans for indicator files: dbt_project.yml for dbt, workflow_settings.yaml or dataform.json for Dataform, Apache Beam imports for Dataflow, PySpark imports for Spark, and deployment.yaml for orchestration. Unambiguous indicators route directly to the matching skill.

Use Cloud Composer when you need to schedule, automate, or coordinate multiple existing pipelines such as dbt and Spark as a unified workflow. Requests mentioning scheduling, cron, or coordinating scripts are routed to the gcp-pipeline-orchestration skill.

Running directly executes the pipeline in the development environment using commands like dbt run or gcloud dataproc jobs submit. Orchestrating deploys the pipeline to a managed service like Cloud Composer and triggers it as part of a larger scheduled workflow.

No. A .py file could be Airflow, Spark, or something else entirely. The Skill confirms with you which type of pipeline you are working with before routing, unless an unambiguous indicator like a PySpark import is found.