What problem does it solve? Setting up data pipeline orchestration on Google Cloud Composer requires correctly authoring two interdependent YAML files (an orchestration pipeline definition and a deployment.yaml), selecting compatible Composer and Dataproc environments, and running validation, deployment, and triggering commands in the right order. This Skill automates that entire workflow so pipelines are created, validated, and run without manual trial and error. ## Core Features & Use Cases - Pipeline Generation & Initialization: Scaffold projects with gcloud beta orchestration-pipelines init and generate orchestration YAML for dbt, Dataform, PySpark, notebooks, Python scripts, and BigQuery SQL actions. - Environment-Aware Deployment: Discover Dataproc clusters and Composer environments via gcloud, verify compatible image versions and PyPI packages, and fill deployment.yaml with real project, region, and bucket values. - Validation, Deploy & Trigger: Validate every environment with gcloud beta orchestration-pipelines validate, deploy with --local, poll for DAG readiness, trigger runs, and pause or unpause pipelines. - Use Case: A data engineer asks to orchestrate a dbt project that runs after a PySpark job on a schedule; the Skill generates both YAML files, validates them against the dev environment, deploys the DAG to Composer, and triggers the first run. ## Quick Start Ask the agent to create and validate a Cloud Composer orchestration pipeline for your dbt or Spark jobs in the dev environment.
