What problem does it solve? Setting up orchestration for data pipelines on Google Cloud Composer requires correctly authoring two interdependent YAML files (an orchestration pipeline definition and a deployment.yaml), matching a strict protobuf-based schema, and running multiple gcloud commands in the right order. This Skill automates that entire workflow and prevents common schema and configuration errors. ## Core Features & Use Cases - Pipeline Generation & Initialization: Scaffolds projects with gcloud beta orchestration-pipelines init and generates orchestration YAML for dbt, Dataform, PySpark, notebooks, Python scripts, and BigQuery SQL actions. - Validation & Error Recovery: Runs gcloud beta orchestration-pipelines validate for every environment and iteratively fixes reported errors until validation passes. - Deploy & Trigger Lifecycle: Deploys pipelines with --local, polls for DAG registration, triggers runs, and supports pause/unpause, with a Python fallback script for triggering via the Airflow API. - Use Case: A data engineer asks to schedule a dbt project to run daily after a PySpark job on Dataproc; the Skill checks Dataproc clusters and Composer environments, generates both YAML files, validates them, and deploys and triggers the pipeline. ## Quick Start Ask the agent to create an orchestration pipeline on Google Cloud Composer that runs your dbt project on a daily schedule, then deploy and trigger it in the dev environment.
