What problem does it solve? Setting up data pipeline orchestration on Google Cloud Composer requires correctly authoring two interdependent YAML files (an orchestration pipeline definition and a deployment.yaml), selecting compatible Composer and Dataproc environments, and running validation, deployment, and triggering commands in the right order. This Skill automates that entire workflow and prevents common schema and configuration errors. ## Core Features & Use Cases - Pipeline Generation & Initialization: Scaffold new orchestration projects with gcloud beta orchestration-pipelines init, or update existing orchestration YAML and deployment.yaml files for dbt, Dataform, PySpark, notebooks, Python scripts, and BigQuery SQL jobs. - Environment-Aware Configuration: Query real Dataproc clusters and Composer environments via gcloud, verify compatible image versions and PyPI packages, and replace placeholders with actual project, region, and bucket values. - Validate, Deploy & Trigger: Run mandatory schema validation per environment, deploy pipelines with --local, poll for DAG readiness, trigger runs, and pause or unpause pipelines, with a Python fallback script for triggering Airflow DAGs directly. - Use Case: A data engineer asks to schedule a nightly dbt run after a PySpark job. The Skill checks for an existing deployment.yaml, lists available Dataproc clusters and Composer environments, generates both YAML files with correct camelCase schema and schedule endTime, validates them, then deploys and triggers the pipeline. ## Quick Start Ask the agent to create an orchestration pipeline that runs your dbt project on a schedule and deploys it to your dev Composer environment.
