What problem does it solve? Setting up orchestration for data pipelines on Google Cloud Composer requires correctly authoring two interdependent YAML files (an orchestration pipeline definition and a deployment.yaml), validating them against a strict schema, and deploying or triggering them with the right gcloud commands. This Skill automates that entire workflow and prevents common schema and configuration errors. ## Core Features & Use Cases - Pipeline Generation & Initialization: Scaffold new orchestration projects with gcloud beta orchestration-pipelines init, or update existing pipeline files referenced by deployment.yaml. - Schema-Guided Authoring: Generate pipeline definitions for dbt, Dataform, PySpark, notebooks, Python scripts, and BigQuery SQL using the bundled protobuf-based schema reference, with correct camelCase field mapping and mandatory schedule endTime values. - Validation, Deployment & Triggering: Validate pipelines per environment, deploy with --local, poll for DAG readiness, trigger runs, and pause or unpause pipelines, with a Python fallback script for triggering Airflow DAGs directly. - Use Case: A data engineer wants to schedule a dbt project to run after a PySpark job on Dataproc. The Skill checks for existing setup, lists available Dataproc clusters and Composer environments, generates both YAML files, validates them, and deploys and triggers the pipeline. ## Quick Start Ask the agent to create an orchestration pipeline that runs your dbt project on a schedule and deploys it to your dev Composer environment.
