What problem does it solve? Setting up data pipeline orchestration on Google Cloud Composer requires correctly authoring two interdependent YAML files (an orchestration pipeline definition and a deployment.yaml), selecting compatible Composer and Dataproc environments, and running validation, deployment, and triggering commands in the right order. This Skill automates that entire workflow and prevents common schema and configuration errors. ## Core Features & Use Cases - Pipeline Scaffolding and Generation: Initializes projects with gcloud beta orchestration-pipelines init and generates orchestration YAML for dbt, Dataform, PySpark, notebooks, Python scripts, and BigQuery SQL actions. - Environment-Aware Deployment: Discovers Dataproc clusters and Composer environments via gcloud, verifies image versions and PyPI packages, and fills deployment.yaml with real project, region, and bucket values. - Validation, Deploy, and Trigger: Validates every environment with gcloud beta orchestration-pipelines validate, deploys with --local, polls for DAG readiness, and triggers runs with a Python fallback script. - Use Case: A data engineer asks to schedule a nightly dbt run after a PySpark job; the Skill generates both YAML files with correct camelCase fields, schedule startTime/endTime, validates them, and deploys to the dev Composer environment. ## Quick Start Create an orchestration pipeline that runs my dbt project nightly on Google Cloud Composer and deploy it to the dev environment.
