What problem does it solve? Setting up data pipeline orchestration on Google Cloud Composer requires correctly authoring two interdependent YAML files (an orchestration pipeline definition and a deployment.yaml), matching strict schema rules, and running multiple gcloud commands in the right order. This Skill guides the agent through the entire lifecycle so pipelines are valid, deployable, and runnable without manual trial and error. ## Core Features & Use Cases - Pipeline Generation & Update: Creates or updates orchestration pipeline YAML files supporting dbt, Dataform, PySpark, notebooks, Python scripts, and inline BigQuery SQL actions, with correct camelCase field mapping and mandatory schedule endTime values. - Environment-Aware Deployment: Discovers compatible Composer environments and Dataproc clusters via gcloud, validates image versions and PyPI packages, and fills deployment.yaml placeholders with real project, region, and bucket values. - Validation, Deploy & Trigger: Runs gcloud beta orchestration-pipelines validate for every environment, deploys with --local, polls for DAG readiness, and triggers runs with a Python fallback script for Airflow DAG triggering. - Use Case: A data engineer asks to orchestrate a dbt project that runs after a PySpark job on a schedule; the Skill scaffolds the pipeline, validates it against the dev environment, deploys it to Composer, and triggers the first run. ## Quick Start Ask the agent to create and deploy an orchestration pipeline on Google Cloud Composer for your dbt, Spark, notebook, or BigQuery SQL jobs.
