gcp-pipeline-resource-provisioning

Define bigquery.dataset and bigquery.table resources in a deployment.yaml file with environment-specific project and region settings, then run gcloud beta orchestration-pipelines deploy. Each resource must include a datacloud label identifying the creating environment.

Declare dataform.repository, releaseConfig, and workflowConfig resources in deployment.yaml, specifying gitCommitish, cron schedules, and service accounts. Validate with gcloud beta orchestration-pipelines validate, then deploy with the deploy command for your target environment.

Run gcloud beta orchestration-pipelines resource-types list to see all supported types. Common ones include BigQuery datasets and tables, Dataform repositories, Dataproc clusters and workflow templates, and BigQuery Data Transfer Service configs.

Never hardcode plain-text secrets in deployment.yaml. Declare them in the secrets block referencing existing Secret Manager versions, and substitute them with {{ SECRET_NAME }} syntax. The framework only references existing secrets and never creates or reads their values.

Use Terraform for general cloud infrastructure like VMs, networks, Kubernetes, and IAM policies, or when infrastructure spans multiple cloud providers. This approach is intended for data pipeline resources like BigQuery, Dataform, Dataproc, and DTS that do not already exist.