gcp-pipeline-resource-provisioning

Define bigquery.dataset and bigquery.table resources under the target environment in deployment.yaml, including schema fields and mandatory labels. Then run gcloud beta orchestration-pipelines validate followed by deploy with the environment flag.

Supported types include BigQuery datasets and tables, Dataform repositories with release and workflow configs, Dataproc clusters and workflow templates, and BigQuery Data Transfer Service configs. Run gcloud beta orchestration-pipelines resource-types list for the full current list.

Declare secrets in the secrets block referencing existing Secret Manager paths, then substitute them with {{ SECRET_NAME }} syntax in resource definitions. Never hardcode plain-text secrets, and create missing secrets manually since the framework only references existing ones.

Use Terraform for general cloud infrastructure like VMs, networks, Kubernetes, and IAM policies, or when infrastructure spans multiple cloud providers. This framework targets data pipeline resources only and should not manage resources already controlled by Terraform.

Validation fails when placeholders like YOUR_PROJECT_ID remain unreplaced, variables are undefined, or resource definitions are syntactically incorrect. Discover actual project, region, and commit SHA values with gcloud and git commands, then rerun the validate command.