skills.rest
arslan9024
arslan9024COMMUNITY
Explore all skills by this author →
GitHub

bigquery-data-transfer-service

Discovers and inspects BigQuery Data Transfer Service configurations for data ingestion pipelines.

Updated Jan 8, 2026
Search Tags:#gcp#pipeline#data ingestion#etl#bigquery#data transfer service#datasource discovery
One-click install
npx skills add https://github.com/arslan9024/White-Caves --skill bigquery-data-transfer-service-arslan9024
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: bigquery-data-transfer-service
Source: https://github.com/arslan9024/White-Caves/tree/main/.agents/skills/bigquery_data_transfer_service
Command: npx skills add https://github.com/arslan9024/White-Caves --skill bigquery-data-transfer-service-arslan9024

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve? Setting up BigQuery ingestion pipelines often leads to duplicate or misconfigured transfers because teams lack visibility into existing Data Transfer Service configurations and required datasource parameters. ## Core Features & Use Cases - Transfer Discovery: Lists and inspects existing DTS transfer configs in a target project and region using bq CLI commands. - Datasource Parameter Inspection: Queries the BigQuery Data Transfer REST API to enumerate required parameters for any data source, including OAuth authorization URL generation for Google data sources. - Guided Provisioning Workflow: Enforces declarative provisioning through a deployment.yaml pipeline framework, with mandatory user confirmation of parameters and transfer run verification. - Use Case: When asked to ingest Google Ads or Cloud Storage data into BigQuery, first check for an existing transfer, discover required parameters, and trigger a monitored manual run before building downstream pipeline tasks. ## Quick Start Ask the assistant to check whether a BigQuery Data Transfer Service config already exists for your project and region before creating a new ingestion pipeline.

Frequently Asked Questions about bigquery-data-transfer-service

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I check for existing BigQuery Data Transfer Service configs?▼

Run bq ls --transfer_config with the target transfer_location and project_id to list existing transfers. Then use bq show --format=prettyjson --transfer_config with the resource name to inspect a specific configuration.

How do I find required parameters for a BigQuery DTS data source?▼

Run the bigquery_dts.py script with your project ID, data source ID, and region. It calls the BigQuery Data Transfer REST API and returns the full parameter specification for that data source.

Can I create BigQuery DTS transfers with bq mk or gcloud commands?▼

No, new transfer configurations must be provisioned declaratively through the pipeline resource provisioning framework using a deployment.yaml file. CLI commands are permitted only for discovery, listing, and triggering manual runs.

How does OAuth authorization work for Google data sources in DTS?▼

For Google data sources like Google Ads configured with end user credentials, the discovery script generates an OAuth authorization URL. The user visits the URL, authorizes access, and provides the versionInfo code for the deployment.yaml definition.

What should I do if a BigQuery DTS transfer run takes too long?▼

Poll the run status every 30-60 seconds for up to 5 minutes. If still in progress after 5 minutes, stop and ask the user for proceed guidance once ingestion finishes before building downstream pipeline tasks.

Related Skills

Explore More Data & Analytics Skills→
BagelHoleBagelHole
46

gcp-audit-logs

Configure GCP Cloud Audit Logs and route them to BigQuery for analysis.

Community
Intermediate
supercent-iosupercent-io

looker-studio-bigquery

Connect BigQuery data sources to Looker Studio dashboards.

Official
Intermediate
tottenjordantottenjordan
1

gcp-pipeline-resource-provisioning

Provisions BigQuery, Dataform, and Dataproc resources declaratively via deployment.yaml files.

Community
Intermediate
SmileAfterBurnSmileAfterBurn

bigquery-data-transfer-service

Discovers and inspects BigQuery Data Transfer Service configurations for data ingestion pipelines.

Community
Advanced