skills.rest
trungenglish
trungenglishCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

bigquery-data-transfer-service

Discovers and inspects BigQuery Data Transfer Service configurations for data ingestion pipelines.

1|Updated Jul 4, 2026
Search Tags:#gcp#pipeline#data ingestion#etl#bigquery#data transfer service#datasource discovery
One-click install
npx skills add https://github.com/trungenglish/SHOPWISE --skill bigquery-data-transfer-service-trungenglish
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: bigquery-data-transfer-service
Source: https://github.com/trungenglish/SHOPWISE/tree/main/.agents/skills/bigquery-data-transfer-service
Command: npx skills add https://github.com/trungenglish/SHOPWISE --skill bigquery-data-transfer-service-trungenglish

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve? Setting up data ingestion into BigQuery often involves unknown transfer configurations, undocumented datasource parameters, and manual OAuth steps. This Skill discovers existing BigQuery Data Transfer Service (DTS) configurations, extracts datasource metadata, and guides the declarative provisioning of new transfers so ingestion pipelines are not duplicated or misconfigured. ## Core Features & Use Cases - Transfer Discovery: Lists and inspects existing DTS transfer configs and run history in a target project and region using bq CLI commands. - Datasource Parameter Inspection: Runs a Python script against the BigQuery Data Transfer REST API to enumerate available data sources and their required parameters, including OAuth authorization URL generation for Google data sources. - Declarative Provisioning Workflow: Enforces creation of transfers through a deployment.yaml-based resource provisioning framework rather than imperative CLI commands, with mandatory user confirmation of parameters. - Use Case: When asked to ingest Google Ads or Cloud Storage data into BigQuery, use this Skill to check whether a DTS transfer already exists, discover required parameters, and trigger and monitor a manual run before building downstream pipeline tasks. ## Quick Start Ask the assistant to check for existing BigQuery Data Transfer Service configs in your project and help set up ingestion for your data source.

Frequently Asked Questions about bigquery-data-transfer-service

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I check for existing BigQuery Data Transfer Service configs?▼

Run bq ls --transfer_config with the target transfer location and project ID to list existing transfers. Then use bq show --format=prettyjson --transfer_config with the resource name to extract full configuration details.

How do I find required parameters for a BigQuery DTS data source?▼

Run the bigquery_dts.py script with your project ID, data source ID, and region. It calls the BigQuery Data Transfer REST API and returns the datasource definition, including all required parameters and OAuth details for Google data sources.

Can I create BigQuery DTS transfers with bq mk or gcloud commands?▼

No, new transfer configurations must be provisioned declaratively through the gcp pipeline resource provisioning framework using a generated deployment.yaml. CLI commands are permitted only for discovery, listing, and triggering manual transfer runs.

Does BigQuery DTS support OAuth for Google data sources like Google Ads?▼

Yes, Google ecosystem data sources require OAuth authorization when using end user credentials. The discovery script generates an authorization URL, and the returned versionInfo code is set as definition.versionInfo in deployment.yaml.

What happens if a BigQuery DTS transfer run takes too long?▼

The workflow polls the run status every 30-60 seconds for up to 5 minutes. If the run is still in progress after that, it stops and asks the user for proceed guidance before continuing with downstream pipeline tasks.

Related Skills

Explore More Data & Analytics Skills→
jyasuujyasuu

gcloud

Automates Google Cloud management tasks via a CLI interface for authentication and resource handling.

Community
Intermediate
1919114514yasenpai-maker1919114514yasenpai-maker

data-autocleaning

Profiles and cleans BigQuery and GCS data using Dataplex scans and SQL transformations.

Community
Advanced
interflownexinterflownex
2

bigquery-data-transfer-service

List and inspect BigQuery Data Transfer Service configurations across GCP projects.

Community
Advanced
1919114514yasenpai-maker1919114514yasenpai-maker

bigquery-data-transfer-service

Discovers and inspects BigQuery Data Transfer Service configurations for data ingestion pipelines.

Community
Advanced