skills.rest
Jatinkrmahato993203
Jatinkrmahato993203COMMUNITY
Explore all skills by this author →
GitHub

bigquery-data-transfer-service

Discovers and inspects BigQuery Data Transfer Service configurations for ingestion pipelines.

Updated Jul 20, 2026
Search Tags:#gcp#data pipeline#data ingestion#etl#gcloud#bigquery#data transfer service
One-click install
npx skills add https://github.com/Jatinkrmahato993203/crime --skill bigquery-data-transfer-service-jatinkrmahato993203
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: bigquery-data-transfer-service
Source: https://github.com/Jatinkrmahato993203/crime/tree/main/skills/bigquery-data-transfer-service
Command: npx skills add https://github.com/Jatinkrmahato993203/crime --skill bigquery-data-transfer-service-jatinkrmahato993203

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve? When building data pipelines on Google Cloud, you often cannot tell whether data ingestion is already managed by an existing BigQuery Data Transfer Service (DTS) configuration, leading to duplicate transfers or misconfigured pipelines. This Skill discovers existing transfer configs, inspects data source parameters, and guides declarative provisioning of new transfers. ## Core Features & Use Cases - Transfer Discovery: Lists and inspects existing DTS transfer configurations in a target project and region using bq CLI commands. - Data Source Parameter Inspection: Runs a Python script against the BigQuery Data Transfer REST API to enumerate required parameters for any data source, and generates OAuth authorization URLs for Google data sources. - Declarative Provisioning Guidance: Enforces creation of transfers through a deployment.yaml resource provisioning framework rather than imperative CLI commands, with mandatory user confirmation of parameters. - Use Case: A user asks to ingest Google Ads data into BigQuery. The Skill checks for existing transfers, discovers the required data source parameters, walks through OAuth authorization, and ensures a successful transfer run before downstream pipeline work begins. ## Quick Start Ask the assistant to check whether a BigQuery Data Transfer Service config already exists for your project and help set up ingestion for your data source.

Frequently Asked Questions about bigquery-data-transfer-service

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I check for existing BigQuery Data Transfer Service configs?▼

Run bq ls --transfer_config with the target transfer location and project ID to list existing transfers. Then use bq show --format=prettyjson --transfer_config with the resource name to inspect a specific transfer's configuration details.

How do I find required parameters for a BigQuery DTS data source?▼

Run the bigquery_dts.py script with your project ID, data source ID, and region. It calls the BigQuery Data Transfer REST API and returns the full parameter specification; omitting the data source ID lists all available sources in that region.

Can I create BigQuery DTS transfers with bq mk or gcloud commands?▼

No, new transfer configurations must be provisioned declaratively through the gcp pipeline resource provisioning framework using a generated deployment.yaml. CLI commands are permitted only for discovery, listing, showing configs, and triggering manual runs.

How does OAuth authorization work for Google data sources in DTS?▼

For Google data sources like Google Ads or YouTube using end user credentials, the discovery script generates an OAuth authorization URL. The user visits the URL, authorizes access, and provides the versionInfo code, which is set as definition.versionInfo in deployment.yaml.

What should I do if a BigQuery DTS transfer run takes too long?▼

Poll the run status every 30-60 seconds for up to 5 minutes using bq ls --transfer_run. If still in progress after 5 minutes, stop and ask the user to confirm once ingestion finishes before continuing with downstream pipeline tasks.

Related Skills

Explore More Data & Analytics Skills→
BagelHoleBagelHole
46

gcp-audit-logs

Configure GCP Cloud Audit Logs and route them to BigQuery for analysis.

Community
Intermediate
YashyasikYashyasik

bigquery-data-transfer-service

Discovers and inspects BigQuery Data Transfer Service configurations for data ingestion pipelines.

Community
Advanced
DingJun1028DingJun1028

gcp-pipeline-resource-provisioning

Provisions BigQuery, Dataform, Dataproc, and DTS resources via declarative deployment.yaml files.

Community
Intermediate
tottenjordantottenjordan
1

bigquery-data-transfer-service

Discovers and inspects BigQuery Data Transfer Service configurations for ingestion pipelines.

Community
Advanced