skills.rest
SmileAfterBurn
SmileAfterBurnCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

bigquery-data-transfer-service

Discovers and inspects BigQuery Data Transfer Service configurations for data ingestion pipelines.

Updated Aug 26, 2026
Search Tags:#oauth#gcp#data ingestion#etl#bigquery#data transfer service#datasource discovery
One-click install
npx skills add https://github.com/SmileAfterBurn/pani-dumka-ai --skill bigquery-data-transfer-service-smileafterburn
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: bigquery-data-transfer-service
Source: https://github.com/SmileAfterBurn/pani-dumka-ai/tree/main/.gemini/skills/bigquery-data-transfer-service
Command: npx skills add https://github.com/SmileAfterBurn/pani-dumka-ai --skill bigquery-data-transfer-service-smileafterburn

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve? When building data pipelines on Google Cloud, it is hard to know whether ingestion is already handled by an existing BigQuery Data Transfer Service (DTS) configuration, and creating new transfers requires discovering datasource-specific parameters, OAuth authorization, and correct project/region settings. ## Core Features & Use Cases - Transfer Discovery: Lists and inspects existing DTS transfer configurations in a target project and region using bq CLI commands. - Datasource Parameter Discovery: Runs a Python script against the BigQuery Data Transfer REST API to enumerate available data sources and their required parameters, including OAuth authorization URLs for Google data sources. - Run Verification: Triggers manual transfer runs and polls their status to confirm successful ingestion before downstream pipeline work. - Use Case: A data engineer asked to "ingest Google Ads data into BigQuery" uses this Skill to check for an existing transfer, discover required parameters, coordinate OAuth authorization, and verify a successful run before building transformations. ## Quick Start Ask the assistant to check whether a BigQuery Data Transfer Service configuration already exists for your project and region before setting up a new data ingestion pipeline.

Frequently Asked Questions about bigquery-data-transfer-service

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I check for existing BigQuery Data Transfer Service configurations?▼

Run bq ls --transfer_config with the target transfer location and project ID to list existing transfers. Then use bq show --format=prettyjson --transfer_config to inspect a specific transfer's configuration details.

How do I find required parameters for a BigQuery DTS data source?▼

Run the bigquery_dts.py script with your project ID, data source ID, and region. It calls the BigQuery Data Transfer REST API and returns the datasource's parameter definitions, and without a data source ID it lists all available sources.

Does BigQuery DTS require OAuth authorization for Google data sources?▼

Yes, when using end user credentials for Google data sources like Google Ads or YouTube, the discovery script generates an OAuth authorization URL. The user visits the URL and provides the versionInfo code, which is set as definition.versionInfo in deployment.yaml.

Can I create BigQuery DTS transfers with bq mk or gcloud commands?▼

No, new transfer configurations must be provisioned declaratively through the gcp pipeline resource provisioning framework with a generated deployment.yaml. CLI commands are permitted only for discovery, listing, and triggering manual transfer runs.

What should I do if a BigQuery DTS transfer run takes too long?▼

Poll the run status every 30-60 seconds for up to 5 minutes using bq ls --transfer_run. If still in progress after 5 minutes, stop and ask the user for proceed guidance once ingestion finishes before continuing pipeline work.

Related Skills

Explore More Data & Analytics Skills→
interflownexinterflownex
2

bigquery-data-transfer-service

List and inspect BigQuery Data Transfer Service configurations across GCP projects.

Community
Advanced
tottenjordantottenjordan
1

data-autocleaning

Profiles and cleans BigQuery and GCS data using Dataplex scans and SQL transformations.

Community
Advanced
aurainfosecaurainfosec

secrets-manager

Store, retrieve, and rotate application credentials in AWS Secrets Manager.

Official
Intermediate
FeexSystemsFeexSystems

bigquery-data-transfer-service

Discovers and inspects BigQuery Data Transfer Service configurations for data ingestion pipelines.

Community
Advanced