skills.rest
Lathika-laa
Lathika-laaCOMMUNITY
Explore all skills by this author →
GitHub

bigquery-data-transfer-service

Discovers and inspects BigQuery Data Transfer Service configurations for ingestion pipelines.

Updated Aug 26, 2026
Search Tags:#gcp#pipeline#data ingestion#etl#bigquery#data transfer service#datasource discovery
One-click install
npx skills add https://github.com/Lathika-laa/Recipe_Box --skill bigquery-data-transfer-service-lathika-laa
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: bigquery-data-transfer-service
Source: https://github.com/Lathika-laa/Recipe_Box/tree/main/.github/.gemini/skills/bigquery-data-transfer-service
Command: npx skills add https://github.com/Lathika-laa/Recipe_Box --skill bigquery-data-transfer-service-lathika-laa

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve? When building data pipelines on Google Cloud, teams often duplicate ingestion work or lose track of existing BigQuery Data Transfer Service (DTS) configurations. This Skill discovers existing transfer configs, extracts datasource metadata, and guides declarative provisioning so ingestion pipelines are managed consistently. ## Core Features & Use Cases - Transfer Discovery: Lists and inspects existing DTS transfer configs and run history using bq CLI commands before creating anything new. - Datasource Parameter Discovery: Runs a Python script against the BigQuery Data Transfer REST API to enumerate available data sources and their required parameters, including OAuth authorization URL generation for Google data sources. - Declarative Provisioning Enforcement: Requires all new transfer configs to be provisioned through a deployment.yaml-based resource provisioning framework rather than imperative CLI commands. - Use Case: A user asks to ingest Google Ads data into BigQuery. The Skill checks for existing transfers, discovers required datasource parameters via the REST API, collects user-confirmed parameters and OAuth authorization, and verifies a successful transfer run before downstream pipeline work. ## Quick Start Ask the assistant to check whether a BigQuery Data Transfer Service config already exists for your project and help set up ingestion for your data source.

Frequently Asked Questions about bigquery-data-transfer-service

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I check for existing BigQuery Data Transfer Service configs?▼

Run bq ls --transfer_config with the target transfer location and project ID to list existing transfers. Then use bq show --format=prettyjson --transfer_config with the resource name to inspect a specific transfer's configuration details.

How do I find required parameters for a BigQuery DTS data source?▼

Run the bigquery_dts.py script with your project ID, data source ID, and region. It calls the BigQuery Data Transfer REST API and returns the full parameter specification, and can list available data source IDs if none is provided.

Can I create BigQuery DTS transfers with bq mk or gcloud commands?▼

No, this workflow requires all new transfer configurations to be provisioned declaratively through a deployment.yaml resource provisioning framework. CLI commands are permitted only for discovery, inspection, and triggering manual transfer runs.

Does BigQuery DTS support Google Ads and YouTube data sources?▼

Yes, Google ecosystem data sources like Google Ads and YouTube are supported but require OAuth authorization when using end user credentials. The discovery script generates an authorization URL, and the returned versionInfo code is added to the deployment configuration.

What happens if a BigQuery transfer run takes too long?▼

The workflow polls the transfer run status every 30-60 seconds for up to 5 minutes. If still running after that, it stops and asks the user to confirm ingestion completion before proceeding with downstream pipeline tasks.

Related Skills

Explore More Data & Analytics Skills→
kubiyabotkubiyabot
16

gcp

Manage Google Cloud resources by wrapping the native gcloud CLI.

Official
Advanced
tottenjordantottenjordan
1

data-autocleaning

Profiles and cleans BigQuery and GCS data using Dataplex scans and SQL transformations.

Community
Advanced
MrJmpl3MrJmpl3

0168-secrets-management

Retrieve and mask secrets from Vault or AWS Secrets Manager in CI/CD pipelines.

Community
Intermediate
1919114514yasenpai-maker1919114514yasenpai-maker

bigquery-data-transfer-service

Discovers and inspects BigQuery Data Transfer Service configurations for data ingestion pipelines.

Community
Advanced