skills.rest
1919114514yasenpai-maker
1919114514yasenpai-makerCOMMUNITY
Explore all skills by this author →
GitHub

bigquery-data-transfer-service

Discovers and inspects BigQuery Data Transfer Service configurations for data ingestion pipelines.

Updated Aug 16, 2026
Search Tags:#oauth#gcp#data pipeline#data ingestion#etl#bigquery#data transfer service
One-click install
npx skills add https://github.com/1919114514yasenpai-maker/focus-quest --skill bigquery-data-transfer-service-1919114514yasenpai-maker
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: bigquery-data-transfer-service
Source: https://github.com/1919114514yasenpai-maker/focus-quest/tree/main/.gemini/skills/bigquery-data-transfer-service
Command: npx skills add https://github.com/1919114514yasenpai-maker/focus-quest --skill bigquery-data-transfer-service-1919114514yasenpai-maker

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve? Setting up BigQuery data ingestion often duplicates existing transfers or misconfigures data source parameters. This Skill discovers existing BigQuery Data Transfer Service (DTS) configurations, extracts transfer metadata, and guides the declarative provisioning of new transfers so ingestion pipelines are not recreated unnecessarily. ## Core Features & Use Cases - Transfer Discovery: Lists and inspects existing DTS transfer configs and run history in a target project and region using bq CLI commands. - Data Source Parameter Inspection: Queries the BigQuery Data Transfer REST API via a Python script to enumerate required parameters for any data source, and generates OAuth authorization URLs for Google data sources. - Guided Provisioning Workflow: Enforces declarative provisioning through a deployment.yaml pipeline framework, mandatory user confirmation of parameters, and monitored manual transfer runs. - Use Case: When asked to ingest Google Cloud Storage or Google Ads data into BigQuery, use this Skill to check whether a DTS transfer already exists, validate its parameters with the user, and trigger a verified run before building downstream pipeline stages. ## Quick Start Ask the assistant to check for existing BigQuery Data Transfer Service configs in your project and set up ingestion for your data source.

Frequently Asked Questions about bigquery-data-transfer-service

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I check for existing BigQuery Data Transfer Service configs?▼

Run bq ls --transfer_config with the target transfer location and project ID to list existing transfers. Then use bq show --format=prettyjson --transfer_config with the resource name to extract full configuration details.

How do I find required parameters for a BigQuery DTS data source?▼

Run the bigquery_dts.py script with your project ID, data source ID, and region to query the BigQuery Data Transfer REST API. It returns the data source definition including all required parameters, and lists available data source IDs when called without one.

Can I create BigQuery DTS transfers with bq mk or gcloud commands?▼

No, new transfer configurations must be provisioned declaratively through the gcp pipeline resource provisioning framework with a generated deployment.yaml. CLI commands are permitted only for discovery, listing, and triggering manual transfer runs.

Does BigQuery DTS require OAuth authorization for Google data sources?▼

Yes, Google ecosystem sources like Google Ads and YouTube require user authorization when using end user credentials. The discovery script generates an OAuth URL; the returned versionInfo code is set as definition.versionInfo in deployment.yaml.

What happens if a BigQuery DTS transfer run takes too long?▼

The workflow polls the run status every 30-60 seconds for up to 5 minutes. If still in progress after that, it stops and asks the user to confirm ingestion completion before continuing with downstream pipeline tasks.

Related Skills

Explore More Data & Analytics Skills→
BagelHoleBagelHole
46

gcp-audit-logs

Configure GCP Cloud Audit Logs and route them to BigQuery for analysis.

Community
Intermediate
gemini-cli-extensionsgemini-cli-extensions
161

data-autocleaning

Automate data quality checks and transformations for BigQuery and GCS pipelines.

Official
Advanced
databrickslabsdatabrickslabs
60

validate-connector-auth

Generate and execute an authentication verification test for a data connector.

Official
Intermediate
FunnelEnvyFunnelEnvy
1

gcp-bq-data-export

Export BigQuery data to Cloud Storage in CSV, JSON, Avro, and Parquet formats.

Official
Advanced