skills.rest
FeexSystems
FeexSystemsCOMMUNITY
Explore all skills by this author β†’
WebsiteGitHub

bigquery-data-transfer-service

Discovers and inspects BigQuery Data Transfer Service configurations for data ingestion pipelines.

1|Updated Aug 30, 2026
Search Tags:#oauth#gcp#data pipeline#data ingestion#etl#bigquery#data transfer service
One-click install
npx skills add https://github.com/FeexSystems/3WM-SONIK-LABS --skill bigquery-data-transfer-service-feexsystems
Or copy as Structured Prompt for Agentβ–Ό
Please help me install this Agent Skill.
Skill: bigquery-data-transfer-service
Source: https://github.com/FeexSystems/3WM-SONIK-LABS/tree/main/.gemini/skills/bigquery-data-transfer-service
Command: npx skills add https://github.com/FeexSystems/3WM-SONIK-LABS --skill bigquery-data-transfer-service-feexsystems

SYSTEM DOCUMENTATION & REQUIREMENTS

πŸ’‘ This Skill includes scripts (resource) components.

What problem does it solve? Setting up BigQuery data ingestion often means hunting for existing transfer configurations, discovering required datasource parameters, and handling OAuth authorization manually. This Skill automates the discovery and inspection of BigQuery Data Transfer Service (DTS) configurations so you avoid duplicate pipelines and misconfigured transfers. ## Core Features & Use Cases - Existing Transfer Discovery: Lists and analyzes existing DTS transfer configs in a project and region, including checking for successful runs and disabled transfers. - Datasource Parameter Inspection: Uses a REST API script to enumerate available data sources and extract the exact parameters required for a new transfer config. - OAuth Authorization Support: Generates OAuth authorization URLs for Google data sources (Google Ads, YouTube) when using end-user credentials. - Use Case: A data engineer needs to ingest Google Cloud Storage data into BigQuery. The Skill checks for existing transfers, discovers required parameters via the REST API, confirms them with the user, and guides declarative provisioning through a deployment.yaml. ## Quick Start Ask the assistant to check for existing BigQuery Data Transfer Service configs in your project and help set up ingestion for your data source.

Frequently Asked Questions about bigquery-data-transfer-service

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I check for existing BigQuery Data Transfer Service configs?β–Ό

Run bq ls --transfer_config with the target transfer location and project ID to list existing transfers. Then use bq show --format=prettyjson --transfer_config with the resource name to inspect a specific transfer's configuration details.

How do I find required parameters for a BigQuery DTS data source?β–Ό

Call the BigQuery Data Transfer REST API dataSources endpoint for your project and region, passing the data source ID such as google_cloud_storage. The response lists all required parameters, and the included discovery script automates this lookup.

Can I create BigQuery DTS transfers with bq mk or gcloud commands?β–Ό

No, new transfer configurations must be provisioned declaratively through the pipeline resource provisioning framework using a deployment.yaml file. CLI commands are permitted only for discovery operations and triggering manual transfer runs.

Does BigQuery DTS require OAuth for Google data sources?β–Ό

Yes, Google ecosystem sources like Google Ads and YouTube require OAuth authorization when using end-user credentials instead of a service account. The discovery script generates an authorization URL, and the returned versionInfo code goes into deployment.yaml.

What happens if a BigQuery DTS transfer run takes too long?β–Ό

Poll the run status every 30-60 seconds for up to 5 minutes using bq ls --transfer_run. If still in progress after 5 minutes, stop and wait for the user to confirm ingestion completion before building downstream pipeline tasks.

Related Skills

Explore More Data & Analytics Skills→
kubonykubony
10

gcp-storage

Manage GCP Cloud Storage buckets and contents using gcloud storage commands.

Community
Advanced
tottenjordantottenjordan
1

data-autocleaning

Profiles and cleans BigQuery and GCS data using Dataplex scans and SQL transformations.

Community
Advanced
interflownexinterflownex
2

bigquery-data-transfer-service

List and inspect BigQuery Data Transfer Service configurations across GCP projects.

Community
Advanced
FunnelEnvyFunnelEnvy
1

gcp-bq-data-export

Export BigQuery data to Cloud Storage in CSV, JSON, Avro, and Parquet formats.

Official
Advanced