skills.rest
1919114514yasenpai-maker
1919114514yasenpai-makerCOMMUNITY
Explore all skills by this author →
GitHub

gcp-dataflow

Guides authoring, packaging, launching, and diagnosing Apache Beam pipelines on Google Cloud Dataflow.

Updated Aug 16, 2026
Search Tags:#gcp#streaming#autoscaling#dataflow#apache beam#pipeline diagnostics#flex template
One-click install
npx skills add https://github.com/1919114514yasenpai-maker/focus-quest --skill gcp-dataflow-1919114514yasenpai-maker
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-dataflow
Source: https://github.com/1919114514yasenpai-maker/focus-quest/tree/main/.gemini/skills/gcp-dataflow
Command: npx skills add https://github.com/1919114514yasenpai-maker/focus-quest --skill gcp-dataflow-1919114514yasenpai-maker

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Building and operating Apache Beam pipelines on Google Cloud Dataflow involves many error-prone decisions: choosing between Google-provided templates and custom code, packaging Flex Templates correctly, configuring required parameters, and diagnosing streaming job health, bottlenecks, and autoscaling behavior. This Skill provides structured workflows for each of these tasks. ## Core Features & Use Cases - Pipeline Authoring: Checks for existing Google-provided Dataflow templates before writing code, and scaffolds new Java, Python, or Go Beam projects with consistent SDK versions. - Flex Template Packaging: Builds Flex Templates with Cloud Build, single Docker image configuration for Python, and secure secret handling via Secret Manager at runtime. - Job Execution & Monitoring: Drafts launch commands with strict parameter validation, requires user confirmation before execution, and monitors job status with console URLs. - Diagnostics & Troubleshooting: Analyzes streaming job health, data freshness, backlog, bottlenecks, parallelism, and autoscaling using Cloud Monitoring metrics, logs, and the Dataflow REST API. - Use Case: A user asks why their streaming pipeline has growing lag. The Skill maps transform names to stages, queries watermark age, bottleneck, and autoscaling metrics, correlates them with worker logs, and produces a structured diagnosis with root causes and remediation steps. ## Quick Start Ask the assistant to create a new Apache Beam Dataflow pipeline from Pub/Sub to BigQuery, or to diagnose why a specific Dataflow job ID is running slowly.

Frequently Asked Questions about gcp-dataflow

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create an Apache Beam pipeline on Google Dataflow?▼

First check whether a Google-provided Dataflow template matches your source and sink by listing the dataflow-templates GCS bucket. If none fits, create a custom pipeline in Java, Python, or Go using the latest Apache Beam SDK, keeping the version consistent across all project files.

How do I package a Dataflow pipeline as a Flex Template?▼

Build a Docker image containing your pipeline code and use Cloud Build with a cloudbuild.yaml to push it, avoiding reliance on local Docker. For Python, use the single Docker image configuration where the launcher and worker share one image, and pass it via sdk_container_image.

How do I diagnose a slow Dataflow streaming job?▼

Check the data watermark age to classify job health, then query job/is_bottleneck, backlog, and parallelism metrics from Cloud Monitoring and correlate them with worker logs. Hot keys, low key cardinality, or blocking external RPCs in DoFns are common root causes.

Why is my Dataflow autoscaling adding workers but backlog keeps growing?▼

This indicates undetected throttling: workers sit idle waiting on downstream IO or hot keys, so CPU stays low while backlog rises and the autoscaler scales to the max limit. Mitigate with worker utilization hints, clamping min/max worker limits, or fixing the IO bottleneck.

Does Dataflow support secrets in Flex Template workers?▼

Yes, but never bake certificates or keys into Docker image layers. Retrieve them at runtime inside DoFn.setup() using the Secret Manager client library, and grant the worker service account the roles/secretmanager.secretAccessor role.

When should I not use this Dataflow skill?▼

Do not use it for general GCP resource management unrelated to Dataflow, issues with other services like GCE or BigQuery unless they directly impact a Dataflow job, or pipeline technologies other than Apache Beam on Dataflow.

Related Skills

Explore More Software Engineering Skills→
markus41markus41
1

gcp

Automate provisioning, deployment, and orchestration of Google Cloud Platform resources via CLI, SDKs, and Terraform.

Community
Advanced
gemini-cli-extensionsgemini-cli-extensions
161

gcp-dataflow

Design, package, deploy, and diagnose Apache Beam pipelines on Google Cloud Dataflow.

Official
Advanced
gemini-cli-extensionsgemini-cli-extensions
161

gcp-data-pipelines

Guide selection and implementation of Google Cloud data pipelines with dbt, Dataflow, Dataform, Spark, BigQuery DTS, and Cloud Composer.

Official
Advanced
gemini-cli-extensionsgemini-cli-extensions
161

gcp-spark

Coordinate Spark workloads on Google Cloud Dataproc and Serverless.

Official
Advanced