skills.rest
FeexSystems
FeexSystemsCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

gcp-dataflow

Guides authoring, packaging, launching, and diagnosing Apache Beam pipelines on Google Cloud Dataflow.

1|Updated Aug 30, 2026
Search Tags:#troubleshooting#gcp#streaming#pipeline#dataflow#apache beam#flex template
One-click install
npx skills add https://github.com/FeexSystems/3WM-SONIK-LABS --skill gcp-dataflow-feexsystems
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-dataflow
Source: https://github.com/FeexSystems/3WM-SONIK-LABS/tree/main/.gemini/skills/gcp-dataflow
Command: npx skills add https://github.com/FeexSystems/3WM-SONIK-LABS --skill gcp-dataflow-feexsystems

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Building and operating Apache Beam pipelines on Google Cloud Dataflow involves many error-prone decisions: choosing between Google-provided templates and custom code, packaging Flex Templates correctly, configuring required parameters, and diagnosing streaming job health, bottlenecks, and autoscaling behavior. This Skill provides structured, step-by-step operational guidance for the entire Dataflow lifecycle. ## Core Features & Use Cases - Pipeline Authoring & Templates: Checks for existing Google-provided Dataflow templates before writing custom code, and guides Java, Python, and Go project setup with consistent Apache Beam SDK versions. - Flex Template Packaging: Enforces best practices such as the Single Docker Image configuration for Python, Cloud Build-based image builds, and secure Secret Manager handling at runtime. - Execution & Monitoring: Drafts validated launch commands with mandatory pre-launch confirmation, then monitors jobs with defined polling intervals and Console URLs. - Diagnostics & Troubleshooting: Performs root cause analysis of streaming and batch jobs using Cloud Monitoring metrics, Cloud Logging queries, and Dataflow REST API job messages, with structured health classifications. - Use Case: A data engineer's streaming pipeline shows growing watermark lag. The Skill maps transform names to stages, queries bottleneck and autoscaling metrics, correlates them with worker logs, and produces a diagnosis with remediation steps. ## Quick Start Help me create and launch a Dataflow Flex Template pipeline that reads from Pub/Sub and writes to BigQuery, then monitor its health.

Frequently Asked Questions about gcp-dataflow

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create an Apache Beam pipeline on Google Dataflow?▼

First check whether a Google-provided Dataflow template in gs://dataflow-templates already matches your source and sink. If not, create a custom pipeline in Java, Python, or Go using the latest Apache Beam SDK version, and package it as a Flex Template for a reproducible launch environment.

How do I build a Dataflow Flex Template for a Python pipeline?▼

Use the Single Docker Image configuration: build one image based on the Apache Beam Python SDK that serves as both the template launcher and worker runtime, then pass it via the sdk_container_image parameter. Prefer Cloud Build over local Docker for building and pushing the image.

How do I troubleshoot a slow Dataflow streaming job?▼

Correlate Cloud Monitoring metrics like job/data_watermark_age, job/is_bottleneck, and job/backlogged_keys with Cloud Logging worker logs and Dataflow job messages. Classify job health by watermark behavior, then analyze bottlenecks, key parallelism, and autoscaling decisions.

Why is my Dataflow autoscaling adding workers but the backlog keeps growing?▼

This indicates undetected throttling: workers sit idle waiting on IO bottlenecks, hot keys, or insufficient parallelism, so CPU stays low while backlog rises and the autoscaler scales to the max limit. Mitigate with in-flight updates to worker limits or utilization hints.

What parameters are required when running a Google-provided Dataflow template?▼

Any parameter in the template metadata JSON without an explicit isOptional: true flag is strictly required by the Dataflow API, even if the description suggests a default. Read the template metadata file, identify all required parameters, and confirm them with the user before launch.

When should I not use this Dataflow guidance?▼

Do not use it for general GCP resource management unrelated to Dataflow, issues with other services like GCE or BigQuery unless they directly impact pipeline execution, or pipeline technologies other than Apache Beam on Dataflow.

Related Skills

Explore More Software Engineering Skills→
ivanshamaevivanshamaev
14

infra-gcp-data-platform-review

Review GCP data platform designs for BigQuery security and cost risks.

Community
Advanced
supercent-iosupercent-io

looker-studio-bigquery

Connect BigQuery data sources to Looker Studio dashboards.

Official
Intermediate
wangx7wangx7

agent-platform-alert-configuration

Configure dynamic Vertex AI Agent Platform alert policies with Terraform and PromQL.

Community
Advanced
trungenglishtrungenglish
1

gcp-data-pipelines

Routes Google Cloud data pipeline requests to the appropriate dbt, Dataflow, Dataform, Spark, or Composer skill.

Community
Intermediate