skills.rest
SmileAfterBurn
SmileAfterBurnCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

gcp-dataflow

Guides authoring, packaging, launching, and diagnosing Apache Beam pipelines on Google Cloud Dataflow.

Updated Aug 26, 2026
Search Tags:#troubleshooting#gcp#streaming#pipeline#dataflow#apache beam#flex template
One-click install
npx skills add https://github.com/SmileAfterBurn/pani-dumka-ai --skill gcp-dataflow-smileafterburn
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-dataflow
Source: https://github.com/SmileAfterBurn/pani-dumka-ai/tree/main/.gemini/skills/gcp-dataflow
Command: npx skills add https://github.com/SmileAfterBurn/pani-dumka-ai --skill gcp-dataflow-smileafterburn

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Building and operating Apache Beam pipelines on Google Cloud Dataflow involves many error-prone decisions: choosing between Google-provided templates and custom code, packaging Flex Templates correctly, validating required parameters, and diagnosing streaming job bottlenecks, backlog, and autoscaling anomalies. This Skill provides structured, step-by-step operational guidance for the entire Dataflow lifecycle. ## Core Features & Use Cases - Pipeline Authoring & Template Selection: Checks the public dataflow-templates bucket for existing Google-provided Classic or Flex templates before writing custom Java, Python, or Go pipelines, and enforces consistent Apache Beam SDK versions. - Flex Template Packaging & Execution: Packages pipelines as Flex Templates using Cloud Build and single Docker image configurations, validates strictly required template parameters, pre-creates BigQuery sinks, and runs jobs with mandatory pre-launch confirmation. - Diagnostics & Troubleshooting: Performs root cause analysis of streaming and batch jobs using Cloud Monitoring metrics, Cloud Logging queries, and the Dataflow REST API, covering watermark health, bottlenecks, hot keys, and autoscaling behavior. - Use Case: A streaming Pub/Sub-to-BigQuery pipeline shows growing data watermark delay. Use this Skill to map transform names to stages, query is_bottleneck and backlog metrics, analyze autoscaling rationale, and receive a structured diagnosis with remediation steps. ## Quick Start Use the gcp-dataflow skill to diagnose why my streaming Dataflow job with ID 2024-05-01_job in project my-project is falling behind.

Frequently Asked Questions about gcp-dataflow

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create an Apache Beam pipeline on Google Dataflow?▼

First check the public dataflow-templates GCS bucket for an existing Google-provided template matching your source and sink. If none fits, create a custom pipeline in Java, Python, or Go using the latest Apache Beam SDK, then package it as a Flex Template for a reproducible launch environment.

How do I build a Dataflow Flex Template with Cloud Build?▼

Provide a cloudbuild.yaml that builds and pushes your Docker image, then register the Flex Template spec in GCS. For Python pipelines with extra dependencies, use the single Docker image configuration where the launcher and worker share one image, and pass it via the sdk_container_image parameter.

How do I troubleshoot a slow Dataflow streaming job?▼

Correlate Cloud Monitoring metrics like data_watermark_age, is_bottleneck, and backlog_bytes with worker logs and Dataflow job messages. Map user-facing transform names to stage IDs via gcloud dataflow jobs describe, then analyze bottlenecks, hot keys, and autoscaling rationale to identify root causes.

Why is my Dataflow autoscaling adding workers but backlog keeps growing?▼

This indicates undetected throttling: workers sit idle waiting on downstream IO, external API quotas, or hot keys with low parallelism, so CPU stays low while backlog rises. Mitigate with in-flight updates to worker utilization hints or min/max worker limits, and fix the underlying IO or key distribution issue.

What parameters are required for Google-provided Dataflow templates?▼

Read the template metadata file from the dataflow-templates bucket; any parameter without isOptional set to true is strictly required by the Dataflow API, even if documentation suggests a default. Pre-create target sinks like BigQuery tables and error tables before launching to avoid runtime failures.

When should I not use this Dataflow guidance?▼

Do not use it for general GCP resource management unrelated to Dataflow, issues with other services like GCE or BigQuery unless they directly impact pipeline execution, or pipeline technologies other than Apache Beam on Dataflow.

Related Skills

Explore More Software Engineering Skills→
ivanshamaevivanshamaev
14

infra-gcp-data-platform-review

Review GCP data platform designs for BigQuery security and cost risks.

Community
Advanced
LAF-USLAF-US

huggingface-trackio

Track ML training experiments and metrics across Python, CLI, and TRL integrations.

Official
Advanced
way-platformway-platform
3

beam-dataflow-python

Guide Apache Beam Python pipeline development for Google Cloud Dataflow.

Official
Advanced
trungenglishtrungenglish
1

gcp-data-pipelines

Routes Google Cloud data pipeline requests to the appropriate dbt, Dataflow, Dataform, Spark, or Composer skill.

Community
Intermediate