skills.rest
Lathika-laa
Lathika-laaCOMMUNITY
Explore all skills by this author →
GitHub

gcp-dataflow

Guides authoring, packaging, launching, and diagnosing Apache Beam pipelines on Google Cloud Dataflow.

Updated Aug 26, 2026
Search Tags:#troubleshooting#gcp#streaming#pipeline#dataflow#apache beam#flex template
One-click install
npx skills add https://github.com/Lathika-laa/Recipe_Box --skill gcp-dataflow-lathika-laa
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-dataflow
Source: https://github.com/Lathika-laa/Recipe_Box/tree/main/.github/.gemini/skills/gcp-dataflow
Command: npx skills add https://github.com/Lathika-laa/Recipe_Box --skill gcp-dataflow-lathika-laa

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Building and operating Apache Beam pipelines on Google Cloud Dataflow involves many error-prone decisions: choosing between Google-provided templates and custom code, packaging Flex Templates correctly, configuring required parameters, and diagnosing streaming job health, bottlenecks, and autoscaling behavior. This Skill provides structured, step-by-step guidance for the entire Dataflow lifecycle. ## Core Features & Use Cases - Pipeline Authoring & Templates: Checks for existing Google-provided Dataflow templates before writing custom code, and scaffolds new Java, Python, or Go Beam projects with consistent SDK versions. - Flex Template Packaging: Enforces best practices such as single Docker image configuration for Python, Cloud Build-based image builds, and secure secret handling via Secret Manager at runtime. - Job Execution & Monitoring: Walks through parameter validation, pre-launch confirmation, job triggering, and console URL generation for monitoring. - Diagnostics & Troubleshooting: Provides a structured RCA workflow using Cloud Monitoring metrics, Cloud Logging queries, and the Dataflow REST API to analyze streaming job health, bottlenecks, hot keys, and autoscaling anomalies. - Use Case: A user's streaming Dataflow job shows growing backlog and rising watermark delay. The Skill maps transform names to stages, queries metrics like job/is_bottleneck and job/backlogged_keys, correlates with worker logs, and produces a diagnosis with root causes and remediation steps. ## Quick Start Ask the assistant to create a new Apache Beam Dataflow pipeline from Pub/Sub to BigQuery, or to diagnose why a specific Dataflow job ID is running slowly.

Frequently Asked Questions about gcp-dataflow

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create an Apache Beam pipeline on Google Dataflow?▼

First check whether a Google-provided Dataflow template in gs://dataflow-templates already matches your source and sink. If not, create a custom pipeline in Java, Python, or Go using the latest Apache Beam SDK, and package it as a Flex Template for a reproducible launch environment.

How do I diagnose a slow or stuck Dataflow streaming job?▼

Correlate Cloud Monitoring metrics like job/data_watermark_age, job/is_bottleneck, and job/backlogged_keys with worker logs from Cloud Logging and job messages from the Dataflow REST API. Common root causes include hot keys, low key cardinality, and blocking external RPCs in DoFns.

Should I use a Flex Template or a Classic template on Dataflow?▼

Flex Templates are recommended for new pipelines because they provide a hermetic, reproducible launch environment packaged as a Docker image. Classic templates are older Google-provided templates whose metadata files end with _metadata in the dataflow-templates bucket.

How do I pass secrets or SSL certificates to Dataflow workers?▼

Never bake secrets into Docker image layers. Retrieve them at runtime inside the DoFn.setup() lifecycle using the Secret Manager client library, or pass comma-separated GCS paths via the extraFilesToStage parameter so files land in /extra_files on workers.

Why does my Dataflow autoscaling keep adding workers without reducing backlog?▼

This typically indicates undetected throttling: IO bottlenecks, hot keys, or insufficient parallelism keep CPU utilization low while backlog grows, so the autoscaler scales up to the max worker limit. Mitigate with worker utilization hints or by clamping min/max worker instance limits.

When should I not use this Dataflow guidance?▼

Do not use it for general GCP resource management unrelated to Dataflow, for issues with other services like GCE or BigQuery unless they directly impact pipeline execution, or for pipeline technologies other than Apache Beam on Dataflow.

Related Skills

Explore More Software Engineering Skills→
AradotsoAradotso
5

google-cloud-data-engineering-hub

Create scalable GCP data engineering pipelines with BigQuery and Dataflow.

Official
Advanced
tottenjordantottenjordan
1

gcp-dataflow

Guides authoring, packaging, launching, and diagnosing Apache Beam pipelines on Google Cloud Dataflow.

Community
Advanced
Fujigo-SoftwareFujigo-Software
24

devops

Automate CI/CD pipelines and infrastructure management for multi-service applications.

Official
Advanced
SmileAfterBurnSmileAfterBurn

gcp-data-pipelines

Routes data pipeline requests to the appropriate Google Cloud tool skill.

Community
Intermediate