skills.rest
tottenjordan
tottenjordanCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

gcp-dataflow

Guides authoring, packaging, launching, and diagnosing Apache Beam pipelines on Google Cloud Dataflow.

1|Updated Mar 19, 2026
Search Tags:#troubleshooting#gcp#streaming#pipeline#dataflow#apache beam#flex template
One-click install
npx skills add https://github.com/tottenjordan/me-skittles --skill gcp-dataflow-tottenjordan
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-dataflow
Source: https://github.com/tottenjordan/me-skittles/tree/main/gemini/gcp-dataflow
Command: npx skills add https://github.com/tottenjordan/me-skittles --skill gcp-dataflow-tottenjordan

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Building and operating Apache Beam pipelines on Google Cloud Dataflow involves many error-prone decisions: choosing between Google-provided templates and custom code, packaging Flex Templates correctly, validating required parameters, and diagnosing streaming job health, bottlenecks, and autoscaling behavior. This Skill encodes those workflows so pipelines are created, launched, and troubleshot correctly the first time. ## Core Features & Use Cases - Pipeline Authoring & Templates: Checks for existing Google-provided Dataflow templates before writing code, and scaffolds new Java, Python, or Go Beam projects with consistent SDK versions. - Flex Template Packaging: Builds Flex Templates with Cloud Build, single Docker image configuration for Python, and secure secret handling via Secret Manager at runtime. - Job Execution & Monitoring: Enforces pre-launch command confirmation, strict parameter validation from template metadata, and structured job monitoring with console URLs. - Diagnostics & Troubleshooting: Performs root cause analysis of streaming and batch jobs using Cloud Monitoring metrics, Cloud Logging queries, and Dataflow REST API, covering watermarks, backlogs, bottlenecks, hot keys, and autoscaling anomalies. - Use Case: A streaming pipeline shows growing backlog and low CPU. The Skill maps transform names to stages, queries job/is_bottleneck and job/backlogged_keys, identifies a hot key bottleneck, and recommends remediation with in-flight updates. ## Quick Start Ask the agent to create a new Dataflow Flex Template pipeline that reads from Pub/Sub and writes to BigQuery, or to diagnose why a specific Dataflow job ID is running slowly.

Frequently Asked Questions about gcp-dataflow

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create a Dataflow Flex Template for an Apache Beam pipeline?▼

Package the pipeline code into a Docker image, preferably built with Cloud Build using a provided cloudbuild.yaml. For Python, use the single Docker image configuration where the launcher and worker share one image, then launch with gcloud dataflow flex-template run.

How do I troubleshoot a slow Dataflow streaming job?▼

Correlate job messages from the Dataflow REST API with Cloud Monitoring metrics and Cloud Logging. Check data watermark age, backlog, job/is_bottleneck with likely_cause, and backlogged keys to identify hot keys, stuck operations, or downstream throttling.

Should I use a Google-provided Dataflow template or write a custom pipeline?▼

First list the public dataflow-templates bucket for Classic and Flex templates matching your source and sink. If a template matches, configure it using its metadata parameters; only build a custom Apache Beam pipeline when no template fits.

Why is my Dataflow autoscaling adding workers but backlog keeps growing?▼

This indicates undetected throttling: workers sit idle waiting on IO or hot keys serialize processing, so CPU stays low while backlog rises. Check job/horizontal_worker_scaling rationale, key parallelism metrics, and clamp worker limits or adjust utilization hints in-flight.

What parameters are required when running a Google Dataflow template?▼

Every parameter in the template metadata JSON without explicit isOptional: true is strictly required by the Dataflow API, even if descriptions suggest defaults. Read the metadata file from the dataflow-templates bucket and supply all required values.

When should I not use this Dataflow skill?▼

Do not use it for general GCP resource management unrelated to Dataflow, issues with other services like GCE or BigQuery unless they directly impact pipeline execution, or pipeline technologies other than Apache Beam on Dataflow.

Related Skills

Explore More Software Engineering Skills→
AradotsoAradotso
5

google-cloud-data-engineering-hub

Create scalable GCP data engineering pipelines with BigQuery and Dataflow.

Official
Advanced
grafanagrafana
213

prometheus

Analyze time-series metrics and create PromQL alerting and recording rules.

Official
Intermediate
awslabsawslabs
881

sdk-getting-started

Validates SageMaker SDK version, AWS region, and execution role before ML operations.

Official
Intermediate
trungenglishtrungenglish
1

gcp-data-pipelines

Routes Google Cloud data pipeline requests to the appropriate dbt, Dataflow, Dataform, Spark, or Composer skill.

Community
Intermediate