skills.rest
Yashyasik
YashyasikCOMMUNITY
Explore all skills by this author →
GitHub

gcp-dataflow

Guides authoring, packaging, launching, and diagnosing Apache Beam pipelines on Google Cloud Dataflow.

Updated Aug 11, 2026
Search Tags:#troubleshooting#gcp#streaming#pipeline#dataflow#apache beam#flex template
One-click install
npx skills add https://github.com/Yashyasik/zexca-api --skill gcp-dataflow-yashyasik
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-dataflow
Source: https://github.com/Yashyasik/zexca-api/tree/main/.gemini/skills/gcp-dataflow
Command: npx skills add https://github.com/Yashyasik/zexca-api --skill gcp-dataflow-yashyasik

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Building and operating Apache Beam pipelines on Google Cloud Dataflow involves many error-prone decisions: choosing between Google-provided templates and custom code, packaging Flex Templates with Docker and Cloud Build, validating strict template parameters, and diagnosing streaming job health, bottlenecks, and autoscaling behavior. This Skill encodes those workflows so pipelines are created, launched, and troubleshot correctly. ## Core Features & Use Cases - Template Discovery & Configuration: Checks the public dataflow-templates bucket for matching Classic or Flex templates, reads template metadata, enforces strict required-parameter validation, and prepares sinks such as BigQuery tables and dead-letter tables. - Custom Pipeline Authoring & Packaging: Sets up Java (Gradle), Python, or Go Beam projects with consistent SDK versions, and packages pipelines as Flex Templates using a single Docker image built with Cloud Build. - Job Execution & Monitoring: Drafts launch commands with real project values, requires user confirmation before execution, and monitors job status with console URLs. - Diagnostics & Troubleshooting: Correlates Monitoring metrics, Cloud Logging, and Dataflow REST API job messages to classify streaming job health, identify bottlenecks and hot keys, and analyze autoscaling behavior. - Use Case: A user asks why their Pub/Sub-to-BigQuery streaming pipeline has growing backlog; the Skill maps transform names to stages, queries watermark, backlog, and bottleneck metrics, and produces a structured diagnosis with remediation steps. ## Quick Start Ask the assistant to create a new Apache Beam Dataflow pipeline from Pub/Sub to BigQuery, or to diagnose the performance of an existing Dataflow job by providing its job ID and project.

Frequently Asked Questions about gcp-dataflow

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create a Dataflow Flex Template for an Apache Beam pipeline?▼

Package the pipeline code into a Docker image, preferably built with Cloud Build using a provided cloudbuild.yaml, and register it as a Flex Template. For Python, use a single image for both the launcher and the worker runtime, passing it via the sdk_container_image parameter at launch.

How do I check if a Google-provided Dataflow template already exists?▼

List the public dataflow-templates bucket with gcloud storage ls: gs://dataflow-templates/latest for Classic templates and gs://dataflow-templates/latest/flex for Flex templates. Match templates by source and sink, then read the template metadata file to identify required parameters.

How do I diagnose a slow or unhealthy Dataflow streaming job?▼

Classify health by data watermark age, then correlate Monitoring metrics like job/is_bottleneck, backlog, and watermark age with Cloud Logging worker errors and Dataflow job messages. Segment the timeline into behavioral phases and corroborate root causes across multiple telemetry sources.

Why is my Dataflow autoscaling adding workers while backlog keeps growing?▼

This indicates undetected throttling: IO bottlenecks, hot keys, or insufficient key parallelism leave workers idle with low CPU while backlog rises, so the autoscaler scales to the max worker limit. Mitigate with in-flight updates to worker utilization hints or min/max worker clamps.

Which Apache Beam SDK version should I use for a new Dataflow pipeline?▼

Search for the most recently released Apache Beam version unless the user already specifies one. Use that same version consistently across Dockerfiles, requirements.txt, and any other files where versions are declared.

When should I not use this Dataflow skill?▼

Do not use it for general GCP resource management unrelated to Dataflow, issues with other services like GCE, GCS, or BigQuery unless they directly impact Dataflow execution, or pipeline technologies other than Apache Beam on Dataflow.

Related Skills

Explore More Software Engineering Skills→
AradotsoAradotso
5

google-cloud-data-engineering-hub

Create scalable GCP data engineering pipelines with BigQuery and Dataflow.

Official
Advanced
runchr-comrunchr-com

metrics-dashboard

Design product metrics dashboards with KPIs, data sources, and alert thresholds.

Official
Intermediate
haryelramalhoharyelramalho

devops-engineer

Automate DevOps infrastructure and deployment workflows for containerized applications.

Community
Advanced
SmileAfterBurnSmileAfterBurn

gcp-data-pipelines

Routes data pipeline requests to the appropriate Google Cloud tool skill.

Community
Intermediate