skills.rest
Jatinkrmahato993203
Jatinkrmahato993203COMMUNITY
Explore all skills by this author →
GitHub

gcp-dataflow

Guides writing, packaging, launching, and diagnosing Apache Beam pipelines on Google Cloud Dataflow.

Updated Jul 20, 2026
Search Tags:#troubleshooting#gcp#streaming#pipeline#dataflow#apache beam#flex template
One-click install
npx skills add https://github.com/Jatinkrmahato993203/crime --skill gcp-dataflow-jatinkrmahato993203
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-dataflow
Source: https://github.com/Jatinkrmahato993203/crime/tree/main/skills/gcp-dataflow
Command: npx skills add https://github.com/Jatinkrmahato993203/crime --skill gcp-dataflow-jatinkrmahato993203

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Building and running Apache Beam pipelines on Google Cloud Dataflow involves many error-prone decisions: SDK version alignment, Flex Template packaging, Docker image configuration, and diagnosing slow or failing jobs. This Skill provides expert operational guidance so pipelines are structured, launched, and debugged correctly. ## Core Features & Use Cases - Project & Pipeline Setup: Guides creation of Java (Gradle), Python, or Go Beam projects with consistent SDK versioning and correct dependency configuration (e.g., SLF4J alignment, no Shadow plugin by default). - Flex Template Packaging: Enforces the Single Docker Image configuration for Python Flex Templates and prefers Cloud Build over local Docker, with ready-to-use cloudbuild.yaml guidance. - Launch & Diagnostics: Detects requirements.txt/setup.py to set correct runner options, and provides a full troubleshooting workflow using Dataflow REST API, Cloud Monitoring metrics, and Cloud Logging queries to root-cause bottlenecks, backlog, and watermark lag. - Use Case: A streaming Dataflow job is falling behind. The Skill maps your transform names to stage IDs, queries metrics like system_lag and is_bottleneck, correlates them with logs, and returns a structured root-cause analysis with recommendations. ## Quick Start Help me create a Python Apache Beam pipeline packaged as a Dataflow Flex Template and diagnose why my streaming job is slow.

Frequently Asked Questions about gcp-dataflow

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create an Apache Beam pipeline for Google Cloud Dataflow?▼

Confirm the language (Java, Python, or Go), then use the latest Apache Beam SDK version consistently across Dockerfiles and requirements files. Structure the project as a Flex Template for a reproducible launch environment.

How do I build a Python Dataflow Flex Template with custom dependencies?▼

Use the Single Docker Image configuration: one image serves as both the Flex Template launcher and the worker SDK container. Install dependencies inside the image and pass it via the sdk_container_image parameter at launch.

Should I use Cloud Build or local Docker for Dataflow template images?▼

Prefer Cloud Build, since local Docker may not be available on the workspace machine. Provide a cloudbuild.yaml for building and pushing images, and share the link to monitor the long-running build operation.

How do I debug a slow or failing Dataflow job?▼

Extract the job ID, project, and time window, then map transform names to stage IDs using gcloud dataflow jobs describe. Query Monitoring metrics like system_lag, backlog_bytes, and is_bottleneck, and correlate them with Cloud Logging errors to identify root causes.

What metrics indicate Dataflow streaming pipeline bottlenecks?▼

Key metrics include per-stage data watermark age, system lag, estimated backlog processing time, is_bottleneck, and backlogged keys. For Streaming Engine jobs, also check persistent state read/write bytes and processing parallelism keys.

When launching a Python pipeline with DataflowRunner, which options are required?▼

Scan the project directory first: if requirements.txt exists, include --requirements_file; if setup.py exists, include --setup_file so local modules are staged. Confirm the final launch command with the user before running.

Related Skills

Explore More Software Engineering Skills→
markus41markus41
1

gcp

Automate provisioning, deployment, and orchestration of Google Cloud Platform resources via CLI, SDKs, and Terraform.

Community
Advanced
AubreyHanAubreyHan

gcp-data-pipelines

Route Google Cloud data pipeline requests to the correct build, deployment, or orchestration skill.

Community
Advanced
first-flukefirst-fluke
1.2k

tf-infra-agent

Automates Terraform provisioning across AWS, GCP, Azure, and Oracle Cloud infrastructure.

Official
Advanced
SmileAfterBurnSmileAfterBurn

gcp-spark

Generates and executes Spark code on Google Cloud Dataproc clusters and serverless batches.

Community
Advanced