skills.rest
SmileAfterBurn
SmileAfterBurnCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

gcp-spark

Generates and executes Spark code on Google Cloud Dataproc clusters and serverless batches.

Updated Aug 26, 2026
Search Tags:#gcp#etl#spark#bigquery#pyspark#iceberg#dataproc
One-click install
npx skills add https://github.com/SmileAfterBurn/pani-dumka-ai --skill gcp-spark-smileafterburn
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-spark
Source: https://github.com/SmileAfterBurn/pani-dumka-ai/tree/main/.gemini/skills/gcp-spark
Command: npx skills add https://github.com/SmileAfterBurn/pani-dumka-ai --skill gcp-spark-smileafterburn

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Writing correct Spark code for Google Cloud requires knowing Dataproc cluster versus serverless execution models, connector configurations for BigQuery, BigLake Iceberg, and Spanner, and avoiding common PySpark pitfalls. This Skill enforces a structured workflow that inspects schemas first, generates validated notebooks, and executes jobs correctly on GCP. ## Core Features & Use Cases - Guided Spark Code Generation: Produces Python notebooks (.ipynb) following a mandatory workflow: schema discovery, source verification, code generation, schema validation, and compilation before execution. - Data Source Integration: Provides verified read/write patterns for BigQuery, BigLake Iceberg catalogs (GCS and S3 storage), Cloud Spanner, Cloud SQL, GCS files, and Pub/Sub. - Dataproc Resource Management: Covers cluster and serverless batch submission, session management, connector dependencies, and IAM requirements via gcloud or MCP tools. - ML on Spark: Includes verified patterns for XGBoost and native Spark ML, with warnings about LightGBM dependency conflicts on Dataproc Serverless. - Use Case: A data engineer needs an ETL pipeline reading CSV files from GCS, enriching them with a BigQuery reference table, and writing results to a BigLake Iceberg table. The Skill generates a validated notebook with the correct catalog configuration and write mode. ## Quick Start Ask the AI to write a Spark ETL pipeline on Dataproc that reads a CSV from a GCS bucket and writes the results to a BigQuery table.

Frequently Asked Questions about gcp-spark

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a PySpark job on Dataproc Serverless?▼

Submit a PySpark script as a serverless batch using gcloud dataproc batches submit pyspark with your script path, project, region, version 2.3, and a --deps-bucket GCS path for dependencies. Expect initial execution to take 10-15 minutes, which is normal behavior.

How do I read and write BigLake Iceberg tables with Spark?▼

Configure the Spark session with the Iceberg REST catalog pointing to https://biglake.googleapis.com/iceberg/v1/restcatalog, choosing GCSFileIO or S3FileIO based on the table's metadataPath. Set the current catalog, then read with spark.read.format("iceberg") or write using the DataFrameWriterV2 writeTo API.

Can I use LightGBM on Dataproc Serverless?▼

LightGBM has dependency conflicts on Dataproc Serverless because the SynapseML wrapper conflicts with Dataproc's internal libraries. Use XGBoost, native Spark ML like GBTClassifier, train on Vertex AI, or use a dedicated Dataproc cluster with autoscaling disabled instead.

Why does my Spark job fail when writing to an Iceberg table?▼

Common causes include a dataframe schema that does not match the target table, a missing namespace, or an unconfigured catalog. Verify schemas with df.printSchema(), create the namespace with CREATE NAMESPACE IF NOT EXISTS, and set the current catalog before writing.

What IAM roles does Dataproc need for Spark jobs?▼

The Dataproc service account needs roles/dataproc.worker for job execution, roles/biglake.admin for Iceberg table management, roles/bigquery.jobUser for query materialization, roles/storage.objectUser for GCS access, and roles/spanner.databaseUser for Spanner writes.

When should I not use Spark on Dataproc?▼

Avoid Spark for generic Python scripts that do not use Spark APIs and for simple SQL queries that can run directly in BigQuery. Spark adds cluster provisioning overhead that is unnecessary for lightweight single-node or query-only tasks.

Related Skills

Explore More Data & Analytics Skills→
trungenglishtrungenglish
1

gcp-spark

Develops and executes Spark code on Google Cloud Dataproc clusters and serverless batches.

Community
Advanced
hmzainjamilhmzainjamil
1

data-pipeline

Build automated ETL pipelines extracting, transforming, and loading data across systems.

Community
Advanced
gemini-cli-extensionsgemini-cli-extensions
161

gcp-spark

Coordinate Spark workloads on Google Cloud Dataproc and Serverless.

Official
Advanced
2nth-ai2nth-ai
1

tech/google/cloud/data

Query BigQuery, publish to Pub/Sub, and run Dataflow pipelines.

Official
Advanced