skills.rest
Jatinkrmahato993203
Jatinkrmahato993203COMMUNITY
Explore all skills by this author →
GitHub

gcp-spark

Develops and executes Spark code on Google Cloud Dataproc clusters and Serverless.

Updated Jul 20, 2026
Search Tags:#gcp#etl#spark#bigquery#pyspark#iceberg#dataproc
One-click install
npx skills add https://github.com/Jatinkrmahato993203/crime --skill gcp-spark-jatinkrmahato993203
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-spark
Source: https://github.com/Jatinkrmahato993203/crime/tree/main/skills/gcp-spark
Command: npx skills add https://github.com/Jatinkrmahato993203/crime --skill gcp-spark-jatinkrmahato993203

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Writing and running Spark jobs on Google Cloud Dataproc involves many error-prone details: configuring Iceberg catalogs, choosing between clusters and serverless batches, wiring up BigQuery or Spanner connectors, and avoiding common PySpark mistakes. This Skill enforces a verified workflow so generated Spark code compiles, matches target schemas, and runs correctly the first time. ## Core Features & Use Cases - Guided Spark Code Generation: Produces Python notebooks (.ipynb) by default for ETL pipelines and ML workloads, with mandatory schema discovery before writing any code. - Data Source Integration: Provides verified patterns for reading and writing BigQuery, BigLake Iceberg catalogs (GCS and S3 storage), Google Cloud Storage, and Spanner. - Dataproc Resource Management: Covers listing and submitting cluster jobs, serverless batches, and interactive sessions via gcloud or MCP tools. - ML on Spark: Includes validated guidance for XGBoost, native Spark ML, and known LightGBM dependency conflicts on Dataproc Serverless. - Use Case: A data engineer needs to build an ETL pipeline that reads CSV files from GCS, enriches them with a BigQuery reference table, and writes results to a BigLake Iceberg table. The Skill generates a notebook with correct catalog configuration, broadcast joins, and schema verification. ## Quick Start Ask the AI to write a Spark notebook that reads a CSV from a GCS bucket and writes it to a BigLake Iceberg table on Dataproc Serverless.

Frequently Asked Questions about gcp-spark

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a PySpark script on Dataproc Serverless?▼

Submit the script as a serverless batch using gcloud dataproc batches submit pyspark with your project, region, version 2.3, and a --deps-bucket pointing to a GCS path. Expect initial execution to take 10-15 minutes, which is normal behavior.

How do I read and write BigLake Iceberg tables from Spark?▼

Configure a SparkCatalog in the Spark session pointing to the BigLake REST endpoint with the correct warehouse and io-impl for GCS or S3 storage. Set the current catalog, then read with spark.read.format("iceberg") and write using the DataFrameWriterV2 writeTo API.

Can I use LightGBM on Dataproc Serverless?▼

LightGBM has dependency conflicts on Dataproc Serverless because the SynapseML wrapper clashes with internal libraries. Use XGBoost, native Spark ML like GBTClassifier, train on Vertex AI, or use a dedicated Dataproc cluster with the SynapseML package instead.

Why does my SparkXGBClassifier fail when I set the objective parameter?▼

SparkXGBClassifier does not allow a custom objective parameter; it is inferred automatically as binary:logistic for classifiers. Setting it raises a ValueError, so omit objective and also disable dynamic allocation for XGBoost workloads.

What IAM roles does the Dataproc service account need for Spark jobs?▼

The service account needs roles/dataproc.worker for job execution, roles/biglake.admin for Iceberg table management, roles/bigquery.jobUser for query materialization, roles/storage.objectUser for GCS access, and roles/spanner.databaseUser for Spanner writes.

When should I not use Spark on Dataproc for data processing?▼

Avoid it for generic Python scripts that do not use Spark and for simple SQL queries that can run directly in BigQuery. Spark adds cluster startup overhead that is unnecessary for lightweight single-node or query-only tasks.

Related Skills

Explore More Data & Analytics Skills→
trungenglishtrungenglish
1

gcp-spark

Develops and executes Spark code on Google Cloud Dataproc clusters and serverless batches.

Community
Advanced
YashyasikYashyasik

gcp-spark

Generates and executes PySpark notebooks on Dataproc clusters and Serverless with BigQuery, Iceberg, and Spanner integration.

Community
Advanced
Datus-aiDatus-ai
1.5k

data-profiler

Profile data quality metrics and value distributions for tabular datasets.

Official
Basic
gemini-cli-extensionsgemini-cli-extensions
161

gcp-spark

Coordinate Spark workloads on Google Cloud Dataproc and Serverless.

Official
Advanced