skills.rest
arslan9024
arslan9024COMMUNITY
Explore all skills by this author →
GitHub

gcp-spark

Generates and executes Spark code on Google Cloud Dataproc clusters and Serverless batches.

Updated Jan 8, 2026
Search Tags:#gcp#etl#spark#bigquery#pyspark#iceberg#dataproc
One-click install
npx skills add https://github.com/arslan9024/White-Caves --skill gcp-spark-arslan9024
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-spark
Source: https://github.com/arslan9024/White-Caves/tree/main/.agents/skills/gcp_spark
Command: npx skills add https://github.com/arslan9024/White-Caves --skill gcp-spark-arslan9024

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Writing correct Spark code for Google Cloud Dataproc requires knowing the right connectors, catalog configurations, IAM roles, and submission commands. This Skill guides the generation, verification, and execution of Spark ETL and ML workloads on Dataproc Clusters and Serverless, preventing common mistakes like schema mismatches, missing imports, and misconfigured BigLake Iceberg catalogs. ## Core Features & Use Cases - Spark Code Generation: Produces Python notebooks (.ipynb) or PySpark scripts following a strict task execution workflow with schema discovery before coding. - Data Source Integration: Reads and writes BigQuery tables, BigLake Iceberg catalogs (GCS and S3 storage), Cloud Spanner, and GCS files (CSV, Parquet, JSON) with verified connector configurations. - Cluster & Job Management: Lists clusters, jobs, batches, and sessions via gcloud or MCP tools, and submits Serverless batches with correct properties for Iceberg, Spanner, and XGBoost. - Use Case: A data engineer needs to build an ETL pipeline that reads a CSV from GCS, enriches it with a BigQuery reference table, and writes the result to a BigLake Iceberg table. The Skill generates the notebook, verifies schemas, compiles the code, and provides the correct batch submission command. ## Quick Start Ask the assistant to write a Spark notebook that reads a BigQuery table and writes the results to a BigLake Iceberg catalog on Dataproc Serverless.

Frequently Asked Questions about gcp-spark

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I submit a PySpark job to Dataproc Serverless?▼

Use gcloud dataproc batches submit pyspark with your script path, project, region, version 2.3, and a --deps-bucket pointing to a GCS path for dependencies. Expect initial execution to take 10-15 minutes, which is normal behavior.

How do I read and write BigLake Iceberg tables from Spark?▼

Configure the Spark session with the BigLake REST catalog URI, warehouse path, and GoogleAuthManager, then set the current catalog with spark.catalog.setCurrentCatalog. Write using the DataFrameWriterV2 writeTo API with modes like createOrReplace or append.

Can I use LightGBM on Dataproc Serverless?▼

LightGBM has dependency conflicts on Dataproc Serverless because the SynapseML wrapper conflicts with internal libraries. Use XGBoost, native Spark ML like GBTClassifier, or train on Vertex AI instead.

Why does my Spark XGBoost training fail with a ValueError?▼

SparkXGBClassifier does not allow setting a custom objective parameter since it is inferred automatically. Remove the objective argument and also disable dynamic allocation with spark.dynamicAllocation.enabled=false.

What IAM roles does the Dataproc service account need?▼

The service account needs roles/dataproc.worker for job execution, roles/biglake.admin for Iceberg tables, roles/bigquery.jobUser for query materialization, roles/storage.objectUser for GCS access, and roles/spanner.databaseUser for Spanner writes.

When should I not use Spark on Dataproc?▼

Avoid it for generic Python scripts that do not use Spark and for simple SQL queries that can run directly in BigQuery. Dataproc is intended for distributed ETL pipelines, ML training, and large-scale data processing.

Related Skills

Explore More Data & Analytics Skills→
trungenglishtrungenglish
1

gcp-spark

Develops and executes Spark code on Google Cloud Dataproc clusters and serverless batches.

Community
Advanced
manutejmanutej
61

apache-spark-data-processing

Teach Apache Spark data processing across RDDs, DataFrames, Spark SQL, streaming, and MLlib.

Community
Advanced
ivanshamaevivanshamaev
14

mlflow-data-pipelines

Configure MLflow tracking and Model Registry for ETL and training runs.

Community
Advanced
ssmleossmleo

bigquery-ai-ml

Run AI and machine learning functions for BigQuery SQL queries.

Community
Advanced