skills.rest
DingJun1028
DingJun1028COMMUNITY
Explore all skills by this author →
GitHub

gcp-spark

Generates and executes PySpark code on Dataproc clusters and Serverless with BigQuery, Iceberg, and Spanner integration.

Updated Aug 17, 2026
Search Tags:#gcp#etl#spark#bigquery#pyspark#iceberg#dataproc
One-click install
npx skills add https://github.com/DingJun1028/esggo-kv --skill gcp-spark-dingjun1028
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-spark
Source: https://github.com/DingJun1028/esggo-kv/tree/main/.agents/skills/gcp-spark
Command: npx skills add https://github.com/DingJun1028/esggo-kv --skill gcp-spark-dingjun1028

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Writing correct Spark code for Google Cloud Dataproc requires knowing exact catalog configurations, connector options, and platform-specific pitfalls. This Skill enforces a verified workflow that inspects schemas before coding, compiles notebooks before execution, and applies proven read/write patterns for BigQuery, BigLake Iceberg, GCS, and Spanner. ## Core Features & Use Cases - Guided Spark Code Generation: Produces Python notebooks (.ipynb) following a mandatory task execution workflow with schema verification and compile checks. - Data Source Integration: Provides verified patterns for reading and writing BigQuery tables, BigLake Iceberg catalogs (GCS and S3 storage), GCS files, and Spanner databases. - Dataproc Resource Management: Covers cluster and serverless batch submission, session management, and job listing via gcloud or MCP tools. - ML Task Guidance: Includes validated patterns for XGBoost and native Spark ML, with warnings about LightGBM dependency conflicts on Serverless. - Use Case: A data engineer needs to build an ETL pipeline reading from BigQuery, transforming data, and writing to a BigLake Iceberg table. The Skill generates a notebook with the correct catalog configuration, schema verification, and DataFrameWriterV2 write mode. ## Quick Start Ask the AI to write a Spark notebook that reads a BigQuery table and writes the results to a BigLake Iceberg catalog on Dataproc Serverless.

Frequently Asked Questions about gcp-spark

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I write Spark code to run on Dataproc Serverless?▼

Generate a PySpark script and submit it with gcloud dataproc batches submit pyspark, specifying --version=2.3 and a --deps-bucket GCS path for dependencies. Expect initial batch execution to take 10-15 minutes, which is normal behavior.

How to read and write BigLake Iceberg tables from Spark?▼

Configure the Spark session with the BigLake REST catalog URI, warehouse path, and GoogleAuthManager, then set the current catalog with spark.catalog.setCurrentCatalog. Write using the DataFrameWriterV2 API such as writeTo().using("iceberg").append(), always surrounding catalog names with backticks.

Can I use LightGBM with Spark on Dataproc Serverless?▼

LightGBM has dependency conflicts on Dataproc Serverless because the SynapseML wrapper conflicts with internal libraries. Use XGBoost with SparkXGBClassifier, native Spark ML like GBTClassifier, or train on Vertex AI instead.

Why does my Spark job fail when writing to an Iceberg table?▼

Common causes include missing catalog configuration, mismatched DataFrame and table schemas, or not setting the current catalog before writing. Verify schemas with df.printSchema() and ensure the namespace exists with CREATE NAMESPACE IF NOT EXISTS.

What IAM roles does Dataproc need for Spark jobs?▼

The Dataproc service account needs roles/dataproc.worker for job execution, roles/biglake.admin for Iceberg table management, roles/bigquery.jobUser for query materialization, roles/storage.objectUser for GCS access, and roles/spanner.databaseUser for Spanner writes.

Related Skills

Explore More Data & Analytics Skills→
trungenglishtrungenglish
1

gcp-spark

Develops and executes Spark code on Google Cloud Dataproc clusters and serverless batches.

Community
Advanced
microwindmicrowind
65

data-pipeline

Ingest, transform, and load data from multiple sources into warehouses.

Official
Advanced
AubreyHanAubreyHan

gcp-spark

Build and validate Spark workflows for Google Cloud Dataproc.

Community
Advanced
2nth-ai2nth-ai
1

tech/google/cloud/data

Query BigQuery, publish to Pub/Sub, and run Dataflow pipelines.

Official
Advanced