skills.rest
Yashyasik
YashyasikCOMMUNITY
Explore all skills by this author →
GitHub

gcp-spark

Generates and executes PySpark notebooks on Dataproc clusters and Serverless with BigQuery, Iceberg, and Spanner integration.

Updated Aug 11, 2026
Search Tags:#gcp#etl#spark#bigquery#pyspark#iceberg#dataproc
One-click install
npx skills add https://github.com/Yashyasik/zexca-api --skill gcp-spark-yashyasik
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-spark
Source: https://github.com/Yashyasik/zexca-api/tree/main/.gemini/skills/gcp-spark
Command: npx skills add https://github.com/Yashyasik/zexca-api --skill gcp-spark-yashyasik

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Writing correct, production-ready Spark code on Google Cloud requires deep knowledge of Dataproc execution modes, connector configurations, and data source schemas. This Skill enforces a structured workflow that discovers schemas first, generates validated PySpark notebooks, and executes them on Dataproc Clusters or Serverless without common mistakes like missing imports, schema mismatches, or misconfigured catalogs. ## Core Features & Use Cases - Guided Spark Code Generation: Produces Python notebooks (.ipynb) following a mandatory workflow: schema discovery, source verification, code generation, schema validation, compilation, and execution. - Multi-Source Data Integration: Provides verified patterns for reading and writing BigQuery, BigLake Iceberg catalogs (GCS and S3 storage), Spanner, Cloud SQL, GCS files, and Pub/Sub. - Dataproc Resource Management: Covers listing and submitting cluster jobs, serverless batches, and interactive sessions via gcloud or MCP tools, with connector dependencies for Spanner, PostgreSQL, Iceberg, and XGBoost. - ML on Spark: Includes verified patterns for XGBoost (SparkXGBClassifier), native Spark ML, and documented workarounds for LightGBM dependency conflicts on Serverless. - Use Case: Ask the AI to build an ETL pipeline that reads a CSV from GCS, enriches it with a BigQuery reference table, and writes the result to a BigLake Iceberg table, then submit it as a Serverless batch. ## Quick Start Use the gcp-spark skill to write a PySpark notebook that reads a CSV from my GCS bucket and writes it to a BigQuery table, then run it as a Dataproc Serverless batch.

Frequently Asked Questions about gcp-spark

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a PySpark job on Dataproc Serverless?▼

Submit a PySpark script as a batch using gcloud dataproc batches submit pyspark with your project, region, version 2.3, and a --deps-bucket for dependencies. Expect initial execution to take 10-15 minutes, which is normal behavior.

How do I read and write BigLake Iceberg tables with Spark?▼

Configure a Spark catalog pointing to the BigLake REST endpoint with the correct io-impl for GCS or S3 storage, then set the current catalog and use spark.read.format("iceberg") or the DataFrameWriterV2 writeTo API. Always look up the table's metadataPath first to determine the storage layer.

Can I use LightGBM on Dataproc Serverless?▼

LightGBM has dependency conflicts on Dataproc Serverless because the SynapseML wrapper conflicts with internal libraries. Use XGBoost, native Spark ML like GBTClassifier, Vertex AI, or a dedicated Dataproc cluster with autoscaling disabled instead.

Why does my SparkXGBClassifier fail when setting the objective parameter?▼

SparkXGBClassifier does not allow a custom objective parameter; it is inferred automatically as binary:logistic for classifiers. Setting it explicitly raises a ValueError, so omit it and also disable dynamic allocation.

What IAM roles does the Dataproc service account need?▼

The service account needs roles/dataproc.worker for job execution, roles/biglake.admin for Iceberg table management, roles/bigquery.jobUser for query materialization, roles/storage.objectUser for GCS access, and roles/spanner.databaseUser for Spanner writes.

When should I not use Spark on Dataproc?▼

Avoid Spark for generic Python scripts that do not use Spark APIs and for simple SQL queries that can run directly in BigQuery. Spark adds cluster provisioning overhead that is unnecessary for lightweight single-node tasks.

Related Skills

Explore More Data & Analytics Skills→
trungenglishtrungenglish
1

gcp-spark

Develops and executes Spark code on Google Cloud Dataproc clusters and serverless batches.

Community
Advanced
microwindmicrowind
65

data-pipeline

Ingest, transform, and load data from multiple sources into warehouses.

Official
Advanced
gemini-cli-extensionsgemini-cli-extensions
161

gcp-spark

Coordinate Spark workloads on Google Cloud Dataproc and Serverless.

Official
Advanced
2nth-ai2nth-ai
1

tech/google/cloud/data

Query BigQuery, publish to Pub/Sub, and run Dataflow pipelines.

Official
Advanced