skills.rest
Lathika-laa
Lathika-laaCOMMUNITY
Explore all skills by this author →
GitHub

gcp-spark

Generates and executes PySpark notebooks on Dataproc clusters and Serverless with BigQuery, Iceberg, and Spanner.

Updated Aug 26, 2026
Search Tags:#gcp#etl#spark#bigquery#pyspark#iceberg#dataproc
One-click install
npx skills add https://github.com/Lathika-laa/Recipe_Box --skill gcp-spark-lathika-laa
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-spark
Source: https://github.com/Lathika-laa/Recipe_Box/tree/main/.github/.gemini/skills/gcp-spark
Command: npx skills add https://github.com/Lathika-laa/Recipe_Box --skill gcp-spark-lathika-laa

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Writing correct Spark code for Google Cloud requires knowing Dataproc submission commands, connector configurations for BigQuery, BigLake Iceberg, and Spanner, and avoiding common pitfalls like schema mismatches and missing imports. This Skill guides the AI through a verified workflow that inspects schemas, generates validated PySpark notebooks, and executes them on Dataproc Clusters or Serverless. ## Core Features & Use Cases - Guided Spark Code Generation: Produces Python notebooks (.ipynb) following a strict workflow: schema discovery, source verification, code generation, schema validation, and compilation checks before execution. - Data Source Integration: Provides verified read/write patterns for BigQuery, BigLake Iceberg catalogs (GCS and S3 storage), Cloud Spanner, GCS files, Cloud SQL, and Pub/Sub. - Dataproc Resource Management: Covers listing and submitting clusters, jobs, serverless batches, and interactive sessions via gcloud or MCP tools, including connector dependencies and runtime properties. - ML Task Support: Includes verified patterns for XGBoost and native Spark ML training, with warnings about LightGBM conflicts on Serverless. - Use Case: Ask the AI to build an ETL pipeline that reads a CSV from GCS, enriches it with a BigQuery reference table, and writes the result to a BigLake Iceberg table, then submit it as a Dataproc Serverless batch. ## Quick Start Use the gcp-spark skill to write a PySpark notebook that reads a CSV from my GCS bucket and writes it to a BigQuery table.

Frequently Asked Questions about gcp-spark

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I submit a PySpark job to Dataproc Serverless?▼

Use gcloud dataproc batches submit pyspark with your script path, project, region, version 2.3, and a --deps-bucket pointing to a GCS path for dependencies. Expect initial execution to take 10-15 minutes, which is normal behavior.

How do I read and write BigLake Iceberg tables from Spark?▼

Configure a Spark catalog pointing to the BigLake REST endpoint with the correct io-impl for GCS or S3 storage, then set the current catalog and use spark.read.format("iceberg") or the DataFrameWriterV2 writeTo API. Always look up the table's metadataPath first to determine the storage layer.

Can I use LightGBM on Dataproc Serverless?▼

LightGBM has dependency conflicts on Dataproc Serverless because the SynapseML wrapper conflicts with Dataproc's internal libraries. Use XGBoost, native Spark ML like GBTClassifier, or train on Vertex AI or a dedicated Dataproc cluster instead.

Why does my SparkXGBClassifier fail with a ValueError?▼

SparkXGBClassifier does not allow setting a custom objective parameter; the objective is inferred automatically from the classifier type. Remove the objective argument and ensure dynamic allocation is disabled with spark.dynamicAllocation.enabled=false.

What IAM roles does the Dataproc service account need?▼

The service account needs roles/dataproc.worker for job execution, roles/biglake.admin for Iceberg table management, roles/bigquery.jobUser for query materialization, roles/storage.objectUser for GCS access, and roles/spanner.databaseUser for Spanner writes.

When should I not use Spark on Dataproc?▼

Avoid Spark for generic Python scripts that do not use Spark APIs and for simple SQL queries that can run directly in BigQuery. Spark adds cluster provisioning overhead that is unnecessary for lightweight single-node tasks.

Related Skills

Explore More Data & Analytics Skills→
trungenglishtrungenglish
1

gcp-spark

Develops and executes Spark code on Google Cloud Dataproc clusters and serverless batches.

Community
Advanced
gemini-cli-extensionsgemini-cli-extensions
161

gcp-spark

Coordinate Spark workloads on Google Cloud Dataproc and Serverless.

Official
Advanced
ivanshamaevivanshamaev
14

mlflow-data-pipelines

Configure MLflow tracking and Model Registry for ETL and training runs.

Community
Advanced
ssmleossmleo

bigquery-ai-ml

Run AI and machine learning functions for BigQuery SQL queries.

Community
Advanced