skills.rest
1919114514yasenpai-maker
1919114514yasenpai-makerCOMMUNITY
Explore all skills by this author →
GitHub

gcp-spark

Generates and executes PySpark notebooks on Dataproc clusters and Serverless batches.

Updated Aug 16, 2026
Search Tags:#gcp#etl#spark#bigquery#pyspark#iceberg#dataproc
One-click install
npx skills add https://github.com/1919114514yasenpai-maker/focus-quest --skill gcp-spark-1919114514yasenpai-maker
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-spark
Source: https://github.com/1919114514yasenpai-maker/focus-quest/tree/main/.gemini/skills/gcp-spark
Command: npx skills add https://github.com/1919114514yasenpai-maker/focus-quest --skill gcp-spark-1919114514yasenpai-maker

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Writing and running Spark jobs on Google Cloud involves many error-prone details: configuring BigLake Iceberg catalogs, choosing correct connector jars, matching DataFrame schemas to destination tables, and submitting jobs with the right gcloud flags. This Skill enforces a verified workflow that prevents common failures like schema mismatches, missing imports, and misconfigured sessions. ## Core Features & Use Cases - Guided Spark Code Generation: Produces Python notebooks (.ipynb) following a mandatory workflow: schema discovery, source verification, code generation, schema validation, and compilation before execution. - Data Source Integration: Provides verified patterns for reading and writing BigQuery, BigLake Iceberg catalogs, Spanner, GCS, Cloud SQL, and Pub/Sub from Spark. - Dataproc Resource Management: Covers listing and submitting clusters, jobs, serverless batches, and interactive sessions via gcloud or MCP tools. - Use Case: A data engineer needs an ETL pipeline that reads CSV files from GCS, enriches them with a BigQuery reference table, and writes results to a BigLake Iceberg table. The Skill generates a validated notebook with the correct catalog configuration, broadcast joins, and write modes. ## Quick Start Ask the assistant to write a Spark notebook that reads a CSV from a GCS bucket and writes it to a BigLake Iceberg table on Dataproc Serverless.

Frequently Asked Questions about gcp-spark

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I write a Spark DataFrame to a BigLake Iceberg table?▼

Configure the Spark session with the BigLake REST catalog, set the current catalog with spark.catalog.setCurrentCatalog, create the namespace if needed, then use the DataFrameWriterV2 API such as df.writeTo(...).using("iceberg").append(). Always verify the DataFrame schema matches the destination table first.

How do I submit a PySpark job to Dataproc Serverless?▼

Use gcloud dataproc batches submit pyspark with your script path, project, region, version 2.3, and a --deps-bucket pointing to a GCS path for dependencies. Expect initial execution to take 10-15 minutes, which is normal behavior.

Can I use LightGBM on Dataproc Serverless?▼

LightGBM has dependency conflicts on Dataproc Serverless because the SynapseML wrapper conflicts with Dataproc's internal libraries. Recommended alternatives are XGBoost via SparkXGBClassifier, native Spark ML like GBTClassifier, or training on Vertex AI.

Why does my Spark job fail when reading CSV files from GCS?▼

Reading CSV without the header and inferSchema options causes the header row to become data and all columns to load as strings. Use spark.read.option("header", "true").option("inferSchema", "true").csv(path) to read CSV files correctly.

When should I not use Spark on Dataproc?▼

Avoid Spark for generic Python scripts that do not use Spark APIs and for simple SQL queries that can run directly in BigQuery. Spark adds cluster provisioning overhead that is unnecessary for lightweight single-node or SQL-only tasks.

Related Skills

Explore More Data & Analytics Skills→
AubreyHanAubreyHan

gcp-spark

Build and validate Spark workflows for Google Cloud Dataproc.

Community
Advanced
ToXMonToXMon

domino-jobs

Automate Domino batch job orchestration, scheduling, and monitoring via API or CLI.

Community
Advanced
ivanshamaevivanshamaev
14

mlflow-data-pipelines

Configure MLflow tracking and Model Registry for ETL and training runs.

Community
Advanced
clawdataclawdata
24

spark

Automate building and running Apache Spark batch and streaming data processing jobs.

Official
Intermediate