skills.rest
trungenglish
trungenglishCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

gcp-spark

Develops and executes Spark code on Google Cloud Dataproc clusters and serverless batches.

1|Updated Jul 4, 2026
Search Tags:#gcp#etl#spark#bigquery#pyspark#iceberg#dataproc
One-click install
npx skills add https://github.com/trungenglish/SHOPWISE --skill gcp-spark-trungenglish
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-spark
Source: https://github.com/trungenglish/SHOPWISE/tree/main/.agents/skills/gcp-spark
Command: npx skills add https://github.com/trungenglish/SHOPWISE --skill gcp-spark-trungenglish

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Writing and running Spark jobs on Google Cloud Dataproc involves many error-prone details: configuring BigLake Iceberg catalogs, choosing the right connectors for BigQuery and Spanner, submitting serverless batches with correct properties, and avoiding common PySpark mistakes. This Skill enforces a verified workflow so generated Spark code compiles, matches destination schemas, and runs correctly on Dataproc. ## Core Features & Use Cases - Guided Spark Code Generation: Produces Python notebooks (.ipynb) by default following a mandatory workflow: inspect schemas first, generate code, verify schema before write, and compile before execution. - Data Source Integration: Provides verified patterns for reading and writing BigQuery tables, BigLake Iceberg catalogs (GCS and S3 storage), Google Cloud Storage files, and Spanner databases. - Dataproc Resource Management: Covers listing clusters, jobs, serverless batches, and sessions via gcloud or MCP tools, plus batch submission with Iceberg, Spanner, and XGBoost configurations. - ML on Spark: Documents verified patterns for XGBoost (SparkXGBClassifier), native Spark ML, and known LightGBM dependency conflicts on Dataproc Serverless. - Use Case: A data engineer needs an ETL pipeline that reads CSV files from GCS, enriches them with a BigQuery reference table, and writes results to a BigLake Iceberg table. The Skill generates a notebook with correct catalog configuration, broadcast joins, and schema verification. ## Quick Start Ask the assistant to write a Spark notebook that reads a BigQuery table and writes the results to a BigLake Iceberg table on Dataproc Serverless.

Frequently Asked Questions about gcp-spark

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a PySpark script on Dataproc Serverless?▼

Submit the script with gcloud dataproc batches submit pyspark, specifying project, region, version 2.3, and a --deps-bucket GCS path for dependencies. Expect initial execution to take 10-15 minutes, which is normal behavior for serverless batches.

How do I read and write BigLake Iceberg tables from Spark?▼

Configure a SparkCatalog with the BigLake REST endpoint, GoogleAuthManager auth, and the correct io-impl based on storage (GCSFileIO for gs: paths, S3FileIO for s3: paths). Set the current catalog, then read with spark.read.format("iceberg") and write using the DataFrameWriterV2 writeTo API.

Can I use LightGBM on Dataproc Serverless?▼

LightGBM has dependency conflicts on Dataproc Serverless because the SynapseML wrapper clashes with internal libraries. Alternatives include XGBoost with SparkXGBClassifier, native Spark ML like GBTClassifier, training on Vertex AI, or using a dedicated Dataproc cluster.

Why does my Spark job fail when writing to an Iceberg table?▼

Common causes include a DataFrame schema that does not match the destination table, a missing namespace, or an unconfigured catalog. Verify with df.printSchema(), create the namespace with CREATE NAMESPACE IF NOT EXISTS, and confirm catalog settings before writing.

Does SparkXGBClassifier support a custom objective parameter?▼

No, SparkXGBClassifier does not allow setting a custom objective; it is inferred automatically as binary:logistic for classifiers and reg:squarederror for regressors. Setting it explicitly raises a ValueError, and dynamic allocation must also be disabled.

Related Skills

Explore More Data & Analytics Skills→
1919114514yasenpai-maker1919114514yasenpai-maker

gcp-spark

Generates and executes PySpark notebooks on Dataproc clusters and Serverless batches.

Community
Advanced
claude-office-skillsclaude-office-skills
368

data-pipeline

Automate ETL workflows for data integration and analytics.

Official
Advanced
ivanshamaevivanshamaev
14

mlflow-data-pipelines

Configure MLflow tracking and Model Registry for ETL and training runs.

Community
Advanced
gemini-cli-extensionsgemini-cli-extensions
161

gcp-spark

Coordinate Spark workloads on Google Cloud Dataproc and Serverless.

Official
Advanced