What problem does it solve? Writing correct Spark code for Google Cloud requires knowing Dataproc execution models, connector configurations for BigQuery, BigLake Iceberg, and Spanner, and avoiding common pitfalls like schema mismatches and missing imports. This Skill enforces a verified workflow so generated Spark code compiles, runs, and reads/writes data correctly on GCP. ## Core Features & Use Cases - Guided Spark Code Generation: Produces Python notebooks (.ipynb) following a mandatory workflow: schema discovery, source verification, code generation, schema validation, compilation, and execution. - Data Source Integration: Provides verified read/write patterns for BigQuery, BigLake Iceberg catalogs (GCS and S3 storage), Cloud Spanner, GCS files, Cloud SQL, and Pub/Sub. - Dataproc Resource Management: Covers listing and submitting clusters, jobs, serverless batches, and interactive sessions via gcloud or MCP tools. - ML on Spark: Includes verified patterns for XGBoost and native Spark ML, with warnings about LightGBM dependency conflicts on Dataproc Serverless. - Use Case: Ask the agent to build an ETL pipeline that reads a CSV from GCS, enriches it with a BigQuery reference table, and writes the result to a BigLake Iceberg table, then submit it as a Dataproc Serverless batch. ## Quick Start Use the gcp-spark skill to write a PySpark notebook that reads a CSV from my GCS bucket and writes it to a BigQuery table, then submit it as a Dataproc Serverless batch.
