What problem does it solve? Writing correct, production-ready Spark code for Google Cloud Dataproc requires deep knowledge of connector configurations, catalog setups, and runtime constraints. This Skill guides the generation, validation, and execution of PySpark notebooks and scripts while preventing common mistakes like schema mismatches, missing imports, and misconfigured Iceberg catalogs. ## Core Features & Use Cases - Spark Code Generation: Produces Python notebooks (.ipynb) for ETL pipelines that read and write BigQuery, BigLake Iceberg, Spanner, GCS, Cloud SQL, and Pub/Sub data. - Dataproc Resource Management: Lists clusters, jobs, serverless batches, and interactive sessions using gcloud or MCP tools, and submits PySpark batch jobs with correct connector dependencies. - ML Workload Guidance: Applies verified patterns for XGBoost and native Spark ML training, including known restrictions like disabled dynamic allocation and prohibited objective parameters. - Use Case: A data engineer needs to build an ETL pipeline reading from a BigLake Iceberg table and writing partitioned results to BigQuery. The Skill inspects schemas, generates a validated notebook, compiles it, and submits it as a Dataproc Serverless batch. ## Quick Start Ask the assistant to write a Spark notebook that reads a BigQuery table, transforms the data, and writes it to a BigLake Iceberg catalog on Dataproc Serverless.
