spark-engineer

Start by checking the Spark UI for shuffle spill, GC time above 10%, and uneven task durations. Tune spark.sql.shuffle.partitions based on data size, enable Adaptive Query Execution in Spark 3.x, filter data before joins, and use broadcast joins for tables under 200MB.

Handle skew by enabling AQE skew join handling with spark.sql.adaptive.skewJoin.enabled, or manually salt skewed keys by appending random bucket values to join keys on both sides. For extremely skewed single keys, isolate them and use a broadcast join separately.

Use DataFrames for structured data and SQL-like operations since they benefit from the Catalyst optimizer and Tungsten execution. Use RDDs only for unstructured data, custom partitioning logic, or fine-grained control over physical data placement.

Cache a DataFrame only when it is reused multiple times in the same job or is expensive to recompute. Use MEMORY_AND_DISK storage level as a safe default, materialize with an action like count(), and call unpersist() when finished to free memory.

Streaming state grows unbounded without watermarks. Add withWatermark() on the event time column so Spark can clean up old state, and consider the RocksDB state store provider for large state. Monitor state rows in the query progress metrics.

Target 128-256MB per partition with 2-4 partitions per executor core. Use repartition() to increase partitions or partition by column, and coalesce() to reduce partitions without a full shuffle, especially after filters that shrink the data.