data-spark

Focus on diagnosis before solutions: use Spark UI to compare Task data volumes, sample to find hot keys, and handle null keys separately. Know when salting, broadcast joins, or AQE skew join apply, including side effects like table inflation and the limitation that AQE skew join only handles sort merge joins.

Core topics include execution model and Stage division, Shuffle mechanics, data skew handling, memory model and OOM troubleshooting, Catalyst optimization, AQE, Structured Streaming state and watermark, Hive/Kafka integration, resource sizing, and PySpark UDF performance.

There is no single correct answer; interviewers evaluate your reasoning. Compare latency requirements, state complexity, team stack, and operational cost. Flink typically wins on low latency and complex stateful processing, while Spark suits micro-batch workloads integrated with existing batch pipelines.

YARN kills containers when total memory including overhead exceeds limits, not just heap. Check memoryOverhead, off-heap usage from Netty buffers or Python worker processes in PySpark, and oversized partitions. Fixes include raising overhead, increasing parallelism, or splitting large partitions rather than only adding heap.

Checkpointing alone does not guarantee end-to-end exactly-once. You also need a replayable source like Kafka with offsets stored in the checkpoint, and an idempotent or transactional sink, such as deduplicating by batchId, to prevent duplicate writes after restarts.

AQE relies on Shuffle stage statistics, so it does not help streaming jobs or queries without Shuffle. It can also regress performance when partition coalescing produces unfavorable sizes. Validate by comparing plans and metrics with AQE toggled off before blaming or crediting it.