skills.rest
Nandansai08
Nandansai08COMMUNITY
Explore all skills by this author →
WebsiteGitHub

spark-etl-debugging

Diagnose Spark job failures and slowdowns using Spark UI task-duration distributions.

1|Updated Jul 3, 2026
Search Tags:#debugging#performance tuning#etl#spark#oom#shuffle#data skew
One-click install
npx skills add https://github.com/Nandansai08/skillz --skill spark-etl-debugging-nandansai08
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: spark-etl-debugging
Source: https://github.com/Nandansai08/skillz/tree/main/skills/data-analytics/spark-etl-debugging
Command: npx skills add https://github.com/Nandansai08/skillz --skill spark-etl-debugging-nandansai08

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Spark jobs that fail, retry, or suddenly crawl are hard to diagnose from driver stacktraces alone, and the wrong fix (like doubling executor memory) wastes cluster budget without addressing skew, shuffle, or memory root causes. ## Core Features & Use Cases - Pathology classification from the Spark UI: Reads the task-duration distribution in the failing stage to separate skew, shuffle-bound, memory, and retry/infrastructure failures before touching any code. - Targeted fix playbooks: Provides ordered fixes for skew (AQE skew joins, NULL/sentinel key handling, broadcast joins, salting), shuffle reduction (pushdown, partition tuning, bucketing), and OOM diagnosis (executor vs driver vs UDF memory). - Regression and verification discipline: Requires what-changed analysis for sudden slowdowns, before/after stage metrics at full production scale, and drift guards for data-shape root causes. - Use Case: A nightly enrichment job that ran in 45 minutes now takes 6 hours and OOMs. Use this Skill to identify a NULL join key affecting 210M rows from the task table, split those rows pre-join, enable AQE skew handling, and restore runtime to 38 minutes. ## Quick Start Diagnose why my Spark job is stuck at stage 12 with 199 of 200 tasks complete and fix the underlying bottleneck.

Frequently Asked Questions about spark-etl-debugging

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I diagnose a slow or failing Spark job?▼

Open the Spark UI, find the long or failed stage, and sort the Tasks table by duration. The distribution reveals the cause: high max versus median duration indicates skew, uniformly slow tasks with heavy shuffle indicate shuffle-bound work, and dying executors indicate memory problems.

How do I fix data skew in a Spark join?▼

First confirm the hot key with a groupBy count ordered descending. Then try AQE skew-join handling, filter or handle NULL and sentinel keys separately, broadcast the smaller side if one fits, or apply salting for genuinely hot keys as a last resort.

Why does my Spark job throw OutOfMemoryError in executors?▼

Most executor OOMs are skew symptoms where one huge partition exceeds memory, so check the task-duration distribution before raising memory. Driver OOM usually means a collect() or toPandas() pulled too much data, and genuine executor pressure may require fewer cores per executor or removing memory-hoarding UDFs.

Should I increase executor memory when a Spark job OOMs?▼

Not before checking the task distribution. A single 38GB partition from skew will OOM regardless of cluster size, so doubling memory only raises cost. Identify whether the OOM is skew, driver-side collection, or genuine executor pressure first.

When should I not use this Spark debugging approach?▼

Do not use it for single-node or OLTP SQL query tuning, which belongs to sql-query-optimization. This workflow starts where the shuffle does and requires access to the Spark UI or History Server for the affected run.

Why did my Spark job suddenly become slow after running fine for a year?▼

A stable job that suddenly crawls almost always means the data changed, not the infrastructure. Ask what changed: input volume, key distribution such as a new whale customer, a nulled upstream column, cluster type, or a recently added UDF.

Related Skills

Explore More Data & Analytics Skills→
yangyongyongyongyangyongyongyong
1

spark-k8s

Diagnose Spark jobs and tasks on Kubernetes via REST APIs.

Community
Advanced
RDEWAIRDEWAI
5

create-gold

Generate PySpark Gold layer artifacts from approved design documents.

Official
Advanced
kiurakkukiurakku
15

pipeline-review

Audits data pipelines for idempotency, reliability, observability, and cost issues.

Community
Intermediate
ivanshamaevivanshamaev
14

trino-query-optimization

Optimizes slow Trino SQL queries via pushdown enforcement and join tuning.

Community
Advanced