Skills
.Work. You
Rest

spark-expert

Community

Master Spark & PySpark performance.

Software Engineering#optimization#data engineering#performance tuning#big data#spark#pyspark#delta lake
Authorbrunoldqueiroz
Version1.0.0
Installs0

System Documentation

What problem does it solve?

This Skill helps users optimize Apache Spark and PySpark jobs, resolve performance bottlenecks, and implement best practices for big data processing.

Core Features & Use Cases

  • Performance Tuning: Provides guidance on optimizing shuffle operations, partitioning, and memory management.
  • Delta Lake Optimization: Offers best practices for Delta Lake OPTIMIZE and MERGE operations, including ZORDER and Liquid Clustering.
  • Code Best Practices: Identifies anti-patterns like withColumn loops and Python UDFs, suggesting efficient alternatives.
  • Troubleshooting: Assists in diagnosing common errors like executor OOMs and slow plan generation.

Quick Start

Use the spark-expert skill to optimize a slow PySpark job that is experiencing shuffle spills.

Dependency Matrix

Required Modules

None required

Components

references

💻 Claude Code Installation

Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.

Please help me install this Skill:
Name: spark-expert
Download link: https://github.com/brunoldqueiroz/marvin/archive/main.zip#spark-expert

Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
View Source Repository

Agent Skills Search Helper

Install a tiny helper to your Agent, search and equip skill from 223,000+ vetted skills library on demand.