Apache Spark Catalyst API

What problem does it solve?

This Skill provides expert guidance for navigating and extending the Apache Spark Catalyst query optimization framework, enabling deeper control over Spark SQL's internal workings.

Core Features & Use Cases

• Query Optimization: Understand and implement custom optimization rules.
• Expression Handling: Create and analyze custom Spark SQL expressions.
• Plan Manipulation: Work with logical and physical query plans.
• Use Case: You need to optimize a complex Spark SQL query by pushing down filters more aggressively or by creating a custom expression to handle a specific data transformation. This Skill will guide you through modifying Catalyst's internal plans and rules.

Quick Start

Use the Apache Spark Catalyst API skill to understand how to create a custom Spark SQL expression.