spark-cli

Author, run, and diagnose Fabric Spark notebooks, Livy sessions, and Materialized Lake Views via REST APIs.

Updated Aug 19, 2026
One-click install
npx skills add https://github.com/paulasilvatech/Fabric-Agentic-SDLC --skill spark-cli-paulasilvatech
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: spark-cli
Source: https://github.com/paulasilvatech/Fabric-Agentic-SDLC/tree/main/.github/skills/spark-cli
Command: npx skills add https://github.com/paulasilvatech/Fabric-Agentic-SDLC --skill spark-cli-paulasilvatech

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Working with Microsoft Fabric Spark through raw REST APIs is error-prone: wrong payload shapes, missed LRO polling, ambiguous workspace/item IDs, and confusion between notebook jobs, Livy sessions, and Materialized Lake View operations. This Skill routes each request to the correct operational mode and executes the documented Fabric API calls against the live workspace. ## Core Features & Use Cases - Notebook authoring and execution: Write PySpark, Scala, SparkR, %%sql, and %%configure cells, save them via updateDefinition, and run named notebooks through the Jobs API. - Livy-session analytics: Create Lakehouse Livy sessions and run interactive ad-hoc PySpark calculations, cross-lakehouse joins, and Delta time-travel queries. - Spark failure triage: Diagnose failed, slow, or throttled notebook, pipeline, and Livy runs using Spark Advisor, monitoring APIs, logs, and resource-usage metrics. - Materialized Lake View lifecycle: Author MLV definitions, review queries for incremental-refresh readiness, create refresh schedules, trigger on-demand refreshes, and classify refresh failures. - Use Case: A user asks "My notebook ETL_Daily failed with an OOM in workspace Production — find out why." The Skill resolves the workspace and notebook, queries Spark Advisor and stage metrics, identifies data skew, and reports the root cause with fix recommendations. ## Quick Start Ask the assistant to write a notebook cell, run a notebook by name, create a Livy session, or diagnose a failed Spark run in your Fabric workspace, naming the workspace and item involved.

Frequently Asked Questions about spark-cli

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a Fabric notebook by name using the REST API?▼

Resolve the workspace and notebook IDs by listing items and filtering with JMESPath, then trigger the run via POST /v1/workspaces/{ws}/items/{notebookId}/jobs/instances?jobType=RunNotebook. Capture the job instance ID immediately and poll status instead of retrying the POST.

How do I schedule a Materialized Lake View refresh in Microsoft Fabric?▼

Create a schedule with POST /workspaces/{ws}/lakehouses/{lh}/jobs/refreshMaterializedLakeViews/schedules using a Cron, Daily, Weekly, or Monthly configuration with a required endDateTime. A schedule refreshes the full MLV lineage by default; use an MLV execution definition to refresh a selected subset.

What is the difference between a Lakehouse Livy session and a notebook Spark session?▼

Lakehouse Livy sessions are created through the public Livy API for ad-hoc interactive PySpark code execution. Notebook Spark sessions are created internally when a notebook runs via the Jobs API and are not managed through the Livy API.

Why did my Fabric Spark notebook fail with an OutOfMemoryError?▼

Query the Spark Advisor API first, since it pre-computes skew detection and task errors without log parsing. Then check stage metrics for data skew (max/median ratio above 3x), disk spill, and executor memory pressure before reading raw driver or executor logs.

Can this skill manage KQL materialized views in an Eventhouse?▼

No. KQL materialized views in an Eventhouse are a hard routing boundary and belong to the eventhouse-cli skill. If that skill is unavailable, the request cannot be completed and no Fabric API calls are made.

Why does GET /materializedLakeViews return 404 in Fabric?▼

The REST discovery endpoint for Materialized Lake Views is not supported. Use Spark SQL discovery instead by running SHOW MATERIALIZED LAKE VIEWS IN <schema> through a Lakehouse Livy session or a notebook cell.