run-on-databricks

What problem does it solve?

Enable execution of PySpark workloads on Databricks when local Docker or development environments cannot handle dataset size, access requirements, or production validation needs. The Skill reduces friction around cluster lifecycle, file transfer, and environment configuration so teams can validate, benchmark, and run integration tests at scale.

Core Features & Use Cases

• Cluster lifecycle management: check cluster status and start clusters using MCP tooling such as cluster_status and start_cluster.
• Deterministic execution: run PySpark code via the execute_code MCP tool and upload required files ahead of execution.
• Validation and artifact handling: compare Databricks outputs against local test expectations and download generated artifacts back to the repository.
• Configuration and security: reads Databricks settings from kyros-agent-workflow/.harnessrc and uses a token supplied via an environment variable while warning to never hardcode credentials.
• Use cases: full-scale ETL on large datasets, integration tests against real data, production validation before PR, Unity Catalog access, and performance benchmarking.

Quick Start

Check the cluster status, start the cluster if necessary, upload any required files, execute your PySpark code on Databricks, validate outputs against local expectations, and download artifacts for review.