skills.rest
AarushiShah
AarushiShahCOMMUNITY
Explore all skills by this author →
GitHub

databricks-synthetic-data-generation

Generate realistic synthetic datasets with Faker and Spark and save them to Databricks volumes.

Updated Mar 24, 2026
Search Tags:#synthetic data#faker#spark#parquet#databricks#unity catalog#test data generation
One-click install
npx skills add https://github.com/AarushiShah/coding-agents-databricks-apps --skill databricks-synthetic-data-generation-aarushishah
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-synthetic-data-generation
Source: https://github.com/AarushiShah/coding-agents-databricks-apps/tree/main/.claude/skills/databricks-synthetic-data-generation
Command: npx skills add https://github.com/AarushiShah/coding-agents-databricks-apps --skill databricks-synthetic-data-generation-aarushishah

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires faker, holidays, numpy, pandas, pyspark.

What problem does it solve? Creating realistic test and demo datasets for Databricks is tedious and error-prone, especially when data needs referential integrity, non-uniform distributions, and time-based patterns that survive downstream aggregation. ## Core Features & Use Cases - Realistic Data Generation: Uses Faker, numpy, and pandas to produce non-linear distributions (log-normal, Pareto, exponential), weighted categories, and coherent row-level correlations. - Databricks-Native Execution: Writes Python scripts locally, executes them on Databricks clusters via MCP tools, and saves raw parquet files to Unity Catalog volumes. - Pipeline-Ready Output: Generates raw transactional data (no pre-aggregated fields) designed as the source for Spark Declarative Pipelines with bronze/silver/gold layers. - Use Case: Generate 2,500 customers, 25,000 orders, and 8,000 support tickets spanning the last 6 months with an incident spike, then feed them into a medallion pipeline and AI/BI dashboard. ## Quick Start Ask the agent to generate synthetic customer, order, and ticket data for the last six months and save it to a schema in the ai_dev_kit catalog on Databricks.

Frequently Asked Questions about databricks-synthetic-data-generation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I generate synthetic data on Databricks with Python?▼

Write a Python script using Faker and pandas to build DataFrames, then execute it on a Databricks cluster with the run_python_file_on_databricks MCP tool. Convert the pandas DataFrames to Spark and save them as parquet files in a Unity Catalog volume.

What libraries are used for synthetic data generation in Python?▼

Faker generates realistic names, companies, and dates, while the holidays library provides country-specific calendars for realistic date patterns. numpy supplies log-normal, Pareto, and exponential distributions, and pandas handles row-by-row generation before Spark saves the output.

Should synthetic data be saved as tables or files on Databricks?▼

Save generated data as parquet files in a Unity Catalog volume rather than tables. This raw data becomes the input for a downstream Spark Declarative Pipeline that builds bronze, silver, and gold layers with aggregations.

Why does my synthetic data look unrealistic after aggregation?▼

Uniform distributions and insufficient row counts cause patterns to disappear after GROUP BY operations. Use log-normal or exponential distributions, weighted categories, and at least 10K-50K rows so trends survive daily, weekly, and per-category aggregation.

How do I fix errors when running a data generation script on Databricks?▼

Edit the local Python file to fix the error, then re-execute with the same cluster_id and context_id returned by the first run. Reusing the context keeps installed libraries like Faker available and runs much faster than starting fresh.

Related Skills

Explore More Data & Analytics Skills→
ThomazRossitoThomazRossito
4

databricks-unstructured-pdf-generation

Generate synthetic PDFs from HTML and upload them to Unity Catalog volumes for RAG evaluation.

Community
Intermediate
databrickslabsdatabrickslabs
38

databricks-spark-declarative-pipelines

Automate creation, configuration, and optimization of Databricks Lakeflow Spark Declarative Pipelines.

Official
Advanced
lucaslessachaveslucaslessachaves
1

databricks-unstructured-pdf-generation

Convert HTML content to PDFs and upload them to Unity Catalog volumes.

Community
Intermediate
rappdwrappdw

synthdata-anonymize

Detect and replace PII in CSV, Excel, or JSON datasets with deterministic Faker values.

Community
Advanced