microsoft-fabric-unified-analytics

Create a Lakehouse and organize data into Bronze (raw files), Silver (cleansed via Dataflow Gen2), and Gold (business KPIs via PySpark notebooks) layers. Each layer stores Delta tables in OneLake, and a Fabric pipeline orchestrates the flow between them.

Create a Dataflow Gen2 in your workspace, connect to the Lakehouse Bronze table, and apply Power Query M transformations such as removing duplicates, handling nulls, and adding computed columns. Set the output destination to a Silver Lakehouse table and publish the dataflow.

Dataflow Gen2 provides low-code visual transformations using Power Query M, suited for standard cleansing tasks. PySpark notebooks offer full programmatic control for complex aggregations like RFM segmentation and large-scale processing, making them better for Gold-layer analytics.

PySpark jobs run out of memory when executor or driver memory is too low for the data volume. Increase spark.executor.memory and spark.driver.memory, raise shuffle partitions, repartition large DataFrames, or process data in batches.

Yes, Power BI connects natively to Fabric semantic models built on Lakehouse Gold tables. You define relationships and DAX measures in the semantic model, then build reports in Power BI Desktop or the service without additional connectors.

Enable schema merging by setting the mergeSchema option to true when appending data, allowing new source columns to be added automatically. Alternatively, use overwriteSchema with overwrite mode to replace the table schema entirely.