fabric-data-factory

Design Microsoft Fabric Data Factory pipelines, Copy Jobs, and Dataflows Gen2 for data ingestion and orchestration.

4|1|Updated May 22, 2026
One-click install
npx skills add https://github.com/ThomazRossito/ai-data-agents --skill fabric-data-factory-thomazrossito
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: fabric-data-factory
Source: https://github.com/ThomazRossito/ai-data-agents/tree/main/plugins/ai-data-agents/skills/fabric-data-factory
Command: npx skills add https://github.com/ThomazRossito/ai-data-agents --skill fabric-data-factory-thomazrossito

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Choosing and configuring the right Microsoft Fabric Data Factory tool for ingestion and orchestration is error-prone, especially with frequent platform changes like the April 2026 deprecation of Dataflow Gen2 Classic and the rename of Data pipeline to Pipeline. This Skill provides up-to-date decision guidance, JSON configuration references, and checklists so pipelines are designed correctly the first time. ## Core Features & Use Cases - Decision Guide: Maps scenarios to the right tool — Copy Activity, Copy Job, Copy Job Activity, Dataflows Gen2, Pipelines, Mirroring, Eventstreams, or Spark notebooks. - Configuration References: Provides JSON examples for pipelines, triggers, Copy Activity sources/sinks, watermark-based incremental loads, error handling, and retry policies. - Breaking Change Awareness: Covers Dataflow Gen2 Classic deprecation, CI/CD-by-default behavior, Fast Copy, Modern Evaluator Engine, and new preview activities like Lakehouse Maintenance, Refresh SQL Endpoint, and dbt Job. - Use Case: When asked to build an incremental ingestion pipeline from Azure SQL into a Fabric Lakehouse, the Skill guides selection of Copy Job Activity with automatic watermark management and provides the pipeline JSON with proper dependencies and retry policies. ## Quick Start Ask the agent to design a Fabric pipeline that incrementally ingests an Azure SQL orders table into a Lakehouse bronze layer and refreshes the semantic model afterward.

Frequently Asked Questions about fabric-data-factory

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I choose between Copy Activity, Copy Job, and Dataflows Gen2 in Microsoft Fabric?▼

Use Copy Activity for granular column mapping and high-throughput loads over 1 GB, Copy Job for guided no-code ingestion with automatic incremental and CDC watermark handling, and Dataflows Gen2 for low-code Power Query transformations with query folding.

How to set up incremental data loading in Fabric Data Factory?▼

Use a Copy Job in Incremental Copy mode, which manages watermarks automatically using ROWVERSION, Datetime, or Date columns. Alternatively, use a Lookup activity in a Pipeline to read the max watermark and pass it to a Copy Activity query.

What changed with Dataflow Gen2 Classic in April 2026?▼

Creating new Dataflows Gen2 without CI/CD and Git integration is no longer available. All new Dataflow Gen2 items include CI/CD by default, and existing Classic items can be converted using the Save As feature.

Does Fabric Data Factory support Change Data Capture replication?▼

Yes, Copy Job supports CDC replication of inserts, updates, and deletes for SQL Server, PostgreSQL, Snowflake, SAP, BigQuery, and Cosmos DB. Mirroring also provides zero-ETL CDC synchronization for relational tables.

Why is my Dataflow Gen2 running out of memory on large datasets?▼

Without staging, Dataflows load data into memory and fail on volumes over roughly 1 GB. Enable staging under Options so the Dataflow uses a Lakehouse as temporary storage, and enable Fast Copy for supported connectors.

When should I use a Spark notebook instead of a pipeline or dataflow in Fabric?▼

Use Spark notebooks for complex large-scale transformations requiring full PySpark control that exceed what Copy Activity mappings or Power Query folding can express. Pipelines remain the orchestration layer that chains notebook activities together.