bi-bigdata-knowledge

Adopt a distributed engine only when scale and operational evidence justify it. This Skill routes you through knowledge entries that define stop rules, so you evaluate data volume, skew, and reconciliation needs before committing to distributed processing.

Data skew in distributed joins is addressed through the partitioning and join routes in the knowledge index. The Skill directs you to the relevant route covering skew detection and mitigation strategies rather than applying generic fixes.

It covers distributed aggregation, partitioning, joins, skew, idempotency, reconciliation, performance, and cost reasoning. Each topic is a selectable route in the knowledge index, so you only load what the current question requires.

Avoid distributed processing when data scale and operational evidence do not justify it. The Skill's stop rules explicitly prevent selecting a distributed engine for workloads that single-node processing can handle reliably.

Idempotency ensures re-running a pipeline produces the same result without duplicates or corruption. The knowledge routes cover idempotent design patterns so retries and backfills remain safe in distributed BI workflows.