architecting-data

Choose a data warehouse for known BI workloads, a data lake for exploratory or ML use cases needing raw data, and a lakehouse for mixed BI and ML workloads with cost optimization. Lakehouses are typically 60-80% cheaper than warehouses at scale.

Apache Iceberg is the default recommendation for new projects due to vendor neutrality and the broadest multi-engine support across Spark, Trino, and Flink. Choose Delta Lake if committed to Databricks, or Apache Hudi for CDC-heavy workloads with frequent upserts.

Data mesh fits large organizations over 500 people with clear domain ownership, mature platform capabilities, and a central team that is a proven bottleneck. A 6-factor readiness assessment scoring 24-30 indicates a strong candidate; below 18, build foundations first.

Medallion architecture organizes lakehouse data into bronze (raw, immutable), silver (cleaned and deduplicated), and gold (business-level aggregates) layers. Use it on lakehouse platforms like Databricks or Iceberg when you want clear data quality layers and reprocessing at any stage.

Dimensional modeling with star schemas fits BI dashboards with known query patterns, while wide denormalized tables suit ML feature stores and data science notebooks. Use normalized 3NF for OLTP systems and Data Vault 2.0 when compliance and full audit trails are required.

A startup stack under $5K per month uses BigQuery or Snowflake for storage, Airbyte or Fivetran for ingestion, dbt Core for transformation, and Metabase or Looker Studio for visualization. This minimizes complexity by relying on managed services.