data-engineering-medallion-pipeline

Clone the repository, run make setup to create the .env file and directories, then run make start to launch all Docker services. Verify container health with make status and access Airflow, Airbyte, MinIO, and Grafana through their local web interfaces.

Bronze models extract typed columns from Airbyte's raw JSONB data as views, Silver models clean, cast, and deduplicate records as tables, and Gold models aggregate business metrics. Run all layers with make dbt-run and validate them with make dbt-test.

Create DAGs that use BashOperator to run dbt run and dbt test commands scoped to each layer, such as dbt run --select silver.*. The bronze DAG uses AirbyteTriggerSyncOperator to start ingestion before validating row counts in PostgreSQL.

Yes, Airbyte connects to MinIO through its S3 source connector by pointing the endpoint to http://minio:9000 with your MinIO credentials and bucket name. Pair it with a PostgreSQL destination using full refresh overwrite sync mode.

Check for import errors with airflow dags list-import-errors inside the webserver container and validate the file syntax with python -m py_compile. DAG parsing failures from syntax errors or missing imports prevent registration.

Define schema tests in schema.yml such as unique, not_null, relationships, and accepted_values on model columns, and write custom singular tests as SQL queries in the tests directory. Run everything with make dbt-test and use --store-failures to inspect failing records.