realtime-cinema-data-engineering-pipeline

Create a Kafka producer that streams JSON events to a topic, then run a Kafka consumer that inserts each event as raw JSONB into a PostgreSQL Bronze table. Use Airflow to transform Bronze data into normalized Silver tables and refresh Gold materialized views.

Medallion Architecture organizes data into Bronze, Silver, and Gold layers. Bronze stores raw ingested events, Silver holds normalized 3NF tables like customers and transactions, and Gold provides materialized views with business-ready aggregates.

Define a DAG with PythonOperator tasks that use PostgresHook to extract unprocessed Bronze records, upsert them into Silver dimension and fact tables, then refresh the Gold materialized view. Schedule the DAG with a timedelta interval such as every five minutes.

Verify the topic exists with kafka-topics --list and check consumer group lag using kafka-consumer-groups --describe. Confirm the bootstrap server address matches and that auto_offset_reset is set to earliest if you need historical messages.

Yes, Streamlit can poll PostgreSQL on a timed loop and re-render Plotly charts with fresh Gold layer data. Use st.cache_resource for the database connection and st.rerun after a sleep interval to refresh metrics every 30 seconds.