skills.rest
reason-machines
reason-machinesOFFICIAL
Explore all skills by this author →
Website𝕏@reasonmachinesGitHub

zomato-ai-data-engineering-pipeline

Builds a batch data pipeline with S3, Snowflake, dbt, Airflow, and OpenAI for food delivery analytics.

5|1|Updated May 16, 2026
Search Tags:#rag#data pipeline#dbt#openai#airflow#medallion architecture#snowflake
One-click install
npx skills add https://github.com/reason-machines/data-skills --skill zomato-ai-data-engineering-pipeline-reason-machines
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: zomato-ai-data-engineering-pipeline
Source: https://github.com/reason-machines/data-skills/tree/main/skills/zomato-ai-data-engineering-pipeline
Command: npx skills add https://github.com/reason-machines/data-skills --skill zomato-ai-data-engineering-pipeline-reason-machines

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires openai, snowflake-connector-python, streamlit, numpy, pandas, apache-airflow.

What problem does it solve? Setting up an end-to-end data engineering pipeline that ingests raw CSV data, models it into analytics-ready tables, orchestrates scheduled runs, and adds AI capabilities like sentiment analysis and text-to-SQL requires coordinating many tools and configurations, which this Skill consolidates into one guided workflow. ## Core Features & Use Cases - Medallion Architecture on Snowflake: Loads CSVs from S3 into a RAW (Bronze) schema via storage integrations, then transforms data through dbt staging views (Silver) and marts (Gold) including incremental fact tables, dimensions, and SCD2 snapshots. - Airflow Orchestration: Provides a daily DAG that reloads raw tables from S3, runs dbt builds, and triggers AI enrichment tasks in sequence. - AI Layer with OpenAI: Enriches reviews with LLM-extracted sentiment and topics, supports RAG-based chat over reviews via Streamlit, and generates SQL from natural language questions. - Use Case: A data engineer wants to build a portfolio or production pipeline processing 10M food delivery orders — this Skill walks them from S3 bucket creation through dbt incremental models to a working RAG chatbot over customer reviews. ## Quick Start Ask the AI to set up the Zomato data pipeline by creating the S3 bucket, Snowflake schemas, and dbt project, then run the Airflow DAG to load and transform the data.

Frequently Asked Questions about zomato-ai-data-engineering-pipeline

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a data pipeline with Snowflake, dbt, and Airflow?▼

Load raw CSVs from S3 into Snowflake RAW tables using COPY INTO with a storage integration, then define dbt staging views and incremental marts for transformation. Orchestrate the full flow with an Airflow DAG that chains a SnowflakeOperator for loading and BashOperators for dbt builds.

How to set up Snowflake S3 storage integration?▼

Create an IAM role and policy granting S3 read access, then create a Snowflake storage integration referencing the role ARN. Retrieve the generated IAM user ARN and external ID with DESC STORAGE INTEGRATION, update the AWS trust policy, and create an external stage pointing at your bucket.

How do dbt incremental models work in Snowflake?▼

Configure the model with materialized='incremental' and a unique_key, then use the is_incremental() macro to filter new rows, such as records where order_date exceeds the current maximum. On subsequent runs dbt merges only new data instead of rebuilding the full table.

Can I use OpenAI to analyze customer reviews in a data pipeline?▼

Yes, call the OpenAI chat API with a prompt asking for JSON output containing sentiment and topic for each review, then insert results into an enriched table. The pipeline processes only unenriched reviews with a configurable sample limit to control API costs.

What are the limitations of the RAG review chat approach?▼

The implementation computes embeddings for every review at query time and compares them in Python, which does not scale beyond small datasets. For production use, store embeddings in a vector index and perform similarity search in the database instead of fetching all rows.

Related Skills

Explore More Data & Analytics Skills→
athina-aiathina-ai
1.1k

client-package-local

Package client strategies, campaigns, content, and leads into a structured local directory with Google Sheets links.

Official
Advanced
tomtoto757tomtoto757
3

ecommerce-data-warehouse

Builds ecommerce data warehouses with star schemas, ETL pipelines, and dbt models for BigQuery.

Community
Advanced
timequitytimequity
6

data-quality

Detect data quality issues in dbt and Great Expectations pipelines.

Official
Advanced
qenex-aiqenex-ai

Data Engineering

Build scalable batch and stream data pipelines with Spark, Airflow, dbt, and Kafka.

Official
Advanced