ai-data-engineering

Build a RAG pipeline by chunking documents at 512 tokens with 50-token overlap, embedding them with Voyage AI voyage-3, storing vectors in Qdrant, and connecting a retriever to a ChatPromptTemplate chain. The skill provides a working basic_rag.py example and streaming variants.

Use 512 tokens with 50-100 tokens of overlap as the default for most RAG systems. Smaller chunks lose context while larger chunks include irrelevant content; code and markdown content benefit from language-aware or header-aware splitters.

Voyage AI voyage-3 scores 69.0 on MTEB, about 9.7% better than OpenAI text-embedding-3-large, making it the choice for production quality. OpenAI text-embedding-3-small costs roughly five times less and suits development or cost-sensitive workloads.

Run the evaluate_rag.py script with a JSON dataset containing question, answer, contexts, and ground_truth fields. It computes faithfulness, answer relevancy, context precision, and context recall, flagging scores below thresholds like 0.8 faithfulness.

Poor retrieval usually stems from wrong chunk sizes, missing overlap, or weak embedding models. Try 512-token chunks with 50-token overlap, switch to hybrid search combining vector and BM25, or add Cohere re-ranking to improve precision.

No, never mix embedding models in the same collection because vector spaces are incompatible. When changing models, delete the collection and re-embed all documents with the new model before re-indexing.