embedding-pipeline-builder

Build a RAG embedding pipeline by loading documents, preprocessing the text, splitting it into chunks with overlap, generating embeddings with a model like text-embedding-3-small, and upserting the vectors into a vector store. The pipeline then supports semantic queries against the indexed chunks.

Use chunk sizes between 500 and 1500 characters to balance context against noise, with 10-20% overlap to prevent context loss at boundaries. Recursive chunking with paragraph, sentence, and word separators preserves natural text structure.

Recursive chunking splits text using a hierarchy of separators like paragraphs and sentences until chunks fit the target size. Semantic chunking embeds each sentence and groups consecutive sentences by cosine similarity, producing topically coherent chunks at higher compute cost.

Yes, the document loader extracts text from PDF files using pdf-parse and from DOCX files using mammoth, alongside plain text, Markdown, and JSON. Web pages are loaded through Readability to strip boilerplate before extraction.

Batch embedding requests with up to 100 chunks per API call and limit concurrency with p-limit, for example to 5 parallel batches. This reduces total API calls while keeping throughput within rate limit boundaries.

Irrelevant results usually come from chunks that are too large, missing overlap, or poorly preprocessed text. Tune chunk size, add metadata filtering, and consider hybrid search combining vector similarity with keyword matching to improve retrieval accuracy.