ai-data-engineering-rag-pipeline

Use the rank-bm25 library's BM25Okapi class with a tokenized corpus built by lowercasing and splitting document text. Call get_scores with a tokenized query, then use numpy argsort to retrieve the top-k ranked documents.

Split documents into document, section, and paragraph levels using newline delimiters, then assign each chunk a deterministic ID from a SHA-256 content hash. Store parent_id references so fine-grained matches can be expanded with parent context at retrieval time.

Recall@10 measures the fraction of relevant documents appearing in the top 10 retrieved results for each query in a golden dataset. Compare retrieved document IDs against the relevant_docs field in a questions.jsonl file and average across all queries.

Yes, normalize BM25 scores and FAISS cosine similarity scores, then combine them with a weighted sum using an alpha parameter. Tune alpha between 0 and 1 to balance keyword matching against semantic similarity for your domain.

The error occurs when query embedding dimensions differ from the index dimension, usually from switching embedding models after indexing. Verify query_emb.shape matches faiss_index.d and re-encode the corpus if you change the sentence-transformers model.

Low recall typically comes from malformed golden datasets, overly aggressive chunk length filters, or poor tokenization. Validate each questions.jsonl line has query and relevant_docs fields, lower minimum chunk length thresholds, and experiment with hybrid alpha values.