doc-knowledge-base

Run the ingest script on each document: it dedups by sha256, uploads the original to R2, extracts text with ebooklib or pymupdf, chunks into ~400-token windows, embeds locally with bge-small-en-v1.5, and inserts into Supabase pgvector tables.

Use the ask_library script with your question and an optional k value. It embeds the query, calls the match_doc_chunks RPC, and prints the top passages with document title, author, and section or page locators for citation.

No. Embeddings use the local BAAI/bge-small-en-v1.5 model and reranking uses a local bge-reranker-base cross-encoder on CPU, so ingest, query, and evaluation make zero metered API calls. Only the optional claim-extraction script uses the Gemini API.

Run eval_retrieval.py for a recall@k acceptance gate against a gold-phrase eval set, or ab_harness.py to compare dense, hybrid rerank, and contextual hybrid arms with context recall, precision, latency, and an explicit ship rule.

PDF extractors split fi/fl/ff ligature glyphs, producing artifacts like prefi ll. The pipeline repairs these with a dictionary-guided ligature fix and de-hyphenation at extraction time, and the eval metrics normalize both sides so retrieval is measured fairly.

Epub is preferred because it yields semantic section-level locators and cleaner text. PDF is best-effort with page-based locators, can carry ligature artifacts, and front-matter pages like tables of contents may add retrieval noise.