skills.rest
Lucien-1127
Lucien-1127COMMUNITY
Explore all skills by this author →
GitHub

markdown-vector-search

Build ChromaDB-based semantic vector search over Markdown file directories.

Updated Jul 6, 2026
Search Tags:#markdown#semantic search#embeddings#rag#vector database#chunking#chromadb
One-click install
npx skills add https://github.com/Lucien-1127/strata-skill --skill markdown-vector-search-lucien-1127
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: markdown-vector-search
Source: https://github.com/Lucien-1127/strata-skill/tree/main/markdown-vector-search
Command: npx skills add https://github.com/Lucien-1127/strata-skill --skill markdown-vector-search-lucien-1127

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires chromadb, openai, tiktoken, python-frontmatter.

What problem does it solve? Searching large collections of Markdown notes or documentation by keyword misses semantically related content. This Skill builds a local semantic search engine over a directory of .md files so you can find relevant documents by meaning, not just exact text matches. ## Core Features & Use Cases - Smart Chunking: Splits Markdown by headings using tiktoken (500 tokens per chunk, 50 token overlap) while parsing YAML frontmatter separately for titles and tags. - Flexible Embedding Providers: Works with any OpenAI-compatible embedding API, including FreeLLM, HuggingFace Inference, OpenAI, and local sentence-transformers. - Incremental Updates: Tracks SHA256 file hashes so unchanged files are skipped on re-indexing, and outputs results as text or JSON for piping into other tools. - Use Case: Index a 100-file documentation vault with ChromaDB, then query it from the CLI to retrieve the top-k most relevant chunks with scores, tags, and summaries. ## Quick Start Ask the agent to build a ChromaDB vector index over your Markdown docs folder and then search it with a natural language query.

Frequently Asked Questions about markdown-vector-search

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build semantic search over Markdown files in Python?▼

Parse each .md file with python-frontmatter, split content into heading-based chunks with tiktoken, embed the chunks through an OpenAI-compatible API, and store them in a ChromaDB PersistentClient. Query by embedding the search string and running a cosine similarity query.

What embedding models work with ChromaDB vector search?▼

Any OpenAI-compatible embedding endpoint works, including text-embedding-3-small (1536 dims), BAAI/bge-m3 via HuggingFace Inference (1024 dims), and local sentence-transformers like all-MiniLM-L6-v2 (384 dims). All chunks in one collection must use the same model.

Can I use a free embedding API instead of OpenAI?▼

Yes. A local FreeLLM API proxy or the HuggingFace Inference API both provide free OpenAI-compatible embedding endpoints. Point the client's base_url at the provider and expect rate limits, so batch requests with retry and backoff.

Why does ChromaDB throw a dimension mismatch error?▼

The error occurs when existing vectors were embedded with a different model than the current one, since dimensions must match exactly. Delete the vector_store directory and rebuild the index with the new embedding model.

Why does ChromaDB deadlock when adding documents in a loop?▼

ChromaDB's PersistentClient can hit a SQLite write lock when collection.add() is called once per file in a loop. Accumulate all embeddings, ids, and metadatas into lists and make a single batched collection.add() call instead.

How do I update a vector index when Markdown files change?▼

Store a SHA256 hash of each file alongside the index and compare hashes on each run. Only re-embed and re-index files whose hashes changed, skipping unchanged files to save embedding API calls.

Related Skills

Explore More Software Engineering Skills→
jander99jander99
1

markdown-editor

Generate and organize Markdown documents with standardized YAML frontmatter and directory structure.

Community
Advanced
itallstartedwithaideaitallstartedwithaidea
31

knowledge-base-rag

Ingest private documents into a vector index and generate cited answers.

Community
Advanced
kmshihab7878kmshihab7878
6

hybrid-search-implementation

Combine vector similarity with keyword matching to improve document retrieval.

Community
Advanced
qdrantqdrant
220

qdrant-hybrid-search-prefetches

Construct Qdrant hybrid search prefetch queries combining BM25 with dense vectors.

Official
Advanced