llm-ops

Implement RAG pipelines, vector databases, prompt engineering, and LLM cost evaluation for production AI systems.

Updated Jun 12, 2026
One-click install
npx skills add https://github.com/bilacchi/agents-skills --skill llm-ops-bilacchi
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-ops
Source: https://github.com/bilacchi/agents-skills/tree/main/skills/llm-ops
Command: npx skills add https://github.com/bilacchi/agents-skills --skill llm-ops-bilacchi

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires anthropic, chromadb.

What problem does it solve? Building a working AI prototype is easy, but making it reliable, scalable, and cost-effective in production is hard. This Skill provides the operational patterns for RAG pipelines, embeddings, vector databases, prompt engineering, semantic caching, cost estimation, and quality evaluation of LLM applications. ## Core Features & Use Cases - RAG Pipeline Implementation: Chunking, embedding, indexing, and querying documents with Chroma, pgvector, Pinecone, Weaviate, or Qdrant, including relevance filtering by distance threshold. - Prompt Engineering & Cost Control: Structured system prompt design, chain-of-thought analysis, semantic caching to avoid duplicate LLM calls, and monthly cost estimation across Claude models. - Quality Evaluation Framework: LLM-as-judge evals scoring responses against criteria like factual accuracy, relevance, and clarity. - Use Case: You need to add document search to your product. Use this Skill to set up a pgvector table with cosine similarity indexes, index your documents with overlapping chunks, and build a query pipeline that injects the top-k relevant chunks into the LLM context. ## Quick Start Ask the assistant to set up a complete RAG pipeline with pgvector for indexing your project documentation and answering questions from it.

Frequently Asked Questions about llm-ops

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a RAG pipeline with Python?▼

Build a RAG pipeline by chunking documents with overlap, embedding and upserting them into a vector database like Chroma, then querying with top-k retrieval and injecting matching chunks into the LLM context. Filter results by distance threshold to exclude irrelevant chunks.

Which vector database should I use for RAG?▼

Chroma suits local development, pgvector fits if you already run PostgreSQL, Pinecone is a managed cloud option starting around USD 70 per month, while Weaviate handles multi-modal data and Qdrant targets high performance. Choose based on hosting model and scale.

How do I set up pgvector for semantic search in PostgreSQL?▼

Enable the vector extension, create a table with a vector(1536) embedding column, and add an ivfflat index using vector_cosine_ops. Query with the cosine distance operator and order by similarity to retrieve the closest matching content.

How can I reduce LLM API costs in production?▼

Reduce costs with a semantic cache that returns stored responses when a new query embedding exceeds a similarity threshold around 0.95, avoiding duplicate API calls. Also estimate monthly spend per model using input and output token pricing before choosing a model tier.

How do I evaluate LLM response quality automatically?▼

Use an LLM-as-judge approach: send the question, expected answer, and actual response to a cheaper model with scoring criteria such as factual accuracy, relevance, and clarity. Request JSON output with a 0-10 score and justification per criterion.

When should I not use RAG for my LLM application?▼

Avoid RAG when the task does not require external knowledge retrieval, when a simpler more specific tool handles the request, or when general-purpose assistance without domain context suffices. RAG adds indexing and retrieval overhead that is unnecessary for self-contained prompts.