rag-observability-evals

What problem does it solve?

This Skill addresses the challenge of treating Retrieval-Augmented Generation (RAG) systems as black boxes, enabling users to monitor, evaluate, and ensure the reliability and quality of RAG outputs in production environments.

Core Features & Use Cases

• Retrieval Quality Metrics: Tracks recall, MRR, citation coverage, and embedding drift.
• Generation Quality Checks: Assesses groundedness, hallucination rates, and instruction adherence.
• Reliability & Cost Monitoring: Analyzes latency, token usage, and cache performance.
• Use Case: A team deploying a RAG-powered customer support bot can use this skill to continuously monitor if the bot's answers are factually supported by retrieved documents and to detect any increase in hallucinations or response latency.

Quick Start

Begin evaluating the RAG system by running the offline evaluation pipeline.