skills.rest
sayalinvidia
sayalinvidiaCOMMUNITY
Explore all skills by this author →
GitHub

rag-eval

Run filesystem-based RAG benchmarks and output NVSkills metrics.

Updated Apr 16, 2026
Search Tags:#rag#benchmarking#evaluation#nlp#information-retrieval#rag-eval
One-click install
npx skills add https://github.com/sayalinvidia/sayali-skills-test --skill rag-eval-sayalinvidia
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: rag-eval
Source: https://github.com/sayalinvidia/sayali-skills-test/tree/main/skills/rag-eval
Command: npx skills add https://github.com/sayalinvidia/sayali-skills-test --skill rag-eval-sayalinvidia

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This skill provides a structured framework to evaluate the quality of filesystem-based RAG deployments by running controlled benchmarks against a corpus/ and train.json layout, enabling predictable comparison of retrieval and answer grounding.

Core Features & Use Cases

  • End-to-end RAG evaluation: assesses retrieval, grounding, and answer accuracy against ground-truth data.
  • Dataset-layout guidance: promotes a consistent corpus/ and train.json organization and conversion practices.
  • Operational workflow: guides users through data preparation, evaluation execution with uv and evaluate_rag.py, and results analysis with NVSkills metrics.

Quick Start

Prepare a corpus/ with PDFs or documents and a train.json at repo root, then run the evaluation against your RAG server to obtain NVSkills metrics.

Frequently Asked Questions about rag-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate RAG retrieval quality and answer accuracy?▼

RAG evaluation benchmarks retrieval quality, context grounding, and answer correctness by running controlled tests against a RAG server using a standardized corpus/ and train.json dataset layout.

How do I benchmark a RAG deployment against a custom dataset?▼

To benchmark a RAG deployment, organize your documents in a corpus/ directory with a train.json file, provide the RAG server host and port, and run the evaluation to receive structured NVSkills metrics.

What data format is needed for RAG evaluation?▼

RAG evaluation requires a filesystem-based dataset organized with a corpus/ directory containing source documents and a train.json file holding ground-truth data for predictable comparison.

Can I use a custom judge model for RAG evaluation?▼

Yes, you can specify an optional RAG_EVAL_JUDGE_MODEL environment variable when running the evaluation to assess answer correctness and grounding against your ground-truth data.

What metrics do I get from RAG benchmarking?▼

RAG benchmarking provides structured NVSkills metrics and per-query data, enabling direct comparisons of retrieval quality, context grounding, and answer correctness across different configurations.

Does RAG evaluation work with PDF documents?▼

Yes, RAG evaluation supports PDFs and other documents placed in the corpus/ directory, allowing the benchmarking process to assess retrieval and answer accuracy against the ground-truth train.json.

Related Skills

Explore More Data & Analytics Skills→
rblake2320rblake2320

rag-eval

Orchestrate filesystem-based RAG benchmarking with corpus/train.json and evaluate_rag.py.

Community
Intermediate
MantisWareMantisWare

eval-rag

Evaluate RAG pipelines with retrieval and generation metrics and bottleneck analysis.

Community
Advanced
AgentientAgentient
2

rag-patterns

Configure Vertex AI RAG pipelines for grounding agent outputs in private data sources.

Official
Advanced
lyndonkllyndonkl
142

graphrag-evaluation

Evaluate GraphRAG systems across knowledge graph completeness, retrieval relevance, and answer correctness.

Community
Advanced