Evaluate RAG retrieval quality with LLM judgments and gap reports.
npx skills add https://github.com/softspark/ai-toolkit --skill evaluate-softspark
Please help me install this Agent Skill. Skill: evaluate Source: https://github.com/softspark/ai-toolkit/tree/main/app/skills/evaluate Command: npx skills add https://github.com/softspark/ai-toolkit --skill evaluate-softspark
Evaluates RAG retrieval quality and leverages LLM-based judgments to quantify answer quality, enabling objective performance tracking.
Run a simple evaluation workflow to score a small set of test queries and generate a gap report.
High-intent search queries and answers about installing and using this skill.
To evaluate RAG retrieval quality, you can use LLM judgments to score faithfulness, relevancy, and context precision across test queries by running the evaluate_rag.py script against a golden_dataset.json file.
Yes, you can integrate RAG evaluation into CI or QA workflows to validate RAG correctness before deployment, using automated metrics to act as a quality gate for your pipeline.
LLM judgment for RAG gap analysis works by processing test queries to identify missing knowledge and potential data sources, generating a report that highlights where your retrieval pipeline fails to find correct information.
To run automated RAG quality metrics, you need python3 installed, the evaluate_rag.py and knowledge_gaps.py scripts, and a golden_dataset.json file located at scripts/golden_dataset.json or a project-specific path.
RAG trustworthiness is assessed using specific metrics that evaluate faithfulness, relevancy, and context precision across multiple queries to quantify answer quality and track performance objectively.
VjayRamDefine DSPy signatures and modules with deterministic evaluation across splits.
ezagent42Parse Markdown inputs into gap-analysis, prd-structure, and task_hints YAML artifacts.
JCETools-PetraDesign RAG pipelines with chunking, embedding, retrieval, and evaluation metrics.
HermeticOrmusEvaluates LLM outputs using automated metrics, human feedback, and LLM-as-Judge techniques.
