skills.rest
agenticgogol
agenticgogolCOMMUNITY
Explore all skills by this author →
GitHub

eval-and-observability

Wires LLM tracing and measured evals for RAG and agent projects.

1|9|Updated Jul 11, 2026
Search Tags:#rag#llm evaluation#observability#tracing#phoenix#eval#ragas
One-click install
npx skills add https://github.com/agenticgogol/Edureka_Coding_Agent_Enabled_Demo_11Jul --skill eval-and-observability-agenticgogol
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-and-observability
Source: https://github.com/agenticgogol/Edureka_Coding_Agent_Enabled_Demo_11Jul/tree/main/.claude/skills/eval-and-observability
Command: npx skills add https://github.com/agenticgogol/Edureka_Coding_Agent_Enabled_Demo_11Jul --skill eval-and-observability-agenticgogol

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Agentic and RAG demos often claim quality without evidence. This Skill ensures every quality claim in a project brief is backed by actual traces and measured evaluation metrics instead of a single happy-path demo run. ## Core Features & Use Cases - Tracing with fallbacks: Wires Phoenix tracing (cloud or local OSS), falling back to structured JSON logs of each LLM, retrieval, and tool span with input, output, latency, and token counts. - Measured evals: Builds a 5-15 case eval set from the brief's Definition of Done and runs Ragas metrics (faithfulness, answer relevancy, context precision/recall) or a scripted LLM-judge/rule-based fallback that works without a paid API key. - Recorded results: Writes actual numbers and pass/fail outcomes into an Eval Results section of the project README. - Use Case: After building an agentic RAG demo, run this Skill to verify groundedness with Ragas faithfulness and produce trace anatomy showing each retrieval and LLM step. ## Quick Start Add tracing and a measured eval pass to my RAG project so its quality claims are verified with real metrics.

Frequently Asked Questions about eval-and-observability

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I add tracing to a RAG or agent project?▼

Wire Phoenix tracing when PHOENIX_COLLECTOR_ENDPOINT is set, otherwise use local OSS Phoenix, and fall back to structured JSON logs capturing input, output, latency, and token count per span. Label each span as llm, retrieval, or tool so the request anatomy is visible.

How do I evaluate RAG answer quality with Ragas?▼

Build a 5-15 case eval set from the brief's Definition of Done, then run Ragas metrics including faithfulness, answer relevancy, and context precision/recall. Record the actual numbers in an Eval Results section of the project README.

Can I run LLM evals without a paid API key?▼

Yes. When Ragas or a cloud LLM judge is unavailable, fall back to a scripted rule-based judge or exact/fuzzy-match checks against expected behavior. Observability and evals must never be skipped just because no cloud key is configured.

What is the difference between Ragas and an LLM-judge eval?▼

Ragas provides standardized metrics like faithfulness and answer relevancy when its dependency is installed. An LLM-judge or rule-based script is the fallback that scores outputs against expected behavior when Ragas cannot run.

When should I skip eval and observability for a demo?▼

Skip only for the simplest single-turn concept demos with no retrieval and no quality claim in the brief, and note the skip in plan.md. Any RAG project or agent claiming reliability, safety, or accuracy requires measured evals.

Does this replace unit tests or end-to-end verification?▼

No. It complements write-and-validate-tests (correctness tests) and run-and-verify (one live end-to-end check) by adding a quality and trust measurement layer on top, not by duplicating them.

Related Skills

Explore More Software Engineering Skills→
The-Skyy-Rose-Collection-LLCThe-Skyy-Rose-Collection-LLC
3

eval-harness

Define, run, and report evaluations with /eval commands.

Official
Intermediate
RCSnyderRCSnyder

deploy

Automate verified software deployment with pre-flight checks and live validation.

Community
Intermediate
ACGSpgpACGSpgp

llm-evaluation

Evaluate LLM applications with automated metrics, human feedback, and benchmarking.

Official
Advanced
matlee0409matlee0409

pretext

Measure text layout data without touching the DOM.

Community
Intermediate