evaluating-llms

Use the RAGAS framework's faithfulness metric, which checks whether each claim in the answer is supported by the retrieved context. Target a faithfulness score above 0.8, and if it fails, adjust prompts to require grounding and citations.

Combine automated metrics like BLEU, ROUGE, and BERTScore for fast feedback with LLM-as-judge scoring on a 1-5 rubric for nuanced quality. Automated metrics alone correlate weakly with human judgment for creative or open-ended generation.

RAGAS is purpose-built for RAG systems with metrics like faithfulness and context relevance, while DeepEval offers general LLM evaluation with pytest integration. Use RAGAS for retrieval pipelines and DeepEval for broader unit-test-style evaluation.

Use lm-evaluation-harness to run standardized benchmarks like MMLU, HumanEval, and GSM8K against both models with identical few-shot settings. Compare scores across multiple benchmarks rather than relying on a single metric.

LLM judges exhibit position bias, verbosity bias, and self-preference bias, and cost $0.01-0.10 per evaluation. Mitigate by randomizing response order, averaging multiple evaluations, and validating scores against human labels on a sample.

Use a layered approach: automated checks on all outputs, LLM-as-judge on a 10% sample, and human review on 1% of edge cases. Track user feedback, latency, and cost metrics with alerting on quality score anomalies.