testing-llm-features

Build an eval suite with fixed golden cases, pinned model versions, temperature 0, and frozen retrieval snapshots so runs are reproducible. Score each case with the cheapest assertion that can fail, starting with schema validation and deterministic checks before LLM judges.

Use promptfoo when the suite will grow and needs matrix runs across prompts and models with built-in assertion types. Use Vitest when evals must share fixtures and helpers with the application, or when the team will not adopt another tool.

Commit a baseline result file and compare each run against it, blocking only when a case that previously passed now fails. Measure the noise band by running the unchanged suite three times so the gate does not fire on ordinary variance.

Use an LLM judge only for genuinely subjective properties like tone or faithfulness, with a written rubric, a different model than the one under test, and validation against 30-50 human-labeled outputs. Below roughly 80% agreement with humans, the judge is noise.

The failure may be in retrieval rather than generation, since a RAG answer is wrong when the right document was never retrieved. Score retrieval independently by snapshotting documents into fixtures so you do not tune the prompt to fix an indexing problem.

Embedding similarity measures semantic closeness, not correctness, so a fluent but wrong answer can score high against a right one. Negation is nearly invisible to embeddings, so use it only as a smoke check, never as the sole assertion on factual claims.