reviewing-ai-output-groundedness

Decompose each output into atomic claims and verify each against the sources actually retrieved for that query, not the whole corpus. Classify every claim as grounded, fabricated, contradicted, conflated, stale, unsupported inference, or unverifiable, then attribute failures to retrieval or generation.

Run three checks per citation: the document exists, it was in the retrieval set for that query, and it supports the specific claim attached to it. The third check fails most often, since a real retrieved document discussing the topic but not the claim passes automated citation-resolution checks.

A working review uses 20-30 outputs: roughly 10 random from real traffic, 6 hard cases, 5 high-stakes cases, and 4 user-reported cases. Below ten outputs the result is a spot check, not a review, and the sample must not carry across model or prompt changes.

No. Eval suites catch structural regressions and approximate faithfulness with a judge, but cannot reliably detect a fluent, well-cited claim that is quietly wrong for the domain. Human review is sampled rather than exhaustive and ends in an attestation with stated limitations, not a score.

Common causes are retrieval failures where the right document was never fetched, generation failures where a retrieved document was misread or contradicted, and coverage gaps where no source exists. Diagnosing which layer failed per claim prevents weeks of prompt engineering spent fixing an indexing bug.

It is sampled, not exhaustive, so results carry explicit limitations such as languages, tenants, or time periods excluded. It also cannot transfer across model or prompt changes, and its output is an attestation with a stated factuality bar, never a percentage presented as a guarantee.