write-judge-prompt

You can evaluate subjective criteria like tone and faithfulness by designing an LLM-as-Judge evaluator. This approach uses a judge prompt to interpret qualities that automated code-based checks cannot handle, requiring human-labeled data to define pass/fail outcomes.

You need an LLM-as-Judge when a failure mode requires interpretation, such as assessing empathetic tone, relevance, or completeness. Code-based evaluation falls short for these subjective qualities, whereas an LLM judge can be designed to interpret and evaluate them.

To create an LLM judge prompt for customer service emails, define the subjective quality to assess, such as empathetic tone. You then design the judge to provide a detailed critique before enforcing a strict binary pass/fail verdict based on human-labeled data.

Designing an LLM-as-Judge evaluator requires human-labeled data and a clear definition of pass/fail criteria. This labeled data grounds the judge's interpretation of subjective qualities, ensuring the structured critique and binary verdict align with expected outcomes.

Yes, an LLM judge can output a detailed critique before a pass/fail verdict. By enforcing structured output, the judge provides its reasoning and interpretation of subjective criteria first, ensuring the strict binary outcome is justified.

Yes, LLM-as-Judge evaluation enforces strict binary outcomes for clear evaluation. The judge provides a detailed critique of subjective criteria before delivering a definitive pass or fail verdict based on the defined human-labeled data.