How to Evaluate LLM Outputs: Evals, Judges, Metrics

Evaluating LLM outputs means checking them against explicit criteria with a repeatable process, not eyeballing a few samples. You combine three layers: automated evals (a fixed test set, a rubric, and pass criteria), LLM-as-judge scoring for qualities that string matching can't capture, and classical metrics like exact match, BLEU, ROUGE, and semantic similarity for cheap, deterministic checks. The practical goal is to catch regressions before users do, at a cost per run low enough that you actually run the suite on every change.

Why LLM Outputs Need Their Own Evaluation Layer

Traditional software tests assert equality: given input X, the function must return Y. LLM outputs are nondeterministic (the same prompt can produce different wording each run) and open-ended (many correct answers exist). A test that demands one exact string will fail on valid variations; a human reading ten samples will miss regressions in the other thousand. That gap is why LLM applications need a dedicated evaluation layer: a harness that runs your system over a fixed set of inputs and scores the outputs against defined criteria, so quality becomes a number you can track over time rather than a vibe you check by hand.

The layer matters most at change points. Swapping the model, editing a system prompt, adding retrieved context, or changing a decoding parameter like temperature (the randomness knob on sampling) can silently break cases that worked before. Without an eval suite, you discover this from user complaints. With one, a prompt edit that drops your pass rate from, say, 90% to 78% on 100 cases is visible in minutes, before deployment.

There is also a cost dimension. LLM calls are billed per token (a token is roughly a word fragment, about 0.75 words in English), and long outputs multiply that cost. Evaluation lets you decide with evidence whether a cheaper model or shorter prompt is acceptable, instead of guessing. Treat the eval suite as part of the product, versioned alongside your prompts, because its test cases encode what "good" means for your use case.

Building Automated Evals: Test Sets, Rubrics, and Pass Criteria

An eval is a procedure: a dataset of inputs, a way to run your system on each, and a scoring rule that yields pass/fail or a graded score. Build it in three parts.

Test sets. Collect inputs that represent real usage: common requests, edge cases (empty input, ambiguous questions, adversarial phrasing), and past failures. Start small; a suite of roughly 30–100 curated cases is a reasonable starting point for a single feature, and growing toward several hundred improves reliability (estimates depend on task diversity). Store each case as input plus, where possible, a reference answer or expected properties. Include a held-out portion you don't tune prompts against, so your scores reflect generalization rather than overfitting to the test set.

Rubrics. A rubric states what a good output must contain or avoid, in checkable terms. Weak rubric: "answer is helpful." Strong rubric: "answer states the refund window in days, cites the policy section, and does not promise outcomes the policy excludes." Break compound criteria into separate yes/no checks so a single failure is attributable. For structured outputs, checks can be programmatic: valid JSON schema, required fields present, no forbidden strings.

Pass criteria. Decide per check whether it is binary (pass/fail) or graded (for example 1–5), and define the threshold. Then define suite-level gates: the overall pass rate must stay above a threshold, and critical checks (safety, format validity) must pass at 100%. Run the suite on every prompt or model change and record results with the version identifier. Here is a minimal shape in Python-like pseudocode:

for case in test_set:
    output = system(case.input)
    results = [check(output, case) for check in rubric_checks]
    record(case.id, results)

pass_rate = passed / total
gate = pass_rate >= 0.90 and all_critical_checks_passed

Deterministic checks (schema validity, forbidden content, required facts) are cheap and should run first; reserve expensive judging for what they cannot capture.

LLM-as-Judge: Prompting, Biases, and Calibration

An LLM-as-judge eval uses a language model to score another model's output against a rubric. It scales where human review doesn't, and it handles open-ended tasks where string matching fails — summarization, tone, helpfulness, instruction-following. The trade-off: you're trusting a model to grade, so the judge needs its own quality control.

Prompting a judge well means being boringly explicit. State the task, paste the rubric verbatim, require a verdict from a fixed set (pass/fail, or a 1–5 scale with each level defined), and ask for a short justification before the verdict — reasoning-first output tends to improve consistency. Return structured output (JSON) so you can parse scores programmatically.

Known biases to design around:

• Position bias: judges favor whichever response appears first (or last). Mitigate by running each pair in both orders and averaging.
• Verbosity bias: longer answers often score higher regardless of quality. Cap length in the rubric or penalize padding explicitly.
• Self-preference: a model tends to rate its own family's outputs higher. Use a different model as judge than the one being evaluated, or ensemble judges.
• Leniency drift: scores creep up over time. Anchor with a few labeled examples at each score level.

Calibration is the step most teams skip: score a sample of, say, 50–100 outputs with both the judge and a human, then measure agreement (e.g., Cohen's kappa, which corrects raw agreement for chance). If the judge disagrees with humans often, fix the rubric before scaling up.

Worked example (estimates). Suppose you run 500 test cases, 10% (50) human-labeled, and the judge agrees with humans on 44 of 50. Raw agreement = 44/50 = 0.88. If humans themselves agree with each other 85% of the time, a common chance-corrected figure gives kappa roughly:

kappa ≈ (0.88 - 0.85) / (1 - 0.85) = 0.03 / 0.15 = 0.20

Low kappa despite high raw agreement — the judge isn't adding much beyond chance. Tighten the rubric and re-check.

Checklist:

• Rubric with defined score levels
• Reasoning before verdict, JSON output
• Both-order runs to cancel position bias
• Judge ≠ model under test
• 50–100 outputs human-labeled for calibration
• Agreement metric tracked over time

Metrics That Matter: Exact Match, BLEU, ROUGE, Semantic Similarity, and Cost

Metrics are cheap, deterministic scores; evals are the harness around them. Match the metric to the task:

• Exact match (EM): output must equal the reference string after normalization. Perfect for classification labels, short factual answers, structured fields. Useless for paraphrase.
• BLEU: n-gram precision between output and reference, originally for translation. Cheap and fast; punishes valid rephrasing.
• ROUGE: n-gram recall, common for summarization. Rewards covering reference content; ignores whether the summary reads well.
• Semantic similarity: embed output and reference with a sentence-embedding model, take cosine similarity. Threshold it (e.g., ≥ 0.85) to turn it into a pass/fail. Catches paraphrase EM misses, but can be fooled by topical overlap without correctness.

Cost accounting. Per-item eval cost is roughly:

cost_per_item ≈ (input_tokens × input_price + output_tokens × output_price) / 1e6
suite_cost ≈ n_items × cost_per_item × n_judge_calls

If a judge call uses ~1,000 input and ~200 output tokens, and (assumed) input costs $0.50 per million tokens and output $1.50 per million:

per call = (1000 × 0.50 + 200 × 1.50) / 1e6 = (500 + 300) / 1e6 = $0.0008
1,000 items, 2 order-swapped calls: 1000 × 0.0008 × 2 = $1.60

Cheap per run, but these compound across CI. Controls: exact-match and embedding metrics first as a filter, run the judge only on ambiguous cases, cache judge results by input hash, and sample rather than judge every case on every commit.

Checklist:

• Deterministic metric where the task allows it
• Semantic threshold tuned on labeled examples
• Judge reserved for cases metrics can't resolve
• Cached results keyed by input hash
• Per-run cost tracked alongside scores

Frequently Asked Questions

What is the difference between an eval and a metric in LLM evaluation?

An eval (short for evaluation) is the whole test procedure: a set of inputs, the expected behavior, and a pass/fail decision produced by running your model against them. A metric is a single number that scores one output, such as exact match or semantic similarity, and an eval usually combines one or more metrics with a threshold to decide whether a case passed. In short, metrics are the measuring instruments, and an eval is the experiment built around them.

When should I use LLM-as-judge instead of exact-match metrics?

Use exact-match metrics when the correct answer is verifiable and narrow, like a classification label, a number, or a fixed format, because they are cheap, fast, and deterministic. Switch to LLM-as-judge when outputs are open-ended (summaries, explanations, tone, helpfulness) and many valid answers exist, since string matching would wrongly fail correct responses. A common pattern is to run cheap metrics first and reserve the judge for cases that pass the basic checks.

How do I reduce bias when using an LLM as a judge?

Reduce position bias (favoring the first or last answer) by swapping answer order and averaging the results, and reduce verbosity bias (favoring longer answers) by explicitly instructing the judge to ignore length. Give the judge a concrete rubric with scored criteria rather than a vague "which is better" prompt, and calibrate it against a small set of human-labeled examples to confirm its verdicts line up with yours. Using a stronger model as judge and keeping the judge prompt fixed across runs also improves consistency.

How many test cases do I need for a reliable eval suite?

There is no universal number, but a practical estimate: to detect a failure rate of roughly 5% with reasonable confidence, you need on the order of a few hundred cases, since small samples produce wide error margins. Start with 20–50 hand-written cases covering your core behaviors and known failure modes, then grow toward 100–300 as you add edge cases over time. Coverage of distinct scenarios matters more than raw count, so prioritize variety over volume.

How do I evaluate LLM outputs while keeping inference costs under control?

Layer your evaluation: run cheap deterministic metrics and cached responses first, and only send ambiguous cases to an LLM judge. Sample rather than judge everything (for example, a few hundred cases per run instead of tens of thousands), use a smaller judge model where its verdicts match a stronger one, and batch judge calls to cut overhead. Reuse a fixed regression set across releases so you are not regenerating test data, and re-run full evaluations only when prompts or models change.