Skip to content
Velloreth
Artificial Intelligence 9 min read

Evaluating AI Outputs: A Practical Framework

A practical framework for evaluating LLM outputs: golden sets, programmatic checks, LLM-as-judge with rubrics, agreement metrics, and evals in CI.

By

“You can just tell it’s better” is the default evaluation method for AI features, and it fails in a specific, predictable way: it confirms whatever you believed going in. The model looks better because you changed the prompt, you remember the three examples where it improved, and the twenty that regressed were never written down.

A practical evaluation framework is not a research project. It is a small set, a scoring method you can repeat, and a habit of running it before every change. The whole thing fits in an afternoon to set up — and it keeps paying off the day someone asks whether the new model is actually better, which is a question you can answer instead of debate.

Decide what “good” means first

Evaluation is impossible until the target is defined, and for AI outputs the target is usually a combination of dimensions:

DimensionThe questionTypical check
CorrectnessIs the answer right?Reference answer, unit tests, DB lookups
FaithfulnessIs every claim supported by the source?Grounding against provided context
Format adherenceDoes it match the schema/template?Parser, JSON schema, regex
Instruction followingDid it obey the constraints?Rule checks per constraint
Safety / policyDid it refuse what it should, allow what it should?Red-team cases, allow/deny lists
Usefulness (subjective)Is it actually good to read or act on?Human rating, LLM judge with rubric

Two habits make this concrete. First, write each dimension as a yes/no question about the output — “does every cited source contain the claim?” rates better than “is the response accurate?” Second, weight them: for a code generator, correctness dominates; for a support reply, tone and policy compliance matter alongside correctness. An unweighted average hides which dimension is slipping.

The evaluation ladder

Not every check deserves the same machinery. Climb the ladder only as far as the task requires:

1. Programmatic checks — free and absolute. Schema validation, required fields, banned strings, length bounds, “did it cite a source that exists.” These are deterministic, instant, and cannot be argued with. Do them first; they will catch the embarrassments before any judge sees the output.

2. Reference-based metrics — cheap when you have ground truth. Exact match for classification, F1 for extraction, token-level overlap or embedding similarity for close-ended generation. The limitation is real: they punish legitimate paraphrases and reward wrong answers that use the right words. Use them where an answer key exists.

3. LLM-as-judge with a rubric — for everything qualitative. Faithfulness, clarity, instruction following. Covered in detail below; this is where most teams over- or under-invest.

4. Human review — for the tail and for calibration. Humans set the standard, resolve ties, and audit the judge. You never need many — a few dozen blind ratings per cycle — but you always need some.

LayerCost per itemDeterministic?Use when
Programmatic~0YesAlways — it is free
Reference metric~0YesAnswer key exists
LLM judgeLow, repeatableNoisyQualitative dimensions
HumanHighCalibratableCalibrating the judge, final sign-off

Build the golden set

The golden set is 50–100 real cases with known-correct expectations. Everything else in this framework depends on it.

Where cases come from:

  • Production traffic, sampled — never only your own test prompts.
  • The documented edge cases (empty input, wrong language, hostile input).
  • Known past failures, each added the moment it ships a fix.
  • Distribution coverage: make sure easy, average, and nasty cases all appear.

What each case contains:

# eval case shape — one file or row per case
id: ticket-triage-042
input: "URGENT: payments API returning 500s since deploy 9f31c"
expected:
  category: incident
  severity: high
  must_contain: ["payments", "500"]
  must_not_contain: ["resolved", "no issue"]
judge_rubric:
  - "Labels the issue as an incident, not a question"
  - "Assigns high or critical severity"
  - "Does not invent a root cause not present in the input"

Note the rubric lives with the case. Per-case criteria outperform a generic global rubric because the specific things that make this answer right are not the things that make every answer right — research on LLM judges consistently finds that instance-specific criteria correlate better with human judgment than a one-size-fits-all prompt.

Keep it honest: hold out a slice the prompt author never looks at while iterating, and version the set in git. An eval set edited to match the current model’s output measures nothing.

Using an LLM as a judge

An LLM judge is a measurement instrument, and like any instrument it needs calibration. The failure modes are well documented:

Validate against humans first. Take 30–50 outputs, have two humans rate them blind with the same rubric, run the judge, and compare. Report an agreement statistic — Cohen’s kappa for categorical verdicts, Spearman for graded scores — not just raw accuracy, because a judge that always says “good” can score well on a skewed set while being useless.

Note that humans disagree too. Human–human agreement is the ceiling you are chasing, not 100%. Rubric work at Microsoft showed individual judges diverge enough that the interesting goal is predicting each rater reliably — combining several specific rubric questions beat a single overall score by a wide margin. If your judge’s agreement is in the same range as two humans rating each other, you have a usable instrument.

Write criteria, not vibes. One study comparing design choices found evaluation criteria had a larger effect on judge–human correlation than including reference answers did. “Is the answer correct?” is a weak criterion. “Does the response address the user’s actual question and avoid unsupported claims?” is a rateable one.

Control the known biases:

  • Position bias — shuffle order in pairwise comparisons, never keep A always first.
  • Length bias — longer answers often score better for no good reason; add “be skeptical of verbosity” or evaluate per-criterion.
  • Self-preference — a judge tends to favor outputs from its own model family; use a different model as judge than as generator where possible.
  • Consistency — run the judge twice on a sample; if verdicts flip, your rubric is underspecified.

A judge prompt that behaves:

JUDGE = """You are grading one answer against a rubric.
For EACH rubric item answer YES or NO, then quote the exact span that justifies it.
Do not grade overall quality. Do not reward length.
If evidence is missing, answer NO.

Rubric:
{rubric}

Input: {input}
Output: {output}

Return JSON: {{"items": [{"id": ..., "verdict": "YES|NO", "evidence": "..."}]}}"""

Requiring a quoted evidence span does two things: it forces the judge to look for proof, and it gives you something to spot-check in seconds.

What to report

A single “quality score” tells you almost nothing. Minimal useful reporting per evaluation run:

MetricWhy
Pass rate per rubric criterionShows which dimension moved
Overall pass rate vs baselineThe headline number
Agreement with human labels (κ or ρ)Whether the judge is trustworthy this cycle
Judge consistency across two runsWhether the rubric is stable
Delta since last releaseDirection, not just level
Failures by input categoryWhere to look next

For pairwise model or prompt comparisons, report win/tie/loss rather than average scores, and keep the comparison blind to the judge.

Watch the noise floor. Nondeterministic generation plus a nondeterministic judge means small deltas are meaningless. Run the eval set more than once; if a claimed 4-point improvement falls inside run-to-run variance, it is not an improvement, it is weather. Changes worth trusting are larger than your own measured variance.

Where evaluation goes wrong

Four patterns account for most dead eval programs:

The set becomes a tutorial. You iterate on prompts until the golden set passes, and the system is now overfit to 80 examples you personally wrote. Production then surprises you. The defense is the held-out slice plus production sampling — you must keep seeing inputs the prompt author has not.

The rubric drifts with the model. Each time the model regresses on a criterion, someone softens the wording until it passes. Rubric changes are legitimate — but they belong in a commit with a note explaining why the old standard was wrong, not in a quiet edit during a release.

One metric gets optimized into the ground truth. If the judge rewards citation presence, the model learns to cite — including sources that do not support the claim. Pair every proxy with the check that catches its characteristic lie: citations get citation-verification, “complete coverage” gets a required-missing-items check.

Evaluation never gates anything. The dashboard exists, numbers are nice, and changes ship without running it. An eval suite that does not block a merge is documentation. Wire it into CI with a threshold, even a crude one, and the framework starts paying for itself.

There is also a trap in over-automating. Some judgments — whether a poem is good, whether advice is wise — resist rubrics and will burn your judge-credibility budget if you force them. For those, sample and rate by hand at low frequency instead. The framework is a tool for deciding; where it cannot decide, that decision is a human one, cheaply made on a small sample.

Running this in practice

The loop that keeps evaluation alive after week one:

  1. Baseline. Run the current system on the golden set, record numbers, commit them. This is what every future change is judged against.
  2. Pre-merge. Run programmatic checks plus the golden set on every prompt or model change. Fail the change on regression beyond a set threshold.
  3. Sample production. Log inputs, outputs, and user behavior; sample a few percent for grading. Production distribution always beats the cases you imagined.
  4. Promote failures. Every escaped failure becomes a golden case within a day. This is the growth mechanism — without it the set rots.
  5. Re-calibrate the judge monthly. Humans rate a fresh sample, recompute agreement, and adjust the rubric where the judge and humans diverge.

Keep the layers separate in your head: programmatic checks gate every run, reference metrics track close-ended tasks, the judge tracks qualitative drift, humans anchor the whole system. None of it is complicated; all of it beats vibes.

The test of the framework is simple. Six weeks from now, someone changes the prompt and asks “is this better?” — and you answer with a number, a per-criterion breakdown, and the three cases where it got worse. That is the entire point.

Frequently asked questions

How many examples do I need in an evaluation set?
Fifty to one hundred well-chosen real cases is enough to guide most early decisions; add cases whenever production reveals a new failure mode. Breadth across the input distribution matters far more than raw count.
Can an LLM judge replace human evaluation?
For routine regression checking, yes — if you validate it against human labels first and keep a human sample in the loop. Humans disagree with each other too, so the realistic goal is matching the agreement level humans achieve with each other.
What metrics should I report for an LLM evaluation?
At minimum: pass rate on your criteria, plus an agreement statistic against human labels such as Cohen's kappa for categorical verdicts or a rank correlation for graded scores. Always state how abstentions and ties were handled.
How often should evaluation sets be updated?
Continuously. Every production failure that your golden set did not catch becomes a new case. Sets that never grow quietly stop measuring your system and start measuring the examples you happened to write first.