How to Use LLM as a Judge: Calibrate and Trust Scores

Key takeaways

• Write judge prompts with anchored rubric levels, evidence-before-verdict output order, and pinned temperature, model version, and output schema.
• Validate a judge against 100-200 human-labeled samples using Cohen's kappa, computed per failure mode rather than as one blended score.
• Split rubrics when failure modes pull against each other, such as separating safety misses from tone overflags, and calibrate each dimension independently.
• Mitigate position bias by swapping pairwise order, verbosity bias by testing short and long answers with identical substance, and self-preference by using a different model family as judge.
• Treat judge-human divergence as data: log both scores, tag the miss by type, and route borderline answers through a confidence-justification checkpoint.

Using an LLM as a judge works when you treat the judge itself as a system under test: write rubric-scoped judge prompts, measure judge-human agreement with Cohen's kappa on a labeled sample, split rubrics so safety misses and tone overflags stop fighting each other, then wire the loop into your eval pipeline. This walkthrough covers each step, including the divergence case where your judge passes a response your human reviewer would flag.

What does LLM-as-a-judge mean and when should you use it?

LLM-as-a-judge means using a model, usually with a written rubric, to score outputs from your pipeline or agent instead of scoring each one by hand. The judge reads the response (and often the retrieved context or trace) and emits a score, a label, or a pairwise preference, which you capture as an eval score in your pipeline.

It is the right approach when deterministic checks cannot express the criterion, such as tone, subtle policy violations, or whether an answer sounds surer than its reasoning allows, and human review is too slow or expensive to run on every eval sample. Use deterministic assertions for anything checkable: schema validity, refusals, exact strings, regex matches. Use human review for calibration, borderline cases, and building the rubric itself.

The standard workflow is documented in Langfuse's LLM-as-a-judge evaluation guide, which covers judge prompt design and score capture. The catch most teams hit: the judge agrees with the literal rubric but misses what a human reviewer knows, which is the divergence problem this article builds toward.

How do you write a judge prompt that scores consistently?

A judge prompt scores consistently when it pins three things down: an explicit rubric, an evidence-before-verdict output order, and fixed inference settings. Change any of those and your eval samples will drift between runs even though nothing about your pipeline changed.

Score with a rubric, not a vibe

Replace "rate the response quality from 1 to 5" with anchored levels. For each score, describe what must be true: a 3 means "correct answer, but skips the user's stated constraint," not "pretty good." Unanchored scales collapse toward the middle because the model has no definition of the extremes.

Split the rubric per failure mode. If one rubric scores subtle coercion and money-related tone in the same pass, the two criteria pull against each other and you get scores that are wrong in both directions at once.

Force evidence before verdict

Require the judge to quote the specific span that justifies each criterion before emitting a score. Verdict-first outputs let the model pattern-match to a number and rationalize afterward. Evidence-first outputs also give your human reviewers something to audit when they disagree with a borderline call.

Fix temperature, model, and output schema

Set temperature to 0 (or as low as your provider allows), pin the judge model and version, and enforce a structured output schema so a score can never arrive as prose. Log all three alongside each score. When scores shift between runs, the first suspect is an unpinned model version, not your rubric.

When should you use a single judge versus pairwise comparison or a panel of judges?

Single-judge scoring fits high-volume regression evals where you need a stable score per sample against a rubric. Pairwise comparison fits A/B testing prompts or models: judges are more reliable at relative calls than absolute ones, though position bias means you should swap order and average, since judges favor whichever answer appears first (Zheng et al., "Large Language Models are not Fair Evaluators"). A panel of judges fits borderline safety calls where one model's blind spots matter, at roughly N times the cost.

ClearRun runs as a single-judge checkpoint and flags when confidence outruns justification; it does not run panels, so for correlated-judge risk you still need diverse models behind the panel.

What are the main biases of LLM judges and how do you mitigate them?

Three biases show up repeatedly: position bias in pairwise comparison (judges favor whichever answer appears first), verbosity bias (longer answers score higher regardless of quality), and self-preference (a model rates its own outputs more favorably).

Mitigations are mechanical. For position bias, run pairwise comparisons in both orders and discard or flag results where the verdict flips. For verbosity bias, strip your rubric of any criterion that correlates with length, and test it: generate one short and one long answer with identical substance and check the scores match. For self-preference, never let a model grade outputs from the same family that produced them; use a different model as judge, or at minimum track agreement rates separately for self-judged samples.

One bias gets less attention: confidence bias, where a judge rewards answers that sound authoritative. That is the failure ClearRun targets, scoring whether an answer's stated certainty is justified by its reasoning rather than its tone.

How do you validate that your LLM judge agrees with human evaluators?

Run a labeled calibration set

Sample 100-200 eval samples your judge will actually see in production, including borderline cases, and have two or more human reviewers label them independently. Record reviewer-to-reviewer agreement first: if humans disagree with each other at 70%, your judge cannot beat that ceiling, and the fix is a sharper rubric, not a better judge. Recalibrate quarterly or whenever you change the judge prompt or model.

Compute agreement, not just accuracy

Raw percent agreement hides chance overlap, especially on imbalanced labels where "pass" dominates. Cohen's kappa corrects for that and is the standard metric for judge-human calibration. As a working rule from the interpretation conventions for kappa, values below 0.60 signal substantial disagreement, 0.60-0.80 is moderate-to-substantial, and above 0.80 is strong definition and interpretation reference for Cohen's kappa. Pair kappa with a confusion matrix so you can see which direction the misses run: a judge that overflags money mentions and misses subtle coercion can post a decent kappa while failing both real jobs.

That directional split matters more than the headline score. Build a per-failure-mode table from your calibration set: for each rubric dimension, compute kappa and the false-positive and false-negative rates separately. A judge scoring 0.85 on tone but 0.45 on coercion is not one mediocre judge, it is one good judge and one broken one wearing the same name. Reporting a single blended number buries exactly the signal you need, which is why teams that only track aggregate kappa keep rediscovering the same miss six weeks later in production complaints.

Split the rubric when failure modes pull against each other

When one dimension's false positives trade off against another's false negatives, stop scoring both with one rubric. Split safety (coercion, policy violation) from tone and appropriateness, run separate judge prompts, and calibrate each against its own human labels. You will get two kappas you can actually move, and thresholds you can tune per dimension without breaking the other.

How do you run LLM-as-a-judge evaluations in Langfuse?

Trace your pipeline first

Before scoring anything, get your actual pipeline into Langfuse as traces. Wrap each LLM call, tool call, and retrieval step so a single eval sample maps to a complete trace with inputs, outputs, and timings. Without traces, your judge scores an output in isolation and you lose the context that explains borderline calls.

Attach judge scores as evaluation scores

Run your judge prompt against trace outputs and record results using Langfuse evaluation scores, keyed to the trace ID. Store the judge's reasoning alongside the numeric score so a human reviewer can audit why a response passed. If you split rubrics into separate dimensions (safety, tone, correctness), attach each as its own score rather than averaging them into one number.

Version judge prompts and compare runs

Keep judge prompts in Langfuse prompt management and tag versions. When you rewrite a rubric, re-score the same eval samples under both prompt versions and compare score distributions run over run. This is how you catch a rubric edit that quietly shifts your judge's thresholds, and it gives you a concrete artifact to bring to the human reviewers when calibration comes up.

How do you turn judge scores into a continuous evaluation pipeline?

A judge score that only runs in a notebook decays the moment your prompts or models change. Treat it like a test suite: run the judge on a fixed set of eval samples in CI, fail the build when agreement with your golden labels drops below a threshold you set, and version the judge prompt itself so a rubric edit is reviewable like any code change. Langfuse's prompt management and experiment versioning support exactly this workflow, tracking which prompt version produced which scores (Langfuse docs). In production, sample a slice of live traces, run the same judge over them, and alert on drift in score distribution rather than on single failures. Route borderline answers through a second checkpoint: ClearRun flags responses whose stated confidence outruns their reasoning and attaches a signed record when the call matters.

Honest gap: this adds latency per sampled trace, and if your sampling rate is low you will miss rare failure modes entirely.

How do you handle judge-human divergence on safety calls?

When the judge passes a response your human reviewer flags, treat the disagreement as data, not noise. Log both scores, tag the miss by type (safety gap like missed coercion versus tone false positive like money mentions), and split the rubric so each failure mode gets its own criterion instead of one score pulling against the other.

Then add a checkpoint for borderline cases: flag answers whose stated confidence outruns their reasoning. That is what ClearRun does. It scores whether an answer's certainty matches its justification, ships as a CLI, SDK, MCP server, and hosted API, and signs a record when the decision matters. Honest limitation: it will not catch a subtle policy violation your rubric never describes, and it adds latency per call. Calibrate it against your reviewer overrides the same way you calibrate any judge.

FAQ

How many labeled examples do I need to calibrate an LLM judge against human reviewers?

Sample 100-200 eval samples the judge will actually see in production, weighted toward borderline cases. Have two or more human reviewers label them independently, and record reviewer-to-reviewer agreement first, since your judge cannot beat that ceiling. Recalibrate quarterly or whenever you change the judge prompt or model.

Can I use the same model as both the generator and the judge?

Avoid it. Self-preference bias means a model rates outputs from its own family more favorably, so scores drift upward without any real quality change. Use a different model family as the judge, or at minimum track agreement rates separately for self-judged samples so the inflation is visible in your calibration data.

What agreement score is good enough before I trust an LLM judge in production?

Use Cohen's kappa, not raw percent agreement, since raw agreement hides chance overlap on imbalanced labels. Below 0.60 signals substantial disagreement, 0.60-0.80 is moderate-to-substantial, and above 0.80 is strong. Also compute kappa per failure mode: a blended 0.85 on tone can hide a 0.45 on coercion.

How do I stop my LLM judge from being too strict or too lenient over time?

Version the judge prompt and re-score the same eval samples under both versions when you edit a rubric, so threshold shifts are visible run over run. In production, sample live traces and alert on drift in the score distribution rather than single failures. Recalibrate against human labels quarterly.

Is LLM-as-a-judge suitable for compliance or audit evidence?

Partially. Judge scores with stored reasoning give auditable records, and checkpoints like ClearRun sign a record when a decision matters. But a judge will miss subtle policy violations your rubric never describes, and aggregate scores bury directional misses. Keep human review for borderline safety calls and calibrate against reviewer overrides.