LLM-as-judge, and its failure modes

You have two hundred answers to mark and an afternoon.

So you hire a marker. Fast, cheap, never bored, works at three in the morning. You hand them your marking scheme and off they go.

Then you spot check their work and find something awkward. They give higher marks to longer answers. They prefer answers written in their own style. And when two answers are equally good, they tend to pick whichever one they read first.

That is a model marking another model's work. It is genuinely useful. It also cheats in four ways that are well documented, and once you know them you can work with it.

The marker you hired

The setup is simple. You send the marker three things: the question, the answer to grade, and your marking scheme. It sends back a score and a sentence of reasoning.

Where it earns its keep is the judgement a rule cannot make. "Is this answer actually supported by the document?" A rule can check whether the word appears. Only a reader can tell you whether the answer twisted what the document said.

Where it lets you down is that it is a model, so it is agreeable, fluent and easily led — the same properties that make the thing under test hard to test.

The five words you will hear

Judge or marker. The model doing the grading.

Rubric. Your marking scheme, written out for the judge to follow.

Pointwise. Grading one answer on its own. Simplest, and what you want most of the time.

Pairwise. Comparing two answers and picking the better. Prone to the position bias above.

Agreement. How often the judge and a human reach the same verdict. This is the only number that tells you whether the judge is usable.

Why you should care about this

Because a judge nobody checked will happily report that a broken feature is improving. The number on your eval is only ever as good as whatever marked it.

That is not a hypothetical. A judge with a lenient streak scores everything 4 out of 5. The team ships prompt changes for a month, the number never moves, and everyone concludes the feature is stable. It was not stable. The marker just was not looking.

There is a second reason, closer to home. When someone shows you a dashboard of AI quality scores, the right first question is "what marked these, and how do we know it is any good?" Almost nobody asks it. Asking it well makes you the person whose numbers get believed — the same instinct as only reporting metrics you can defend.

How you test the marker

You test it the way you would test any new tool: against a known-good answer key.

1. Mark fifty by hand first. Yes, fifty, and yes, by hand. Use your golden dataset so the cases are the ones you care about. This is the boring afternoon that makes everything afterwards trustworthy.

2. Run the judge on the same fifty. Now you have two columns.

3. Count the disagreements, and read them. Agreement above roughly 80% on a binary pass/fail is usable. Below that, the judge is not broken — your rubric is vague. Read the ten cases where you differed and you will usually find the missing sentence.

4. Fix the rubric, not the model. Judges follow instructions literally. "Is the answer good?" is unanswerable. "Does every factual claim appear in the supplied document? Answer yes or no, then quote the sentence" is answerable.

5. Test it with deliberate junk. Feed it an answer padded with a hundred harmless extra words. Feed it a confident, well-written, completely wrong answer. If both score well, you have measured the length bias in your own setup, which is far more persuasive than reading about it here.

6. Swap the order. Where you compare two answers, run each pair both ways round. If the winner changes, position bias is real in your setup and you should switch to grading one at a time.

7. Re-check whenever anything changes. New judge version, new rubric, new answer format — re-run the fifty. Agreement decays quietly, and it is the same drift problem that moves the feature you are testing.

Try this today

Take ten answers your feature has already produced. Mark them yourself, pass or fail, before you look at anything else. Then ask a model to mark them with a deliberately vague rubric, then again with a sharp one.

The two rows are the whole lesson. Nothing changed except the wording of the instruction, and the marker went from useless to useful. Whenever a judge looks stupid, suspect the rubric first.

Now read the one case you disagreed on. Nine times out of ten it is a case where you would struggle to explain the rule, which means the rule needs writing down.

How to show you know it

An agreement number with your name on it. "I hand-marked fifty; the judge agrees on 88%, and here are the six it got wrong." That single sentence is worth more than any tool evaluation.

A rubric that got sharper. Keep both versions and the agreement score for each. It shows the fix for a bad judge is usually writing, not technology.

A bias you found in your own setup. Padding an answer and watching the score go up takes fifteen minutes and makes the point permanently, to everyone in the room.

Knowing when not to use it. "We use a rule for the refund window because a rule is exact, and the judge only for groundedness." Choosing the cheaper tool where it works is the mark of someone who has done this before.