Rubrics, scoring and inter-rater agreement

Watch gymnastics at the Olympics. Several judges, sitting apart, not allowed to talk. Their scores land within a fraction of each other.

That is not telepathy. They share a rubric. Start value for the routine, named deductions with fixed amounts — a quarter off for a bent knee, half for a step on landing.

Ask five people "was that a good routine?" and you get five answers. Give them the deduction list and they converge.

Every argument about AI quality is that first question. A rubric is how you turn it into the second one.

What a rubric actually is

A short list of checks, each one thing, each answerable by someone who has never met you.

Here is the difference in one pair of lines.

Bad: "The answer should be helpful and accurate."

Good: "States the refund window in days. Cites the current policy document. Contains no promise of a refund outside that window. Does not name any other customer."

Four checks. Yes or no on each. Two people would tick the same boxes, and so would a model — which is the entire reason the second version is worth writing.

The four words you will hear

Rubric. The list of checks.

Weight. How much each check is worth. Most rubrics do not need weights — a plain count is fine, and simpler to defend.

Blocking check. A line where failure fails the whole case regardless of the rest. Data leaks and safety refusals belong here.

Inter-rater agreement. How often two markers reach the same verdict. This is the number that tells you the rubric works.

Why you should care about this

Because without a rubric, "quality" is decided by whoever is most senior in the meeting.

You have seen this argument. Someone says the new prompt is better, someone else says it is worse, both are looking at the same four answers, and the change ships or does not ship based on tone of voice. A rubric ends that in an afternoon.

It also gives you something rarer: a way to prove that a judgement is stable. Anyone can show a test that passes. Showing that two independent markers agree 90% of the time on what "grounded" means is a different kind of evidence, and it is the thing that makes a model marker usable at all — see the marker who likes long answers.

How you write one and prove it works

1. Write the checks from failures you have seen. Not from an ideal. Go through the last twenty bad answers in your golden dataset and turn each into a line. A rubric grown from real failures covers the right things automatically.

2. Keep it to five to seven lines. Longer rubrics do not get followed — by people or by models. If you need more, you probably have two rubrics: one for safety, one for quality.

3. Mark the blocking lines. Usually two or three. "Leaked another customer's data" is not a deduction, it is a fail.

4. Score plainly. Count of checks passed, out of the total, and feed that into the eval you already run. Reach for weights only when someone can explain why one line is worth double, and write the reason down next to it.

5. Now test the rubric on people. Give the same ten answers and the rubric to a colleague, separately. Compare, line by line, not just on the totals — two markers can agree on 4/5 while disagreeing about which line failed, and that gap is where the ambiguity lives.

6. Rewrite the lines you disagreed on. Every disagreement is a sentence that needs sharpening. This is the loop, and two or three passes usually gets a rubric to the point where it stops causing arguments.

7. Re-check it when the product changes. New policy, new feature, new refusal rule — the rubric needs a line, or an old line is now wrong.

Try this today

Ten answers, one colleague, twenty minutes.

Write five checks. Mark ten answers each, separately, with no talking. Then put the two sheets side by side.

Two ambiguous lines out of five is a completely normal first attempt. Rewrite those two, mark ten fresh answers, and watch the agreement climb. That climb is the deliverable — it is the evidence that your definition of quality is now shared rather than personal.

How to show you know it

A rubric and an agreement number. "Five checks, two of them blocking; two of us agree on 9 of 10 answers." Very few people can produce that pair of things, and it settles arguments before they start.

The before-and-after of one line. The vague version, the disagreement it caused, the sharpened version. It shows you understand that the fix is writing.

A rubric that names its blocking lines. It proves you can tell the difference between a quality problem and a safety problem, which is the distinction most quality dashboards blur.

A note on what it does not measure. "This says nothing about latency, cost or tone." The same honesty that makes session notes useful rather than decorative.