Groundedness, faithfulness, hallucination rate

A school essay with citations gets marked twice. Once for whether the argument is any good, and once for whether the quotes are real.

The second check is mechanical and merciless. The teacher takes the quote, opens the book, and looks. Either the sentence is there or the student invented it.

That is groundedness. Every claim in the answer, checked against the material the answer was given.

It is the most useful number on this layer, and the most inconsistently defined. Five tools will give you five different figures for the same answers, all of them called the same thing.

The essay and the book

The claim is the unit. Not the answer, not the sentence — the claim.

"Refunds take 14 days and are issued to the original card" is two claims. One might be in the document and the other invented, and any number that treats the whole answer as pass-or-fail loses that completely.

So you go claim by claim. For each one, open the pile of documents the model was given, and try to point at the sentence that supports it. Three outcomes:

Supported. You found it. Good.

Contradicted. The document says something different. This is the worst kind, because the answer sounds authoritative and the source was right there.

Not in the pile. Nothing supports it. The model filled the gap itself. This is the classic hallucination.

The five words you will hear

Claim. One checkable statement. Splitting answers into claims is most of the work.

Grounded / faithful. Supported by the supplied material. The two words are used interchangeably, and that is fine.

Hallucination rate. The share of claims not supported. Ask whether it is measured per claim or per answer, because the two numbers differ by a lot.

Citation. A pointer from a claim to its source. If your feature emits these, checking becomes almost free.

Abstention. Saying "I do not know". A feature that abstains when the pile is empty is doing the right thing, and a groundedness score that punishes it is misconfigured.

Why you should care about this

Because it is the number that converts "the AI made something up" into evidence.

"It hallucinated" is an opinion. "Of the 63 claims across these 20 answers, 7 were not supported by the retrieved documents, and here they are" is a finding. One of those gets fixed.

And it is the number executives ask for. When somebody senior asks "how often does it make things up?", the honest answer is a rate with a method attached. Being the person who can produce both is worth a great deal — and being the person who can explain why the vendor's number is not comparable is worth more, which is the same discipline as choosing metrics you can defend.

How you measure it

Start by hand. Twenty answers, one afternoon, and you will understand this better than any dashboard could teach you.

1. Save the pile with the answer. You need the exact documents the model was handed. Without them there is nothing to check against, and this is usually the blocker — ask for it early.

2. Split the answer into claims. One statement per line. Numbers, dates, names, policies, causal statements. Ignore politeness and filler.

3. Mark each claim supported, contradicted, or not-in-pile. Quote the supporting sentence when you find it. Quoting keeps you honest.

4. Report two numbers, not one. Claims supported out of total claims, and answers containing at least one unsupported claim. The first is the quality number, the second is what a customer actually experiences.

5. Separate the abstentions. Count "I do not know" answers on their own. They are correct behaviour when the pile is empty and a problem when it is not.

6. Then automate the boring part. Once you have done twenty by hand, a judge model with a sharp rubric can score the rest — and you have the fifty hand-marked cases you need to check it against, as in the marker who likes long answers.

Try this today

Take five answers your feature produced, along with the documents each was given. Split and mark. It takes about half an hour and it will change what you file bugs about.

Look at what that finds. The answer reads beautifully, it is right about the headline number, and it contradicts the policy on the deadline while inventing a support escalation that does not exist. No exact-match test would ever have caught it, and neither would a quick skim by a busy human.

Now do the same five again with the old policy document removed from the shelf. Watch the contradiction disappear. That is your bug report writing itself: the fix is in the index, not the model.

How to show you know it

A rate with a method. "70% of claims supported across 27 claims in 5 answers; one contradiction. Method: manual, claim level, against the retrieved pile." Anybody can quote a percentage; almost nobody attaches the method.

A contradiction, with both sentences. The answer's claim and the document's line, side by side. It is the single most persuasive bug you can file on a RAG feature.

The two-number habit. Per-claim and per-answer rates together. It shows you know that one bad claim ruins a whole answer for the person reading it.

A question about someone else's number. "Is that per claim or per answer, and against the retrieved documents or the whole knowledge base?" Vendors rarely volunteer this, and the answer changes the figure enormously.