RAG testing: retrieval precision and recall

Think about a fishing net.

Cast a small, careful net and you bring up three fish, all of them the ones you wanted — but the fish you actually needed swam past the edge. You missed it.

Cast an enormous net and you definitely catch that fish, along with two boots, a shopping trolley and forty other fish. Somewhere in that pile is your answer.

Those two problems have names. Missing the fish you needed is poor recall. Hauling up the trolley is poor precision.

Every retrieval bug you will ever find is one of those two.

Two numbers, plain English

Recall. Of the questions where a correct document exists, how often did it come back? Ten out of ten is the goal. This is the number to fix first.

Precision. Of the documents that came back, how many were actually relevant? Lower precision is survivable — a model can ignore a boot — but a pile of near-duplicates crowds out the right page and confuses the answer.

You will see both written as "at k", as in recall@5. The k is just how many documents the net is allowed to bring up. Recall@5 means "was the right document in the top five".

The four words you will hear

Top-k. How many documents the lookup returns. Three to ten is typical.

Recall@k. Was the right document inside those k.

Rank. Where in the pile it landed. First is much better than fifth.

Reranker. A second pass that reorders the pile so the best document rises. Improves rank without changing what was caught.

Why you should care about this

Because it turns the vaguest complaint in AI into a two-column table.

"The chatbot is unreliable" is unactionable. "The correct document was missing for 6 of 20 questions, all six about the new pricing page" points at an indexing job and a person who can fix it before lunch.

It is also the cheapest testing on this layer. You do not need the model, a judge, a rubric or a threshold. You need the lookup and a list of questions. That means you can run it early, run it often, and run it on every content change — which is exactly when things break.

How you test it

1. Build the question-to-document list. Twenty real questions, and for each one the document that should answer it. This is the same extra column from the RAG topic, and it is the whole test suite. Pull the questions from tickets and search logs, not imagination — the rule from building a golden dataset.

2. Ask for a way to run the lookup alone. Usually a debug endpoint or a script that already exists. Without it you are testing the whole pipeline and guessing which half failed.

3. Measure recall@k, then rank. Two counts: how often the right document appeared, and how often it was first. Both from the same run.

4. Vary the wording, not the meaning. Take one question and write it five ways a real person would — "money back", "refund", "cancel my order", "get charged twice", plus one with a typo. A net that only catches the official phrasing will look excellent in testing and fail in production.

5. Add the six cases that break real systems. Near-duplicate documents. A superseded version alongside the current one. An answer that sits in the middle of a long PDF. A question needing two documents at once. A question with no answer anywhere. A question about content that should be invisible to this user.

6. Re-run on every content change. New article, edited policy, re-index, chunk-size tweak. In a RAG system, publishing content is a release, and this suite is its regression pack.

Try this today

Twenty questions, one spreadsheet, no model involved.

That pattern line is the finding. Not "retrieval is weak" — a specific, reproducible gap with four examples attached, and a fix that lives in the lookup rather than the prompt.

How to show you know it

Recall and rank, with the pattern. Two numbers and one sentence about what the misses have in common. The sentence is the part that gets it fixed.

A permission miss, if you find one. File it as a security bug, not a quality one, and expect it to jump the queue.

A content-change run. "Recall was 78% before the re-index and 91% after." It proves the suite is doing its job, and it makes you the person who can answer "did that help?"

The honest caveat. "This measures the lookup only. It says nothing about whether the answer used the document well." That second half is groundedness, and knowing the boundary between the two is the mark of someone who has actually run both.