RAG systems explained for testers

RAG has a frightening name and a simple idea.

Before the app answers your question, something goes off and finds a few documents. Those documents get pasted in next to your question. Then the model writes an answer using them.

That is all "retrieval-augmented generation" means: we looked things up first.

The name matters far less than one habit it gives you. When the answer is wrong, your first question is no longer "why did the AI make that up?" It is "what did we hand it?"

The librarian and the student

Imagine an open-book exam.

A student sits at a desk. They are not allowed to walk to the shelves. A librarian has ten seconds to grab five books, drops them on the desk, and the student writes the answer using only those five.

The student is the model. The librarian is the retriever. The five books on the desk are the context. The shelves are your documents — your help centre, your policy PDFs, your wiki.

Now, two things can go wrong, and they look identical on the screen.

The librarian brings the wrong books. The student reads them honestly and writes a confident, wrong answer. The student was never given a chance.

The librarian brings the right books and the student misreads them. Now it really is the model's fault.

Everyone blames the student. In production, it is usually the librarian.

One more detail. Books do not go on the shelf whole. They get cut into pages first, because the desk is small. That cutting-up is called chunking, and it decides what can ever be found. If the answer sits in a sentence that got split down the middle, no librarian on earth will bring it in one piece.

The five words you will hear

You do not need the maths. You need to know which part of the library each word is talking about.

Chunk. A page torn out of a document. Everything is stored as chunks, because the desk is small.

Embedding. A way of turning a chunk into numbers so the librarian can match on meaning, not just words. It is why "money back" can find a page that only ever says "refund".

Top-k. How many books the librarian is allowed to bring. Usually three to ten. If the answer is in book eleven, nobody will ever see it.

Reranker. A second, slower librarian who reorders the pile so the best book ends up on top. Not every app has one.

Grounded. An answer where every fact can be pointed at in the pile. Ungrounded means the model filled a gap itself. This word will come up in every meeting about RAG, and now you can use it correctly.

Why you should care about this

Because "the AI made it up" sends the bug to the wrong people.

If the right document never arrived, the fix is search: the indexing job, the chunk size, the filter that quietly excluded the new policy. That work belongs to whoever owns the index. Prompt changes will not help, no matter how many you try.

If the right document did arrive and the answer still ignored it, the fix lives with the prompt or the model. Different team, different change, different test.

There is a second reason, and it is the one that gets people promoted. RAG is now sitting on top of the most sensitive content a company has — support histories, HR policies, pricing, contracts. The question "can this feature quote a document at a customer that it should never have been shown?" is a testing question. Nobody else in the room is going to ask it.

How you test it

The whole method is one extra column in your test cases.

Normally you write the question and the answer you expect. For RAG you also write which document should answer it. That column is the trick. It splits one vague bug into two precise ones.

Then work in this order.

1. Run the lookup on its own. Ask your question and look at the pile of documents that came back, before you read a single word of the answer. Was the right one in there? Write down yes or no. That is the single most valuable number in RAG testing.
2. Check where it landed. Was the right document first, or buried under four near-duplicates? Models lean on what is at the top of the pile, the same way you skim the first page of search results.
3. Now read the answer. For every fact in it, point at the sentence in the pile it came from. If you cannot, the model added something. That is a real hallucination, and now you can say so with evidence.
4. Ask something the documents cannot answer. A good feature says it does not know. A bad one guesses politely. This test takes ten seconds and fails a surprising number of apps.
5. Ask a question in the wrong words. Real users write "money back" when your policy says "refund". If the librarian only matches words, this is where it falls over.
6. Change the content and test again. Add a document, edit one, delete one. In RAG, publishing a help article is a release. Almost nobody treats it as one.

Keep the results in a plain table: question, document that should answer it, documents that actually came back, verdict. Twenty rows will teach you more about your feature than a week of reading its code — the same way reading your run history teaches you more about flaky tests than theorising about them.

Six bugs you can go and find this week

These are the ones that turn up again and again. None of them needs special tooling.

The stale answer. Two versions of a document, the old one wins. Test it by editing a document and asking the same question straight away.

The confident nothing. Ask about a product you discontinued, or a country you do not ship to. A weak app invents a plausible policy rather than saying it does not know.

The wrong customer's document. If your content is per-customer or per-team, ask as user A about something only user B should see. This is the bug that ends up in a newspaper, and it is often the easiest to find.

The synonym miss. Your document says "invoice", the customer says "bill". Try five real-world wordings of the same question and see how many find the right page.

The long-document miss. The answer is in the middle of a forty-page PDF. Chunking often loses exactly that. Ask something that can only be answered from page 23.

The empty pile. Ask a very short question — one word, a product code. Weak lookups return nothing useful and the model answers from thin air anyway.

Try this today

Twenty minutes, three documents, five questions.

Write three short notes. Make two of them nearly identical on purpose — an old policy and a new one — because that is the bug you are hunting. Then write five questions, one of which your notes cannot answer.

Run each question through the lookup and record what came back. Do not read the answers yet.

Two numbers fall out. How often the right note was found at all, and how often it was found first. Watch what happens to the second number when you add a near-duplicate. That is the whole lesson, and you will not forget it.

Then read the answers and mark each fact against the pile. Where the answer used the old policy while both were on the desk, you have found the classic bug in your own toy — and you now know exactly what it looks like in the real product.

How to show you know it

A table, not an opinion. Twenty questions, the expected document for each, and the two numbers from the exercise above. Anyone can argue with "retrieval seems weak". Nobody argues with "the right document was missing in 6 of 20 cases, and here they are".

A bug report that splits the two halves. "Retrieval miss — the current policy was not in the returned set. With it pasted in by hand, the answer is correct three times out of three." That report gets picked up by the right person on the first read, which is the whole skill described in anatomy of an LLM feature.

One question in a design review. "What happens when someone edits a help article — how long until the answer changes, and who tests that?" Teams often have not decided. Asking early is worth more than any bug you file later, and it is the same instinct as asking what to automate first instead of automating whatever is nearest.