Chunking, ranking and citation checking

Somebody takes a cookbook and cuts it into index cards, because the cards are easier to file.

Cut between recipes and every card makes sense. Cut every three inches with scissors and you get a card holding the last two ingredients of one recipe and the first three steps of the next.

Now ask the kitchen for "the cake recipe". They hand you a card. It is not wrong exactly. It is unusable.

That is chunking. It happens to every document before anybody can search it — the shelf in the librarian and the student is made of these cards — it is invisible in the interface, and it quietly decides which questions your feature can ever answer.

Cards, order, and receipts

Three ideas, one after the other.

Chunking is the cutting. Roughly a page at a time, sometimes by heading, sometimes just by character count. The last option is common and it is where the trouble lives.

Ranking is the order the cards come back in. The model reads the top of the pile most carefully, exactly like a person skimming search results, so a correct card in fifth place often loses to a confident wrong one in first.

Citations are the receipts — a pointer from each claim back to the card it came from. Not every feature emits them. Every feature should, because they make checking groundedness almost free.

Why you should care about this

Because this is the failure nobody in the room can see.

The prompt looks fine. The lookup returned something. The answer reads well. The actual cause is a cut through the middle of a policy sentence three weeks ago, in a pipeline nobody has looked at since it was built.

And it is testable from the outside, without reading any code. You choose questions whose answers live in awkward places, and the awkward places tell you where the scissors went wrong.

How you test it

1. Ask questions whose answers sit in hard places. The middle of a long PDF. Inside a table. Step four of a seven-step process. The footnote. The second column. Each of these probes a different cutting mistake.

2. Ask for the chunks, not the answer. Same trick as retrieval testing: get the retrieved text itself and read it as the model got it. Half-sentences, orphaned headings and stray page numbers are all visible immediately.

3. Check whether the whole answer needs two chunks. "Compare the 2024 and 2026 refund windows" needs both. Many systems can only bring one, and the answer silently covers half the question.

4. Watch the rank. When the right chunk is present but the answer is wrong, look at where it sat. If the wrong chunk was first, that is a ranking bug, and it has a different fix from a chunking bug.

5. Verify every citation. Click it. Open the source. Does the quoted sentence exist, and does it say what the answer claims? A citation that points at the right document but the wrong sentence is worse than no citation, because it buys trust it has not earned.

6. Re-test after any content change. New heading structure, a converted PDF, a Word file exported differently — all of these change the cutting, and none of them looks like a software release. Keep the probe questions in your golden dataset so they run every time.

Try this today

Pick your longest, ugliest document — the one with tables and numbered steps. Write four questions against its awkward parts.

Four questions, four different root causes, none of which would be visible from the answers alone. That table is the most useful thing you can hand the person who owns the ingestion pipeline.

How to show you know it

A probe set for awkward places. Four or five questions aimed at tables, lists, footnotes and cross-document comparisons. Reusable on every document type you ingest.

A verdict column that names the layer. Chunking, retrieval or ranking. Three different owners, three different fixes.

One broken citation. Find a citation whose sentence does not support the claim and the room's trust in the feature recalibrates immediately.

A question about the ingestion job. "What splits these documents, and when did it last run?" Nobody asks, and the answer is frequently "a script from March that nobody owns."