Anatomy of an LLM feature

An LLM feature looks like one thing from the outside. Text goes in, text comes out, and when the answer is wrong there is one obvious verdict: the AI got it wrong.

Inside, it is six things in a row. A system prompt. The context assembled for this particular run. Whatever was retrieved to go with it. The tools the model was allowed to call. The model itself. And the code that tidies the answer before anyone sees it.

Six places for a bug to live. Your first job in any AI investigation is to say which one — because "the AI is wrong" is not a bug report, and everything useful you do afterwards depends on getting this right.

Why this matters in 2026

Two years ago an LLM feature was a demo tucked behind a flag. Now it is in the product path: the support reply, the summary on the dashboard, the code review comment, the agent that files the ticket. The person asked to test it is usually you, and usually with no new tooling.

The old skill transfers. The old oracle does not. Classic testing compares an output to an expected value; here the answer is different every run, so the question changes shape. It stops being "is this string correct" and becomes "which part of this machine produced this string". That is localisation, and it is the same instinct as reading a stack trace — except nothing prints a stack trace for you.

Security made it urgent rather than merely useful. The 2026 OWASP lists for LLM and agentic applications are organised almost entirely by where a failure enters the system: retrieved content, tool calls, excessive permissions, memory. If you cannot say which layer handled the untrusted text, you cannot test any item on either list.

Then there is the unglamorous reason. Each layer has a different owner. Wrong standing instructions is usually a product decision. A missing document is the search index. A malformed tool argument is a backend engineer. Confidently wrong reasoning from correct material is a model choice, a guardrail, or an eval threshold nobody has set yet. Naming the layer is what routes the bug to someone who can fix it.

The mental model: a new hire with a folder

Picture a capable new hire in their first week. Pinned above the desk are the standing instructions. In front of them is today's request. Somebody has dropped a folder of documents on the desk. They have a phone and the keys to a filing cabinet. They have their own judgement. And the front desk retypes their letter before it goes out.

That is the whole feature. Everything on that desk is a layer, and each one fails in its own recognisable way.

The prompt: the standing instructions

The system prompt — role, rules, tone, what to refuse, what shape the answer takes. It is identical on every run, which makes it the easiest layer to test and the easiest to forget. Failure looks like the feature being wrong the same way every time: always too long, always answering something it should decline, always ignoring a rule that is genuinely in the text but ninth in a list of nine.

How you tell: the failure reproduces across inputs that have nothing else in common.

The context: what was on the desk this run

Everything assembled for this one call: the user's message, the conversation so far, retrieved documents, tool results, injected facts like today's date or the customer's plan. Most teams have no single place that shows it. Failure looks like an answer that is right about the wrong thing — last quarter's price, a different customer's order, the half of a document that survived truncation.

How you tell: print the exact context and read it as the model received it. Half of all reported hallucinations die right here.

Retrieval: the folder somebody fetched

In a RAG feature, a retriever picks documents before the model runs at all. It fails two ways: the right document was never fetched, or it was fetched and five near-duplicates crowded it out. Either way the model answered honestly from what it was handed, which is why the output is indistinguishable from invention.

How you tell: check whether the correct source is in the retrieved set before you read the answer. If it is not there, no amount of prompt work will ever fix that answer.

Tools: the phone and the keys

Anything the model can call — a search, a database query, sending the email, creating the ticket. Three ordinary failures: the wrong tool, the right tool with wrong arguments, and the right call whose result came back empty or stale. The fourth is the serious one: the call that worked exactly as designed and should never have been permitted.

How you tell: log every call with its arguments and its result, and check those independently of the final text. A good-looking answer can sit on top of a bad call.

The model: the judgement

What is left once the desk is right — the reasoning, the wording, the refusal, the decision. This layer is genuinely non-deterministic: same input, different output. It also shifts underneath you when the provider ships a new version. Failure looks like a wrong answer drawn from material that was correct, complete and sufficient.

How you tell: you have ruled out the other five. That is the whole payoff of localising — this is the only layer where "the model got it wrong" is a true sentence.

Post-processing: the front desk

Parsing, JSON extraction, truncation, markdown stripping, safety filters, retries, fallbacks. Cheap code that nobody thinks of as part of the AI, and therefore nobody tests. Failure looks like a good answer arriving broken: cut mid-sentence, citations stripped, a refusal rewritten into a friendly non-answer, a valid response dropped because a parser expected different formatting.

How you tell: compare the raw response against what the user actually saw. Those two differ far more often than teams expect.

What changes when it is an agent

Everything above describes one pass. An agent runs the same six layers in a loop: it answers, calls a tool, reads the result, and goes round again — and each turn's output becomes part of the next turn's context.

Two things follow, and both are testing problems rather than engineering ones.

The first is that errors compound instead of showing up. A slightly wrong retrieval on turn two is quietly treated as fact for the next eleven turns, and the answer you eventually see has no visible connection to the mistake that caused it. Localising means reading a sequence, not a snapshot: which turn first contained the wrong thing, and what the agent did with it afterwards.

The second is that two new questions become worth asking on every run. Did it stop? — agents get stuck retrying the same failing call until the budget runs out, and the loop is a layer of its own. And what could it have done at its worst? — the same six layers now have credentials, so a tool call that fires when it should not is no longer a wrong answer but a wrong action.

Both questions have their own nodes further along this layer — trajectory testing, tool misuse and loop detection, blast radius. All of them are downstream of this one. An agent trace is just the single-pass trace repeated, so if you cannot read one turn cleanly, fourteen of them will tell you nothing at all. Work through the map in order and this is the node the rest of L7 stands on.

How to actually learn it

There is one hour of reading worth doing before you touch anything, and it is not a course.

Read one provider's documentation on messages and tool use, end to end. Anthropic's tool-use overview is the shortest complete version: what a request carries, how a tool call comes back, what you have to send next. You are reading for the request and response shape, not for tips.

Then read Building effective agents for the layers that sit above a single call — the loop, the planning, the memory. It is short and it is the vocabulary the rest of this roadmap layer assumes.

Then stop reading and instrument something, which is the next section. Come back to the OWASP GenAI lists afterwards; once you can name the layers they read as a test charter rather than a glossary.

Four things to skip. Prompt-engineering certificate courses — they teach you a prompt, not a system. Building a transformer from scratch, which is genuinely interesting and will not help you here. Tokenizer internals, until a context limit actually bites you. And any vendor webinar whose diagram is their own product.

What you cannot get from reading is the trace. An hour spent reading real inputs and outputs beats a day of theory, for the same reason that reading your run history beats theorising about flaky tests.

Build this: a trace you can read

The exercise is one file. Take an LLM feature — your product's, or forty lines of your own — and make it print every layer for a single run. Then break each layer on purpose and watch what the output does.

Run it once and read the trace file rather than the console. Then run four experiments, one at a time.

1. Break retrieval. Delete the note that answers the question and leave everything else alone. Watch what the answer becomes. Most people are surprised: it looks exactly like a hallucination, and this is the shape of a retrieval bug in your own feature.
2. Break the context. Truncate it mid-sentence, as a real context window would. Ask the same question again.
3. Break the prompt. Bury the "not in the notes" rule at the bottom of a list of nine rules and see whether it survives.
4. Isolate the model. Ask the same question five times with everything else fixed, and diff the answers. Whatever varies is the model layer. Whatever is stable is every other layer.
5. Add a tool. Give it one function — a lookup, a fake refund call — and log the name and arguments it gets called with. Then ask a question that does not need it. Watching a model reach for a tool it should have left alone, on material that gave it no reason to, is the fastest education in this layer that exists.

If you already own an LLM feature at work, run the exercise against that instead of the toy. The deliverable is the same either way: one trace file per run that a colleague could read without you sitting next to them. It is the AI-era equivalent of a test record a release can rest on.

How to prove you know it

Three artefacts, in ascending order of how much they are worth.

A bug report that names the layer. This is the cheapest thing on the list and the one that changes how people treat you:

Five lines, and every one of them answers a question the person fixing it would otherwise have to ask you. The evidence line proves the document was absent rather than ignored. The "not the model" line is the part people skip, and it is the part that stops the bug bouncing between two teams for a week: you have already run the experiment that rules out the layer everyone blames first.

The harness in a public repository. Forty lines of code and five committed traces, with anything sensitive stripped. It demonstrates something no certificate does — that you have looked at real inputs and outputs and know what you are looking for. The same principle as keeping charters and notes from an exploratory session: the record is the proof.

An interview answer that starts in the right place. Asked how you would test an AI feature, do not start with evals. Start here: "First I would want one full trace — prompt, context, retrieved documents, tool calls, raw response. Until I can tell those apart, every bug I file will just say the AI is wrong."

Then name what you would test at each layer, and what you would measure. That answer sounds like someone who has actually done it, because it is what doing it feels like — the same way a story turned into cases someone else can run reads differently from a story someone only talked about.

It also sets you up for the layers either side of this one. Reviewing what a model wrote is its own skill — see generate freely, merge carefully — and choosing which number to report is the whole of QA metrics worth reporting.