L6Core4 min read

How LLMs work — enough to test with them

Tokens, context windows, temperature, and why the same prompt gives different answers. Not a research course: the specific mental model you need to predict where a model will be unreliable, which is the whole basis of testing one.

A large language model (LLM) predicts the next piece of text, over and over, until it has produced an answer.

That is the whole mechanism. It is not looking anything up, and it holds no database of facts. It has learned statistical patterns from an enormous amount of text, and it uses them to continue whatever you gave it.

Two consequences follow, and they are the only two you strictly need.

The same input can produce a differently worded answer each time, because the next piece of text is sampled rather than fixed.
It will produce a fluent answer whether or not it has the information, because producing text is the only thing it does.

Everything else on this layer builds on those two facts.

The six terms you will hear

Token. A chunk of text, roughly three quarters of a word. Models read and write in tokens, and limits are counted in tokens rather than characters.
Context window. The total tokens a model can consider in one call, covering your instructions, the conversation and any documents. Anything beyond it is invisible.
Prompt. Everything you send. Usually a system instruction plus the user's message.
Sampling, and temperature. How the next token is chosen from the candidates. Lower settings are more repetitive, higher ones more varied. This is the direct cause of run-to-run variation.
Training and inference. Training built the model, once. Inference is one call to it now. Your prompt never trains it.
Weights and model version. The learned parameters, frozen in a version. A new version behaves differently, which is why you pin it.

Why a tester needs this

Because these facts explain the failures you will be asked to investigate.

For example, a feature stops mentioning a policy that appeared in the answer last week. Nothing in your code changed. Either the wording sampled differently, or the document fell outside the context window, or the model version moved. Knowing those three candidates is the difference between an investigation and a shrug.

It is also why the oracle problem arrived. The model produces plausible text by construction, so plausibility proves nothing, which is the point made in test oracles.

What this means for your testing

Never assert exact wording. Assert the facts and the shape instead, as in why assertions break.
Run anything important several times. One pass tells you one sample. Report a rate.
Count the tokens when things get long. A long conversation or a large document silently pushes earlier material out of the window.
Pin the model version and record it with results. A result without a version is not comparable to anything.
Expect confident gaps. Ask a question the material cannot answer and see whether it declines or invents. That test takes ten seconds and finds real problems.
Stop asking why it said that. Nobody can answer. Ask what it was given instead, which is the layer approach from the anatomy of an LLM feature.

The model is not looking anything up. It is continuing your text in the most plausible way it can, which is why plausibility is worth nothing as evidence.

A worked example

Ten minutes with any model, and it makes the two facts concrete.

llm-basics-probe.txt

PROBE 1  the same question, five times, nothing else changed
  "How long do refunds take under our policy?"  (no documents supplied)

  run 1  "Refunds are typically processed within 14 days."
  run 2  "Most refunds are completed in 5 to 7 business days."
  run 3  "Our policy allows up to 30 days for refunds."
  run 4  "Refunds usually take about two weeks."
  run 5  "Refunds are issued within 10 business days."

  every answer is fluent. three different numbers. no document was
  supplied, so all five are invented, and none is marked as a guess.
  -> plausibility is not evidence

PROBE 2  the same question with the policy supplied
  runs 1 to 5 all state 14 days. wording differs every time:
    "within 14 days", "in 14 days", "up to two weeks (14 days)"
  -> the fact is stable, the wording is not. assert the fact.

PROBE 3  the context window
  pasted a 40 page policy, then asked about page 2
  correct answer. then pasted 12 more documents and asked again
  -> "the documents do not mention a refund window"
  nothing changed except how much came before it. the page fell out
  of the window.

PROBE 4  what it does not know
  "What is our refund window for enterprise customers?"
  (no such policy exists anywhere in what I supplied)
  4 of 5 runs invented a plausible answer. 1 said it could not find it.
  -> a 20 percent refusal rate on an unanswerable question is a finding

Probe 4 is the one to show people. The system will invent a policy four times out of five and sound the same as when it is right. That is not a bug in the model, it is the mechanism working as designed, and it is why testing these features is a different job.

How to show you know it

A five-run wording table. It settles the "just write a test that checks the text" conversation in one screenshot.
A context window demonstration. Showing an answer degrade purely because more material was added is memorable.
An invented answer with a rate. "Four in five runs invented a policy that does not exist."
Correct vocabulary. Saying token, context window and model version accurately marks you out immediately, and none of it requires maths.

Questions

Do I need to understand transformers or neural networks?

No. The six terms above cover professional work. Model internals matter if you are training or tuning one, which is a different job from testing a product built on one.

Does using it teach the model our data?

Not the model itself. Your prompt goes into one call, and providers differ on whether they retain it for other purposes. That is a contract and configuration question, and it is worth reading the answer for your own provider.

Can I make it deterministic?

You can reduce variation with settings, and you cannot rely on it. A provider update moves behaviour regardless. Assert properties instead, which is the durable answer.

Why does it never say it does not know?

Because producing text is the only action available to it. Declining has to be trained or instructed, and even then it is a tendency rather than a rule, which is why you test for it deliberately.