The model-bump problem

A bakery has made the same loaf for years. Same recipe, same oven, same timings.

One Tuesday the flour supplier changes the mill. Nobody tells the baker, because nothing on the invoice changed. The bread comes out slightly different — a little denser, a little paler. Regulars notice before the baker does.

That is a model upgrade. Your prompt did not change. Your code did not change. The ingredient underneath changed, and behaviour moved with it.

It is a dependency upgrade with no changelog, and the only thing that catches it is a set of loaves you bake every week and compare.

What actually changes

Wording and length. Answers get shorter or longer, more or less formal. Harmless unless something downstream parses them.

Refusals. The boundary moves in both directions. Something previously refused is now answered, or the reverse — which is the toast problem from guardrails and over-refusal.

Instruction following. Newer versions often follow instructions more literally. Prompts written to shout at an older, more reluctant model can start over-triggering.

Tool use. Agents may reach for tools more or less often, take different paths, or use different numbers of steps.

Format. The shape of structured output can drift — which is the one that breaks code rather than annoying users.

Why you should care about this

Because it is the one release nobody in your team is running.

Every other change goes through review, testing and a deploy. This one arrives from outside, on someone else's schedule, and the first signal is often a customer complaint or a quietly failing parser.

And it is the clearest possible argument for the eval suite. Every other justification for building one is about catching your own mistakes. This one is about catching someone else's, and it lands with people who have never cared about evals before.

How you handle an upgrade

1. Find out what you are on, and pin it. Exact version, written down, in config rather than folklore. If you are on a floating alias, that is finding number one.

2. Know when it moves. Provider release notes, deprecation emails, someone subscribed to them. A calendar reminder beats hoping.

3. Run the same set on both versions. Your golden dataset, same cases, same scorer, several runs each. Old version and new, side by side.

4. Compare case by case, not just the totals. This is the part people skip. A score moving 87% to 89% can hide four cases that broke and six that got better. The four are your report.

5. Check the non-quality axes. Latency, cost per answer, output length, token usage, refusal rate, average step count for agents. These often move more than quality does, and they hit budgets and timeouts.

6. Re-run the security set. Injection and jailbreak results shift with model versions, in both directions — which is why every result you record should carry a version stamp.

7. Then re-tune the prompt, deliberately. Instructions written to force an older model are the most likely thing to misfire. Change one thing at a time and re-run.

8. Keep the old version available if you can. A rollback path turns a crisis into a decision.

Try this today

Find out which version you are on. If nobody knows, you have your first finding and it took four minutes.

Then write the comparison table you will need when the day comes.

The headline says the upgrade is an improvement. The detail says there is a blocker, a latency risk that will trip a timeout, and a policy question for someone else to answer. Same data, three very different conversations — and only the second one is useful.

How to show you know it

A before-and-after by case. The table above. It is the single most valuable thing a tester produces around a model upgrade.

A blocker found before release. The JSON parser case is the classic: nothing about quality, everything about shape.

The non-quality axes. Latency and cost movements are what operations and finance actually feel, and almost nobody reports them alongside quality.

A version stamp on every result you publish. It quietly proves you understand that a result without a version is not a result.