Offline evals vs online evals

Passing your driving test does not make you a good driver. It makes you a person who drove one route, once, with an examiner watching.

What actually tells you how you drive is the dashcam on your commute for a month.

You need both. The test stops you getting on the road while you still mount kerbs. The dashcam shows you the junction you misjudge every Tuesday in the rain.

Offline and online evals are exactly this pair, and most teams only ever build the first one.

Two different questions

Offline asks: did we break anything we already knew about? Fixed inputs, an answer key, a score you can compare with last week. It runs in the pipeline. It is repeatable, cheap and completely blind to anything not on the list.

Online asks: what is actually happening out there? Real users, real questions, no answer key. You cannot mark it the way you mark a test paper, so you look for signals instead.

The trap is treating the offline score as the health of the feature. It is the health of your list.

The four words you will hear

Offline eval. Fixed dataset, run before release.

Online eval. Measurement of live traffic after release.

Proxy signal. Something you can count that stands in for quality — retries, escalations, thumbs-down.

Sampling. Reading a small random handful of real conversations by hand, every week. Unglamorous, and the single highest-value online habit there is.

Why you should care about this

Because the questions real people ask are not the questions you wrote down.

A team I would describe as careful had ninety offline cases and a healthy score. Their online data showed a quarter of real conversations were in a language nobody had tested, because the product had launched in a new market and nobody told QA. Offline was green the entire time. It was answering the question it was given.

There is also a career argument. Offline evals are becoming standard. Reading production behaviour and turning it into new test cases is still rare, and it is the closest thing on this layer to classic testing skill — reading run history to find the real pattern, applied to conversations.

How you set up both halves

Offline, in the pipeline. Your golden dataset, the rubric, a threshold, run on every prompt change and every model bump. Blocking on the safety cases, reporting on the rest. That is the practice track, and the eval topic covers it end to end.

Online, in three steps that cost almost nothing.

1. Count the proxies. Escalations, retries, refusals, empty retrievals — an empty pile is a lookup failure, not a model failure. If your team has any logging at all, these are usually one query away. Put them on a chart with a date axis and look at it weekly.

2. Read ten conversations a week, by hand. Random ten, not the ten that were complained about. This is the habit that finds the things no metric names — the tone that reads as sarcastic, the answer that is technically right and useless, the question type nobody anticipated.

3. Feed every surprise back into offline. New case, straight onto the route. This is the loop that makes the whole system get better instead of just getting measured.

Two practical warnings. Watch what you log — real conversations contain real customer data, and "we needed it for testing" is not a lawful basis. And do not let an online number become a target on its own: refusal rate can be driven to zero by a feature that cheerfully answers everything, including the things it should decline.

Try this today

Ask for one number: how many AI conversations last week ended with a human taking over. Then ask for ten random transcripts.

Read the ten. Mark each one good, poor, or "would have been better as a plain search box". Then write down every question type you did not have in your offline set.

Four new case types out of ten conversations, and an escalation rate that moved five points while every offline test stayed green. That is a normal first sample, and it is why this habit is worth twenty-five minutes a week forever.

How to show you know it

A weekly sample with notes. Ten conversations, three verdicts, and the list of question types you were missing. Cheap to produce, and almost nobody does it.

A chart with a date on it. Escalations or retries over eight weeks, with a marker where a release went out. It converts "users seem happier" into something a room can discuss.

Offline cases with an online origin. "These six cases came from production samples in August." That single line proves the loop is running.

A privacy sentence. "We sample from redacted transcripts and keep them for thirty days." Being the person who asked before reading customer conversations is worth more than any metric here.