L6Go deeper4 min read

Measuring whether AI actually helped

Perceived speed-up and real speed-up diverge sharply, and there is now research showing they can point in opposite directions. How to measure honestly, so the decision to keep or drop a tool is evidence rather than mood.

Two numbers matter and they are not the same number.

Perceived speed-up is how much faster the work felt. Real speed-up is how much faster it was, measured on the clock.

There is now published research where experienced developers using AI tools believed they were meaningfully faster on a set of tasks, and the measured time went the other way. Whatever you make of any single study, the finding is a warning worth taking: enthusiasm is not evidence, in either direction.

So if you are going to claim a benefit, or defend a tool, or decide whether to keep paying for one, you need a measurement somebody could disagree with.

The terms you will hear

Baseline. The measurement before the change. Without it there is nothing to compare.
Cycle time. How long a piece of work takes end to end, not how long the typing took.
Review time. The often invisible half. Generated work moves cost from writing to reading.
Escape rate. Defects reaching production. The quality counterweight to any speed claim.
Confounder. Anything else that changed at the same time, which is usually several things.

What to measure

Four numbers, and you need all four to make a claim that holds.

Time per unit of work. Time to write and land ten test cases, or to triage one failed run, or to migrate one file. Pick something you do repeatedly.
Review time separately. The most common error in these claims is counting generation and ignoring review, which is where the cost went.
A quality signal. Escape rate, defects found later, or the break-test result from reviewing generated tests. Speed without this is meaningless.
Cost. Tokens, subscriptions, and the engineering time to keep any tooling alive.

How to run an honest comparison

Choose one repeated task. Narrow enough to time, common enough to get a sample. Test case drafting, failure triage and migration are all good candidates.
Baseline it for two weeks without changing anything. Record start and end times, and the review time separately.
Change one thing. One tool, one workflow, for the same task. Changing two makes the result uninterpretable.
Measure the same way for two weeks. Same task type, same recording, same people if you can.
Compare all four numbers. Time, review time, quality, cost. Report them together or not at all.
Name the confounders. A new joiner, a quiet fortnight, a different feature area. Naming them is what makes the number credible.
Decide, and write down why. Keep, drop or change. A recorded decision beats a rolling assumption, and it is the same discipline as an accepted known issue.

If the only evidence is that it feels faster, you have measured your enthusiasm. That is a real thing, and it is not a business case.

A worked comparison

For example, here is one small team measuring one task honestly.

did-it-help.txt

TASK      drafting test cases for a story, from criteria to merged cases
PERIOD    2 weeks baseline, 2 weeks with a model, same tester, 9 stories each

                              BASELINE        WITH A MODEL
  stories                     9               9
  median drafting time        38 min          11 min      -71 percent
  median review time          6 min           24 min      +300 percent
  median total per story      44 min          35 min      -20 percent
  cases produced per story    5.2             8.4
  cases surviving review      5.0             5.6
  break test (defect caught
    by the new cases)         5 of 5          5 of 6      slightly worse
  cost                        0               about 3 GBP for the fortnight

WHAT THE TESTER SAID BEFORE SEEING THE NUMBERS
  "it is at least three times faster"

WHAT THE NUMBERS SAY
  20 percent faster end to end, not 300 percent. drafting collapsed and
  review grew. the extra cases were mostly rejected, so producing more
  was not the benefit.

CONFOUNDERS NAMED
  the second fortnight had two smaller stories
  the tester got better at prompting during the period, so a repeat
  would probably show a slightly larger gain

DECISION
  keep it for drafting, because 20 percent is real and the cost is
  trivial. do not claim a speed-up in the sprint review. revisit in
  a quarter with a bigger sample.

The gap between "three times faster" and twenty per cent is the whole reason to measure. Twenty per cent is still worth having, and it is a number that survives being questioned.

How to show you know it

A before-and-after with four columns. Time, review, quality, cost. Very few people bring all four.
The perception gap. Recording what somebody predicted before showing them the measurement is the most persuasive artefact here.
Named confounders. It signals that you understand the limits of your own study.
A decision either way. "We kept it for drafting and dropped it for triage" reads as judgement rather than advocacy.

Questions

Is it not obvious that these tools save time?

On some tasks, plainly. On others the time moves rather than disappearing, from writing into reviewing. Which is which varies by task and by team, and that is exactly why a small measurement beats an assumption.

Two weeks is a small sample. Is it worth doing?

Yes, as long as you say it is small. A rough measurement with named limits is far better than a survey, and much better than nothing.

What if the numbers say it did not help?

Then you have saved money and learned something. Report it plainly. Being the person whose numbers are trusted matters more than being the person whose tool won.

Which single number should I report to leadership?

None on its own. Time and quality together, with cost beside them, is the smallest honest report, and it is the same principle as the metrics worth reporting.