What to Automate First in Testing: A Scoring Method

Automation order shouldn't be set by whoever's loudest

"What should we automate next?" usually gets answered one of two ways: by whoever complained most in the last standup, or by whatever broke last week. Both feel responsive. Both are just intuition, and intuition is a poor prioritiser — it over-weights the recent, the loud, and the memorable, and quietly ignores the boring high-value stuff nobody's shouting about.

There's a better way, and it fits on one page. Score each candidate on a few factors — most of which you can pull from data you already keep — add them up, and automate down the ranked list. It isn't perfect science, and it isn't meant to be. It's a way to make the decision explicit and repeatable instead of a vibe.

The real value isn't agreeing that this is sensible. It's running it on your own suite this week, on real candidates, and letting the ranking tell you something your gut wouldn't. So here's the method, worked all the way through.

The four factors

Score each candidate test on four things. The first three make a test more worth automating; the fourth makes it less, so it counts against.

• Execution frequency. How often you run, or would run, this check. Something you'd run on every deploy pays back automation many times over; something you run once a quarter barely does.
• Cost of the bug escaping. How bad it is if this breaks in production. A broken login or checkout is a different category of pain from a misaligned banner. The worse the escape, the more you want a machine guarding it.
• Stability of the surface. How settled the thing under test is. A stable API or a long-unchanged flow stays automated cheaply. A screen redesigned every sprint means rewriting the test every sprint.
• Cost to automate. How much effort to build it. This one counts against the score, because an expensive test has to clear a higher bar to be worth it.

A simple, transparent way to combine them: score each factor 1 to 5, then compute frequency + escape cost + stability − cost to automate. Higher means automate sooner. You can weight the factors differently for your context — these are your dials, not laws.

A worked example, with the arithmetic

Take five candidates and score each factor from 1 to 5, where 5 strongly favours automating. Remember cost to automate is subtracted. The score is frequency + escape cost + stability − cost to automate.

• Login smoke check — runs on every deploy (frequency 5), a broken login is a disaster (escape 5), the login form barely changes (stability 5), trivial to script (cost 1). Score: 5 + 5 + 5 − 1 = 14.
• Checkout with a saved card — run often (4), a lost sale is expensive (escape 5), the flow is fairly settled (stability 4), moderate effort (cost 2). Score: 4 + 5 + 4 − 2 = 11.
• Password reset — run regularly (3), a real problem if broken (escape 4), stable (4), moderate effort (cost 2). Score: 3 + 4 + 4 − 2 = 9.
• Admin bulk export — rarely used (frequency 1), moderate impact (escape 3), stable (4), fiddly to build (cost 3). Score: 1 + 3 + 4 − 3 = 5.
• New promo banner — shown to everyone (frequency 3), low impact if wrong (escape 2), redesigned almost weekly (stability 1), fiddly (cost 3). Score: 3 + 2 + 1 − 3 = 3.

Ranked, you automate in this order: login (14), checkout (11), password reset (9), admin export (5), promo banner (3).

The interesting one is the promo banner. It's the most visible thing on the list — exactly what a stakeholder would point at and say "automate that." It ranks last, because its surface churns weekly, and automating it buys a permanent maintenance bill for a low-stakes check. The score just protected you from the loud, obvious, wrong choice. That's the whole point of writing it down.

Where the inputs come from

The method is only as honest as its inputs, so be clear about which are data and which are estimates.

• • Execution frequency comes from run history — how often this check has actually run, or how often you'd run it. That's data you already have, if you kept it.
  • Cost of the bug escaping comes partly from defect records — has this area caused incidents before, and how bad were they — and partly from judgement about blast radius.
  • Stability of the surface comes from change history: how often has this screen or API actually changed in the last few months?
  • Cost to automate is the one you have to estimate, because it hasn't happened yet. So estimate it honestly and conservatively, and don't dress a guess up as a measurement. The goal isn't fake precision; it's to stop pretending you have none. Three factors grounded in data and one honest estimate beats four gut feelings every time.

The tie-break: stability wins

Two candidates will often land on the same score. When they do, break the tie with stability of the surface, and automate the steadier one first.

The reason is maintenance. A test on a churning surface costs you upkeep for as long as it exists — every redesign breaks it, and you pay again. A test on a stable surface, once automated, keeps paying back with almost no further work. Between two equal-value candidates, the stable one gives you the value sooner and the bill later, or never. Stability isn't just a scoring factor; it's the thing that decides whether an automated test ends up an asset or a liability over its life.

Re-score, or the ranking rots

A ranking you compute once is wrong within a quarter. This isn't a flaw in the method; it's just true of the inputs. Frequencies shift as the product changes. A churning surface settles, or a stable one gets rewritten. New features arrive with their own candidates, and an incident can turn a "moderate escape cost" into a "never again."

So treat the ranking as a living thing, not a monument. Re-score on a cadence — each quarter is a reasonable default — or whenever the product changes materially. This is the same discipline as regression suite maintenance: prioritising and pruning are two sides of keeping a suite honest, and both rot the moment you stop doing them.

A prioritisation you set once and never revisit isn't a data-driven ranking anymore. It's last quarter's intuition, wearing a spreadsheet's authority — which is arguably worse than an honest guess, because it looks objective.

Honest limits, and where a record helps

Be clear about what this is. It's not validated science, and the specific weights aren't derived from anyone's data — they're a structure for making a judgement call explicit, repeatable, and handable to someone else. Adjust the factors to your context; the value is that it beats "whoever's loudest," not that it's exact.

Where a tool helps is upstream of the score. The frequency comes from run history, and the escape cost leans on defect and case records — so keeping a clear, queryable record is what makes the inputs real rather than invented. Tesbo keeps that case and run record; it doesn't run this scoring for you today, and it doesn't run your tests — the framework and CI stay yours. The record feeds the decision; the decision is still yours to make.

Run it this week

The whole method fits on a page. List your top twenty automation candidates, score each on the four factors, rank them, and automate down the list. Re-score next quarter. That's it.

It won't be perfect, and it doesn't need to be. It needs to be better than whoever spoke last and more repeatable than a hunch — and because it's written down, you can hand it to a colleague and get the same answer, which is the one thing intuition can never promise. Prioritising regression tests stops being an argument and becomes a calculation you can check.