Test Automation Strategy: Three Questions, Not a Tool

A strategy is not a tool decision

Ask most teams for their test automation strategy and you'll get the name of a tool. "We're a Playwright shop." "We use Cypress." That's not a strategy. That's a purchase.

A tool tells you how you'll write automated tests. A strategy tells you which tests are worth writing, who keeps them alive, and which ones you throw away. Those are the decisions that actually determine whether your suite helps you ship or slowly becomes a thing everyone ignores. The tool is a detail you can only choose sensibly once you've made them.

Here's the whole idea in one line: a test automation strategy is a standing answer to three questions — what earns automation, who maintains it, and what gets deleted. A team that can answer those three needs far less ceremony than the consultancies sell. A team that can't will not be saved by any tool, however good.

This is the pillar for that argument. We'll take the three questions in turn, deal honestly with the famous pyramid, and end with where documented test cases fit — because automation with no case behind it can't be traced to anything.

The three questions

Everything in a real strategy comes back to three questions. Answer them once, write the answers down, and most day-to-day automation decisions stop being arguments.

What earns automation? Not everything should be automated, and "automate everything" is a slogan, not a plan. You need a rule for which tests are worth the cost.

Who maintains it? Every automated test is a standing commitment. Something has to break, get noticed, and get fixed — by a named person, on work time. If nobody owns that, the suite rots.

What gets deleted? Suites grow forever unless someone is allowed to remove tests. A suite that only grows gets slower, noisier, and less trusted until people stop reading it at all.

Notice what's not on the list: the tool. Tool choice is downstream of all three. You can't sensibly pick a framework until you know what you're automating, who's maintaining it, and how ruthlessly you'll prune. Pick the tool first and you've answered the easy question while leaving the hard ones open.

Question 1: what earns automation

The honest selection rule is a combination of three factors. No single one decides it; you weigh them together.

How often the test runs. A check you run on every deploy pays back automation quickly — you're buying back that manual effort many times over. A check you run once a quarter barely does. Frequency is the clearest signal that a test is worth the up-front cost.

What the bug would cost if it escaped. Some failures are cosmetic. Others lose money, corrupt data, or lock users out. A rare check on the payment path can be worth automating even if you run it seldom, precisely because the cost of missing the bug is enormous. Frequency tells you how often automation pays; escape cost tells you how badly you need the safety net.

How stable the thing under test is. This is the factor teams forget, and it's the one that quietly wrecks suites. Automating a screen that gets redesigned every month means signing up to repair that test every month. A stable surface — an API contract, a settled workflow — earns automation. A surface still churning through redesigns does not yet, no matter how important it is.

Put them together and the strongest candidates are obvious: run often, expensive to get wrong, and stable enough to stay put. Those get automated first. When two candidates look equal, let stability break the tie — a valuable test on shifting ground will cost you more in upkeep than it ever returns.

The pyramid: what it got right, and where shapes replaced thinking

You can't write about automation strategy without the test automation pyramid, so let's be fair to it.

The pyramid says: have lots of fast, cheap, low-level tests (unit tests) at the base, fewer integration tests in the middle, and only a thin layer of slow, expensive end-to-end tests at the top. What it got right is durable: fast, stable checks should outnumber slow, brittle ones, and pushing a check down to the lowest level that can catch the bug is usually cheaper to run and less flaky.

Here's the honest objection. Somewhere along the way, the shape became the point. Teams now argue about whether they should follow the pyramid, the "testing trophy," or the "honeycomb" — as if picking the right silhouette were the strategy. It isn't. The shape is a heuristic about cost, not a quota to fill.

A payments API and a marketing site do not want the same mix, and neither should contort itself to match a diagram drawn for someone else's product. Use the pyramid as a nudge — prefer the cheap, stable check; be suspicious of a suite that's all slow end-to-end tests — and then think about your own system instead of your silhouette.

Question 2: who maintains it

Here is the cost nobody puts on the invoice. Writing an automated test is the cheap, one-time part. Keeping it alive is the recurring bill, and it arrives every month whether you budgeted for it or not.

An automated test is a standing liability as much as an asset. It breaks when the UI changes. It goes flaky and needs debugging. It fails for a reason that turns out to be the test's fault, not the product's, and someone loses an afternoon proving that. Multiply by a few thousand tests and maintenance is not a footnote — it's a significant, ongoing slice of your team's week.

Strategies that ignore this look great in year one and collapse in year two. The suite that was going to save everyone time is now a part-time job nobody signed up for, and because no one owns it, it slides. Tests that break stay broken. Red becomes normal. The signal you paid for stops meaning anything.

So maintenance has to be a first-class line in the plan, with two parts. First, an owner: a named person or team responsible for the suite's health, not "whoever's free." Second, budgeted time: regular, expected hours for keeping tests green. Regression suite maintenance isn't a failure state you fell into; it's the price of having a suite at all, and a real strategy prices it in from the start.

Question 3: what gets deleted

Selection decides what goes in. Deletion decides what comes out — and it's the half almost everyone skips.

Left alone, a suite only grows. Every sprint adds tests; nothing removes them. Over a couple of years you get a suite that takes an age to run, throws off failures nobody investigates, and contains tests for features that no longer exist. The slow, quiet result is the worst outcome in testing: people stop trusting the suite as a whole and start ignoring red, because they've learned that half of it is noise.

A deletion policy prevents that. It's permission — and an expectation — to remove tests that no longer earn their place:

• Duplicates that check what another test already covers.
• Dead tests for features that were removed or rewritten.
• Permanently flaky tests nobody trusts and nobody's going to fix.

That last one is worth saying plainly: a test nobody trusts is worse than no test. It costs maintenance, adds noise, and trains the team to ignore failures. Deleting it is not giving up — it's removing something that was actively lying to you. A suite you prune stays fast, honest, and trusted. A suite that only grows becomes wallpaper.

Where documented cases fit

There's a piece underneath all of this that automation strategy discussions tend to skip: the documented test case.

An automated test is an implementation. It runs some steps and checks a result. But on its own, in a repo, it can't tell you what requirement it was protecting or why it exists. Delete the intent and you're left with code that fails one day, and an engineer reverse-engineering "what was this even supposed to prove?" from the assertions.

The fix is to keep the documented case as the source of truth and treat the automated test as one way of executing it. The case says what the behaviour should be and which requirement it ties to; the automation implements that. Now a failing test points back to a specific case and expected result, and every check in your suite is traceable to something a human decided mattered.

This is the job Tesbo is built for, and it's worth being precise about the boundary. Tesbo helps you manage and draft those documented cases, with a person approving each one before it enters the trusted record. It does not run, schedule, or execute your automated tests — the framework, the runner, and the CI pipeline stay entirely yours. It keeps the record the automation is built from; it doesn't replace the runner that executes it.

Maturity, honestly

You'll see "test automation maturity model" charts that rank teams from ad-hoc up to fully automated, and it's tempting to treat climbing them as the goal. Be careful. Automating more is not the same as maturing.

A mature automation practice isn't the one with the highest automation percentage. It's the one that can answer the three questions without a meeting: it knows what earns automation, it budgets maintenance, and it deletes without drama. A team that prunes and traces its cases is more mature than one that automated everything and now can't tell which failures matter. Chase the three answers, not a level on someone's ladder.

The strategy fits on one page

Strip away the ceremony and a real test automation strategy is short enough to fit on a single page:

• What earns automation: run frequency, cost of the bug escaping, and stability of the surface — weighed together, with stability breaking ties.
• Who maintains it: a named owner and budgeted time, because upkeep is the real recurring cost.
• What gets deleted: duplicates, dead tests, and permanently flaky ones — pruned on purpose so the suite stays trusted.
• What ties it together: every automated test traces back to a documented case, so a failure means something and a requirement is never left unguarded.

Answer those, and the tool becomes what it always should have been — an implementation detail. The strategy was never the framework. It was the thinking the framework can't do for you.