When Not to Automate a Test (and What to Do Instead)

The cost that isn't on the invoice

Automating a test has a price you get billed for — the time to write it — and a price you don't: the time to keep it alive. That second cost is invisible when you make the decision and very real by year two.

For most tests, it's still worth it. This isn't an argument against automation; a good automated suite is one of the best investments a team makes. It's an argument about the exceptions — because there's a whole class of tests where the hidden upkeep costs more than the manual run it was supposed to replace.

Naming that class precisely is more useful than one more nudge to automate more. "Automate everything" is the easy advice, and it's the advice a vendor is supposed to give. This is the other half: when not to automate a test, and what to do with it instead.

The four kinds of test not to automate

Four categories come up again and again. If a test lands in one of them, automating it usually costs you more than it returns.

Run-once tests. A check you'll perform a handful of times in its whole life — a one-off data migration, a launch-day verification. The manual run is cheap and finite. Automating it means paying the authoring cost, plus upkeep, for something you were never going to run enough to pay it back.

Fast-changing surfaces. A screen or flow that gets redesigned every week or two. Automate it and you're not maintaining a test occasionally — you're rewriting it almost every time it runs. You'd spend more time fixing the test than it spends protecting you.

Quality judgements. "Does this layout look right? Does the tone feel off? Is this the kind of result a user would trust?" A machine can check that an element exists or a value matches. It cannot judge whether something looks or feels correct. Those checks need eyes, and forcing them into automation gives you a green tick that misses the actual question.

Things cheaper to verify by looking. Some checks a person answers in three seconds with a glance. Wrapping that in a script — writing it, maintaining it, debugging it when it flakes — costs far more than the glance ever will, for no extra safety.

Authoring is the small number

Here's where the usual cost comparison goes wrong. People weigh "time to write the automated test" against "time to run it by hand," decide the automation pays off after a few runs, and stop there. That comparison leaves out the expensive part.

Authoring is a one-time cost, and it's the small one. Maintenance is the recurring cost, and over a test's life it's usually the large one. Every UI change that breaks a selector, every flaky failure someone has to debug, every framework upgrade — that's the real bill, and it arrives again and again for as long as the test exists.

So the honest question was never "is automating cheaper than running this once?" It's "will this test run often enough, on stable enough ground, to earn years of upkeep?" A test that runs constantly on a settled surface easily clears that bar. A run-once or fast-changing one doesn't come close.

Exploratory testing is work automation can't do

One category deserves to be stated without hedging, because it's not "manual testing we haven't gotten around to automating." It's a different job that automation structurally cannot do.

Automation checks what you told it to check. It verifies expectations you already had and wrote down. Exploratory testing is a person moving through the product, following hunches, trying the thing nobody scripted, and noticing the problem no test case anticipated. A machine can confirm a suspicion you've already encoded; it cannot have a new one.

That means exploratory testing finds a class of bug your automated suite never will — the ones outside every expectation you thought to write. Don't try to automate it away, and don't treat it as a backlog of not-yet-automated cases. Protect time for it on purpose. It's the part of testing that pays off precisely because it isn't scripted.

The automation-percentage trap

Somewhere a target gets set: "we should have 80% of our tests automated." It sounds like progress. It quietly makes things worse.

Once the percentage is the goal, people automate to hit it — including the run-once tests and the churning surfaces that should never have been automated. The number goes up; the suite gets slower, flakier, and less trusted. You've optimised for a metric that measures activity, not value.

A smaller automated suite of well-chosen tests beats a big one padded with tests that shouldn't exist. Measure whether your automation catches real problems cheaply, not what fraction of some list is green. The percentage is a vanity number, and chasing it is how good suites go bad.

What to do with the ones you don't automate

Deciding not to automate a test is not deciding to skip it. That's the mistake — "we won't automate this" quietly becomes "we'll wing it," and the check stops happening at all.

The right move is to document the case properly and run it deliberately. Write the steps and a specific expected result, the same as any good case, so anyone can run it and reach the same verdict. Then run it on purpose — on a schedule, or at the trigger that matters: before a release, or whenever that area of the product changes.

A well-documented case that runs when it should is worth more than a flaky automated one that runs constantly and gets ignored. This is where the documented record earns its keep: the case exists, it's reviewed, and it's traceable to a requirement, whether a machine runs it or a person does. Tesbo manages and drafts those cases with a person approving each one; it doesn't run them for you — the manual run, like the automated one, stays yours.

The honest summary

Automation is worth it for tests that run often, sit on stable ground, and check something a machine can actually judge. For the run-once, the fast-changing, the judgement calls, and the glance-and-done, the upkeep outweighs the payback.

Name those tests, don't automate them, document them, and run them on purpose. That isn't being anti-automation. It's what keeps the automated suite you do build small enough to trust — which is the only kind worth having.