AI in Software Testing: What Actually Works

Sold twice, delivered once

AI in software testing has been sold to you before. The record-and-playback wave promised testing without coding. The "autonomous testing" wave promised testing without testers. The category has been sold at least twice and delivered maybe once, and every new pitch arrives with the last one's broken promises still fresh.

So let's not do promises. Let's sort the claims into three piles instead: the ones that genuinely work today, the ones that work only under conditions nobody mentions in the demo, and the ones that are marketing.

There's a single thread that decides which pile a claim lands in, and it's worth stating up front. Where AI in testing works, a person still accepts the output. Where the pitch is "no human needed," you're looking at the marketing pile. AI changed what gets typed. It did not change who's accountable for what ships.

Pile one: genuinely working

Start with what actually works, because it's real and it's useful. Models are reliably good at a specific shape of task: high volume, lots of permutations, consistent format, and the tedious cases a tired person skips.

Drafting test cases from a user story works. So does laying out every combination — each payment method across each country across each account type — without getting bored at row twelve. So does keeping hundreds of cases in an identical shape, and producing the boring negative and boundary cases (empty field, expired card, maximum length) that are easy to leave out at the end of the day. Summarising a wall of failure logs into a readable paragraph works too.

Notice what these have in common. Each is a bounded, reviewable task where a human reads the result and accepts or rejects it. The model does the legwork; the person keeps the judgement. That combination is what makes pile one trustworthy — not the cleverness of the model, but the fact that nothing it produces is trusted until someone says so.

Pile two: works, under conditions nobody states

The middle pile is the tricky one, because the claims are true — with a caveat the demo skips. The clearest example is self-healing test automation.

The pitch: when the UI changes and a locator breaks, the test heals itself, re-points to the new element, and keeps passing. And it does. The unstated condition is what matters. A test that quietly re-points itself can keep going green while no longer checking what it was written to check. Healing the locator is not the same as confirming the behaviour is still correct — and a test that can't fail isn't testing anything, it's just reassuring you.

So self-healing belongs in pile two, not pile three: it's a genuine capability with a real caveat. It moves up into pile one the moment a person confirms each heal was correct, and slides toward useless the moment healing runs unattended. Same feature, two very different piles, decided entirely by whether a human is still in the loop.

Pile three: marketing

The third pile is the one that keeps the category's reputation where it is. It's the pitch that AI will autonomously find your bugs, test the whole application end to end with no one steering, and let you retire the QA team.

Here's why it's marketing, not product. A model produces output that is plausible — fluent, well-structured, confident — whether or not it's correct. The thing that made pile one work was a person accepting or rejecting that output. "Autonomous QA" removes exactly that person. What you're left with is a system that generates plausible results and ships them unexamined, which is not testing; it's automated guessing with good presentation.

You can spot the pile-three pitch by what it asks you to give up. It asks you to stop reviewing. Anything that positions the human as the bottleneck to eliminate, rather than the check that makes the output usable, is selling the pile that never delivers.

The failure that decides everything

Underneath all three piles is one failure mode, and understanding it explains the whole sorting. A model's characteristic mistake is not random garbage. It's confident, well-formatted output describing behaviour the product doesn't actually have.

It writes a clean test case for a button that was never built. It heals a locator to the wrong element and reports success. It summarises a failure with a cause that sounds right and isn't. In every case the wrong answer looks exactly like a right one — same tone, same structure, same confidence. That's what makes it dangerous.

And it's why a machine can't police itself here. The model has no reliable way to flag which of its outputs are the invented ones; they don't look different from the inside. Only a person who knows the product can tell the plausible-and-true from the plausible-and-false. The error mode is human-shaped, so the check has to be human too.

Review is the design, not a safety feature

Once you accept that the failure is plausible-and-wrong, one conclusion follows: review can't be a feature bolted on at the end. It has to be the load-bearing part of the design.

The shape that works is the one pile one already showed us. Generate freely, because generation is cheap and reversible. Then route everything through a review queue where a person accepts each item before it enters the trusted record. Nothing joins the suite on the model's say-so. Generation becomes the commodity; the moment of human acceptance becomes the actual product.

This is the design Tesbo is built around. It drafts and manages the cases that populate the record, and a person approves each one before it's trusted — nothing enters unreviewed. It doesn't run, schedule, or execute your tests; the framework and CI stay yours. The point isn't that the AI is careful. It's that the AI isn't trusted to be.

What doesn't change

For all the movement, two parts of the job don't move at all, and they're the two that mattered most.

The first is deciding what's worth testing. A model will draft a hundred cases; it can't tell you which ten protect the thing that would actually hurt if it broke. Prioritisation by risk is judgement, and it stays with a person.

The second is harder and more important: knowing when the requirement itself is wrong. A model will write a flawless test for a mistaken spec, because it has no way to know the spec is mistaken. Noticing that the thing you were asked to build is the wrong thing is not a task you can hand to a system that only knows what it was told.

This is also the honest answer to "will AI replace QA engineers." The part of the job that was always transcription — typing up cases, formatting, filling in permutations — is genuinely being automated, and pretending otherwise helps no one.

But the part that was judgement — what to test, whether the requirement is right, what a real user would actually do — is not. The role shifts toward the judgement and away from the typing. That's a real change, and for people who defined their value by the typing, an uncomfortable one.

Whose keys, whose data

There's a question that belongs at the front of any AI-in-testing decision, not the appendix: whose model, whose key, whose data, and what leaves the network. Your test cases describe exactly how your product behaves, and generating them means sending that description to a model. Where it goes is a first-order concern.

The honest architecture is plain. You bring your own AI provider key, so you choose the model and pay for usage directly. Your data stays inside your own AI account, and the tool vendor doesn't train anything on it or ship a model of its own.

With Tesbo, that's the arrangement on every plan — your keys, your model, your data — and self-hosting tightens the boundary further. The details of what a security reviewer should ask are their own discussion, but the principle is simple: the account your test data passes through should be yours.

A quick way to place any claim

You don't need the three piles memorised to sort a new claim. Three questions do it.

Is the task bounded and reviewable? Can a person actually read the output and judge it — a case, a permutation set, a summary? If it's small enough to check, it's a candidate for pile one. If the output is too large or too fast for anyone to review, be suspicious.

Does a human accept the result before it's trusted? Find the exact point where a person says "yes, this can be used." If there's a clear accept step, the claim can work. If the answer is "the AI handles it end to end," you've found pile three.

What happens when it's wrong and nobody's watching? For anything the tool does unattended, ask what a plausible-and-wrong output costs — a self-healed test that stops testing, a case for a feature that doesn't exist. If a silent error ships, the task needed a human it didn't have.

Run any pitch through those three and it sorts itself. The impressive demo isn't the evidence. The answer to "where's the human, and what does a quiet mistake cost?" is.

The one test for any AI testing tool

Put the piles back together and you have a single question to carry into any evaluation of AI testing tools: where does a person accept the output, and what happens to your data on the way?

A tool whose answer is "the AI just handles it, trust it" is selling pile three, however good the demo looks. A tool built around human acceptance, with your own keys and your own data, is selling pile one — the pile that actually delivers.

AI genuinely changed testing: it took over the typing, the permutations, the first drafts. It did not take over the judgement, and it did not move the accountability. Someone still has to accept what ships — so make sure your tools are built like they know that.