Chaos Testing Isn't About Breaking Things. It's About Being Right

An SDET on a payments team spends a Tuesday afternoon watching a chat thread fill up with theories about what happened during last month's outage. The database failover took 40 seconds instead of the 4 everyone assumed, and nobody had ever actually tested it. Someone eventually says the phrase "we should do some chaos testing," and the thread goes quiet, because half the room pictures something reckless: pulling a plug in production and hoping for the best. That reputation is the reason a lot of teams never start. This post is for the SDET who has to explain to a skeptical lead why chaos testing is closer to a disciplined test case than a stunt, and what has to be true before anyone runs one against anything real.

The part everyone remembers is the easy part

Ask ten engineers what chaos testing is and most will describe breaking something on purpose: killing a server, cutting a network link, throttling a database. That description is not wrong, but it describes the tooling, not the discipline. Injecting a failure takes one command. Anyone can do it. The actual skill, the part that separates a chaos experiment from vandalism, is what happens before you touch anything: you write down what you expect the system to do, in specific terms, before you find out.

That correction matters because it changes what "doing chaos testing" actually requires. It is not a tool you install. It is a prediction you are willing to be wrong about, followed by a check on whether you were.

A chaos experiment is a test case with an unusual precondition

Here is the reframe that makes chaos testing click for anyone who already writes test cases: it has the same four parts your ordinary test cases have, just applied to a system instead of a screen.

• Steady state: what "normal" looks like right now, in numbers. Checkout completes in under 2 seconds for 99% of requests. Order volume holds around 1,200 per minute.
• Hypothesis: what you expect to happen when you break one specific thing. "If the primary database node dies, failover completes within 10 seconds and checkout error rate stays under 1%."
• Injected failure: the actual precondition. Kill the primary node. Throttle the payment gateway to 500ms latency. Drop the connection between two services for 90 seconds.
• Verification: did steady state hold, or didn't it? Not a vibe check. The same metrics you used to define steady state, checked again.

A regular test case says: given this input, expect this output. A chaos experiment says: given this failure, expect this behavior. Same shape, same rigor, same requirement that someone wrote the expected result down before running it. The team that treats it as a documented case instead of an ad hoc drill is the team that can tell you, six months later, exactly what they tested and what they found.

The blast radius question has to be answered out loud

The reason chaos testing earns its reckless reputation is that some of it really has been reckless: engineers running failure injection against production with no scoping, no rollback plan, and no one else in the room. That is a legitimate risk, and a post about chaos testing that skips past it does readers a disservice.

Before any experiment against a system real customers touch, three questions need real answers, not assumed ones:

• Who has to sign off before this runs? On a payments team that is usually more than one person, and it should include whoever owns the on call rotation for that service.
• What is the smallest blast radius that still tells you something? Killing one node in a 12 node cluster during a low traffic window at 3am on a Tuesday teaches you almost as much as killing it during peak, with far less exposure.
• What is the abort condition, and who is watching for it? If checkout errors cross 5%, someone specific is watching that dashboard and someone specific has the authority to stop the experiment immediately.

Most teams start chaos experiments in staging entirely, or in production against internal, non customer facing services, and only move to customer facing systems once the process itself is trusted. That progression is not caution for its own sake. It is how you build the muscle for answering the blast radius questions correctly before the stakes are high.

What survives the experiment matters more than the experiment itself

Here is the failure mode that quietly wastes the most effort: a team runs a genuinely well designed chaos experiment, learns something real (the failover takes 40 seconds, not 4), fixes the immediate issue, and never writes any of it down as a repeatable case. Eighteen months later, after two rounds of turnover, a new database migration reintroduces the same slow failover, and nobody remembers it was ever a known risk. The experiment happened. The finding evaporated.

The fix is unglamorous: the hypothesis and the result get written up as a test case the suite actually keeps, not a slide in a postmortem deck that gets archived and forgotten. "When the primary database node fails, checkout error rate must stay under 1% and failover must complete within 10 seconds" is a test case. It can be rerun after the next infrastructure change. It can be assigned an owner. It can show up in a traceability report next to the requirement it protects. An experiment whose finding never gets written down as something the team will check again was theatre, however good the tooling that ran it.

Naming the tools honestly, and where Tesbo fits

Tools like Gremlin, Chaos Mesh, and Litmus exist specifically to inject the failures: killing pods, throttling networks, exhausting CPU, simulating region outages. That is real, useful engineering, and if you are running chaos experiments, one of those, or something like them, is doing the actual breaking.

Tesbo is not one of those tools, and it should not be described as one. Tesbo does not inject failures, orchestrate chaos experiments, or monitor systems while an experiment runs. What Tesbo is built for is the part right after the injection tool has done its job: capturing the hypothesis, the precondition, and the expected result as a documented test case that lives alongside the rest of your suite, so the next engineer who touches that system inherits the finding instead of rediscovering it. The injection tool causes the failure. The test case is what makes the finding permanent.