End-to-End Testing: What It Proves That Nothing Else Does

Search for end to end testing and you will be told it is the layer closest to the user. That is true, and it is also why so many teams over-invest in it. A suite of 340 end-to-end tests takes 55 minutes and fails about one run in four. Nobody can say which failures are real, so the habit becomes re-running it.

On a Thursday, three days before a release, somebody finally reads those results properly. Two genuine defects are sitting under eleven timeouts. That suite is not too slow. It is too large. This post covers what end-to-end testing genuinely proves, why it collects flakiness, and how to choose the handful of journeys that earn a permanent case.

What end-to-end testing is

End-to-end testing exercises a complete user journey through a running system, using the same interfaces a person or a calling service would use.

No mocks in the middle. The browser talks to the real front end, which calls the real API, which reads the real database, which triggers the real background job. If a queue sits between two services, the test waits for the queue.

That is the definition. The more useful question is what it buys you that a cheaper layer does not.

The one thing it proves

Every other layer tests a part against an assumption about its neighbours. A unit test assumes the function it calls behaves. An integration test assumes the contract it was written against is the contract in production.

End-to-end testing is the only layer that checks the assumptions themselves.

That matters because most expensive production incidents are not logic bugs. They are wiring bugs. Consider the failures that only appear once everything is connected:

• A config value that is correct in staging and wrong in production
• A service returning the right data in a shape the caller stopped expecting three deploys ago
• A background job that never runs because nobody deployed the scheduler
• A migration that ran on one replica and not the other
• An expired credential in a service nobody has touched in eight months

Every one of those passes a full unit and integration suite. Only a journey through the running system catches them.

A wiring bug, and the only test that would have caught it

Here is the shape of the problem, from a payments team on a fortnightly release.

A payment service publishes a message when a transfer completes. A ledger worker consumes it and credits the recipient. Somebody adds a retry to the publisher, because a timeout was occasionally losing messages. The retry is correct. The consumer is correct. Both teams have full test suites and both suites pass.

In production, roughly one transfer in 900 is published twice, and the ledger credits it twice. Six days pass before a customer notices. Reconciling the 41 affected accounts by hand then takes two people most of a Friday, about seven hours each. The unit tests could not see it, because neither service is wrong on its own. The integration tests could not see it, because each was written against the contract rather than against the running pair.

A single end-to-end case would have caught it in one run: send a payment, then assert that exactly one ledger entry exists. Not that the balance is correct, which it briefly is under some timings, but that the count of entries is one. That is a nine word assertion protecting two weeks of trouble.

Why this layer collects flakiness

An end-to-end test has more ways to fail than any other kind, and only one of them is a real defect.

It depends on a deployed environment, network timing, background jobs completing, third party sandboxes being up, and test data that no other test has already consumed. Each is a source of failure unrelated to the code under test.

This is why a 340 case suite fails one run in four. The product is not broken 25 percent of the time. It is that 340 cases each carrying a small chance of environmental noise adds up to a suite that is almost never fully green.

The consequences are worse than lost time:

• Real failures get re-run rather than read
• The suite stops being a gate and becomes a report nobody opens
• Engineers learn that red means nothing, and that habit transfers to other suites

None of that is fixed by better tooling. It is fixed by having fewer end-to-end tests, each worth investigating when it goes red. Flaky tests are a design problem rather than bad luck, and at this layer the design decision that matters most is how many cases you allow.

Test data is the other half of the problem

Most advice on end-to-end flakiness stops at waits and retries. The larger cause is usually shared, mutable test data.

Two cases both use the account with the 1,000.00 balance. One spends 250.00. Run them in the order the suite was written and both pass. Run them in parallel, which the pipeline started doing when somebody sharded the suite to get it under an hour, and one fails perhaps a third of the time.

Three approaches work, in descending order of reliability:

• Each case creates the data it needs and cleans up after itself
• Each case gets a dedicated account from a pool, never shared within a run
• Cases share fixtures but are pinned to run serially, which costs you the parallelism

The first is the most work and the only one that survives sharding. Whichever you pick, the choice belongs on the case as a precondition, so the next person can tell whether a failure is a defect or a collision.

Choosing which journeys earn a case

Coverage is the wrong criterion here, and it is the one most teams use. Coverage thinking produces an end-to-end case per feature, which is how you reach 340.

Two better questions:

Which assumption does this remove? If a case would fail only when the logic is wrong, and never when the wiring is wrong, it belongs at a cheaper layer. A case that fails when the payment service cannot reach the ledger belongs here.

What does this journey earn or protect? Follow the money and the obligations. The journey that takes payment, the journey that provisions an account, the journey that produces a record you are legally required to keep.

For a payments product that is roughly six journeys:

• Sign up, verify, and reach a usable account
• Add a funding source and have it confirmed
• Send a payment and see it settle in the recipient balance
• Hit the daily cap and be refused
• Request a refund and see the ledger reconcile
• Export a statement covering a period that includes all of the above

Six cases, not sixty. Each crosses at least three services. Each fails for a reason worth an engineer's morning.

The reasoning that governs what to automate first applies with more force at this layer, because the maintenance cost per case is the highest in the suite.

What a written case looks like

The third journey, written out.

Title: A payment sent between two accounts settles in the recipient balance

Preconditions: Two verified accounts exist, created by this case rather than shared. The sender has a confirmed funding source and a balance above 1,000.00 GBP. The recipient balance is recorded before the test starts. The ledger service and the notification worker are both running.

Steps:

1. Sign in as the sender
2. Open the payment form and select the recipient
3. Enter 250.00 and submit
4. Wait for the confirmation reference to appear
5. Sign in as the recipient

Expected result: The recipient balance has increased by exactly 250.00. The sender balance has decreased by 250.00 plus the tier fee shown at submission. Exactly one ledger entry exists for the transfer. The recipient has one notification. No error appears in either account's activity log.

The line that matters is "exactly one ledger entry". That is the assertion from the retry incident above, and no per service test will ever make it. Naming what must not have happened is what turns a click-through into a check, and it is the same discipline a good API test case needs at a lower layer.

What end-to-end testing is not

Three things get filed here that do not belong.

• A slow integration test. If it does not cross a real boundary, it is an integration test with a long setup.
• A performance check. Timing assertions at this layer fail for environmental reasons and get re-run. Keep them separate.
• A smoke suite. A smoke check answers whether the build is worth testing, runs in minutes, and is deliberately shallow.

That third confusion is common and expensive. Teams wire a 55 minute end-to-end suite into the merge check. Developers start merging around it, and within a month it is advisory.

Where it sits

End-to-end testing is the most expensive layer to write, the slowest to run, and the hardest to keep honest. It is also the only one that tells you the system works rather than that the parts do. The wider map of testing types covers how the layers relate.

The way to hold both truths at once is to keep the suite small enough that every failure gets read. Six well chosen journeys that always get investigated are worth more than 340 that get re-run.