The QA mindset: risk, not coverage

Risk-based testing means deciding what to test by what it would cost to be wrong, rather than by trying to cover everything evenly.

Risk in testing has two parts. Impact is how bad the failure would be. Likelihood is how probable it is. You rank by impact first, because you can live with a likely failure that costs nothing, and you cannot live with a rare one that costs the company.

Compare that to coverage, which measures how much of the product your tests touch. Coverage is a number about your effort. Risk is a number about the customer.

Two testers get the same feature and the same week.

The first writes 140 test cases and runs them all. Every field, every button, every combination of the three dropdowns. The report says 140 of 140 passed. It reads beautifully.

The second writes 30 cases. Twenty of them are about the payment step, because that is where money moves. Five are about the refund path, because a mistake there cannot be undone. Five cover the rest of the form.

Ten days after release, a customer is charged twice. The first tester had a case for it, somewhere in the 140, and it passed on the wrong data. The second tester found it before release, because she spent half her week in the one place where being wrong was expensive.

That is the difference between coverage and risk.

The terms you will hear

• Risk. Impact multiplied by likelihood. In practice you score both as high, medium or low.
• Impact, or consequence. What it costs if this breaks: money, trust, data, a regulator.
• Likelihood. How probable the failure is. New and rushed code is more likely than untouched code.
• Coverage. The share of the product, code or requirements your tests exercise.
• Residual risk. What you knowingly leave untested. Worth writing down, because silence reads as coverage.

Why coverage misleads

Coverage is easy to count, which is why it gets reported. It tells you how much of the surface you touched. It says nothing about whether you touched the parts that matter.

Say a checkout page has forty fields and one of them is the amount charged. Testing thirty-nine fields thoroughly and the amount casually gives you 97% coverage and a company-ending bug. Nobody is lying in that report. The number is simply answering a question nobody should care about.

The four questions

Take any release and ask these, in this order.

• What touches money? Payments, refunds, discounts, invoices, anything with a number a customer would notice.
• What cannot be undone? Emails sent, records deleted, orders placed, messages posted.
• What changed? New code, changed code, changed dependencies, changed configuration.
• What has broken before? Your run history is the best predictor you have of where the next bug lives.

Anything appearing in two of those lists is where your week goes. In practice that is usually three or four areas out of twenty.

How you do it in twenty minutes

1. List the areas in the release. Not test cases. Areas, at the level of "checkout", "refunds", "account settings".
2. Score consequence, high, medium or low. Ask what a customer, a colleague or a regulator would say if it were broken for a day.
3. Score likelihood the same way. New code and rushed code are more likely. Untouched code is less likely.
4. Sort by consequence first. A high consequence and low likelihood item still outranks a low consequence and high likelihood one, because you can live with the second being wrong.
5. Spend your time in the top third. Deliberately. Write it down so it is a decision instead of a drift.
6. Say in the summary where you did not spend it. "The catalogue pages got a smoke check only." That sentence is what makes the whole approach honest, and it belongs in the release record.

A worked risk table

Twenty minutes for the gift-card release at Willow Books, a small online bookshop. This is the entire artefact.

Look at the bottom two blocks. The plan is visible, the trade is explicit, and the assumption is stated. If it turns out the release did touch search, nobody has to guess whether you knew.

How to show you know it

• A one-page risk table for a real release, with the time you actually spent beside it.
• A high-consequence area you argued up the list. For example, insisting on time for refunds when the story sounded small.
• A stated non-coverage. The smoke-only line. It reads as confidence rather than a confession.
• A prediction that came true. "I flagged the money maths as the highest risk, and that is where both defects were." Keep that; it is your evidence next time.