Choosing what goes in the regression pack

A regression pack, also called a regression suite, is the set of tests you re-run every release to check that what worked still works.

Test selection is the decision about which tests to run this time. Doing it by hand is judgement. Doing it automatically from the code that changed is called test impact analysis.

The problem is growth. Every release adds cases and almost nobody removes them. The pack goes from two hours to a day to two days, and then it quietly stops being run in the week it matters most. That decay has a name in practice: suite rot.

So the real skill is not adding tests. It is deciding what earns a place, and being willing to delete.

The terms you will hear

• Regression testing. Re-running tests to confirm that existing behaviour still works after a change.
• Regression pack, or suite. The chosen set of those tests.
• Test selection. Choosing a subset to run for a given change.
• Test impact analysis. Tooling that picks that subset from the code diff.
• Quarantine. Moving an unreliable test out of the blocking run while it is investigated.
• Suite rot. The slow decay of a pack into something too slow and too noisy to trust.

Why it matters

Because a pack nobody runs provides no protection at all, and it costs real time to maintain.

For example, a team with 400 cases finds the run takes two days. Under pressure they run "the important half", chosen from memory, differently each time. Nobody can say afterwards what was covered. The pack still costs a day a month to maintain and it has stopped being evidence of anything.

Cutting that pack to 120 deliberate cases that run in four hours is a quality improvement, even though it looks like doing less testing.

How to choose what stays

1. Score each case by consequence. Would a customer notice, would money be wrong, is it irreversible. Same lens as risk over coverage.
2. Score by change frequency. Areas touched every sprint deserve permanent cover. A feature untouched for two years rarely does.
3. Score by history. Anything that has broken before earns a place, because that is your best predictor of the future.
4. Delete duplicates. Suites accumulate three versions of the same journey through slightly different screens. Keep the best one.
5. Delete cases for features nobody uses. Check the usage numbers rather than guessing. It is the biggest single saving in most packs, and it is the what quality means question applied to your own suite.
6. Quarantine the unreliable ones. A case that fails randomly trains people to ignore red, which costs more than the case is worth. Investigate it with your run history, and either fix it or drop it.
7. Set a time budget and hold it. For example, four hours. Adding a case then means removing one, which forces the conversation to be about value.

A worked scoring

Here is a real cull, done in an afternoon, on a pack that had grown past its usefulness.

The last block matters. Cutting a pack is a bet, not a certainty, and saying so out loud is what makes the decision defensible when one of the deleted cases would have caught something.

How to show you know it

• A cull with numbers. Before, after, and the time the run now takes. Very few testers have done this deliberately.
• A deletion you can justify. "Seventy-one cases for a feature four people used in six months."
• A time budget your team agreed. It changes every future conversation about adding cases.
• An honest miss. Naming the case you deleted that would have caught something is more credible than pretending the cull was free.