State transition and pairwise testing

Two techniques for the bugs that are not in any single input.

State transition testing is used when a feature has modes. You list the states it can be in, the events that move it between them, and which of those moves should be impossible. The bugs are usually in the moves, not the states.

Pairwise testing, also called all-pairs or combinatorial testing, is used when several independent options multiply into too many combinations to try. It selects a small set of cases that still covers every pair of values at least once.

A vending machine has states.

Empty. Coins inserted. Item selected. Dispensing. Out of stock. The interesting bugs are never in a state, they are in the moves between states. Press refund after selecting. Insert a coin while it is dispensing. Select an item that sold out one second ago.

Now think about a pizza order form instead. Size, crust, four toppings, delivery or collection, a discount code. Nothing about the order matters, but the combinations run into the thousands.

Two different problems, and two techniques that handle them.

The terms you will hear

• State. A mode the thing can be in: draft, paid, shipped, refunded.
• Event, or trigger. The action that moves it between states: pay, cancel, refund, time passing.
• Transition. One state plus one event, and the state it lands in.
• Invalid transition. A move that should be refused. Where most defects in this technique are found.
• Combinatorial explosion. The number of combinations growing faster than anyone can test.
• Pairwise, or all-pairs. A set of cases covering every pair of option values at least once.

Order matters: state transition testing

Some features have modes, and the mode changes what everything else does. An order that is draft, paid, shipped, or refunded. A subscription that is trialling, active, past due, or cancelled.

The method is simple.

1. List the states. Usually five to eight for one feature.
2. List the events that move between them. Pay, ship, cancel, refund, time passing.
3. Draw the grid. States down the side, events across the top.
4. Mark the moves that should work, and the ones that should not.
5. Test both. The allowed ones must work. The forbidden ones must fail cleanly, and that is where the bugs are.

Too many combinations: pairwise

Say a checkout has five choices: payment type (4 options), delivery (3), currency (2), member or not (2), and discount code or not (2). That is 4 times 3 times 2 times 2 times 2, which is 96 combinations. Add one more choice and you are past 200.

Nobody tests 200 combinations. So people test the four they can think of and hope.

Pairwise is the middle path. Most combination bugs involve just two values interacting, for example Apple Pay with a foreign currency. Pairwise picks a small set of cases that covers every possible pair at least once. For the 96 above, that is usually around a dozen cases.

You do not calculate it by hand. Tools do it, including a free one in most languages. The skill is knowing when to reach for it, and being able to say what it does and does not cover.

How to use them on a real story

1. Ask whether the feature has modes. If the word "status" appears anywhere, draw the state grid.
2. Ask whether choices multiply. If there are three or more independent options, count the combinations before testing anything.
3. Do the forbidden moves first. They are quick, usually unspecified, and they find real defects. They are also good exploratory charters.
4. Generate a pairwise set for the rest. A dozen cases beats guessing four.
5. Add the realistic combination. The one your best customers actually use, whether or not the tool picked it.
6. Say what you covered. "All pairs covered, no three-way combinations" is honest and useful in the release summary.

A worked example

Here is both techniques on the gift-card feature, in about fifteen minutes.

Two defects, and neither was in any single input. One came from a forbidden state move, the other from a three-way pile-up that a dozen well-chosen cases happened to include.

How to show you know it

• A state grid with the forbidden moves marked. It reads as senior work and takes ten minutes.
• A bug from an illegal transition. Refunding a voided card, or its equivalent in your product.
• A pairwise set with the count. "48 combinations, 12 cases, all pairs covered." Precise, and honest about the limit.
• An unspecified transition you turned into a question. As with naming the oracle, the gap is the finding.