L4 · Quality in the pipeline
L4Go deeper5 min read

Feature flags and progressive delivery

Once release is separate from deploy, testing moves. You are now verifying flag combinations and behaviour for a cohort rather than a single build — and the number of combinations grows faster than most teams notice.

For most of testing's history, "deployed" and "released" meant the same thing. The moment code hit production, every user saw it. Feature flags break that link. Code can sit deployed and dormant behind a flag for weeks, then turn on for one percent of users, then a named beta group, then everyone. All without another deploy. That is progressive delivery, and it moves the hardest part of testing from "does this build work" to "does this behaviour work for this cohort, in this combination."

Deploy and release are now two different events

A feature flag is a runtime toggle, usually backed by a service like LaunchDarkly, Split, or an in-house config store, that decides which code path executes for a given user or request. The code for both the old and new behaviour ships in the same deploy. Nothing about the binary changes when a flag flips.

Only the runtime decision changes. That means turning a feature off again is a config change, not a rollback, and it takes seconds rather than a full redeploy.

This is genuinely useful for testing. A team can deploy a risky feature to production behind a flag that is off for everyone, verify nothing broke for the default population, then turn it on for internal users only, then a small beta cohort, then a wider rollout. Each step is testable in production with real traffic before the blast radius grows.

It pairs naturally with canary releases and automated rollback, since a flag can gate a feature within a build the same way a canary gates the build itself.

The combinatorics problem nobody plans for

Here is where flags become a QA headache instead of a QA convenience. One flag doubles the states a page can be in: on or off. Five independent flags on the same page produce thirty two possible combinations, and most teams do not track which combinations are even reachable, let alone test them.

Consider a retail platform team that ran into exactly this. They had four active flags on the product page: a redesigned image carousel, a new recommendation widget, an updated price display for a currency migration, and an experimental "buy now" button placement. Each shipped clean in isolation.

The combination of the currency migration flag and the new price display flag, both on, caused a rounding difference. It showed two different totals on the same page for international users. It survived three sprints because the flag combination matrix was never tested as a matrix, only as four separate features.

Strategies that keep this manageable

Testing every combination exhaustively does not scale past two or three flags. A few practical constraints keep the surface honest instead:

  • Flag registries with owners and expiry dates: every flag has a named owner and a planned removal date. This keeps stale flags from silently accumulating and multiplying the combination space forever.
  • Kill switches tested separately from rollout flags: an emergency off-switch needs its own test, confirming it actually disables the feature under load, not just in a clean staging request.
  • Default-state regression coverage: the pre-release suite always runs with every experimental flag at its default (usually off), so the baseline experience never silently regresses while attention is on the new flag.
  • Pairwise testing for flags known to touch the same surface: when two flags render to the same page or the same code path, test all four combinations of on/off explicitly rather than assuming isolation held.
  • Flag state visible in bug reports: a reproduction step is useless if it does not name which flags were active, since the same page in a different flag state is effectively a different page.
flag-matrix.test.ts
const flagStates = [
  { newLayout: false, newRecs: false },
  { newLayout: true, newRecs: false },
  { newLayout: false, newRecs: true },
  { newLayout: true, newRecs: true },
]

for (const state of flagStates) {
  test(`product page renders with flags ${JSON.stringify(state)}`, async () => {
    await setFlags(state)
    await expect(page).toHaveNoConsoleErrors()
  })
}

What this means for the pre-release suite

A test suite that only exercises the flag states that existed when it was written goes stale fast, because flags change more often than the code around them. Treating flag state as a first-class test parameter, the way test data management treats fixture state, keeps the suite honest as flags roll forward and get retired.

The goal is not covering every combination forever. It is knowing which combinations are actually reachable in production right now, and testing those on purpose rather than by accident.

FAQ

Questions people ask

How many feature flags is too many to test reliably?

There is no fixed number, but once flags on the same page or workflow exceed two or three, exhaustive combination testing stops scaling and you need pairwise testing or explicit interaction tests instead.

Should the default-off state of a flag always be part of regression testing?

Yes. The default state is what most users see, and it is the state most likely to silently regress while attention is focused on the new flagged behaviour.

Do feature flags remove the need for canary releases?

No, they solve different problems. A flag controls which code path runs within a deployed build, a canary controls which build a user gets. Many teams use both together.

What happens to old flags that nobody removes?

They accumulate and multiply the number of reachable combinations, making the app harder to reason about and test. A registry with owners and expiry dates is the usual fix.