L3Core4 min read

Test data: factories, seeding, anonymisation

Most flakiness is a data problem wearing a costume. Data built per test rather than shared, cleaned up predictably, and never a copy of production with real names in it. Unglamorous, and it decides whether the suite is trusted.

Test data is the state a test needs before it can run: the customer, the order, the gift card with a balance.

Three approaches, and most teams end up with all three.

A factory or builder creates data in code, with valid defaults and overrides for the field under test. A seed loads a fixed known dataset before a run. Anonymised production data is a copy of real data with the identifying parts replaced.

Getting this wrong produces two familiar problems. Tests that pass only on one machine, and a spreadsheet of real customer emails on somebody's laptop.

The terms you will hear

Factory, or builder. Code that produces an object with defaults and overrides.
Seed data. A fixed dataset loaded before tests run.
Fixture. The mechanism that provides data to a test and cleans up after.
Anonymisation, or masking. Replacing identifying values so people cannot be recognised.
Synthetic data. Generated from scratch, never derived from real people.
Data leakage. Real personal data ending up somewhere it should not, including your laptop.

Which approach for what

Use a factory for anything the test changes. The customer it logs in as, the order it refunds, the card it spends. Created per test, deleted or ignored afterwards, so tests never collide. This is the default and it should cover most of your suite.

Use a seed for reference data everything shares. The product catalogue, tax rates, country lists, roles. Stable, read-only, loaded once.

Use anonymised production data for volume and shape, in a separate performance or exploratory environment. Never as the source for a functional test, because it changes underneath you and you cannot reason about it.

Why it matters

Because bad test data is the most common reason a suite cannot be trusted or cannot be run.

For example, a suite depends on customer id 4021 existing with two orders. It passes for a year. Then somebody refreshes staging, the id changes, and forty tests fail for a reason unrelated to the product. The team spends a day and concludes that the tests are unreliable, which is technically true and not the real diagnosis.

How to get it right

One builder per domain object, with valid defaults and one override per test. This is the architecture pattern doing double duty.
Create per test, not per suite. Shared data is the main source of order dependence and parallel collisions.
Never hardcode an id. Create the thing and use what comes back.
Make the awkward cases first-class. A card with 0.01 left, an order with 400 items, a name with an apostrophe, a customer in a different time zone. Name them and keep them.
Seed only reference data, and version the seed with the code so a refresh is reproducible.
Mask properly, or generate synthetically. If the copy can be re-identified by joining an email to a postcode, it is not masked.
Set a retention rule for test data, and delete on schedule. Then write it in the strategy so it is a rule rather than a habit.

If a test depends on data somebody else created, it is not a test. It is a report on the state of a database at a moment nobody recorded.

A worked setup

For example, here is one small team's data approach after a clean-up.

test-data-plan.txt

BEFORE
  staging seeded once in March, by hand
  suite depended on customer 4021 and card GIFT-DEMO-1
  a refresh in July broke 41 tests
  a "customers.xlsx" export from production sat in the shared drive

AFTER

  BUILT PER TEST (in code, deleted after)
    customers        aCustomer().withEmail().withOrders(n).build()
    gift cards       aGiftCard().withBalance().expired().voided().build()
    orders           anOrder().withTotal().paidBy('card' | 'gift').build()
    named awkward cases kept as builder presets:
      cardWithOnePenny, orderWith400Items, customerInAuckland,
      nameWithApostrophe, addressOver400Chars

  SEEDED (reference only, versioned in the repo)
    12 products, 4 tax rates, 3 roles, 1 currency
    reloaded by a script on every environment refresh, ~4 seconds

  ANONYMISED COPY (separate environment, monthly)
    used for performance runs and exploratory sessions only
    masking: emails replaced, names replaced, addresses replaced,
      postcodes truncated to the area, dates of birth shifted by a
      random offset per person, card details removed entirely
    joined-field check: can a row be re-identified from what remains?
      reviewed by whoever owns the data, not by the tester alone
    retention: 30 days, deleted by a scheduled job

  RULES WRITTEN DOWN
    no production export leaves production unmasked, ever
    no test depends on an id somebody else created
    every awkward case is a named preset, not a magic value

  RESULT
    the next environment refresh broke 0 tests
    the spreadsheet was deleted, and the export path was closed

The line that mattered most was the last one. The technical changes made the suite reliable, and closing the export path removed the risk that no test failure would ever have shown.

How to show you know it

A builder with named awkward presets. cardWithOnePenny is more useful than any amount of documentation.
A refresh that broke nothing. The proof that no test depends on somebody else's data.
A masking review by the data owner. Showing you did not sign off your own anonymisation is the professional move.
A retention rule with a deletion job. Rules without a job attached are wishes.

Questions

Is it ever acceptable to test against production?

Read-only smoke checks against production are normal and useful. Creating or modifying data there is not, and using a production copy for functional tests is a different and worse idea because it moves under you.

How much data does a test need?

The minimum that makes the case real. One customer, one card, one order. Large datasets belong in performance environments, not in functional tests.

Our staging has no data. Where do I start?

A seed script for reference data and one builder for your most-used object. That combination usually unblocks most of a suite in a day.

Is generated synthetic data good enough?

For functional testing, usually better, because you control it. Where you need realistic distributions, such as performance work, an anonymised copy is more faithful, and it comes with the obligations above.