L3Core4 min read

Determinism: clocks, seeds, network control

Freeze time, seed the randomness, control the network. Almost every intermittent failure traces back to one of those three being left to chance — and every technique here transfers directly to testing AI systems in L7.

A deterministic test gives the same result every time it runs on the same code. Same inputs, same environment, same answer.

That sounds obvious and most suites are not deterministic, because four things vary underneath them without anybody choosing it.

Time. The real clock. Dates roll over, tokens expire, "today" changes.
Randomness. Random test data, random ids, unseeded shuffles.
The network. Third parties, latency, rate limits, someone else's sandbox.
Order and concurrency. Which test runs first, which worker gets which row.

Flakiness is the symptom. Non-determinism is the cause. Controlling these four is how you stop treating flaky tests and start preventing them.

The terms you will hear

Frozen clock. Replacing the system time with a fixed value for the test.
Seed. A fixed starting value that makes a random generator repeatable.
Stub, mock, fake. Stand-ins for a real dependency, in increasing order of behaviour.
Hermetic test. One that depends on nothing outside itself.
Contract test. How you keep a stub honest against the real service.
Time zone drift. The most common date failure, and it usually appears in one office only.

How to control each source

Time. Freeze it. Set a fixed instant in setup so a date test gives the same answer in January and in July. Then add one deliberate test for the boundary you care about, such as midnight or a month end.

Randomness. Seed it. If your data builder generates names or ids, give it a fixed seed so a failure is reproducible. A test that fails once with an unrepeatable value is nearly worthless.

The network. Stub it for the suite and check the stub separately. Never let a third party's availability decide whether your pipeline is green, and never let its rate limit decide either.

Order and concurrency. Give each test its own data. A per-test customer, a per-test card, a per-worker schema. It is the highest-value change in most suites, because it removes two causes of flakiness outright.

Why it matters

Because a non-deterministic suite spends your attention rather than saving it.

For example, a date-handling defect only appears on the last day of a month. With a real clock, the suite catches it once a month, on a day when somebody is probably shipping. With a frozen clock and a deliberate month-end case, it is caught on every run, forever, in milliseconds.

The second reason is diagnosis. When a test can fail for four reasons that have nothing to do with the product, every failure costs a triage. Remove the four and a failure means one thing.

A worked setup

For example, here is the control applied to one suite, with what it fixed.

deterministic-setup.ts

import { test as base, expect } from '@playwright/test'
import { aCustomer, aGiftCard } from './builders'

// One fixed instant for the whole suite. Any date logic now behaves the
// same in January as in July, and the same on a developer machine as in CI.
const FIXED_NOW = new Date('2026-09-04T10:00:00.000Z')

// One seed, so generated names and codes repeat. A failure you cannot
// reproduce is nearly worthless.
const SEED = 20260904

export const test = base.extend({
  // Time: frozen before the page loads, so client-side dates match too.
  page: async ({ page }, use) => {
    await page.clock.setFixedTime(FIXED_NOW)
    await use(page)
  },

  // Data: created per test, never shared. This removes order dependence
  // and lets the suite run in parallel safely.
  customer: async ({ request }, use, testInfo) => {
    const built = aCustomer().withSeed(SEED + testInfo.workerIndex).build()
    const created = await request.post('/test/customers', { data: built })
    await use(await created.json())
  },

  card: async ({ request, customer }, use) => {
    const built = aGiftCard().forCustomer(customer.id).withBalance(25.0).build()
    const created = await request.post('/test/gift-cards', { data: built })
    await use(await created.json())
  },

  // Network: the payment provider is stubbed for the suite. A separate,
  // small spec runs against the real sandbox on a schedule.
  stubPayments: [async ({ page }, use) => {
    await page.route('**/api.stripe.com/**', (route) =>
      route.fulfill({ status: 200, json: { id: 'pi_test', status: 'succeeded' } }),
    )
    await use()
  }, { auto: true }],
})

export { expect }

Four lines of intent, and four failure modes disappeared.

Date tests stopped failing overnight and at month end.
Parallel workers stopped colliding on the same seeded card.
A third-party outage stopped turning the pipeline red.
A failure became reproducible, because the seed is fixed.

What to leave non-deterministic on purpose

Not everything should be frozen. Keep a small, separate set of tests that runs against reality.

A handful hitting the real payment sandbox, on a schedule rather than on every commit.
One or two that use the real clock, to catch anything your freeze hid.
Anything checking a third party's actual contract, which is the placement question all over again.

Label them clearly, run them separately, and accept that they will occasionally fail for reasons outside your control. That is their job.

How to show you know it

A frozen clock plus a deliberate boundary test. It shows you removed the variation without losing the coverage.
A seed in the output. Printing the seed on failure makes every failure reproducible, which reviewers notice.
A per-test data fixture. The change that removes order dependence, built with the fixtures and builders from earlier in this layer.
A named real-world suite. "These six tests hit the real sandbox nightly." It proves the stubs have a check on them.

Questions

Does freezing time hide real bugs?

It can, which is why you add deliberate boundary cases and keep a small real-clock suite. Freezing removes noise; the boundary cases keep the coverage.

Is mocking the network cheating?

For most tests it is the correct choice, because you are testing your code rather than a third party. The risk is stub drift, so pair it with contract tests or a small suite against the real sandbox.

How do I make tests safe to run in parallel?

Per-test data and no shared mutable state. If two tests can touch the same row, they will, and the failure will look like flakiness rather than a design problem.

What about randomised or property-based testing?

Valuable, and it should still print the seed. Random exploration with a recorded seed is deterministic when you need to reproduce, which is the whole trick.