Large-scale migration with agents

Test migration is moving an existing suite to a different framework, language or pattern without changing what it checks.

It is mechanical, repetitive, and enormous. Three hundred files of the same transformation is the kind of work that gets postponed for two years, because nobody can justify three weeks of typing that adds no coverage.

This is the strongest case for a coding agent in testing. The transformation is well defined, the correct outcome is known, and there is an obvious oracle: the migrated suite must pass and must assert the same things.

The risk is equally clear. At three hundred files nobody reviews every line, so the method has to be a sampling strategy rather than blind trust.

The terms you will hear

• Batch. A group of files migrated and reviewed together.
• Error rate. The share of files needing correction, measured on your first fully reviewed batch.
• Sampling. Reviewing a defined subset of later batches rather than everything.
• Assertion parity. The migrated test checks exactly what the original did.
• Parallel run. Old and new suites both running, results compared, until trust is earned.

How to run one

1. Write down what must not change. Assertions, test names, the number of tests. Anything else is fair game.
2. Migrate one file by hand first. You need the target pattern before an agent can copy it. This also tells you the awkward parts.
3. Do one batch of ten, and review all ten. This gives you an error rate and a list of its habits, which is worth more than any estimate.
4. Fix the prompt, not the output. If it added fixed waits in three of ten, say so explicitly for the next batch.
5. Then scale in batches of twenty to fifty. Review a sample, plus every file in a high-consequence area.
6. Diff assertion counts mechanically. A script comparing the number of assertions per file before and after catches silent weakening at scale, which no sampling will.
7. Run both suites in parallel for a few releases. Compare failures. Retire the old one when the new one has caught the same things, which is the record-keeping habit applied to a migration.

Why the sampling matters

Because full review does not scale and blind trust is unsafe, so you need a defensible middle.

For example, on three hundred files, reviewing everything costs about a week of reading, which defeats the exercise. Reviewing nothing means shipping an unknown number of weakened tests. Reviewing a measured sample, plus everything on the money paths, plus a mechanical assertion diff, costs about a day and gives you a number you can quote.

That number is what makes the migration defensible when somebody asks whether the new suite is as good as the old one.

A worked migration

For example, here is a real one, 312 files from Selenium to Playwright.

The batch-one error rate of thirty per cent is the most important number there. It told the team the habits to correct before touching the other three hundred files, and it dropped to ten per cent immediately after one instruction change.

How to show you know it

• A measured error rate. From a fully reviewed first batch, with the habits it revealed.
• A mechanical assertion diff. A short script that compares assertion counts before and after. It catches at scale what sampling cannot.
• A sampling strategy, written down. Random sample plus all high-consequence files plus a mechanical check. That is a defensible method.
• A parallel run with an outcome. "Both suites for four releases, the new one caught two extra defects." That is how you retire the old suite without an argument.