L6 · AI as your instrument
L6Core4 min read

Coding agents in the QA loop

Claude Code, Copilot, Cursor and the rest, used on real testing work. Where they genuinely save hours — scaffolding, migration, repetitive fixes — and where handing over control costs more than it saves.

A coding agent is a tool that reads your repository, writes code, runs commands and iterates on the result. Claude Code, Copilot's agent mode and Cursor are the common examples.

The difference from a chat window is autonomy. A chat suggests text you paste. An agent edits files, runs the suite and keeps going until it believes the task is done.

For testing work that changes the economics of three specific jobs, and leaves everything else roughly where it was. Knowing which is which is the whole skill. Point one at the wrong task and it produces a large, confident mess that takes longer to review than to write yourself.

The terms you will hear

  • Coding agent. A tool that edits files and runs commands in a loop until a task is done.
  • Autonomy level. How much it does before asking. From suggesting a line to opening a pull request.
  • Tool call. One action it takes: read a file, run the suite, edit a line.
  • Diff. The change it produced. The only thing worth reviewing.
  • Guardrail. A limit you set: which directories, which commands, no pushing to main.
  • Human in the loop. A required review point before the change lands.

Where it genuinely pays

  • Scaffolding. Twenty page objects from twenty pages, fixtures, test data builders, the boilerplate around a new suite. Mechanical, verifiable, dull.
  • Mechanical migration. Moving a suite from one framework to another, or renaming a pattern across 300 files. This is the strongest case, and it has its own topic later on this layer.
  • Repetitive fixes. The same selector change in 40 tests. The same assertion updated after a deliberate behaviour change.
  • Reading unfamiliar code. "Which tests cover the refund path, and what do they assert?" Faster than grepping, and worth checking.

Where it costs more than it saves

  • Deciding what to test. It will produce cases for whatever you point at, with no view on what matters. That judgement is risk work.
  • Anything with an unstated expectation. With no oracle, it writes an assertion that matches current behaviour, which locks in the bug.
  • Flaky test investigation. It will add waits and retries until the test passes, which hides the cause instead of fixing it.
  • Wide refactors of a suite you do not know well. The diff becomes too large to review honestly, and an unreviewed diff is not a saving.

How to work with one

  1. Give it one task with a definition of done. "Convert these 30 Selenium tests to Playwright. The suite must pass. Do not change any assertion."
  2. Set guardrails first. Which folders, which commands, no force pushes, no changes to production config.
  3. Give it a way to verify itself. A failing test, a lint rule, a type check. An agent with a check to satisfy is dramatically more useful than one working blind.
  4. Read the diff, always. Look for weakened assertions, added waits, deleted tests and changed expected values. It is reviewing what a machine wrote, applied to code.
  5. Keep the change small enough to review. If you would not review a human's pull request of that size, do not accept it from an agent either.
  6. Re-run the suite yourself. Trust the run you performed, which is the same rule as checking the real state for any agent.
  7. Record what it did. A one-line note in the pull request saying what was generated and what you changed afterwards.

An agent will always tell you it finished. The diff tells you what it actually did, and those two are different documents.

A worked example

For example, here is one real migration task, with the review that mattered.

agent-migration-review.txt
TASK  convert 30 Selenium tests in tests/legacy/ to Playwright.
      done means: all 30 pass, no assertion changed, no test deleted.
GUARDRAILS  only tests/, may run the suite, may not touch src/ or CI config

WHAT IT DID  41 minutes, 312 tool calls, 34 files changed

REVIEW OF THE DIFF (35 minutes, the part that matters)
  27 files   clean mechanical conversion. correct waits, correct
             selectors, assertions untouched.                    accepted
   2 files   added page.waitForTimeout(3000) where the original
             had an explicit wait for an element                 rejected,
             fixed by hand. this is the flake pattern, not a fix.
   1 file    changed expect(total).toBe('35.99') to
             toContain('35')                                     rejected,
             this is the weakened assertion. the exact bug this
             review exists to catch.
   1 file    deleted a test it described as "duplicate"
             it was not a duplicate, it covered the tax path      restored
   3 files   converted correctly but left the old file in place   deleted

OUTCOME
  saved roughly two days of mechanical work
  cost 35 minutes of review, and 3 of the 34 files needed correction
  net: clearly worth it, and only because the diff was read

WHAT I WROTE IN THE PULL REQUEST
  "30 tests converted with a coding agent. I reviewed all 34 files.
   Three changes were rejected: two added fixed waits, one weakened an
   assertion. One deleted test was restored."

Three problems in thirty-four files, and all three were the kind that make a suite quietly worthless. The saving was real, and it existed only because somebody read the change.

How to show you know it

  • A review note like the one above. It shows you use the tool and do not trust it.
  • A weakened assertion you caught. The signature finding of this topic.
  • A task you refused to delegate. "I did not ask it to decide which cases to write, because that is the judgement I am paid for."
  • A time comparison with the review cost included. Two days saved, thirty-five minutes spent. Honest arithmetic beats enthusiasm.

Questions

Will this replace test automation engineers?

It removes a large share of the typing, not the judgement. Deciding what to test, what an assertion should be, and whether a suite is trustworthy is the job. Those are the parts an agent is worst at.

How large a change should I accept?

One reviewable pull request. If the diff is too big to read honestly, split the task. Accepting an unreviewed diff means the saving is imaginary.

It says the suite passes. Is that not enough?

No, because it may have changed the assertions to make it pass. Run it yourself and read the diff for weakened checks, deleted tests and added waits.

What should I never let it do?

Touch production configuration, change CI gates, push to a protected branch, or "fix" flaky tests unsupervised. Those four cause the most damage for the least benefit.