L1 · Running the work
L1Core4 min read

Drafting QA documents with AI

A model will produce a test plan in thirty seconds, and it will be generic, over-long and confidently wrong about your system. What to feed it — the spec, the architecture, last release’s incidents — and which judgements have to stay yours, because those are the parts anyone is paying for.

A model will produce a test plan in thirty seconds. It will be well structured, complete looking, and wrong about your system in ways you have to know the system to spot.

Two words explain why. The prompt is what you ask for. The context is the material you supply with it. A model given a prompt and no context can only write the average of every test plan on the internet, which is exactly what a template farm would give you.

The useful frame is a ghostwriter who has never met you. Fluent, fast, and unable to know that your refund window is 60 days, that scheduled delivery has no monitoring, or that last release broke on Safari.

Feeding it that material is the work. The judgements below are the part nobody is paying a model for.

The terms you will hear

  • Prompt. The instruction you give the model.
  • Context. The material you supply alongside it, such as a story, a spec or an incident list.
  • Hallucination. A confident, invented specific. The main failure mode for documents like these.
  • Grounding. Making the output traceable to supplied material rather than to the model's memory.
  • Review. Reading the draft against your own knowledge, which is where the value is added.

What it is good at, and not

Good at: structure, first drafts, rewriting for a reader, turning your rough notes into prose, listing the things you may have forgotten, and converting a plan into a checklist.

Bad at: knowing anything specific about your system, judging what matters, deciding what to leave out, and being honest about gaps. It will fill an unknown with something plausible rather than saying it does not know.

For example, ask for a test plan for a gift-card feature with no context and you will get load testing, security testing and browser compatibility as scope items. All reasonable in general. Your strategy may put two of them explicitly out of scope, and the model has no way to know that.

How to do it properly

  1. Write the scope yourself, from the tickets. This is the part that must be true, and it takes ten minutes.
  2. Give it real context. The stories, your test strategy, the architecture in three sentences, the defects from the last two releases, the environments you actually have.
  3. Ask for a draft, not a document. "Draft the approach and environments sections from this material. Leave anything you cannot support as a question."
  4. Delete every generic line. If a sentence would be true for any company, it is noise. This alone halves most drafts.
  5. Keep the four judgements. Scope, risk, exit criteria and the recommendation. These come from knowing the system and being accountable, so they stay yours.
  6. Check every specific against reality. Numbers, environment names, tool names, dates. Invented specifics are the main failure mode, and they read exactly like the true ones.
  7. Never paste sensitive material. No customer data, no credentials, no unreleased commercial detail. Use structure and fake values instead, which is the same rule as what never goes in a prompt.

A worked before and after

For example, here is the generic draft and the version that survived review, for the same release.

ai-draft-review.txt
WHAT THE MODEL PRODUCED WITH NO CONTEXT (extract)
  Scope: functional testing, integration testing, performance testing,
  security testing, usability testing, cross-browser testing,
  regression testing, user acceptance testing.
  Risks: insufficient time, resource constraints, unclear requirements,
  environment instability, scope creep.
  Exit criteria: all test cases executed, zero open defects,
  100% requirements coverage, sign-off obtained.

  verdict: true of every company, therefore useless to this one.
  the exit criteria are unholdable. the risks name no owner and no
  response. nothing here is checkable.

WHAT I FED IT INSTEAD
  - the 6 gift-card stories, pasted in full
  - our test strategy v3 (including the out-of-scope list)
  - three sentences on architecture: Next.js, Node API, Postgres, Stripe
  - the 6 defects from the previous release
  - "we have one tester, a weekly release, and no load testing"

WHAT IT PRODUCED THEN (extract, after my edits in [brackets])
  Scope: buying a card (4 amounts, card and Apple Pay), delivery of the
  code by email, redemption full and partial, balance check, refund of a
  gift-card order, admin void. Chrome, Safari, iOS Safari, Android Chrome.
  Out of scope: performance [correct, per strategy], expiry handling
  [correct], physical cards, non-GBP currencies.
  Risks: partial redemption maths [it ranked this first, which matched
  my own view], refunds against gift-card orders, cron worker with no
  monitoring [it picked this up from the defect list].

WHAT I STILL WROTE MYSELF
  the risk responses and owners, the exit criteria, my leave dates,
  and the recommendation. The model had no basis for any of them.

TIME  25 minutes total, against about 90 minutes writing it from scratch.

The interesting part is the middle. Given the strategy and the defect list, it identified the same top risk a tester would. Given nothing, it produced a page that could have been about any product in the world.

How to show you know it

  • A before and after like the one above. It demonstrates judgement rather than tool use.
  • A generic line you deleted. "Zero open defects" is the classic, and knowing why it is wrong is the point.
  • An invented specific you caught. A tool you do not use, an environment that does not exist, a number nobody agreed.
  • A stated boundary. "Drafting is assisted. Scope, risk, exit criteria and the recommendation are mine." That sentence is worth having ready, because you will be asked.

Questions

Is it acceptable to use AI for QA documents at all?

Yes, for drafting and rewriting, with the specifics checked and the judgements kept. What is not acceptable is shipping a document you have not verified, and that was true of templates long before models existed.

Will it write test cases too?

It will write plausible ones quickly, and they need the same review. Generated cases tend to assert weakly and cover the happy path, which is the ground generate freely, merge carefully covers.

What must never go in a prompt?

Customer data, credentials, keys, unreleased commercial information, and anything under an agreement you have not read. Use structure and fake values, and check your company's policy before you paste anything.

Does this make the documents topics obsolete?

The opposite. You can only review a draft if you know what a good plan looks like, so the value moves from typing to judging. That is why the documents come earlier on this layer than this topic does.