Most QA roadmaps were written before any of this existed. This one starts with the fundamentals that still hold, and carries through to testing the systems everyone is now shipping — evals, prompt injection, agent behaviour, and the regulation arriving behind them.
Ticks are stored in this browser only — nothing is sent anywhere, and there is no account to make.
The thinking that survives every tooling shift — risk, oracles, and how to tell whether something is actually right.
Writing code stopped being the bottleneck, so shipping more of it stopped being the win. Quality is now about whether the thing does what someone actually needed — a judgement no generator makes for you. Start here; it reframes everything below.
Coverage counts what you tested. Risk asks what it would cost to be wrong. Teams that optimise the first ship suites that are enormous and reassure nobody. Learning to rank by consequence is the single highest-leverage habit in testing.
Did we build it right, and did we build the right thing. The distinction sounds like exam vocabulary until an AI feature passes every assertion and is still useless to the person using it — at which point it becomes the most practical idea here.
Where testing sits changes completely between a two-week sprint and twenty deploys a day. You do not need the certification; you need to recognise which model you are in, because it decides when your work happens and what it can block.
The techniques that turn an infinite input space into a short list worth trying. Unglamorous and permanently useful — and the thing that lets you tell whether a generated test suite is thorough or merely long.
For features where the bug is not in any one input but in the order they arrived, or in a combination nobody tried. Pairwise in particular turns a combinatorial explosion into a few dozen cases with almost no loss of power.
How much of the inside you are allowed to look at, and what each choice buys you. Mostly settled ground — but worth knowing precisely, because most modern testing is grey box and most people describing it get the term wrong.
Does it work, versus how well does it work under conditions nobody planned for. The second category is where the expensive failures live and where the tooling is weakest, which is why four of these layers exist.
An oracle is whatever tells you the output was correct. For most of testing history that was a hardcoded expected value. AI features have no such value — the answer is different every run — so the oracle problem went from theory to the central practical question of the job.
Structured investigation, not clicking around. Charters, timeboxes, notes. It is the skill that finds what no written case anticipated, and it is the one part of testing that has gotten more valuable as automation got cheaper, not less.
How testing actually happens on a team — the sprint, the regression cycle, and the documents everybody asks you to produce.
The most-asked question in testing and the least written-about answer. What you do on day one of a two-week sprint versus day nine, how you keep up with stories that land on the last afternoon, and why “we’ll test it at the end” is how a sprint quietly starts failing.
Testing starts before the code exists. Sitting in refinement, turning a vague story into acceptance criteria somebody could actually verify, and asking the one question that stops a week of work going the wrong way. The cheapest testing you will ever do.
Steps, data and a stated expected result — written so a colleague, a contractor or a model can run it without asking what you meant. Most test cases fail that bar, which is why suites get rewritten every time somebody leaves.
The same bug, two reports: one is closed as cannot-reproduce, the other gets a fix that afternoon. The difference is steps, environment, evidence and a plain statement of what you expected instead. This is the most visible thing you produce all week.
Severity is how bad it is, priority is when it gets fixed, and they are set by different people — confusing the two causes most of the arguments in a bug tracker. How a triage meeting should actually run, and what to do with the bugs everyone knows will never be fixed.
What actually happens in the days before a release: what you re-run, in what order, how long it takes, and what you do when it fails on the final afternoon. Written as a procedure you could hand to somebody, because “do a regression” is not one.
A regression suite that only ever grows eventually stops being run at all. Deciding what earns its place — by risk, by change frequency, by what has broken before — and being willing to delete cases is what keeps the cycle finishable. Test impact analysis is the automated version of this same judgement.
Terms teams use interchangeably and mean differently, which is how a release ends up signed off by nobody. What each check is for, how long it should take and who owns it — enough to settle the vocabulary with your team in a single meeting.
The standing document: what this organisation tests, at which level, with what tooling, and what it deliberately does not test. It changes maybe twice a year and it settles a hundred arguments. Most teams have never written one down, which is precisely why they keep having the same argument.
The document people are asked for and dread. Scope, approach, environments, data, risks, schedule and sign-off — for one release or one project, not for eternity. Learn the sections, learn which ones you are allowed to cut, and learn to write it in an afternoon rather than a fortnight.
The conditions under which testing may start, and the conditions under which it may stop. Agreed in advance they are protection; invented at the end they are an argument you lose. A short topic, and the part of a test plan that does the most real work.
How long the testing will take, asked at the worst possible moment. Breaking the work down, sizing it against something you have actually done before, and stating the assumption that would make the number wrong. Roughly right and explicit beats precise and quietly optimistic.
Answering “which tests cover this requirement, and did they pass” without a week of archaeology. A matrix in a spreadsheet is the old form and still works; links in your tracker are the better one. Dull until an auditor, a customer or a regulator asks — and under the AI rules, more of them now do.
What was tested, what was found, what is still open, and whether you would ship it. One page, written for people who will not read a second. Sign-off is a judgement you are putting your name to rather than a rubber stamp, and the wording matters on the day something goes wrong.
Where cases, runs and results live, and how they join up with the tickets. The tools differ far less than their vendors claim; what matters is deciding what you keep in there and what belongs in the repository beside the code.
A model will produce a test plan in thirty seconds, and it will be generic, over-long and confidently wrong about your system. What to feed it — the spec, the architecture, last release’s incidents — and which judgements have to stay yours, because those are the parts anyone is paying for.
The price of admission now. A tester who cannot read a diff cannot review what an agent wrote.
Not four languages badly. Pick the one your team ships in and get genuinely fluent — types, async, error handling, packaging. Everything in the automation layers assumes it, and reviewing AI-written code demands it.
Not just commit and push. Rebasing, resolving conflicts, reading history to find when something broke, and leaving review comments that change the code. Testing work now arrives and departs as pull requests.
The genuinely new core skill. When an agent drafts the tests, your value is deciding whether the draft is right — which means reading unfamiliar code quickly and spotting what a diff quietly changed. Almost nobody teaches this; everybody now needs it.
Requests, responses, idempotency, caching headers, what a 409 actually means. Every API test, every network-tab debugging session and every mock you write sits on this. Shallow knowledge here shows up as flaky tests you cannot explain.
Two shapes that break REST-shaped test tooling. GraphQL has one endpoint and no status codes worth asserting on; gRPC is binary and needs generated stubs. Learn how each fails, because the failure modes are what you will be testing.
Where test suites go to die. Understanding token lifetimes, refresh flows and scopes is the difference between a login helper that works for two years and one that breaks every sprint — and it is also half of the security layer.
Reading the database is how you confirm what the UI claimed, seed a scenario, and find the row that made the test flake. Joins, transactions, isolation levels. Absent from most QA roadmaps and present in almost every real testing job.
Once work happens in the background, "assert immediately after the click" stops being valid. Learn at-least-once delivery, ordering guarantees and eventual consistency, or you will spend years adding sleeps to hide the symptoms.
What the page is actually made of, and when it is ready. Client versus server rendering, hydration, shadow DOM. Nearly every "flaky UI test" is a misunderstanding at this level rather than a problem with the test framework.
Reading the network tab, throttling a connection, replaying a request, catching a console error the test swallowed. This is the fastest feedback loop you own, and speed of diagnosis is most of what makes a tester look senior.
How you get the same environment on your laptop and in CI, and how you stand up a dependency the tests need. You do not need to write production images; you need to read a Compose file and know why the container cannot reach the database.
Where the thing runs, how it gets its configuration, and how credentials reach it without being committed. Enough to debug an environment-only failure and to avoid being the person who pasted a production key into a test fixture.
Writing a test is the commodity half. Designing a suite that is still trusted in three years is not.
The pyramid is advice, not physics, and it was written before the tooling changed. What matters is matching test level to where your system actually breaks. Learn both shapes and the argument between them, then decide deliberately.
The difference between a suite that survives three years and one that gets deleted. Setup that composes, data built rather than hardcoded, and abstractions that hide the page without hiding the intent. This is the craft half of automation.
Auto-waiting, tracing, parallel workers, and a debugging story the previous generation never had. Learn it properly — locators, fixtures, projects, the trace viewer — because it is now the reasonable default and the baseline others are judged against.
You will inherit these. Enough to read an existing suite, keep it running, and make an honest case for or against migrating. Not somewhere to invest deeply if you are starting today.
Faster, steadier and closer to the logic than driving a browser. Asserting against a schema rather than a hand-written body is what stops the suite from breaking every time a field is added, and catches the changes that actually matter.
The answer to "integration tests across six services are too slow and too flaky". Each side verifies its half of the agreement independently. Hard to introduce socially, enormously valuable once it lands.
MSW, WireMock, and knowing when a fake helps versus when it quietly makes the test meaningless. The judgement call — what to stub and what to leave real — matters far more than the tool.
A different set of problems: device farms, permissions, gestures, build pipelines that take twenty minutes. Learn one native framework properly before reaching for a cross-platform one.
The tests developers own — which is exactly why you should be able to read and improve them. A well-placed component test removes ten end-to-end ones, and knowing where that trade lands is a quality-engineering skill.
Catches what assertions cannot: layout that broke, a component that vanished, a theme that inverted. Also the category most prone to becoming noise, so the real lesson is threshold tuning and review discipline.
Most flakiness is a data problem wearing a costume. Data built per test rather than shared, cleaned up predictably, and never a copy of production with real names in it. Unglamorous, and it decides whether the suite is trusted.
A suite nobody trusts is worse than no suite, because it costs money and blocks releases while catching nothing. Learn the actual causes, how to quarantine without hiding, and why blanket retries are how a team stops noticing real bugs.
Freeze time, seed the randomness, control the network. Almost every intermittent failure traces back to one of those three being left to chance — and every technique here transfers directly to testing AI systems in L7.
Where most QA careers stall and most suites quietly die. Everything here is about a suite surviving contact with CI.
Not which CI tool — they are broadly the same. What matters is where tests run in the pipeline, what they block, how failures are reported, and how the whole thing stays under the runtime budget a team will actually tolerate.
A forty-minute suite gets skipped; a four-minute one gets trusted. Splitting work across workers, balancing shards, and knowing what your pipeline costs per run. This is often the highest-impact work available to a test engineer.
Run the tests the change could plausibly break rather than all of them. Mature tooling exists now and almost nobody in QA is using it. Enormous lever on feedback speed once a suite gets large.
A fresh environment per pull request, destroyed on merge. Removes the shared-staging queue, the "someone else broke it" failure, and most environment-only bugs. Changes what is even possible to test before merge.
Once release is separate from deploy, testing moves. You are now verifying flag combinations and behaviour for a cohort rather than a single build — and the number of combinations grows faster than most teams notice.
Ship to one percent, watch the metrics, roll back automatically if they move. Testing becomes something that continues after deploy, and defining the signal that triggers a rollback is a quality decision.
A trace tells you which of eleven services was slow; a log tells you why. Being fluent here turns "it failed sometimes" into a specific, fixable claim — and it is the same skill L7 needs for reading agent behaviour.
The grown-up version of "is it good enough": an explicit reliability target and an agreed allowance for falling short. Gives quality arguments a number instead of an opinion, which is how they start being won.
Some things are only true in production — real data, real scale, real third parties. Running safe, continuous checks there catches what no pre-release environment could, and is now standard practice rather than heresy.
What happens when it breaks anyway. Being useful during an incident, then finding the real cause rather than the nearest human, is the fastest way for a tester to become someone the engineering team relies on.
Which numbers describe quality honestly and which get gamed within a month. Test-count and coverage-percentage targets are the classic own goals; change failure rate and time to restore are the ones worth defending.
Performance, security, accessibility and privacy — the four that get people fired, and the four AI is worst at unsupervised.
Scripting a realistic load, running it somewhere that can generate it, and reading the result without fooling yourself. k6 is the reasonable modern default; the concepts transfer to whichever your team already runs.
An average response time hides the users having the worst experience, and those are the ones who leave. p95, p99, and why tail latency is where the real story lives. Short topic, changes how you read every performance result afterwards.
Different questions need different shapes of load. Does it leak over eight hours, survive a launch spike, or degrade gracefully at the limit — three tests, not one, and picking the wrong shape answers a question nobody asked.
What a real user experiences before the page is usable. LCP, INP, CLS, and how to measure them in a way that reflects real devices rather than your laptop on office wifi.
The baseline vocabulary. Injection, broken access control, misconfiguration — enough to recognise a class of problem in a pull request and to write a test that proves it. You are not becoming a pentester; you are becoming hard to slip past.
APIs fail differently to web pages: broken object-level authorisation, mass assignment, unrestricted resource consumption. Since most of what you test is an API, this list is often more directly useful than the web one.
Automated scanning of the code, the running app and the dependencies. The practical skill is triage — most findings are noise, and a team that cannot separate the real ones stops reading the reports entirely.
Most of what you ship was written by strangers. Knowing what is actually in the build, where it came from, and how fast you could replace a compromised package is now a mainstream quality concern rather than a niche one.
How credentials reach a test run without living in the repository, and what to do when one leaks. Also the discipline that makes the AI layers safe — see what never goes in a prompt.
Automated tooling finds perhaps a third of real accessibility problems. The rest needs someone who has actually navigated the product by keyboard and heard it read aloud. High-value, still scarce, increasingly a legal requirement.
Accessibility became enforceable rather than aspirational for a large class of products in the EU. Enough to know whether it applies to what you ship, and what evidence you would need if asked.
Restoring a production dump into staging is the most common serious data-protection failure in engineering, and testers do it more than anyone. What you may hold, for how long, and how to anonymise so it is still useful.
Deliberately breaking a dependency to see whether the system degrades or collapses. You do not need a chaos platform to start — killing one container during a load test teaches most of the lesson.
Using AI to do the testing job better. This is the layer that makes an engineer who already has a job faster at it.
Tokens, context windows, temperature, and why the same prompt gives different answers. Not a research course: the specific mental model you need to predict where a model will be unreliable, which is the whole basis of testing one.
Vague prompts give plausible, useless tests. Learn to supply the spec, the constraints, the existing patterns and the definition of done — the same inputs you would give a new colleague, which is exactly why testers are good at this.
Claude Code, Copilot, Cursor and the rest, used on real testing work. Where they genuinely save hours — scaffolding, migration, repetitive fixes — and where handing over control costs more than it saves.
When a machine writes the test, the specification becomes the artefact that matters. Learning to write one precisely enough to generate from — and to review against — is the highest-leverage writing skill in modern QA.
Generated tests are confident, well-formatted, and frequently assert nothing meaningful. The failure modes are specific and learnable: tautological assertions, mocked-away logic, coverage of the happy path only. This is the job now.
Not to run the session, but to widen it. Generating edge cases you had not considered, adversarial inputs, and personas whose behaviour differs from yours. Best used as a second brain, not a replacement for the first.
Point a model at the code and the suite and ask what is untested and would matter. It is unreliable on "would matter" and quite good at "untested", and the combination still beats reading the report yourself.
The most oversold idea in test automation and not entirely worthless. Understand what it actually does, the failure mode where it happily heals past a genuine bug, and when a stable test id is simply the better answer.
Two hundred red tests, one cause. Grouping failures, summarising logs and proposing likely culprits is where AI earns its place in a pipeline today — measurable, low-risk, and it gives back the hours triage used to eat.
Moving nine hundred tests from one framework to another used to be a quarter of work nobody would fund. It is now days — if you can define the transformation precisely and verify the result at scale, which is the actual skill.
Small internal tools: a triage bot, a flake classifier, a release-note drafter. Enough API fluency to build the thing your team needs and nobody sells. This is where a test engineer starts producing leverage rather than output.
Giving an agent safe, structured access to your CI, your test results and your bug tracker instead of pasting logs into a chat window. The plumbing that turns a clever assistant into part of the workflow.
Where enthusiasm meets the invoice. Model calls in a pipeline are slow, priced per token and not reproducible — three properties CI is built to reject. Knowing how to place them anyway is what makes the difference between a demo and a practice.
Customer data, credentials, unreleased material, anything under an NDA. The rules are simple, the pressure to bend them is constant, and testers touch more sensitive data than almost anyone. Learn this before the tooling, not after.
Perceived speed-up and real speed-up diverge sharply, and there is now research showing they can point in opposite directions. How to measure honestly, so the decision to keep or drop a tool is evidence rather than mood.
Every company is shipping an LLM feature and almost nobody knows how to test one. The largest layer here, on purpose.
Prompt, context, retrieval, tools, model, post-processing. Knowing which layer produced a bad answer is the first move in every AI investigation, and it is the difference between a useful bug report and "the AI is wrong".
Retrieval-augmented generation, in plain terms: the model is handed some documents before answering. Most "hallucinations" in production RAG are actually retrieval failures — the model answered faithfully from the wrong page — and that distinction drives every test you will write.
What changes when the model stops answering and starts acting: it has goals, credentials, tools and multiple steps. Every property that makes an agent useful also makes it a new category of thing to test.
The same input gives a different output, and the model changes underneath you. `expect(response).toBe(...)` is simply not available. Understanding precisely why is what makes the rest of this layer make sense.
The replacement for assertions: a dataset of inputs, a way of scoring outputs, and a threshold you agree to hold. Same instinct as a regression suite, different machinery. If you learn one thing in this layer, learn this.
The eval is only as good as the examples in it. How to collect real cases, cover the failure modes, keep it from going stale, and resist the temptation to fill it with cases the system already passes.
Using a model to grade another model’s output. Practical and unavoidable at scale, and it has documented biases — toward longer answers, toward its own phrasing, toward whichever option came first. Usable once you know them; misleading if you do not.
Turning "is this a good answer" into something two people, or two models, would score the same way. Borrowed from social science, and the piece most teams skip on the way to numbers they cannot defend.
Offline runs against a fixed dataset before release; online measures real interactions after. You need both, they answer different questions, and teams that only do the first are consistently surprised by production.
The metrics that ask whether the answer was supported by the source rather than invented. Definitions vary between tools, so learn what each is actually computing before quoting the number to anyone.
For agents, the question is not whether the text reads well but whether the job got done and the right tools were called with the right arguments. A different measurement problem to text quality, and usually a more tractable one.
Test the retriever separately from the generator. If the right document never arrived, no amount of prompt work fixes the answer — and this split is what makes RAG debugging systematic rather than superstitious.
How documents get cut up decides what can ever be retrieved, and citation checking is the cheapest strong signal that an answer was actually grounded. Practical, high-yield testing work.
Judging the path, not just the destination. An agent that reaches the right answer after fourteen wasted calls and one dangerous one has not passed. Learning to assert over a sequence of steps is the core new automation skill here.
Agents call the wrong tool, call the right one with wrong arguments, and get stuck repeating themselves until the budget runs out. All three are testable, and all three are common enough to deserve dedicated cases.
A user typing instructions that override the system prompt. The canonical LLM vulnerability, still unsolved in the general case, and the first thing to try against any feature that takes free text.
The dangerous one. Instructions hidden in content the system retrieves — a document, a web page, a ticket comment — so the attacker never touches your interface. Any RAG or agent feature that reads untrusted content needs testing for this.
Getting a model past its own guardrails, and getting it to leak what it was given — system prompts, other users’ context, retrieved documents. Techniques evolve constantly; the categories are stable enough to test against.
The question of what an agent could do at its worst, not what it usually does. Permissions, spend limits, irreversible actions and human checkpoints. Mostly a design review, and testers are unusually well placed to run it.
The shared vocabulary for AI security risk, refreshed for 2026 from real incident data. Learn the list, then use it as a test charter — it is the closest thing this field has to a standard checklist.
The companion list for systems that act: planning, tool use, identity, memory, inter-agent communication, rogue agents. It extends the LLM list rather than replacing it — agents inherit every model-level risk and add their own.
Structured adversarial testing: a charter, an attack taxonomy, a record of what worked. The closest thing in this layer to exploratory testing, and the place a strong manual tester has an immediate advantage.
Blocking harmful output is half the problem. A model that refuses reasonable requests fails users just as surely, and it fails quietly because nobody files a bug saying "it was too careful". Test both directions.
The provider ships a new version and your carefully tuned behaviour shifts — sometimes better, sometimes not, always differently. This is the AI equivalent of a dependency upgrade with no changelog, and the eval suite is the only defence.
What you may need to document, and by when. The EU timeline has already shifted once, so learn the shape of the obligations rather than memorising dates — and note that most of the evidence they ask for is testing evidence.
What stays scarce once writing tests gets cheap. The actual answer to “will AI take my job”.
Deciding where to spend testing effort, writing it down, and defending it. The skill that separates someone who executes a plan from someone who makes one — and the one that survives every change in tooling.
Turning "there are eleven open bugs" into "here is what could go wrong at launch, how likely it is, and what it would cost". Quality decisions get made in that language, and testers who cannot speak it get overruled by people who can.
Your bug reports, specs and prompts are now the interface to both your colleagues and your tools — a vague spec produces a vague test whether a person or a model writes it. Clear writing became a technical skill.
The cheapest bug is the one argued out of existence in a meeting. Getting into the room, asking the questions that surface unstated assumptions, and doing it without becoming the person who blocks everything.
When developers write most of the tests, your leverage is making their tests better — through review, pairing and patterns rather than by writing more yourself. The shift from doing quality to enabling it.
Treating the tooling, fixtures, environments and pipelines as a product with users who are your colleagues. The clearest path from senior tester to a role that scales past your own hands.
Where the roles are heading and what each actually asks for day to day. Honest about which are growing, which are consolidating, and what the AI-quality speciality currently pays for.
Credentials matter less each year; demonstrable work matters more. A public repository, an eval suite someone can run, a written breakdown of a bug you found. Especially true for the AI layers, where nobody has a certificate yet.
The roadmap is a living document, not a snapshot. New topics are added as the field moves, and guides are revised when the thing they describe changes — which, in the AI layers, happens often. Anything published in the last month is flagged New; anything revised in the last two, Updated. The version stamp moves once a year.