CI/CD for test suites

Continuous integration (CI) is the practice of merging often and having every merge automatically built and tested. Continuous delivery (CD) extends that so every passing build is releasable.

For a tester, the pipeline is where your work either protects a release or gets quietly skipped. Three decisions matter, and they are yours to influence.

What blocks. Which failures stop a merge or a deploy, and which only report.

What runs where. Fast checks on every commit, slower ones later, expensive ones on a schedule.

Who fixes. A red pipeline needs an owner, or it stays red until somebody switches the notification off.

The terms you will hear

• Stage, or job. One step in a pipeline, such as lint, unit, integration.
• Gate. A stage whose failure blocks progress.
• Required check. A gate enforced by the repository, so a pull request cannot merge without it.
• Artifact. Something a stage produces and passes on, such as a build or a test report.
• Fail fast. Ordering so the cheapest, most likely failure runs first.
• Non-blocking, or advisory. A stage that reports and does not stop anything.

How to stage a suite

Order by cost, so a broken build is known in seconds rather than in twenty minutes.

1. Static, in seconds. Types, lint, formatting, schema validation. No execution.
2. Unit and component, in a minute or two. Cheap, precise, and where most logic defects surface.
3. Integration, in a few minutes. Real database, stubbed third parties, per mocking.
4. A handful of end-to-end, in a few more. The critical journeys only, per pyramid or trophy.
5. Nightly, everything expensive. Full browser matrix, performance, mobile real devices, red-team suites.

Why it matters

Because suite duration quietly sets team behaviour.

For example, a pipeline taking 35 minutes produces larger, less frequent merges, because nobody wants to wait twice. Larger merges are harder to review and harder to diagnose when they break. Cut the same pipeline to six minutes and merges get smaller, which makes every failure easier to attribute.

That is why suite duration is not a testing detail. It shapes how the team works, and arguing for it is a legitimate part of the job.

What good practice looks like

1. Publish the report as an artifact, so a failure can be read without re-running anything.
2. Keep traces and screenshots on failure only. Enough to diagnose, no storage cost on green runs.
3. Make one retry visible. Retries are acceptable while flakiness is being fixed, and only if they are counted, per flakiness.
4. Name an owner for red. Whoever merged, by default. A pipeline everybody owns is one nobody fixes.
5. Separate infrastructure failures from test failures. A farm timeout is not a defect and should not read like one.
6. Set and defend a duration budget. For example ten minutes to merge. Adding a test then means removing one.
7. Run the same commands locally. If a developer cannot reproduce a CI failure with one command, the pipeline is a black box and diagnosis costs hours.

A worked pipeline

For example, here is one small team's layout after a clean-up.

The thing to notice is that the suite did not shrink. It was re-ordered, and only four stages kept the right to block.

How to show you know it

• A staged pipeline with three gates. Being able to explain what earns the right to block is the core judgement here.
• A duration budget. "Six minutes to merge, so a new test displaces an old one."
• A separated infrastructure category. It keeps failure numbers honest.
• A merge frequency change. Showing that a faster pipeline changed how the team works is the strongest argument you will ever make for suite hygiene.