An Open Source QA Stack for a Team of Six: The Joins Nobody Writes Down

Picture a team of six. One product, a 900 case test suite, a release every Friday, and nobody whose job title includes the word ops. Somebody on that team has spent an evening searching for the best open source QA stack and found five separate lists, each naming the same five products. None of them says what happens when you actually wire those five products together. This post is not another list. It is one setup, built for that exact team. It covers the parts that usually get left out: how a case becomes a run, how a run becomes a result, how a result becomes an issue, and how that issue finds its way back to the case that should have caught it sooner. It is one defensible setup among several, not the only correct one, and it says so honestly.

The team this is written for

Six people. One product. A 900 test suite that nobody has fully read end to end in months. Weekly releases on a fixed cadence, usually Thursday or Friday. No dedicated infrastructure or ops person, so whoever sets this up also has to be the one who keeps it running, quietly, alongside their actual job.

That last constraint changes every decision below. A team of sixty can afford a platform engineer to babysit four self hosted services. A team of six cannot. Every choice in this post gets weighed against one question. Who answers the page at 6pm on a Thursday when the release is already running late.

The four jobs, not the four products

It helps to stop thinking in product names for a moment and think in jobs instead. A QA stack, no matter which vendor or open source project fills each slot, has to do four things:

• Write and keep test cases: the actual documented steps, expected results, and history of who changed what
• Run tests: execute the suite, whether that is a human clicking through steps or a script in CI
• Track issues: capture what went wrong, in enough detail that someone can act on it
• Report what happened: turn a pile of results into something a release manager can read in two minutes

Anyone can name a tool for each job. Kiwi TCMS or TestLink for cases, a CI runner for execution, an issue tracker for defects, a dashboard or spreadsheet for reporting. The naming part is easy. The part that actually determines whether this stack survives contact with a real Thursday release is what happens between the boxes.

The joins are the actual content

Here is where most self assembled stacks quietly fail. It is rarely the tools themselves. It is the four seams connecting them.

Case to run. When someone kicks off a run, does the tool know which cases belong in it. Or does a person have to remember and select them by hand every single time. On a 900 case suite, hand selection means someone eventually forgets a case. It slips out of a release without anyone noticing until a customer finds the gap.

Run to result. A result has to land against the specific case and the specific run, with a timestamp. It cannot just be a message posted to a chat channel that scrolls away. If results live only in a build log, nobody can answer whether a feature was tested in the last three releases without an archaeology project.

Result to issue. A failing result should create or link to an issue automatically, or close to it. The alternative is someone manually retyping the failure into a tracker a day later. That is where detail gets lost and where the same bug gets logged twice by two different people.

Issue back to case. This is the join everyone forgets. When a bug is fixed, does anything point back to the test case that should have caught it. Without that link, the case never gets strengthened, and the same defect reappears two releases later. Without this loop, the suite never actually learns from its own misses.

Each of these four joins is usually a webhook, an API call, or a shared identifier field between two open source projects that were never designed with each other in mind. That gluing work, more than any single tool choice, is what a team of six is actually signing up for.

What this actually costs to run yourself

Someone should say the number out loud before adopting this. Running four self hosted services, patching them, backing them up, and gluing the joins together realistically costs a team of six somewhere between 8 and 15 hours a month once the initial setup is done. That is not a one time cost. It recurs every month, quietly, whether or not anyone is tracking it.

Where do those hours actually go. A rough breakdown for a typical month looks like this:

• Two to four hours on security patches and version upgrades across the four services
• One to two hours resolving a broken webhook or a failed sync between two of the joins
• Two to three hours on backup verification, because a backup nobody has restored is not a backup
• Two to four hours answering the inevitable question of why a case did not show up in last week's run

The honest question to ask in a planning meeting is not whether the team can self host this. It is who, by name, owns it. Not a team, a person. If nobody in the room can answer that question with a name, the stack does not have an owner. It has a hope.

Do not self host every layer

The honest recommendation, and the one that gets pushback in planning meetings, is to not self host all four jobs. Pick the one or two layers where control genuinely matters, usually the case data itself because that is your institutional knowledge, and pay for the rest.

A six person team paying for a hosted issue tracker and a hosted CI runner while self hosting only the case management layer is a completely reasonable setup. It concentrates the operational burden on the piece that matters most. It also buys back the hours everywhere else.

Self hosting everything because it is technically possible is how a team of six spends fifteen hours a month on server maintenance instead of testing. That trade rarely gets made on purpose. It usually happens by accretion, one self hosted service at a time, until nobody remembers deciding to run four of them.

Think about it the way a team already thinks about its product infrastructure. Nobody self hosts every dependency in a codebase just because the source is available. Some libraries get vendored because control matters. Most get pulled from a registry because the maintenance is somebody else's job. A QA stack deserves the same reasoning.

Where AI fits, and the question it raises

Several of these tools, and most commercial alternatives, now offer an AI layer that drafts cases or suggests coverage. The moment you add that layer to a self hosted stack, a new question appears. Whose key is calling which model, and where does your test case content go when it does.

That question deserves its own answer rather than a rushed paragraph here. We have covered it in detail in a separate post on bring your own key arrangements for self hosted teams, and the short version is that the arrangement matters as much as the feature itself. A self hosted stack that quietly ships case content to a third party model provider has not actually kept the data where it looked like it was staying.

What to do first

A team of six cannot adopt four new tools in one quarter alongside a weekly release schedule. Pick the join that hurts the most today, usually case to run or result to issue, and fix that one connection before touching anything else. Everything else in this post can wait a release or two.

Write down, this week, who owns each of the four jobs and each of the four joins. That single document, more than any tool choice, is what will still be true in six months.