Open Source or Commercial Test Management: A Four Question Method

Somebody on a QA team has to make this call at least once a year: keep using the open source test management tool, or pay for a commercial one. The conversation usually turns into a values argument, open source good, vendor lock in bad, or a price argument, the free tool costs nothing so why pay. Both framings miss the actual question. This is a decision about which risks a team would rather hold, and it can be answered with four questions in an afternoon, in an order specific enough that a person can walk into a meeting and explain the answer to whoever has to sign off.

Why the usual framing fails

Most teams argue this as values versus price. Open source feels aligned with engineering culture, so it wins the values argument by default. Commercial tools have a subscription line item, so they lose the price argument before anyone reads the feature list. Neither framing asks the question that actually matters: which risk is the team more willing to carry, the risk of running infrastructure themselves, or the risk of depending on a vendor's roadmap and pricing.

A team that frames it as values versus price ends up defending a decision it made for the wrong reason. It defends that decision badly when a CFO or a security reviewer asks a harder question later.

Question one: is there a hard requirement

Start here because it eliminates the fastest. Is there a contract clause, a regulator, or a security review that forces test data onto infrastructure the company itself controls. If a customer contract specifies that test artifacts for their account cannot leave your network, that answer is already made. Self hosting is not a preference in that case, it is a term you already signed.

Most teams do not have this constraint. If yours does not, move to question two.

Question two: does anyone actually want to own the database

Name the person. Not a team, a person, with hours actually allocated on their calendar for patching, backups, and version upgrades. A QA lead who says the team would figure that out later has just answered the question. The honest answer is no.

A team of eight running a self hosted Kiwi TCMS instance without anyone named for this job will find out who owns it during the first outage, at the worst possible time, usually a release night.

Question three: what does leaving actually cost

Do not estimate this. Test it. During any trial of a commercial tool, or at any point with an existing open source instance, export a sample of 50 cases along with their run history and see what actually comes out. If attachments, custom fields, or run history do not survive the export cleanly, that is the real switching cost, not a number in a sales conversation.

A team that discovers during a trial that run history does not export learns something a spreadsheet estimate never would have told them.

Question four: where does the AI call go

Most tools in this category now offer an AI feature that drafts or suggests cases. Whichever tool a team picks, ask where that feature sends data and whose account is billed for it. This question applies equally to open source tools with an AI plugin and to commercial tools with a built in assistant. Neither category gets a pass on this one by default.

The arithmetic that decides the rest

Once the first three questions are answered, most remaining cases come down to hours versus dollars. Two worked examples show where the line actually sits.

A six person team spending roughly 10 hours a month on self hosted maintenance, at a fully loaded cost of 60 dollars an hour, is spending about 600 dollars a month in labor to avoid a subscription. If the commercial tool's per seat price for six seats comes in under 600 dollars a month, the arithmetic already favors paying for it, regardless of any values argument either way.

A thirty person team spread across three product lines often needs enough customization, permission structures, and integrations that a commercial tool's per seat price climbs past what ten hours of a shared ops person's time would cost. At that size, self hosting a well maintained instance frequently comes out ahead on pure arithmetic, especially if the team already has someone named for the job from question two.

The crossover point is rarely obvious in advance. It has to be calculated for the actual team size and the actual hourly cost, not assumed from a general rule either way.

The four outcomes, stated without preference

Running this method honestly produces one of four outcomes, and a credible method has to be able to reach all four:

• A hard requirement forces self hosting regardless of cost or preference
• Nobody will own the database, so a commercial tool wins by default
• The export test reveals a hidden lock in risk that changes the calculus
• The arithmetic favors paying a commercial vendor once labor hours are counted honestly

Two of these four outcomes end in paying a commercial vendor. A method that always lands on open source was never really a method. It was a conclusion looking for a justification.

The choice is reversible, if you check now

Whichever way this goes, the decision does not have to be permanent if the team checks one thing at selection time: can a full export, including run history and attachments, actually be produced on demand. A tool that passes that test today keeps the door open regardless of which option gets chosen. A tool that fails it turns today's choice into next year's migration project.