Who Runs the AI When You Self Host Your Test Cases

A head of quality self hosts a test management tool for one clear reason: so the data stays on infrastructure the company controls. Then a security reviewer asks a simple question about a new AI feature that drafts test cases, and the honest answer turns out to be uncomfortable. That AI call leaves the building. It carries exactly the test case content the self hosting decision was meant to protect. This is the collision self hosted AI test cases run into, and this post settles the four arrangements a reader will meet and why bring your own key is the honest answer rather than a marketing one.

The collision nobody notices until someone asks

A team picks a self hosted tool because the data should stay put. That decision feels complete. Then a feature ships that drafts a test case from a short description, or suggests missing coverage for a flow, and it works by calling a language model somewhere. The server that model runs on is very likely not the server the team just spent a quarter self hosting everything else on.

Most teams do not notice this until a security review, a customer questionnaire, or a new head of quality asks where the AI feature actually sends data. The answer is frequently unknown even to the person who turned the feature on, because the toggle in the settings page rarely says.

Here is a concrete version of it. A QA lead at a company running a self hosted TestLink instance enables an AI assist plugin two weeks before a SOC 2 readiness review. The plugin drafts cases for a new refund flow. The auditor's first question is which provider processed those prompts, and nobody in the room knows the answer.

Three questions that settle it for any tool

For any AI feature in any tool, self hosted or not, three questions get you a real answer fast:

• Whose account is billed for the model call: the vendor's account, or yours
• Which provider actually receives the prompt: OpenAI, Anthropic, a vendor's own model, or something running locally
• Whether the feature can be switched off entirely, with the outbound call actually stopping rather than just hiding a button

A vendor that cannot answer all three in a sentence each has not thought this through, or does not want to say.

Why test case content is genuinely sensitive

It is easy to assume test case content is boring. It is not customer data, and it is not a password. But a test case describes exactly how a product behaves, step by step, including the paths that are not public yet. A case for an unreleased billing flow, or for a permission check that has not shipped, tells a reader precisely what the product does before a competitor or an attacker would otherwise know.

Think of a case titled "Verify refund exceeds original charge is blocked at the gateway, not just in the UI". That single line reveals a fraud control that has not been announced anywhere else. Handing that line to an unknown third party is a real disclosure, not a hypothetical one.

The four arrangements you will actually meet

Every AI feature in a QA tool falls into one of four arrangements, and each has a different consequence for where your data ends up:

• Vendor owned key, vendor account: the vendor calls the model using their own account, and your data flows through infrastructure and terms you did not negotiate
• Vendor owned key, billed to you: same data flow as above, except the invoice for the model usage lands on your bill instead of theirs
• Your own key, your own provider account: you provide the API key, the call goes to a provider you already have a relationship with, under terms you already agreed to
• A model running inside your boundary: no outbound call happens at all, because the model itself lives on infrastructure you control

The difference between the first two arrangements is only who pays. The data still leaves through the vendor's relationship with the model provider either way. Teams frequently assume paying the bill themselves changes the residency, and it does not.

Bring your own key, plainly

Bring your own key means you supply the credential, and the model call goes to a provider account that already exists inside a relationship your legal and security teams have already reviewed. It does not create a new third party. It routes the call through one you already trust enough to use elsewhere.

If a security team already approved sending certain data to OpenAI or Anthropic for another internal tool, routing the test case drafting feature through that same approved account adds no new vendor to the list a reviewer has to sign off on. That is the entire value of the arrangement.

We have written about this arrangement in more depth in a dedicated post, and the short version repeats here on purpose. The key matters more than the interface sitting on top of it.

The air gapped edge case

Some teams cannot make any outbound call at all, by policy or by network design. For those teams there are only two honest options. The AI feature is switched off completely, or the model itself runs inside the network boundary with no call ever leaving it. There is no third option where the call quietly happens anyway on a different port. If a vendor claims otherwise, that claim deserves scrutiny.

What to ask a vendor in writing

Ask these questions over email, not on a call, so an evasive non answer is visible in writing:

• Which specific model provider receives the prompt when this feature is used
• Can this feature be disabled at the account level, and does disabling it stop the outbound call entirely
• If we provide our own API key, does that key fully replace any vendor owned key for this feature
• Where is the prompt logged, and for how long

A vendor who answers all four in the first reply has clearly thought about this before you asked. A vendor who answers in generalities, or points only to a general privacy policy, has not.

The human line that does not move

Whatever arrangement a team lands on, one thing stays constant. A generated test case is a draft. It is not correct just because a model wrote it, and it does not enter the record until a person reads it and approves it. Bring your own key solves where the data goes. It does not solve whether the case is any good. That still needs a human, every time, on every case, before it becomes part of the documented suite that an auditor or a teammate will later rely on.