L6Core4 min read

Cost, latency and determinism in CI

Where enthusiasm meets the invoice. Model calls in a pipeline are slow, priced per token and not reproducible — three properties CI is built to reject. Knowing how to place them anyway is what makes the difference between a demo and a practice.

A pipeline wants three things from every step: fast, free, and the same answer every time.

A model call offers none of them.

Latency. A call takes seconds, sometimes tens of seconds. Multiply by a suite and the pipeline stops being something people wait for.

Cost. Priced per token, so the bill scales with how often the suite runs. A suite on every pull request can quietly cost more than the tool it tests.

Determinism. The same input gives a differently worded answer, so a naive assertion fails at random and teaches everybody to ignore red.

None of that means keep AI out of CI. It means place it deliberately, and the placement is the skill.

The terms you will hear

Token cost. Charged per input and output token, at different rates.
Latency, p95. How long the slowest calls take. The number that decides pipeline patience.
Caching. Reusing a previous response for an identical request.
Rate limit. The provider's cap on calls per minute. Hit it and a whole run fails at once.
Threshold gate. Blocking on a score rather than on a pass or fail.
Nightly, or scheduled run. Where expensive checks live instead of on every commit.

Where to put what

On every commit, cheap and deterministic. Shape checks, forbidden-content checks, parser tests, anything that needs no model at all. Most of what people call AI testing is actually this, and it belongs here.
On every pull request, a small sample. Ten to twenty eval cases, cached where possible. Enough to catch an obvious regression in prompt or retrieval.
Nightly, the full eval set. The whole golden dataset, several runs per case, with the score recorded and compared.
On demand, the expensive ones. Red-team suites, long agent trajectories, anything measured in minutes rather than seconds.
Never in the blocking path, the noisy ones. A judge with 80 per cent agreement makes a poor gate. Report it, watch it, and gate only once it has stopped surprising you.

Why it matters

Because the failure is not technical, it is social. A slow, expensive, flaky pipeline step gets disabled, and then nothing is tested at all.

For example, a team added 120 eval cases to every pull request. Runs went from four minutes to nineteen, the monthly bill arrived at four figures, and two cases failed randomly most days. Within three weeks somebody added a skip flag "temporarily". Nobody removed it. The evals still existed and had stopped protecting anything.

The same 120 cases, split across commit, pull request and nightly, with caching and a spend cap, would still be running.

A worked placement

For example, here is a real split, with the numbers that justified it.

ci-placement.txt

BEFORE  all 120 eval cases on every pull request
  pipeline time        4 min  ->  19 min
  cost per PR run      about 1.40 USD, roughly 60 runs a day
  monthly bill         about 2,500 USD
  random failures      2 to 4 per run
  outcome              skipped after 3 weeks

AFTER  same 120 cases, placed by cost and stability

  every commit (no model calls at all, 40 seconds)
    JSON shape valid on 30 stored responses
    forbidden-content regexes on the same 30
    prompt template renders and stays under the token budget
    parser handles the 12 malformed responses we have collected
    -> free, instant, and catches most real breakage

  every pull request (18 cases, cached, 90 seconds)
    the 12 must-refuse cases      gate: all must pass
    6 highest-risk quality cases  gate: 5 of 6
    cache hit rate about 70 percent on a normal day
    cost per run about 0.06 USD

  nightly (all 120 cases, 5 runs each, 22 minutes)
    reported as a score, not a gate
    compared with yesterday, and a drop of more than 5 points pages
    the on-call
    cost about 9 USD a night

  weekly (red-team suite, 40 minutes)
    run on a schedule, reviewed by a person

  GUARDRAILS
    spend cap on the API key, alert at 70 percent
    per-run token budget, the job fails if exceeded
    provider rate limit respected with a concurrency of 4

  RESULT
    pull request time 4 min -> 5 min 30
    monthly bill about 320 USD
    random failures on the blocking path: zero in 6 weeks
    still running today, which is the only metric that matters

The line to notice is the first block. Most of what the team wanted from those evals is delivered by checks that use no model at all, and those run in forty seconds for nothing.

How to show you know it

A placement table. Which checks run where, and why. It reads as engineering judgement rather than enthusiasm.
A no-model-needed list. Showing that shape, forbidden content and parser checks cover a large share is the most useful insight here.
A cache hit rate. Cheap to measure, and it turns a cost argument into arithmetic.
A spend cap you set before the bill. Nobody regrets this one.

Questions

Should model calls ever block a merge?

Yes, for the stable ones. Safety and refusal cases should be near-perfect and rarely wobble, so they make good gates. Quality scores from a judge are better reported than enforced until you trust them, which is the rollout in evals.

How do I stop random failures on the blocking path?

Only gate on properties that do not vary, run each gated case more than once, and require a rate rather than a single pass. Anything still flaky after that belongs off the blocking path.

Is caching honest, given the model might answer differently?

For a pipeline, yes, and it is the point. You are checking that your code and prompts behave, not resampling the model. Bypass the cache on the nightly run, where variation is what you want to measure.

How do I estimate the cost before committing?

Count tokens on ten representative cases, multiply by the published rate, then by how often the suite will run. It is arithmetic, and it takes fifteen minutes.