Thinking in percentiles, not averages
An average response time hides the users having the worst experience, and those are the ones who leave. p95, p99, and why tail latency is where the real story lives. Short topic, changes how you read every performance result afterwards.
An average response time is one of the most confidently misleading numbers in software. It tells you almost nothing about the users having the worst experience, and those are exactly the users who leave a bad review, abandon a cart, or quietly stop opening the app. This is a short topic, but it changes how you read every performance test result you look at afterwards, from a k6 load test to a production dashboard.
Why the average lies
Imagine a hundred requests to an endpoint. Ninety-five come back in 100 milliseconds. Five come back in 3 seconds, because they hit a cold cache or a lock contention edge case. The average across those hundred requests is around 240 milliseconds, a number that looks perfectly healthy on a dashboard.
But nobody experienced 240 milliseconds. Ninety-five people had a fast response, and five people waited three seconds. The average describes a request that never actually happened. It is a statistical artifact, not a user experience.
This matters more as traffic grows. At web scale, "5% of requests" is not a rounding error, it is thousands of real people every hour having the slow experience while your dashboard reports green.
Percentiles describe the distribution, not a fiction
A percentile answers a different, more honest question: what value did N% of requests fall at or below. The 95th percentile, written p95, is the response time that 95% of requests were faster than. The remaining 5% were slower.
The higher the percentile, the closer you get to describing your worst experience rather than your typical one. That is deliberate. A typical experience takes care of itself. A team that only tracks p50 will ship a system that feels fast in every demo and slow to a meaningful slice of real traffic, and never see it in their own metrics.
Why the tail is where the real story lives
Tail latency, the far right edge of the distribution, is not noise to be averaged away. It usually points at a specific, fixable cause: a cold cache path, a database query that occasionally scans instead of using an index, a downstream service with its own bad tail, a garbage collection pause, a lock that only contends under concurrent load.
Chasing p50 down from 100ms to 80ms rarely changes how anyone feels about your product. Chasing p99 down from 3 seconds to 400ms usually does, because that is the tail your most active, most valuable users are most likely to hit repeatedly.
This is also why a performance test that reports only an average is close to useless as a release gate. A threshold like p(95)<500 catches a regression that an average would smooth over completely.
Setting thresholds that mean something
A good performance threshold is expressed as a percentile, not an average, and picked with the user experience in mind rather than a round number that feels safe.
export const options = {
thresholds: {
http_req_duration: [
'p(50)<200', // typical request stays snappy
'p(95)<500', // the tail stays acceptable
'p(99)<1500', // even the worst case has a ceiling
],
http_req_failed: ['rate<0.01'],
},
}Setting three thresholds instead of one is deliberate. p50 catches a general slowdown across the board. p95 catches a tail that is creeping wider even while the typical request still looks fine. p99 catches the rare but real worst case, the one a support ticket eventually gets written about.
Reading a distribution, not a single number
When you look at a performance result, resist the pull of a single headline figure. A histogram or a percentile table tells you the shape of the experience. A tight cluster with a short tail is healthy. A tight cluster with a long thin tail stretching out to several seconds means a specific subset of requests is hitting a specific problem.
That shape is often more diagnostic than the number itself. A p99 that is ten times the p50 says: something specific and reproducible is going wrong for a minority of requests, go find it. A p99 that is twice the p50 says: the system degrades smoothly under load, which is a much healthier signal even if the absolute number is similar.
FAQ
Questions people ask
Why not just use p99.9 or p100 to be extra safe?
Extremely high percentiles get noisy fast, one slow outlier from a bad network blip can dominate them, and p100 is just your single slowest request, which tells you almost nothing repeatable. p95 and p99 are the sweet spot: high enough to catch real tail problems, stable enough to trust run over run.
How many requests do I need before a percentile is meaningful?
As a rule of thumb, you need at least 20 requests to say anything about p95, and more like 100+ for it to be stable, because p95 of a 20-request sample is just your second-worst request. p99 needs proportionally more, at least a few hundred requests, before it stops being dominated by noise.
Should I track percentiles outside of performance testing too?
Yes. Production observability tools (Grafana, Datadog, and similar) should show p50/p95/p99 on every key endpoint by default. The same principle that makes averages misleading in a load test makes them misleading on a live dashboard.
What's the difference between latency percentiles and load profiles?
Percentiles describe how response times are distributed within one test. Load profiles describe the shape of traffic you throw at the system to produce that distribution: soak tests for leaks over hours, spike tests for launch traffic, stress tests for finding the breaking point.