Latency Percentiles for API Comparison: Why the Median Misleads

Latency percentiles for API comparison look like a reporting detail until two providers swap places when you change which percentile you sort by. The median describes a typical call. An agent that makes many calls per task does not experience typical calls. It experiences its slowest one, and that one comes from the tail.

A median describes the call you rarely notice

The median, or p50, is the latency half of all calls beat. It is stable, easy to estimate from a modest sample, and a fair description of what one call usually costs in time. It says nothing about the other half.

The higher percentiles describe the slow side of the distribution. A p95 is the latency that 95 percent of calls beat, so one call in twenty is slower. A p99 leaves one call in a hundred above it. Between the median and the p99 of a single endpoint, the gap can be large: cold starts, queueing behind a slow upstream, a retry inside the provider, or a garbage-collection pause all land in the tail without moving the median at all.

For a single interactive call, the median is a reasonable summary. Agent workloads are rarely a single call.

Chained calls make the tail the common case

An agent task typically chains several tool calls, and the slowest of them sets the pace. The arithmetic is simple. If each call independently has a five percent chance of exceeding its p95, the chance that at least one call in a task does is:

• One call: five percent.
• Five calls: about 23 percent.
• Ten calls: about 40 percent.
• Twenty calls: about 64 percent.

At p99 the same task of twenty calls still has roughly an 18 percent chance of at least one call landing in the slowest one percent. These figures assume independent calls. Real slowdowns are often correlated, because one overloaded upstream slows many requests at once, which changes the shape of the problem but rarely makes it smaller.

The consequence for comparison is direct. For a workflow of many calls, a provider's p95 is not an outlier it will occasionally show. It is something the task will hit on a large share of runs.

Rankings can flip with the percentile

Two candidates can order differently depending on which point of the distribution you compare.

One provider might have a fast median and a long tail: most calls return quickly, a few take many times longer. Another might be a little slower on a typical call but tightly bounded, with almost no gap between its median and its p99. Sorted by median, the first wins. Sorted by p99, the second wins. Neither ranking is wrong. They answer different questions, and the buyer has to decide which question its workload is asking.

This is why a latency number in a comparison is incomplete without its percentile, in the same way it is incomplete without its vantage point. The question of what the clock includes and the question of which percentile is reported are separate, and both change the result.

Which percentile belongs behind which decision

Match the statistic to the way the workload waits.

• Estimating throughput or the typical cost of waiting: the median is the right input.
• A hard per-call ceiling on a step that a user or another process waits on: a tail percentile, usually p95, and p99 when the task chains many calls or the deadline is strict.
• A deadline for the whole task: divide it into a per-call budget first, then compare candidates on the tail percentile that matches the number of calls.
• Batch work nobody waits on: compare on the median for cost and throughput, and use the tail only to set a timeout.

A hard ceiling is a filter, not a preference. It should remove candidates before price is considered, which is the pattern described in constraint-based API filtering. The point here is narrower: a ceiling set against a median will pass candidates that routinely break it in practice. The same logic applies when the ceiling comes from a contract, which is where SLA-aware provider decisions start from.

Tail percentiles need evidence in the tail

The higher the percentile, the more calls it takes to estimate. A median can be estimated from relatively few observations. A p99 needs enough calls that the slowest one percent contains more than a handful of them, or a single slow request can move it sharply.

That makes tail comparisons the first place thin data misleads. A new provider with a short history may show an excellent p99 simply because none of its few calls happened to be slow. The rules in sample size in API benchmarks apply with extra force here: a comparison should say when a tail percentile is not yet supported, rather than reporting a number that looks precise and is not.

Before trusting any latency figure, ask four things of it:

• Which statistic it is: mean, median, p95 or p99.
• Where it was measured: at the provider, at a gateway, or at the buyer.
• Over what window it was collected.
• How many calls it rests on, and whether that is enough for the percentile shown.

Where Apiosk fits

Apiosk measures latency only for listings it has proxied enough calls of, and reports it in `/v1/compare` beside price, measured success rate and input compatibility, each scored 0–100 relative to the candidates in that comparison. Every response carries the weights that produced the score and each candidate's contribution per dimension. Where a candidate has not been measured, latency is dropped from its weighting and named rather than scored as zero.

A buyer sets a hard ceiling once with `max_latency_ms`, and `/v1/decide` removes any candidate that exceeds it, naming that constraint as the reason. Preferences such as `optimize_for=latency` change the ordering among the survivors, not the filter. Before treating a ceiling as a tail guarantee, confirm which latency statistic a comparison reports for each candidate. The full parameter list is in the Apiosk documentation.