Sample Size in API Benchmarks: When to Show Results

Sample size in API benchmarks decides whether a measured figure is useful evidence or a precise-looking accident. A provider should not be ranked on one quick response, and a buyer should not be shown a success rate that has not survived enough representative calls.

The difficult part is that no universal request count solves the problem. Ten calls can reveal a broken integration. Ten thousand calls can still describe the wrong workload. A defensible benchmark connects the count to the metric, the request population, the observation window and the amount of uncertainty the buying decision can tolerate.

A sample count is necessary but not sufficient

Every benchmark figure comes from a set of observations. The count tells the reader how much evidence sits underneath the result, but it does not say whether that evidence represents future requests.

Suppose an endpoint completes 500 identical cached lookups from one region. That sample may describe cached lookups on that route well. It does not establish latency for large payloads, cold connections or another region. Adding another 500 identical calls narrows uncertainty around the same limited workload; it does not broaden the claim.

Before a figure is eligible for display, the measurement record should identify:

• The number of qualifying calls and excluded calls.
• The start and end of the observation window.
• The request classes, input shapes and response conditions represented.
• The measurement location and route.
• The timeout, retry and failure treatment.
• The statistic to be shown and the uncertainty rule applied to it.
• Any endpoint or schema change that made older observations ineligible.

These fields turn a count into provenance. They also make it possible to distinguish a thin sample from a biased one.

Latency and success rate need different evidence rules

Latency is a distribution. A median describes its middle, while p90 or p95 describes a slower part of the observed requests. Tail percentiles need observations in the tail. If only a few calls support that region of the distribution, the label can imply more stability than the data provides.

This is why measured API latency needs context beyond a millisecond value. The display rule can require a minimum qualifying count, coverage across an observation window and a stability check between consecutive windows. The exact threshold is a methodology choice, not a universal constant. It should be declared rather than guessed after seeing the result.

Success rate estimates a proportion. A record of nine successes from ten calls produces a percentage, but the next call can move it sharply. As the qualifying count grows, a confidence interval around the rate usually narrows. The eligibility rule should therefore bind the visible figure to a tolerated interval width or another declared uncertainty test.

The definition of success must also stay fixed. A transport-level response, a valid schema and a task-acceptable result are different outcomes. Apiosk measures success rate for listings it has proxied enough calls of, but it does not measure result quality. API success-rate measurement must not be presented as proof that the returned content was correct or useful.

Use a predeclared display rule

A benchmark becomes vulnerable when the operator decides whether to publish after seeing which provider won. The safer sequence is to define eligibility first, collect observations second and calculate the comparison only after the rule passes.

A practical display rule can combine four gates:

• Count: enough qualifying observations exist for this metric.
• Coverage: observations span the request classes and time window named by the methodology.
• Comparability: candidates were observed through a sufficiently consistent path and workload.
• Stability: adding a recent batch does not move the reported statistic beyond the declared tolerance.

None of these gates needs to promise that the next call will match the history. They establish a narrower claim: the displayed figure has enough relevant evidence for the benchmark's stated use.

Set the rule per metric. A coarse median used to filter obviously slow candidates can tolerate different uncertainty from a p95 used to enforce a tight latency ceiling. A low-stakes catalogue preview can use a different maturity state from a production routing decision. What matters is that the rule is visible, consistent and applied before the outcome is known.

Show the state before the number qualifies

An immature measurement should not leave an unexplained blank. The response should say that the dimension is unmeasured or has insufficient qualifying history, then preserve the observations for later evaluation without turning them into a ranking input.

Do not replace the missing figure with zero. Zero is a measured value and would penalize the provider for the comparison layer's lack of evidence. Do not substitute the provider's advertised number into a column of proxied measurements. The timing boundary and workload may differ. Do not copy an average from the candidate set; that awards evidence the endpoint has not earned.

The correct treatment is the same principle used for unmeasured dimensions in API comparison: name the omission and drop it from the weighting. The remaining score is narrower because it uses fewer dimensions. Its weights and each candidate's contribution must travel with it so a buyer can see exactly what produced the result.

This interim state is not a verdict on the endpoint. It is a verdict on the evidence. Providers can improve the state by keeping schemas current and allowing representative routed traffic to accumulate. Buyers can proceed with the narrower comparison, run a controlled evaluation or require more evidence before spending.

Large samples still need a fair comparison

Count does not repair mismatched conditions. One provider measured during quiet hours and another during a traffic peak do not form a clean head-to-head benchmark. Neither do a small lookup request and a large generation request merely because both use HTTP.

Neutral comparison matters because API providers cannot compare themselves under one shared observation path. Each provider sees its own infrastructure and selects its own reporting boundaries. A comparison layer sees routed buyer calls, but it must still describe what those calls represent and keep external or incompatible evidence out of the score.

Freshness matters too. An endpoint release, routing change or revised input schema can make an old sample less representative. A rolling window can remove stale evidence, but it may also reduce the qualifying count. The honest result can move from measured back to insufficient. Measurement maturity is a state to maintain, not a badge earned forever.

Where Apiosk fits

Apiosk uses measured latency and measured success rate only for listings it has proxied enough calls of. When a candidate lacks sufficient measurement, the dimension is dropped from the weighting and named rather than scored zero. Price, settlement rail and input compatibility remain available where the listing and request support them.

`GET /v1/compare?candidates={id},{id}` returns a 0–100 score relative to the candidates in that one comparison. The score is never returned alone: the active weights and each candidate's contribution per dimension accompany it. Result quality and provider terms are not measured anywhere and cannot be inferred from a larger operational sample.

If you operate an endpoint, list your API on Apiosk with accurate inputs and current gateway terms. Representative proxied calls can then build the evidence needed for measured dimensions. Until they do, the comparison keeps the gap visible instead of manufacturing certainty.