API-infrastruktur og backendutvikling

Norsk

API

Designing p99 latency budgets for API gateways

An API that answers in 40 ms on average can still feel slow if one request in a hundred takes two seconds. At scale, that one request is not rare: a page that fans out to twenty calls hits the slow tail on almost every load. Latency budgets turn this into an engineering constraint you can design and test against.

Publisert 7 min lesetid

Dette innholdet er foreløpig kun tilgjengelig på engelsk.

Why the tail matters more than the mean

Latency distributions for network services are skewed. Most requests complete quickly, but a long tail is caused by cache misses, lock contention, garbage-collection pauses, slow disks, noisy neighbours and retries. The arithmetic mean blends these together and moves very little when the tail gets worse.

Users experience the tail directly. A single page or mobile screen often triggers many API calls; if any one of them is slow, the whole interaction is slow. The probability that at least one of n independent calls is slower than the p99 is 1 − 0.99^n. For 20 calls that is roughly 18%, for 100 calls about 63%. Dean and Barroso described this effect in The Tail at Scale, and it is the reason serious latency objectives are expressed as percentiles.

For a gateway, the consequence is simple: optimise and measure the p99 (and for high-volume routes the p99.9) per route, and treat the median as a health signal rather than a target.

From an SLO to a per-route budget

A latency budget starts from a service-level objective that matters to the business, for example “99% of POST /orders requests complete within 300 ms at the edge, measured over 28 days”. The gateway team then allocates that 300 ms across everything on the request path:

Stagep99 allocationNotes
TLS and connection handling15 msAssumes connection reuse; new handshakes are tracked separately
Authentication and authorisation10 msLocal JWT verification with cached keys
Rate limiting5 msShared counter store in the same zone
Schema validation10 msBody size capped
Network to upstream10 msSame region
Upstream service220 msIncluding its own dependencies
Reserve30 msAbsorbs variance; never allocated permanently

Percentiles do not add up exactly – the p99 of a sum is not the sum of p99s – so the table is an engineering allocation, not a statistical identity. It still works as a contract: each stage owner knows their share and is alerted when they exceed it.

Where gateways lose time

In practice, the gateway’s own stages rarely consume much of the budget when they are configured well. Tail latency usually comes from a few recurring sources:

  • Remote token introspection. Calling an authorisation server for every request adds a network hop and couples your tail to theirs. Local verification of signed tokens, with key caching and short token lifetimes, removes it.
  • Connection churn. Opening new upstream connections under load adds handshake time and can exhaust ephemeral ports. Keep pools warm and size them to peak concurrency.
  • Unbounded retries. A retry after a timeout doubles the time spent and adds load to an already struggling service. Retries need a per-request budget and should apply only to idempotent requests.
  • Queueing. When a worker pool is saturated, requests wait before any work starts. Queueing time is invisible in upstream metrics but obvious at the edge.
  • Large payloads. Parsing and validating multi-megabyte bodies on the hot path blocks event loops and workers. Cap sizes per route.

Enforcing the budget with timeouts

A budget that is not enforced is only a hope. Each hop needs a timeout derived from the remaining budget, and the gateway should propagate a deadline so that downstream services can stop work that can no longer complete in time. gRPC carries deadlines natively; for HTTP, a header such as a request deadline or remaining-time value, agreed between teams, serves the same purpose.

Timeouts should decrease along the call chain. If the gateway gives up after 300 ms but the upstream keeps working for two seconds, capacity is wasted on answers nobody will receive. Pair timeouts with circuit breakers so that a failing dependency is shed quickly instead of consuming the budget of every request.

Hedged requests – sending a second copy of a request after the p95 has elapsed and taking whichever answers first – can cut the tail for read-only, idempotent calls. They increase load, so cap them to a small percentage of traffic and never use them for writes without idempotency keys.

Measuring correctly

Measure at the edge, per route and per client class, using histograms rather than pre-computed percentiles so that values can be aggregated across instances. Record queueing time separately from service time. Load tests must use an open workload model – requests arrive at a fixed rate regardless of how fast the system answers – otherwise a slow system slows the test down and hides its own tail; this distortion is known as coordinated omission.

Finally, compare the budget with production data continuously. A budget that is met in a test but missed every Monday at 09:00 is telling you about capacity or batch jobs, not about the gateway.

Budgets for different classes of traffic

Not every caller needs the same promise. Interactive clients, partner integrations and batch jobs share the gateway but have different tolerances. A common pattern is to classify traffic at the edge – by client credential, route or a priority header that only trusted callers may set – and give each class its own objective and its own capacity.

Batch and back-office traffic should be rate-limited separately and, where possible, routed to separate upstream pools so that a nightly export cannot push interactive requests into the tail. Partner traffic often deserves a stricter, contractual objective with its own dashboards, because partners notice degradation before your internal monitoring does. When capacity is short, shedding the lowest class first – with a clear 429 or 503 and a Retry-After value – protects the budget of the traffic that matters most.

Eksempel: A checkout route that missed its p99

A route with a 300 ms p99 objective measured 280 ms in tests but 900 ms in production during peaks. Edge histograms showed that authentication occasionally took 400 ms: the gateway called a remote introspection endpoint whose own p99 degraded under load. Switching to local JWT verification with keys cached for ten minutes and a five-minute token lifetime removed the remote call. A second issue was a retry policy of three attempts on timeouts for a non-idempotent POST; retries were disabled for that route until idempotency keys were introduced. The p99 returned to 260 ms at peak, with the reserve intact.

Vanlige fallgruver

  • Setting objectives on the average or median and declaring success.
  • Computing percentiles per instance and then averaging them across the fleet.
  • Closed-loop load tests that suffer from coordinated omission.
  • Retrying non-idempotent requests, or retrying without a per-request budget.
  • Upstream timeouts longer than the gateway timeout, so abandoned work continues.

Spørsmål og svar

Should every route have the same budget?

No. Budgets should reflect what the route does and what the user is waiting for. A search suggestion and a batch export have very different needs; give each route its own objective.

Is p99 enough, or do we need p99.9?

For routes with high volume or heavy fan-out, p99.9 reveals problems that p99 hides. Start with p99 everywhere and add p99.9 where traffic and impact justify it.

Do hedged requests make sense for writes?

Only when the write carries an idempotency key and the service guarantees that duplicate requests have a single effect. Otherwise hedging can create duplicates.

Sjekkliste

  • Define a percentile-based objective per critical route.
  • Allocate the budget across stages and keep a reserve.
  • Verify tokens locally; avoid remote calls on the hot path.
  • Derive timeouts from the remaining budget and propagate deadlines.
  • Restrict retries to idempotent requests with a retry budget.
  • Record histograms at the edge and separate queueing time.
  • Load-test with an open workload model at expected peak.

Primærkilder

  • Dean, J. and Barroso, L. A., “The Tail at Scale”, Communications of the ACM 56(2), 2013
  • RFC 9110 – HTTP Semantics (idempotent methods, status codes)
  • RFC 7519 – JSON Web Token (JWT)
  • Beyer, B. et al. (eds.), Site Reliability Engineering, O’Reilly, 2016 – chapters on service level objectives

Alle artikler