High-throughput edge routing & connection pooling
API gateways & edge infrastructureArchitectural bottleneck
Sudden concurrent webhook bursts push p99 latency far beyond budget. Each request opens a new upstream connection, so TCP and TLS handshakes, ephemeral-port pressure and blocking upstream handling pile up exactly when traffic peaks.
Engineering approach
- Terminate client connections at an asynchronous edge proxy and keep warm keep-alive pools to each upstream, sized from peak concurrency.
- Use HTTP/2 multiplexing towards upstreams that support it, so bursts share connections instead of opening new ones.
- Route on headers without buffering bodies; cap body size per route and stream payloads upstream.
- Bound queues and set per-route timeouts so overload is shed early with 503 and Retry-After instead of stalling.
How success is measured
- p95/p99 edge latency per route within the agreed budget during synthetic burst tests
- Upstream connection reuse ratio and new-handshake rate during bursts
- No dropped or reset connections at the target burst size
- Queueing time at the edge reported separately from upstream time