API-taristu ja taustasüsteemide arendus

Eesti

Skaleerimine

Autoscaling latency-sensitive services on Kubernetes

Autoscaling promises capacity that follows demand. For latency-sensitive APIs it often delivers something else: CPU throttling that inflates the tail, scale-ups that arrive after the spike, and cold pods that serve their first requests slowly. Getting it right means choosing resources, signals and timing deliberately.

Avaldatud 6 min lugemist

See sisu on praegu saadaval ainult inglise keeles.

Requests, limits and throttling

Each container declares CPU and memory requests, which the scheduler uses to place pods, and optional limits, which cap usage. Memory limits protect nodes from runaway processes and are generally advisable. CPU limits are more subtle. The kernel enforces them through a quota per scheduling period; a multi-threaded service that bursts – for example during garbage collection or a burst of requests – can exhaust its quota early in a period and then wait, idle, until the next period. Average CPU usage looks low while p99 latency suffers.

For latency-sensitive services, a common approach is to set CPU requests accurately from measurements and to omit CPU limits or set them generously, relying on requests and node capacity for fairness. Monitor throttling metrics such as container_cpu_cfs_throttled_periods_total; any significant throttling on a latency-critical service is a finding.

Choosing the scaling signal

The Horizontal Pod Autoscaler (HPA) adjusts replica counts to keep a metric near a target. CPU utilisation – relative to requests – is the default, but it is not always the best signal:

  • Gateways and API services: requests per second per pod, or in-flight requests, track load directly and react before CPU saturates.
  • Workers consuming queues: consumer lag or queue depth per pod reflects work waiting, which CPU does not show. KEDA provides scalers for common brokers and can scale to zero for intermittent workloads.
  • CPU-bound services: CPU utilisation remains a reasonable signal, provided requests are accurate.

Whatever the metric, the target must leave headroom. Scaling at 80% of capacity leaves almost no buffer for the time it takes to add pods.

Timing: the race against the spike

A scale-up is the sum of several delays: metric collection and aggregation, the autoscaler’s evaluation interval, scheduling, image pull, container start, application warm-up and readiness. Together these can take a minute or more. If traffic doubles in twenty seconds, the existing pods must carry it alone during that time.

Reduce each delay where possible – small images, pre-pulled base layers, fast start-up, warm caches – and size the minimum replica count for the fastest realistic ramp. For predictable peaks such as business-hour starts or scheduled campaigns, scheduled scaling in advance is simpler and more reliable than reacting. HPA behaviour settings allow fast scale-up and slow scale-down, which avoids oscillation when load fluctuates.

Node capacity and the cluster autoscaler

Pods can only scale if nodes have room. When they do not, the cluster autoscaler or a node provisioner adds nodes, which takes minutes. Keeping a small buffer of spare capacity – for example with low-priority placeholder pods that are evicted when real pods need the space – lets latency-sensitive services scale immediately while new nodes arrive in the background.

Pod disruption budgets and topology spread constraints keep replicas spread across nodes and zones, so that scaling down or draining a node never removes too many replicas of one service at once.

Warm-up, readiness and shutdown

A new pod that receives full traffic before it has loaded configuration, opened connection pools and warmed caches or JIT-compiled code will serve its first requests slowly. Readiness probes should report ready only when the service can meet its latency objective; some teams additionally ramp traffic to new pods gradually at the load balancer.

Scaling down needs equal care. On termination, a pod should stop accepting new connections, finish in-flight requests within a grace period and then exit. Without a pre-stop delay, endpoints may still route traffic to a pod that has already begun shutting down, producing errors that look random.

Vertical sizing and right-sizing

Horizontal scaling assumes that each pod is sized sensibly. The Vertical Pod Autoscaler can recommend requests based on observed usage; running it in recommendation mode and applying its suggestions through normal deployments avoids pods being restarted unexpectedly. Revisit requests after significant code changes, runtime upgrades or traffic shifts – sizes chosen two years ago rarely match today’s workload.

Verifying with load tests

Autoscaling configurations must be tested with traffic that ramps the way production does: sudden spikes, steady growth and slow decline. Measure p99 latency and error rate throughout, not only at steady state, and record how long each scaling step takes. Use an open workload model so that slow responses do not reduce the arrival rate and hide the problem.

Observability for scaling decisions

Autoscaling is a control loop, and control loops need instrumentation. Dashboards for a latency-sensitive service should show, on one time axis, the scaling signal and its target, desired and actual replica counts, pending pods, CPU throttling, request rate and p99 latency. With that view, it is easy to see whether a latency spike happened before scaling began, while pods were pending, or after new pods became ready but were still warming up.

Alert on symptoms first – latency and error rate against objectives – and use scaling metrics for diagnosis. Useful additional alerts include an HPA pinned at its maximum replica count, pods pending for longer than a few minutes, and sustained throttling on critical workloads. Each of these means that the system is no longer able to buy its way out of load and that someone needs to look at capacity, limits or efficiency.

Finally, keep an eye on cost. Scaling that never comes down, or minimum replica counts set for a peak that happens once a year, are easy to miss without a regular review.

Näide: A gateway throttled at 30% CPU

An API gateway ran with a CPU limit of one core per pod and an HPA target of 70% CPU. Average usage stayed around 30%, yet p99 latency doubled during traffic bursts. Throttling metrics showed that pods hit their quota in short bursts. The team removed the CPU limit, set requests from measured usage, switched the HPA to scale on requests per second per pod with a target at 60% of tested capacity, and added two placeholder pods of spare node capacity. Under the same burst pattern, throttling disappeared, scale-ups completed in about 40 seconds and p99 latency stayed within budget.

Levinud lõksud

  • Tight CPU limits on latency-sensitive services.
  • Scaling only on CPU for queue workers or I/O-bound services.
  • Targets so high that no headroom remains during scale-up.
  • Readiness probes that pass before the service is warm.
  • No graceful shutdown, causing errors on every scale-down.

Küsimused ja vastused

Should we never set CPU limits?

Not never – they are useful for batch workloads and multi-tenant fairness. For latency-critical services, measure throttling and prefer accurate requests over tight limits.

HPA or KEDA?

HPA for resource and request-based scaling of services; KEDA when the right signal comes from an external system such as a queue, or when scaling to zero is useful. KEDA drives an HPA under the hood.

Can we combine HPA and VPA?

Not on the same resource metric for the same workload. A common pattern is HPA on a custom metric with VPA in recommendation mode for sizing.

Kontrollnimekiri

  • Set CPU and memory requests from measurements.
  • Monitor CPU throttling on latency-critical services.
  • Choose a scaling signal close to user impact.
  • Leave headroom in targets and size minimum replicas for ramps.
  • Keep spare node capacity for immediate scale-ups.
  • Make readiness reflect warm-up and implement graceful shutdown.
  • Load-test realistic ramps and measure p99 throughout.

Esmased allikad

  • Kubernetes documentation – Horizontal Pod Autoscaling
  • Kubernetes documentation – Resource Management for Pods and Containers
  • KEDA documentation – Scalers and ScaledObject specification
  • Linux kernel documentation – CFS bandwidth control

Kõik artiklid