Capacity Planning for Bursty Workloads Is Queueing Theory, Not Averages

Two services both peak at ten times their mean. The first is an e-commerce checkout path that ramps over forty minutes as a promotion goes live. The second is a webhook receiver that goes from 200 to 20,000 requests per second in under three seconds when a partner's batch job fires. Identical peak-to-mean ratio, completely different infrastructure problems, and any planning approach built on that ratio will get one of them badly wrong.

What separates them is the relationship between two timescales: how long the burst takes to arrive, and how long you take to add capacity. Everything else in capacity planning for spiky workloads follows from that comparison.

The provisioning latency budget

Write down what it actually takes to serve one more unit of load, end to end. Not the autoscaler's reaction time — the whole chain.

Sum it. A typical Java service on a cluster that needs to add a node lands somewhere around 4 to 7 minutes from "load arrived" to "new replica serving at full rate." A Go service on a cluster with headroom and a pre-pulled image can be at 20 seconds.

Now compare against how fast the burst arrives:

• Ramp time > 3× provisioning latency: reactive autoscaling works. This is the promotion case — forty minutes of ramp against five minutes of provisioning is comfortable. Tune the HPA and move on.
• Ramp time ≈ provisioning latency: reactive autoscaling works with static headroom to cover the gap. Most real services live here.
• Ramp time < provisioning latency: autoscaling cannot help you. The webhook case. Capacity must already exist when the burst arrives, or the burst must be absorbed by a buffer.

That third category is the one people keep trying to solve with better autoscaler tuning, and it is not solvable that way. No amount of stabilisation window adjustment makes a five-minute provisioning chain respond to a three-second spike.

Sizing headroom with queueing theory instead of a percentage

The usual approach is "leave 30 percent headroom," picked because it sounds prudent. The number should come from your latency target and your service time distribution.

For a rough M/M/c model, the relationship between utilisation ρ and queueing delay is the part that matters:

ρ = 0.50  →  wait ≈ 1.0 × service_time
ρ = 0.70  →  wait ≈ 2.3 × service_time
ρ = 0.80  →  wait ≈ 4.0 × service_time
ρ = 0.90  →  wait ≈ 9.0 × service_time
ρ = 0.95  →  wait ≈ 19  × service_time

If your service time is 40ms and your p99 latency budget is 250ms, you have about 210ms for queueing, which puts your ceiling near ρ = 0.83. Real systems with many servers and less variable service times do better than this approximation, and systems with heavy-tailed service times do considerably worse — but the shape is right, and the shape is what people get wrong. The cost of the last 10 percent of utilisation is not linear.

Two refinements matter in practice. Variability in service time makes everything worse: if your service time distribution has a long tail (a few requests that take 20× the median), the effective queueing delay at a given utilisation is much higher than the exponential assumption suggests. And more parallel servers helps — twenty replicas at 80 percent queue less than four replicas at 80 percent, because a burst is less likely to find every server busy. That is an argument for horizontal spread over vertical size in bursty systems, independent of cost.

Four ways to hold capacity, priced

Static overprovisioning. Run more replicas than steady state needs. Simple, reliable, and you pay for it 24 hours a day. For a service with a 10× peak lasting two hours daily, provisioning for peak costs roughly 5× what the average load justifies.

Pause pods (low-priority placeholders). Run pods that do nothing at a negative priority class. They occupy node capacity, so the nodes exist and are warm. When real work arrives, the scheduler preempts them instantly and the real pods start on already-running nodes. You pay for the nodes but the capacity is genuinely reusable, and preemption takes about a second rather than the two minutes a node scale-up takes.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: overprovisioning
value: -10
globalDefault: false
description: "Placeholder pods; evicted the moment real work needs the room"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: capacity-reserve
spec:
  replicas: 8
  template:
    spec:
      priorityClassName: overprovisioning
      terminationGracePeriodSeconds: 0
      containers:
      - name: pause
        image: registry.k8s.io/pause:3.10
        resources:
          requests: { cpu: "3", memory: 12Gi }

This is the single most underused technique for bursty workloads on Kubernetes. It converts a node provisioning problem into a pod scheduling problem, and pod scheduling is two orders of magnitude faster.

Scheduled scaling. If the burst is predictable by clock — market open, a nightly batch, a weekly report run — scale before it rather than in response to it. A CronJob that patches minReplicas fifteen minutes ahead is unglamorous and works better than any reactive scheme.

schedule: "45 8 * * mon-fri"
command:
- kubectl
- patch
- hpa/checkout-api
- --type=merge
- -p
- '{"spec":{"minReplicas":40}}'

Predictive scaling. Forecast load from history and provision ahead of it. Worth it when the pattern is strong and regular but not clock-aligned, and it demands a fallback — a forecast that misses a novel event has to degrade to reactive scaling rather than to an outage. Most organisations get 90 percent of the value from the scheduled version at 5 percent of the complexity.

Make the burst someone else's problem

The cheapest capacity is capacity you do not provision, and the way to get there is to decouple arrival from processing.

For anything that does not need a synchronous response — webhooks, event ingestion, uploads, notification fan-out — accept the request, write it to a durable buffer, return 202, and process at whatever rate you can sustain. A 20,000 RPS spike lasting three seconds is 60,000 messages. A consumer fleet processing 2,000 per second drains that in thirty seconds. You have converted a 10× capacity problem into a thirty-second latency problem, and thirty seconds of processing lag on an async workload is usually invisible.

The ingestion tier still has to absorb the spike, but an ingestion tier that only validates and enqueues has a service time in single-digit milliseconds, so it needs a fraction of the capacity the processing tier would have needed. Scale the buffer's consumers on queue depth rather than on CPU — the external-metric HPA configuration for exactly this shape is worth setting up properly.

For synchronous paths, the equivalent moves are load shedding and admission control. Under overload, a system that sheds 20 percent of requests with a fast 429 and serves the rest within SLO is strictly better than one that queues everything and misses SLO on 100 percent — and far better than one that collapses. Concurrency limiting at the edge, with a queue bounded well below the point where latency exceeds the client's timeout, is the mechanism. An unbounded queue in front of an overloaded service does not improve anything; it converts a fast failure into a slow one while consuming memory.

Sizing the number, and testing it

A workable procedure:

1. Get the arrival distribution at one-second resolution for 30 days. Not per-minute averages — a per-minute average hides exactly the spike you are planning for. This alone changes people's understanding of their own traffic.
2. Take the p99.9 of one-second arrival rate, not the maximum. The max is often a single anomalous scrape; the p99.9 is the burst you should survive.
3. Measure the maximum sustainable throughput per replica at your latency SLO — load test until p99 exceeds budget, then back off 15 percent.
4. Compute the replica count needed for p99.9 arrivals at your target utilisation from the table above.
5. Compare that to the steady-state count. The difference is what your headroom strategy has to cover, and the four options above are how you cover it.

Then test it, because none of this is real until it survives a burst. Load tests that ramp gently over ten minutes validate nothing about burst behaviour — the autoscaler keeps up, everything looks fine, and you learn nothing. Test with a step function: idle to full burst rate in under a second, held for two minutes. What you are watching for is the shape of the p99 curve during the first 60 seconds and whether it recovers or diverges.

Two things this test reliably exposes. Connection pool exhaustion downstream — your service scales, the database's connection limit does not, and the burst turns into a wave of connection errors. And cold-start cliffs, where the new replicas that arrive are slower than the warm ones for their first minute, so adding capacity briefly makes p99 worse before it makes it better.

One last consideration: bursty workloads and cheap capacity pull in opposite directions. Headroom is the thing spot instances are worst at holding, because reclamation removes capacity precisely when the market is tight — which correlates with when everyone else needs it too. Keep the headroom tier on committed capacity and let the elastic tier above it run on interruptible instances, rather than the reverse. And whichever autoscaler you use, the provisioning latency in the table above is a property you can measure and improve; the choice between them moves that number by roughly a factor of two.