Spot Instances for Stateful Workloads Are Defensible, With Conditions

The reflex is to sort workloads into two bins — stateless goes on spot, stateful goes on-demand — and stop thinking. That heuristic is a proxy for the thing that actually matters, and it is a bad proxy. What matters is how long the service is degraded when capacity disappears without warning, and whether that degradation is visible to a user. A Kafka broker with 200 GB of local state and a six-hour rebuild time fails that test badly. A Redis read replica behind a client that retries against a sibling fails it not at all, and it is just as stateful.

Spot capacity is 60 to 90 percent cheaper than on-demand, and for a cluster where stateful services are 40 percent of spend, that is not a rounding error. The engineering question is whether you can drive recovery time low enough and interruption correlation low enough that the expected cost of interruptions is well below the discount. For a surprising number of stateful workloads, you can.

Model the interruption, not the vibe

Start with three numbers you can actually obtain.

Interruption rate per instance-hour. AWS publishes a Spot Placement Score and an interruption frequency rating per instance type per region in the Spot Instance Advisor, bucketed as <5%, 5-10%, 10-15%, 15-20%, and >20% monthly. Those are monthly interruption probabilities per instance, not per hour, and the spread across instance types within the same family is enormous — I have seen r6i.4xlarge at under 5 percent in one AZ while r6i.8xlarge in the neighbouring AZ sat above 20.

Recovery time to full service. Not time to pod Ready. Time until the replacement is serving at the same capacity and correctness as the thing it replaced. For a stateful service this includes volume attach, data load or replica catch-up, cache warm, and any leader election. Measure it by actually killing a pod in a load test and watching the SLI, not by reading the readiness probe.

Correlation. This is the one that kills people. Spot reclamation is driven by capacity pressure in a specific capacity pool — the tuple of instance type, availability zone, and (on AWS) operating system. If all six of your Cassandra nodes are i3en.2xlarge in us-east-1a, they are one pool, and a pool-level reclamation event takes all six inside two minutes. Your quorum is gone and your two-minute interruption notice bought you nothing.

The expected annual cost of interruption is then roughly:

E[cost] = interruptions_per_year
        × P(correlated | interruption)
        × recovery_minutes
        × cost_per_minute_of_degradation

Compare that against the discount. If a workload costs $180k/year on-demand and $54k on spot, you have a $126k budget for interruption pain. If a single interruption costs you fifteen minutes of degraded read latency and you expect thirty a year, the question is whether 7.5 hours of annual degradation is worth $126k. Often it plainly is. Sometimes it plainly is not. Either way you have a number instead of a policy.

Diversification is the primary control

Everything else on this list is secondary to spreading across capacity pools. The interruption probability of any single pool is not something you control; the probability that all your pools go at once is entirely something you control.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: stateful-spot
spec:
  template:
    spec:
      requirements:
      - key: karpenter.sh/capacity-type
        operator: In
        values: ["spot"]
      - key: karpenter.k8s.aws/instance-family
        operator: In
        values: ["r6i", "r6a", "r7i", "r7a", "m6i", "m6a", "m7i"]
      - key: karpenter.k8s.aws/instance-cpu
        operator: In
        values: ["8", "16"]
      - key: topology.kubernetes.io/zone
        operator: In
        values: ["us-east-1a", "us-east-1b", "us-east-1c"]
      taints:
      - key: capacity-type
        value: spot
        effect: NoSchedule

Seven families × two sizes × three zones is 42 distinct capacity pools. Karpenter's allocation strategy will spread across them, and the probability of a simultaneous reclamation across a meaningful fraction of 42 pools is low enough to stop worrying about. Contrast with a hand-rolled ASG pinned to one instance type in one AZ, which is how most people's first spot experiment is configured and why most people's first spot experiment goes badly.

Then enforce the spread at the workload level, because a diverse node pool does not help if the scheduler happens to pack all your replicas onto two nodes:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: cassandra
spec:
  replicas: 9
  template:
    spec:
      tolerations:
      - key: capacity-type
        value: spot
        effect: NoSchedule
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels: { app: cassandra }
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels: { app: cassandra }
      terminationGracePeriodSeconds: 100

Both constraints use DoNotSchedule rather than ScheduleAnyway. Soft spread constraints are advisory and the scheduler will cheerfully ignore them under pressure, which means they provide no guarantee at exactly the moment you need one. Hard constraints can leave pods Pending, and that is the correct failure — a Pending pod is a visible problem, an accidentally co-located quorum is an invisible one. This does interact with bin-packing efficiency, and the tension is real; the packing side of that argument is worked through elsewhere.

Use the two minutes

AWS sends a Spot interruption notice roughly two minutes before reclamation, and a rebalance recommendation earlier than that with no fixed guarantee. Karpenter subscribes to these via an SQS queue and starts a graceful drain immediately. Cluster Autoscaler needs the separate AWS Node Termination Handler. Either way, the node is cordoned and pods are evicted with their grace period — which only helps if the application does something useful with the signal.

Two minutes is enough time to do real work for a stateful service:

• Leader handoff. A PostgreSQL primary running under Patroni can execute a controlled switchover in 5 to 15 seconds. Wire preStop to trigger it rather than letting the failover detector notice the corpse 30 seconds later.
• Checkpoint flush. Flink, Spark, and most stream processors can be told to take a savepoint. A 90-second grace period covers most state sizes under a few GB.
• Connection draining. Stop accepting new connections, let in-flight requests finish, deregister from service discovery before the endpoint controller notices.
• Hinted handoff / streaming. Cassandra's nodetool drain flushes memtables so the replacement does not have to replay commit logs.

lifecycle:
  preStop:
    exec:
      command:
      - /bin/sh
      - -c
      - |
        curl -s -XPOST localhost:8008/switchover \
          -d '{"leader":"'"$POD_NAME"'"}' || true
        sleep 5
        pg_ctl stop -m fast -D "$PGDATA"
terminationGracePeriodSeconds: 100

Keep the grace period comfortably under 120 seconds. The kernel does not care about your terminationGracePeriodSeconds; when the reclamation lands, the instance is gone.

Where the state actually lives decides most of this

The classification that predicts spot suitability better than "stateful vs stateless" is where the durable copy of the data sits.

The last row is worth dwelling on, because it is the common case and the framing is backwards. Teams say the database cannot run on spot because failover takes four minutes. But a four-minute failover is a problem regardless of spot — instances fail, AZs degrade, kernels panic. If your recovery time is bad enough to disqualify a 70 percent discount, the recovery time is the defect. Fixing it earns you the discount as a side effect.

The blend, and the honest caveat

Do not go all-spot on anything with a quorum. Run a mixed pool where a floor of on-demand capacity holds the minimum viable set and spot carries everything above it. For a nine-node Cassandra ring with RF=3, three on-demand and six spot means no realistic reclamation pattern breaks quorum, and you still capture two-thirds of the discount. Express it with a second NodePool at higher weight for on-demand and a minReplicas-style split across two StatefulSets, or with capacity-type-aware topology spread.

The caveat: spot pricing is a market, and markets change. Instance types that were 80 percent off in 2024 have compressed considerably as GPU-adjacent demand pushed general-purpose capacity tighter in some regions. Re-derive your discount quarterly rather than assuming the number from your original business case still holds, and fold it into whatever capacity model you use for headroom. If the discount on a family drops below about 40 percent, the interruption engineering usually stops paying for itself, and the right answer is a Savings Plan on on-demand capacity instead — which is a commitment-portfolio decision rather than a scheduling one.