VPA and HPA Will Fight Each Other Unless You Separate Their Signals

Put a Horizontal Pod Autoscaler on CPU utilisation and a Vertical Pod Autoscaler in Auto mode on the same Deployment, and you have built a feedback loop with two controllers writing to inputs the other one reads. HPA scales replicas to hold utilisation near a target. Utilisation is usage divided by request. VPA changes the request. The denominator moves out from under the controller that is trying to stabilise the ratio.

The oscillation goes like this. Traffic is steady. VPA observes that pods are using 300m against a 500m request and drops the request to 350m. Utilisation, which HPA computes as usage over request, jumps from 60 percent to 86 percent. HPA is targeting 70, so it adds replicas. More replicas means the same total work spread thinner, so per-pod usage falls to 210m. VPA sees the new usage and drops the request again. Repeat. In a cluster I looked at where this was live on a queue consumer, the Deployment went from 8 replicas to 31 over about forty minutes on flat input traffic, and the only thing that stopped it was VPA hitting its minAllowed floor.

The rule, stated plainly

Two autoscalers can coexist on one workload if and only if they act on independent signals. The Kubernetes VPA documentation says this too, in a sentence that is easy to skim past. In practice it collapses to three configurations that are safe and one that is not:

The second row is the one most teams should reach for first, because it requires no change to how they already scale. VPA takes memory, which HPA was never going to manage well anyway — memory-based HPA is a bad idea for a separate reason, namely that most runtimes do not release memory when load drops, so a memory-triggered scale-up never scales back down.

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: orders-api
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: orders-api
  updatePolicy:
    updateMode: "Auto"
  resourcePolicy:
    containerPolicies:
    - containerName: api
      controlledResources: ["memory"]     # CPU deliberately excluded
      minAllowed:  { memory: 512Mi }
      maxAllowed:  { memory: 8Gi }
    - containerName: envoy
      mode: "Off"                          # never resize the sidecar

The controlledResources: ["memory"] line is the whole trick. HPA keeps CPU utilisation as its control signal and the request it divides by never moves.

The third row is better, and harder

Scaling on a business-meaningful external metric rather than CPU is the correct end state for most services, and it happens to make the VPA conflict disappear entirely. If HPA is targeting a Kafka consumer lag of 1000 messages or 40 requests per second per pod, then VPA can resize both CPU and memory freely — the two controllers are operating on genuinely different quantities.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: orders-worker
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: orders-worker
  minReplicas: 4
  maxReplicas: 120
  metrics:
  - type: External
    external:
      metric:
        name: kafka_consumergroup_lag
        selector:
          matchLabels:
            consumergroup: orders-worker
            topic: orders
      target:
        type: AverageValue
        averageValue: "1000"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 30
    scaleDown:
      stabilizationWindowSeconds: 600
      policies:
      - type: Percent
        value: 10
        periodSeconds: 60

Note the asymmetric behavior block, which matters more than people expect. Scaling up should be immediate and aggressive — the cost of an extra pod for ten minutes is cents, the cost of a backlog is a customer. Scaling down should be slow and cautious, because a premature scale-down during a lull immediately precedes the next spike. The default 300-second down-stabilisation is usually too short for anything with a bursty arrival pattern; 600 to 900 seconds is a better starting point, and the arrival-rate modelling that tells you which is worth doing properly for your top few services.

Why in-place resize does not fully rescue Auto mode

Historically the killer objection to VPA in Auto mode was that changing a pod's resources required recreating the pod. VPA would evict, the ReplicaSet controller would create a replacement with the new request, and you would take a rolling restart of your fleet on the autoscaler's schedule rather than yours. With a PodDisruptionBudget and a well-behaved application this is survivable. With a JVM that takes 90 seconds to warm up, or a service that holds long-lived gRPC streams, it is a recurring low-grade outage.

In-place pod resize (the InPlacePodVerticalScaling feature, beta from 1.33) changes the mechanics: CPU requests can be adjusted on a running pod without recreation. It is a genuine improvement and it removes the restart objection for CPU. It does not remove the control-loop objection. An in-place CPU request change still moves HPA's denominator, and it moves it faster than before, which makes the oscillation tighter rather than gentler. Memory resize also remains constrained — shrinking memory in place is not generally permitted, since you cannot reclaim pages the process is holding.

Recommender-only is the highest-value configuration

Most of the value in VPA is the recommender, and most of the risk is the updater. Running VPA with updateMode: "Off" across every workload in the cluster costs one controller, writes nothing, and gives you a continuously-maintained, per-container estimate of what each workload actually needs.

kubectl get vpa -A -o custom-columns=\
'NS:.metadata.namespace,\
NAME:.metadata.name,\
CPU:.status.recommendation.containerRecommendations[0].target.cpu,\
MEM:.status.recommendation.containerRecommendations[0].target.memory'

The recommender uses a decaying histogram of usage samples with a half-life of 24 hours, and targets roughly the 90th percentile for CPU and the 90th with a safety margin for memory. That is a reasonable default, but understand what it implies: a workload with a monthly batch peak will have that peak decayed almost entirely out of the histogram by the time the next one arrives, so the recommendation will be too low. For anything with a cycle longer than about a week, override with minAllowed sized to the known peak, or exclude it from automation and size it by hand from max-over-time on the real window.

Feed those recommendations into a weekly job that opens pull requests against the manifests when declared requests drift more than 40 percent from the recommendation. This converts autoscaling from a runtime behaviour into a reviewed change, which is what you want for anything that alters scheduling. The engineer who owns the service sees the diff, approves it or explains why not, and the cluster's allocation stays honest. The end-to-end version of this workflow — sequencing, verification, and who signs off — is written up separately.

The failure mode nobody warns you about

A VPA whose maxAllowed exceeds the largest allocatable capacity on any node in the cluster will happily recommend a request that cannot be scheduled anywhere. The updater evicts the pod, the replacement goes Pending with Insufficient cpu, and if that Deployment is the only thing keeping a queue drained you now have an incident caused by your cost optimisation tooling.

Always set maxAllowed below the allocatable capacity of your smallest relevant node type, accounting for system reserved and any DaemonSets. On a 16-core node with typical reservations you have roughly 15.2 cores allocatable, and three DaemonSets taking 600m collectively, so a ceiling of 12 cores is defensible and 16 is not. Pair that with a PodDisruptionBudget on every VPA-managed Deployment — the updater respects them, and without one it will evict past your availability floor during a resize sweep.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: orders-api
spec:
  minAvailable: 80%
  selector:
    matchLabels:
      app: orders-api

If you take one thing away: decide which controller owns which dimension before you install either one, write that decision into the manifests as an explicit controlledResources list, and treat any workload with both a CPU-based HPA and an unrestricted VPA as a production incident waiting for a traffic pattern to trigger it.