Training Clusters and Inference Fleets Are Not the Same System

A training job is a batch computation with a deadline measured in days, an appetite for every FLOP you can give it, and a hard dependency on all-reduce bandwidth between accelerators. An inference deployment is a queueing system with a latency budget measured in milliseconds, a bursty arrival process, and — for most model sizes — no cross-device communication at all. The only property they share is that both use GPUs, and organisations keep concluding from that single shared property that they should share a platform.

The result is a cluster with an expensive InfiniBand fabric that inference pods never touch, an autoscaler tuned for job queues that cannot respond to a traffic spike, and a scheduler configured for gang admission that makes rolling out a new model version take eleven minutes. Everyone is unhappy and the hardware bill is 40 percent higher than it needs to be.

The interconnect is the fork in the road

Training a model that does not fit on one device requires gradient synchronisation every step. For a 70B-parameter model in bf16 with data parallelism, that is roughly 140 GB of gradients all-reduced per step across the world size. At a step time of a few hundred milliseconds, the interconnect is not a supporting component — it is the thing that determines whether you get 60 percent model-FLOPs utilisation or 25 percent.

This is why training clusters are built the way they are: NVLink and NVSwitch inside the node giving 900 GB/s all-to-all on H100 systems, and RDMA over InfiniBand or RoCE between nodes at 400 Gb/s per rail, eight rails per node, with a rail-optimised fat-tree topology so that the k-th GPU in every node lands on the same leaf switch. On AWS this is EFA with GPUDirect and a placement group; the whole point is to keep collectives off the host CPU and off the general VPC network.

Inference for a model that fits on one or two devices needs none of it. The network carries request bodies and response tokens — kilobytes, not gigabytes. Standard 25 to 100 Gb/s VPC networking is over-provisioned for the job. Building an inference fleet on training-class networked instances is paying a substantial premium for idle silicon.

The nuance: large-model serving with tensor parallelism across 4 or 8 GPUs does need fast intra-node links, because every token generated requires an all-reduce across the tensor-parallel group. That argues for NVLink within the node, not for InfiniBand between nodes. The distinction matters for instance selection — you want a full 8-GPU NVLink domain, and you do not want the EFA-heavy variant.

Utilisation targets are opposites, and this is the deepest split

A training cluster should run at 90 percent or higher, because there is always another job in the queue. Idle GPU time in a training cluster is pure waste, and the correct response to a gap is to admit the next job. Preemption, backfill, and long queues are features.

An inference fleet running at 90 percent is a fleet about to violate its latency SLO. Queueing theory is unforgiving here: as utilisation approaches capacity, wait time rises hyperbolically. For an M/M/1 approximation, expected wait scales as ρ/(1−ρ) — at 50 percent utilisation you wait one service time, at 90 percent you wait nine, at 95 percent nineteen. Real serving systems with continuous batching behave better than M/M/1 but the shape holds. Sizing an inference fleet at 60 percent steady-state utilisation is not waste, it is the latency budget, and someone will try to "optimise" it every quarter.

Two systems, opposite objectives, one scheduler. It does not work.

What actually determines inference cost

Autoregressive generation splits into two phases with completely different hardware characteristics, and understanding the split is the difference between a serving fleet at $0.40 per million tokens and one at $2.00.

Prefill processes the entire prompt in one forward pass. It is compute-bound and parallelises well across the sequence. Decode generates one token at a time, and each step must read the full model weights plus the KV cache from HBM to produce a single token. Decode is memory-bandwidth-bound, and arithmetic intensity is terrible — you move gigabytes to compute a few megaflops.

Three consequences follow.

First, batching is the primary cost lever, and only for decode. Batching 32 requests means one weight read serves 32 tokens instead of one. Continuous (in-flight) batching, where new requests join the running batch at token boundaries rather than waiting for a batch window, is the single largest throughput improvement available — commonly 3 to 6× over static batching. If you are running anything other than vLLM, TensorRT-LLM, SGLang, or an equivalent with continuous batching and paged attention, that is the first thing to fix and nothing else on this list comes close.

Second, the KV cache is your real memory constraint, not the weights. For a 70B model with grouped-query attention, per-token KV cache is on the order of 150 to 320 KB depending on configuration. A 32K-token context is 5 to 10 GB per concurrent request. On an 80 GB device holding 140 GB of weights across two-way tensor parallelism, the cache determines your concurrency ceiling, and paged attention exists specifically because naive contiguous allocation wasted 60 to 80 percent of it to fragmentation.

Third, prefill and decode interfere. A long prefill occupying the GPU stalls decode for every request in the running batch, which shows up as inter-token latency spikes on requests that were doing fine. Chunked prefill splits long prompts into pieces interleaved with decode steps; disaggregated serving goes further and runs prefill on separate hardware from decode, which is increasingly how large deployments are built.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-serve-70b
spec:
  replicas: 6
  template:
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
        - --model=/models/llama-70b
        - --tensor-parallel-size=2
        - --max-model-len=16384
        - --gpu-memory-utilization=0.92
        - --enable-chunked-prefill
        - --max-num-batched-tokens=8192
        resources:
          limits:
            nvidia.com/gpu: 2
        volumeMounts:
        - { name: models, mountPath: /models, readOnly: true }
        - { name: shm, mountPath: /dev/shm }
      volumes:
      - name: shm
        emptyDir: { medium: Memory, sizeLimit: 16Gi }

The /dev/shm mount is not optional with tensor parallelism — NCCL uses shared memory for intra-node communication and the 64 MB Kubernetes default will produce a hang that looks like a model loading failure.

Scaling signals

CPU utilisation is meaningless for a GPU serving pod. GPU utilisation as reported by nvidia-smi is nearly as bad, since a decode step at batch size 1 shows near-100 percent duty cycle while wasting most of the device.

Scale on the queue. Time-to-first-token and the number of requests waiting for admission are the signals that correlate with user experience:

metrics:
- type: Pods
  pods:
    metric: { name: vllm_num_requests_waiting }
    target:
      type: AverageValue
      averageValue: "4"
behavior:
  scaleUp:
    stabilizationWindowSeconds: 0
    policies: [{ type: Pods, value: 3, periodSeconds: 60 }]
  scaleDown:
    stabilizationWindowSeconds: 900
    policies: [{ type: Pods, value: 1, periodSeconds: 300 }]

Scale down slowly. A serving replica takes 3 to 8 minutes to become useful — node provisioning, then pulling a container image that is often 10 to 20 GB, then loading 140 GB of weights into HBM. That startup cost inverts the usual autoscaling intuition and is why keeping weights on a local NVMe cache and pre-pulling images onto warm nodes matters more here than anywhere else. The same reasoning that governs provisioning latency budgets for bursty CPU workloads applies, with every constant an order of magnitude larger.

The organisational version of the same argument

Separate the pools. Separate the quotas. Separate the on-call. A training cluster is a batch system whose failure mode is a delayed research result; an inference fleet is a production service whose failure mode is a customer-visible outage. Those need different change management, different SLOs, and usually different people.

The one thing worth sharing is the model artifact pipeline — registry, weight storage, evaluation gates — because that is genuinely common and duplicating it produces version skew between what was trained and what is served. Everything below that line should be two systems.

Where they do meet, expensively, is capacity planning: both compete for the same constrained accelerator supply, and the quota mechanics that arbitrate between them are where the actual politics live. Getting the cost model right for each — with the fixed/marginal split made explicit, because it is far more extreme here than on CPU infrastructure — is a prerequisite for having that argument productively.