GPU Sharing in Multi-Tenant Clusters: Four Mechanisms, Three of Them Traps

Kubernetes treats a GPU as an integer. You request nvidia.com/gpu: 1, the device plugin advertises however many the node has, and the scheduler hands you the whole thing. There is no 500m for accelerators, no oversubscription, no burst. A notebook pod running an idle Jupyter kernel holds an H100 as completely as a job saturating every SM on the die.

This is why GPU clusters routinely run at 15 to 30 percent average utilisation while engineers queue for capacity. The scheduler is doing exactly what it was told. The question is which sharing mechanism you put underneath it, and the four available options differ so much in isolation guarantees that treating them as interchangeable performance knobs will produce an incident involving one team's out-of-memory error killing another team's six-day training run.

The four mechanisms

Time-slicing

The device plugin advertises one physical GPU as N logical replicas. The CUDA driver context-switches between processes. Configuration is two lines:

apiVersion: v1
kind: ConfigMap
metadata:
  name: device-plugin-config
  namespace: gpu-operator
data:
  shared: |-
    version: v1
    sharing:
      timeSlicing:
        resources:
        - name: nvidia.com/gpu
          replicas: 4

What you get: four pods on one GPU, each believing it has a whole one. What you do not get: memory isolation, compute isolation, fault isolation, or any fairness guarantee. All four processes share the same 80 GB of HBM with no enforcement. Process three allocates a 60 GB tensor and processes one, two, and four hit CUDA OOM. There is no accounting, no quota, no error attribution — the pod that dies is not the pod that caused it.

Time-slicing is correct for exactly one situation: interactive development environments belonging to a single team that already shares a blast radius, where the alternative is a fleet of idle notebooks each holding a $30,000 device. It is malpractice for anything with an SLO or anything crossing a team boundary.

MPS (Multi-Process Service)

MPS runs a daemon that multiplexes CUDA contexts from several processes into one, so kernels from different clients execute concurrently rather than by context switch. This is a genuine throughput improvement over time-slicing for small kernels that individually cannot fill the GPU — the classic case being many small inference requests.

Newer MPS versions support memory limits per client via CUDA_MPS_PINNED_DEVICE_MEM_LIMIT and compute limits via CUDA_MPS_ACTIVE_THREAD_PERCENTAGE, which is more control than time-slicing offers. But fault isolation remains weak: an unrecoverable fault in one client can take down the MPS server and every client attached to it. And the memory limit is enforced by the runtime, not by hardware, so it constrains well-behaved processes rather than defending against hostile ones.

MPS is the right choice for consolidating many small inference workloads owned by one team, where throughput matters more than isolation.

MIG (Multi-Instance GPU)

MIG is hardware partitioning, available on A100, H100, H200, and Blackwell datacenter parts. The GPU is physically divided into up to seven instances, each with its own dedicated SMs, its own slice of L2 cache, its own memory controllers, and its own HBM allocation. The partitions are enforced in silicon. A fault in one instance does not affect another. Memory is genuinely separate.

# enable MIG and apply a mixed profile
nvidia-smi -i 0 -mig 1
nvidia-smi mig -i 0 -cgi 3g.40gb,2g.20gb,1g.10gb,1g.10gb -C

# resulting node advertises distinct resource names
kubectl get node gpu-node-01 -o jsonpath='{.status.allocatable}' | jq
# {
#   "nvidia.com/mig-3g.40gb": "1",
#   "nvidia.com/mig-2g.20gb": "1",
#   "nvidia.com/mig-1g.10gb": "2"
# }

Two real constraints. First, reconfiguring the partition layout requires draining every workload on the device — MIG geometry is not a runtime knob, so you are committing to a shape for a shift or a day, not per-pod. Second, the profiles are discrete and not all combinations are valid; you cannot express "40 percent of a GPU," you can express 3g.40gb. Third, and least discussed: MIG instances cannot communicate over NVLink with each other, so any workload needing multi-GPU collectives is disqualified entirely.

MIG is the only mechanism on this list that gives you isolation strong enough to put in front of another team, and that makes it the default for multi-tenant inference on capable hardware.

Whole-device allocation

One pod, one GPU, no sharing. For any large training job this is not a fallback, it is the correct answer — the job will saturate the device and any partitioning would cost you NVLink bandwidth you need. The efficiency work here is not sharing, it is queueing.

Sharing is the second problem. Queueing is the first.

Even with perfect partitioning, the default Kubernetes scheduler will destroy a shared GPU cluster, for a specific reason: it schedules pods independently. A distributed training job needing 16 GPUs arrives as 16 pods. The scheduler places 11, runs out of capacity, and leaves 5 Pending. Those 11 pods sit holding $300,000 of hardware, doing nothing, waiting for peers that will never arrive because a stream of single-GPU notebook pods keeps consuming the capacity that frees up. This is textbook resource deadlock and it happens within days of a GPU cluster becoming popular.

Gang scheduling is not optional. Kueue is the Kubernetes-native answer and it should go in before the second team arrives:

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: research
spec:
  namespaceSelector: {}
  cohort: shared-gpu
  resourceGroups:
  - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
    flavors:
    - name: h100
      resources:
      - name: "nvidia.com/gpu"
        nominalQuota: 32
        borrowingLimit: 16     # may borrow idle capacity from the cohort
        lendingLimit: 24       # but keeps 8 reserved for itself
      - name: "cpu"
        nominalQuota: 512
      - name: "memory"
        nominalQuota: 4Ti
  preemption:
    reclaimWithinCohort: Any
    withinClusterQueue: LowerPriority

Three properties make this work. Jobs are admitted all-or-nothing, so no partial gang holds capacity. Quota is per-tenant, so a single team cannot starve the cluster. And the cohort with borrowing limits gives you the thing that makes quota politically survivable — idle capacity flows to whoever needs it, but a team that lends can reclaim its nominal quota by preemption when it comes back.

That last property is what separates a system people accept from one they route around. Rigid quota in a GPU cluster produces hoarding: teams keep jobs running on capacity they do not need because releasing it means queueing to get it back. Borrowing with guaranteed reclaim removes the incentive.

Preemption needs checkpointing to be honest

Preempting a training job that cannot resume is not reclamation, it is destruction. Before enabling preemption, verify that every job class in the cluster checkpoints at an interval you can afford to lose — 15 to 30 minutes is a reasonable target for large runs — and that resumption actually works, tested, not assumed.

The economics: a checkpoint of a 70B-parameter model in bf16 with optimizer state is on the order of a terabyte, and writing it to object storage takes minutes. Checkpointing every 10 minutes on a job with a 4-minute checkpoint cost means you spend 40 percent of your cluster writing checkpoints. Checkpointing every 4 hours means a preemption costs you up to 4 hours of a multi-node job. Sharded and asynchronous checkpointing narrows this considerably and is worth the integration effort on any cluster where preemption is a real policy.

What to actually build

For a cluster serving multiple teams with mixed workloads, the configuration that holds up:

• Training pool: whole devices, MIG disabled, NVLink intact, gang-scheduled through Kueue with cohort borrowing, preemption enabled for jobs below a priority threshold, mandatory checkpointing verified in CI.
• Inference pool: MIG-partitioned on a small number of standard profiles, exposed as distinct resource names so teams request a size rather than a fraction, no preemption, autoscaled on queue depth.
• Interactive pool: time-sliced, small, aggressively reaped — a hard TTL on idle notebook pods, measured by GPU utilisation rather than by kernel liveness, because a Jupyter kernel is always alive.

Three pools, three mechanisms, three sets of expectations. Attempting one homogeneous pool that serves all three is the mistake, and it is the same structural mistake as trying to run training and serving on one platform.

Finally, instrument utilisation properly. nvidia_gpu_duty_cycle tells you whether any kernel was resident, which is close to useless — a kernel using 3 percent of the SMs reads as 100 percent busy. Use DCGM's SM occupancy and memory bandwidth utilisation instead. The gap between "GPU allocated" and "GPU doing useful work" is typically the largest single line item in an AI infrastructure budget, and you cannot bill for it, prioritise against it, or build unit economics on top of it until you can measure it.