How Multiple Teams Can Share One GPU with KAI Scheduler and vCluster

When several trusted internal AI teams need separate Kubernetes API servers, role-based access control (RBAC), and custom resource definitions (CRDs) while sharing one GPU, vCluster can isolate their control planes and KAI Scheduler can allocate GPU scheduling quotas. In an official tutorial published on August 3, 2026, NVIDIA demonstrated this setup by running three tenant clusters on one L40S.

Shared nodes provide logical isolation only. Tenants still share the host kernel, physical network, storage infrastructure, and GPU. When real Node synchronization is enabled, selected GPU Node objects appear in every vCluster, so this design does not isolate node information. Use vCluster private nodes from the start for untrusted tenants; private nodes require vCluster Platform, whose free mode is sufficient, and an existing shared-nodes vCluster cannot be migrated in place. The NVIDIA L40S specifications list “Multi-Instance GPU (MIG) Support” as “No.” Hard memory isolation therefore requires a MIG-capable GPU and a new resource and admission design.

If these boundaries are acceptable, verify the GPU runtime first, then install KAI, create the queues, and deploy the vClusters. Stop the rollout if admission, networking, reclaim, or memory-limit validation fails.

Use shared nodes only for trusted internal teams

The three teams in NVIDIA's example share one physical node and GPU. Only their virtual control planes and namespaced workload views are isolated.

Requirement	Choice
Trusted internal teams need separate API servers, RBAC, CRDs, and `cluster-admin`	Start with vCluster shared nodes
Teams need guaranteed GPU quota and may borrow idle capacity	Use hierarchical KAI Scheduler Queues
Trusted teams still need cross-tenant traffic restrictions	Enable vCluster-managed host NetworkPolicies and verify that the container network interface (CNI) enforces them
Tenants are untrusted and require node, kernel, network, and storage isolation	Create vCluster private nodes through vCluster Platform; shared nodes cannot be converted in place
Hardware-enforced GPU memory isolation is required	L40S cannot provide it; move to a MIG-capable GPU and allocate MIG resources through the scheduler
Multiple CUDA processes may compete on one device and no performance service-level agreement (SLA) is tied to the requested fraction	GPU sharing can be evaluated

Before rollout, prove all of the following:

Each team can see only its own namespaced workloads from its vCluster
Visible fields on the selected Node match the approved redaction policy
All three queues receive their guarantees and can borrow and reclaim capacity as designed
Host NetworkPolicies block cross-tenant and unapproved public traffic
RuntimeClass, GPU environment variables, and ephemeral containers cannot bypass the KAI Queue
nvidia-smi succeeds inside every GPU test container; a Running Pod alone is insufficient
L40S memory overruns are handled by application limits or HAMi software limits; hard isolation uses different hardware

Pin the reproduction versions and the CDI/NRI contract

NVIDIA's August 3, 2026 tutorial uses this combination:

Component	Validated version
Operating system	Ubuntu 24.04.4 LTS
Kubernetes distribution	MicroK8s 1.36.2
GPU	1 NVIDIA L40S
GPU Operator	26.3.3
KAI Scheduler	0.16.4
vCluster CLI	0.35.1

Use these exact versions for this reproduction. As of 2026-08-04, the latest releases were KAI Scheduler 0.17.0 and vCluster 0.36.1. A validated combination is not the same as the latest combination. Read the release notes and run preproduction regression tests before upgrading. KAI 0.17.0 adds delayed preemption and Dynamic Resource Allocation (DRA)-based extended resources, so an upgrade requires new reclaim-timing and host-admission tests.

Check the CLI, nodes, storage classes, and GPU Operator Pods:

vcluster --version
kubectl get nodes -o wide
kubectl get storageclass
kubectl get pods -n gpu-operator-resources

vcluster --version must report 0.35.1. Some distributions install GPU Operator in gpu-operator instead. Before continuing, GPU nodes must be Ready, a default StorageClass usable by vCluster must exist, and the NVIDIA device plugin, container toolkit, and validator must not be failing continuously.

This path pins GPU Operator 26.3.3. Its runtime contract is cdi.enabled=true, cdi.nriPluginEnabled=false, with the nvidia RuntimeClass retained. GPU Operator 25.10.0 and later use the Container Device Interface (CDI) by default, while the Node Resource Interface (NRI) plugin remains optional. Run:

kubectl get clusterpolicy
kubectl get clusterpolicy cluster-policy \
  -o jsonpath='{.spec.cdi.enabled}{"\t"}{.spec.cdi.nriPluginEnabled}{"\n"}'
kubectl get runtimeclass \
  -o custom-columns=NAME:.metadata.name,HANDLER:.handler

The two values must be true and false, respectively, and the RuntimeClass list must include nvidia. If the ClusterPolicy is not named cluster-policy, replace it with the name returned by the first command. When CDI is false, correct it according to the GPU Operator 26.3 CDI documentation and rerun the validator.

For the pinned MicroK8s setup, log in to every GPU node and read containerd's effective merged configuration:

sudo /snap/microk8s/current/bin/containerd \
  --config /var/snap/microk8s/current/args/containerd.toml \
  config dump | grep -E 'default_runtime_name[[:space:]]*='

The default runtime must be runc or another audited non-NVIDIA handler. Stop if the command has no output or reports nvidia, then verify the node's actual containerd path and imported files. On other Kubernetes distributions, follow the GPU Operator troubleshooting guide and inspect the effective configuration with containerd config dump or crio status config.

GPU Operator 26.3 no longer configures the NVIDIA handler as the default runtime, and the admission guard below relies on that behavior. If a node was manually changed to use the NVIDIA handler by default, an image-baked NVIDIA_VISIBLE_DEVICES value will not appear in the Pod spec and admission cannot detect it. Stop and restore the normal default handler using the procedure for the active runtime. If the RuntimeClass list also contains nvidia-cdi, nvidia-legacy, or custom names, record them in the change plan. The policy below sends every explicit RuntimeClass through the queue guard so aliases fail closed.

Stop here when NRI is true. NRI removes the nvidia RuntimeClass, while KAI Scheduler 0.16.4 defaults admission.gpuFractionRuntimeClassName to nvidia; continuing would inject a nonexistent RuntimeClass into fractional GPU Pods. This design also relies on GPU management containers requiring the nvidia RuntimeClass when NRI is disabled, which closes unaccounted device access. Merely setting KAI's RuntimeClass to an empty value is therefore unsafe. Clusters that require NRI need a separate host-admission or runtime policy and are outside this pinned path.

If NRI is enabled and no other workload depends on it, disable it through the pinned ClusterPolicy:

kubectl patch clusterpolicy cluster-policy --type merge \
  -p '{"spec":{"cdi":{"enabled":true,"nriPluginEnabled":false}}}'
kubectl get clusterpolicy cluster-policy \
  -o jsonpath='{.spec.cdi.enabled}{"\t"}{.spec.cdi.nriPluginEnabled}{"\n"}'
kubectl get runtimeclass nvidia

Wait for the container toolkit, device plugin, and validator to recover before continuing. The KAI installation explicitly sets cdiEnabled=true; GPU reservation Pods still do not need binder.runtimeClassName. These are different RuntimeClass settings and must not be conflated. NVML in later logs means NVIDIA Management Library.

When using MicroK8s, enable DNS and storage for vCluster PVCs:

microk8s enable dns
microk8s enable hostpath-storage

hostpath-storage is suitable only for this single-node reproduction. Use a dynamic StorageClass that meets recovery requirements on multi-node or production clusters and skip the second command.

Install KAI Scheduler from its Open Container Initiative (OCI) Helm chart under the runtime contract above. In addition to GPU sharing, block ordinary containers from setting NVIDIA_VISIBLE_DEVICES and pin CDI and RuntimeClass behavior explicitly:

helm upgrade -i kai-scheduler \
  oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler \
  -n kai-scheduler --create-namespace \
  --version v0.16.4 \
  --set "global.gpuSharing=true" \
  --set "global.blockNvidiaVisibleDevices=true" \
  --set "binder.cdiEnabled=true" \
  --set "admission.gpuFractionRuntimeClassName=nvidia" \
  --set "defaultPriorityClasses.enabled=true"

Check the deployment and KAI Config:

kubectl get pods -n kai-scheduler
kubectl get configs.kai.scheduler kai-config -o yaml
kubectl get configs.kai.scheduler kai-config \
  -o jsonpath='{.spec.admission.gpuSharing}{"\t"}{.spec.admission.blockNvidiaVisibleDevices}{"\t"}{.spec.admission.gpuFractionRuntimeClassName}{"\t"}{.spec.binder.cdiEnabled}{"\n"}'
kubectl get priorityclass train inference

The JSONPath command must print true true nvidia true in that order. The last command must find the chart-managed train and inference PriorityClasses. Correct the Helm values before proceeding if any value differs; later Pod tests are not a substitute for configuration validation.

The output should include these controllers and services:

Path	Components
Admission and grouping	admission, pod-grouper, podgroup-controller
Scheduling and binding	scheduler, binder
Configuration and queues	operator, queue-controller

All long-running components must be Running. A reservation Pod is created after a shared-GPU request arrives; it must not repeatedly fail with RuntimeClass or NVML errors. A workload without schedulerName: kai-scheduler remains on the default kube-scheduler and never enters this KAI queue path.

Set the organization's guaranteed quota and hard limit

In KAI Scheduler, quota is the guaranteed allocation and limit is the hard ceiling; limit: -1 means unlimited. NVIDIA's example configures the parent queue as quota: 1, limit: -1. Physical capacity constrains a single-GPU environment, but the same configuration can consume more than one GPU when copied to a larger cluster. The following create-queues.yaml sets the parent limit to 1 so the organization can use no more than one GPU.

apiVersion: scheduling.run.ai/v2
kind: Queue
metadata:
  name: ml-org
spec:
  resources:
    gpu: { quota: 1, limit: 1, overQuotaWeight: 1 }
---
apiVersion: scheduling.run.ai/v2
kind: Queue
metadata:
  name: team-nlp
spec:
  parentQueue: ml-org
  priority: 100
  resources:
    gpu: { quota: 0.33, limit: 1, overQuotaWeight: 1 }
---
apiVersion: scheduling.run.ai/v2
kind: Queue
metadata:
  name: team-vision
spec:
  parentQueue: ml-org
  priority: 100
  resources:
    gpu: { quota: 0.33, limit: 1, overQuotaWeight: 1 }
---
apiVersion: scheduling.run.ai/v2
kind: Queue
metadata:
  name: team-recommender
spec:
  parentQueue: ml-org
  priority: 100
  resources:
    gpu: { quota: 0.33, limit: 1, overQuotaWeight: 1 }

According to the KAI 0.16.4 Queue documentation, each child queue has a 0.33 guarantee and may borrow up to 1 when sibling queues have no demand. overQuotaWeight controls the relative allocation of capacity above the guarantee among queues of equal priority. The Queue field priority orders queues; it is not the Pod's Kubernetes PriorityClass.

Apply the queues and inspect the hierarchy:

kubectl apply -f create-queues.yaml
kubectl get queues
kubectl describe queue ml-org

Do not copy the three-way split mechanically into production. Derive quota, limit, priority, and overQuotaWeight from minimum throughput, peak windows, and workload priorities, then set the parent limit to the organization's actual GPU budget.

Create the vClusters and limit Node information exposure

In vCluster 0.35.1, setOwner only controls whether synchronized host objects are attached to the vCluster Service for garbage collection. It does not reconstruct virtual Job, Deployment, or ReplicaSet owner chains on the host. With true, KAI can mistake the vCluster Service for a physical Pod's top-level workload. The following vcluster.yaml sets it to false, so the bare Pods used below remain independent scheduling units.

The same configuration enables host-managed isolation controls. The baseline Pod Security Standard (PSS) rejects privileged containers, hostPath, and similar escape capabilities. The default NetworkPolicy permits only workloads from the same vCluster, the virtual control plane, and DNS; public workload egress is disabled here. ResourceQuota and LimitRange constrain CPU, memory, ephemeral storage, and object counts so GPU scheduling does not leave another noisy-neighbor path. Save this as vcluster.yaml:

experimental:
  syncSettings:
    setOwner: false
policies:
  podSecurityStandard: baseline
  networkPolicy:
    enabled: true
    workload:
      publicEgress:
        enabled: false
  resourceQuota:
    enabled: true
    quota:
      requests.cpu: "4"
      requests.memory: 8Gi
      requests.storage: 20Gi
      requests.ephemeral-storage: 20Gi
      limits.cpu: "8"
      limits.memory: 16Gi
      limits.ephemeral-storage: 40Gi
      services.nodeports: 0
      services.loadbalancers: 0
      count/pods: 40
      count/services: 30
      count/secrets: 100
      count/configmaps: 100
      count/persistentvolumeclaims: 10
  limitRange:
    enabled: true
    default:
      cpu: "2"
      memory: 2Gi
      ephemeral-storage: 4Gi
    defaultRequest:
      cpu: 100m
      memory: 256Mi
      ephemeral-storage: 1Gi
sync:
  toHost:
    networkPolicies:
      enabled: false
    resourceClaims:
      enabled: false
  fromHost:
    runtimeClasses:
      enabled: false
    nodes:
      enabled: true
      clearImageStatus: true
      selector:
        labels:
          nvidia.com/gpu.present: "true"
    priorityClasses:
      enabled: true

These numbers are reproduction ceilings, not production capacity derived from the current machine. ResourceQuota also counts the vCluster control-plane Pod and PVC. Before creation, inspect GPU-node allocatable values and current use, reserve capacity for system components and recovery, then adjust each quota to the team's requirements:

kubectl get nodes -l nvidia.com/gpu.present=true \
  -o custom-columns=NAME:.metadata.name,CPU:.status.allocatable.cpu,MEMORY:.status.allocatable.memory,EPHEMERAL:.status.allocatable.ephemeral-storage
kubectl top nodes

kubectl top requires Metrics Server. Use equivalent data from the existing monitoring system if it is unavailable. The combined request ceilings of all three tenants plus host-system headroom must not exceed node allocatable capacity.

This configuration does not use NVIDIA's selector.all: true. The vCluster Node synchronization documentation explains that all: true exposes every host Node to each tenant. The label selector above synchronizes only GPU nodes and applies the same selector to Pods synchronized to the host. clearImageStatus: true clears status.images, but the selected Node's name, scheduling labels and taints, capacity, and some runtime information can still be visible in all three vClusters. If those fields are sensitive, redact them with patches, disable real Node synchronization, or use private nodes.

priorityClasses.enabled: true synchronizes host PriorityClasses read-only into the tenant cluster. Without it, Pods that reference train or inference cannot synchronize to the host. All PriorityClasses are synchronized by default. Add a selector according to the vCluster PriorityClass documentation if the list must be narrowed, and keep both required classes in scope.

sync.fromHost.runtimeClasses.enabled: false does not expose host RuntimeClasses to tenants. KAI host admission injects the nvidia RuntimeClass into fractional GPU Pods, so tenants do not need to select a host handler directly. The server-side dry run on the host below validates the RuntimeClass bypass case.

policies.networkPolicy is managed on the host by the vCluster Helm release and does not depend on a tenant administrator preserving an object inside the virtual cluster. sync.toHost.networkPolicies.enabled: false prevents tenant-created NetworkPolicies from reaching the host; those virtual objects are not an effective network boundary. The host policy allows workloads from the same vCluster to communicate. Kubernetes NetworkPolicy permissions are additive, so even if synchronization were enabled, a tenant default-deny could not cancel an existing host-managed allow rule. For application segmentation within a tenant, use an administrator policy supported by the CNI or redesign the host rules instead of enabling NetworkPolicy synchronization on this configuration.

Create the three tenant clusters:

vcluster create team-nlp \
  --namespace vcluster-team-nlp \
  --values vcluster.yaml --connect=false
vcluster create team-vision \
  --namespace vcluster-team-vision \
  --values vcluster.yaml --connect=false
vcluster create team-recommender \
  --namespace vcluster-team-recommender \
  --values vcluster.yaml --connect=false
vcluster list
vcluster connect team-nlp --namespace vcluster-team-nlp -- \
  kubectl get priorityclass train inference
kubectl get resourcequota,limitrange,networkpolicy -n vcluster-team-nlp
kubectl get resourcequota,limitrange,networkpolicy -n vcluster-team-vision
kubectl get resourcequota,limitrange,networkpolicy -n vcluster-team-recommender

setOwner: false is not a general default. vCluster serializes virtual Pod owner references into annotations on the physical Pod, while KAI 0.16.4 pod-grouper traverses real host metadata.ownerReferences and their parent objects. This acceptance path covers bare Pods only. Do not extrapolate it to Jobs, Deployments, PyTorchJobs, or gang scheduling; those workloads need a separately designed and validated host PodGroup contract.

Disabling the owner reference also removes part of the garbage-collection protection. After component upgrades or vCluster deletion and recreation, inspect the corresponding host namespaces for orphaned Pods or PodGroups and reconfirm that bare Pods still have no physical owner reference:

kubectl get pods,podgroups -n vcluster-team-nlp
kubectl get pods,podgroups -n vcluster-team-vision
kubectl get pods,podgroups -n vcluster-team-recommender

Submit a privileged hostPath Pod from the NLP vCluster to prove that baseline prevents it from reaching the host. Save this manifest as invalid-privileged-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: invalid-privileged-hostpath
spec:
  containers:
    - name: test
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      securityContext:
        privileged: true
      volumeMounts:
        - name: host-dev
          mountPath: /host-dev
  volumes:
    - name: host-dev
      hostPath:
        path: /dev

vcluster connect team-nlp --namespace vcluster-team-nlp -- \
  kubectl get namespace default \
  -L pod-security.kubernetes.io/enforce
vcluster connect team-nlp --namespace vcluster-team-nlp -- \
  kubectl apply -f invalid-privileged-pod.yaml
vcluster connect team-nlp --namespace vcluster-team-nlp -- \
  kubectl get pod invalid-privileged-hostpath
kubectl logs -n vcluster-team-nlp statefulset/team-nlp \
  -c syncer --since=5m
kubectl get pods -n vcluster-team-nlp -o name | \
  grep -F invalid-privileged-hostpath
vcluster connect team-nlp --namespace vcluster-team-nlp -- \
  kubectl delete -f invalid-privileged-pod.yaml

The first command must report baseline. Pod Security Admission in the tenant API normally rejects the privileged Pod directly, in which case the tenant-side kubectl get returns NotFound. If the tenant API accepts the object first, the syncer must log a Pod Security Standard rejection. In either path, the host grep must have no output and exit with status 1. Stop immediately if a physical Pod exists. When the tenant API rejects the Pod directly, NotFound from the final delete command is expected.

Verify that host NetworkPolicy isolates the tenants

The vCluster shared-nodes guide warns that some CNIs accept NetworkPolicy objects without enforcing them. Establish reachability within Vision first, then connect from NLP to the same Pod IP to prove that host policy blocks cross-vCluster traffic:

vcluster connect team-vision --namespace vcluster-team-vision -- \
  kubectl run np-web --image=nginx:1.29-alpine --port=80
vcluster connect team-vision --namespace vcluster-team-vision -- \
  kubectl run np-client --image=busybox:1.36 --restart=Never -- sleep 3600
vcluster connect team-nlp --namespace vcluster-team-nlp -- \
  kubectl run np-client --image=busybox:1.36 --restart=Never -- sleep 3600
vcluster connect team-vision --namespace vcluster-team-vision -- \
  kubectl wait --for=condition=Ready pod/np-web pod/np-client --timeout=120s
vcluster connect team-nlp --namespace vcluster-team-nlp -- \
  kubectl wait --for=condition=Ready pod/np-client --timeout=120s
VISION_POD_IP="$(kubectl get pods -n vcluster-team-vision \
  -l vcluster.loft.sh/managed-by=team-vision,run=np-web \
  -o jsonpath='{.items[0].status.podIP}')"
test -n "$VISION_POD_IP"
vcluster connect team-vision --namespace vcluster-team-vision -- \
  kubectl exec np-client -- wget -qO- -T 3 "$VISION_POD_IP"

The same-tenant baseline must return the NGINX page. Before testing public egress, prove that the NLP Pod can resolve a public name through vCluster DNS. Otherwise, a failed wget proves only that DNS is broken, not that egress is blocked:

vcluster connect team-nlp --namespace vcluster-team-nlp -- \
  kubectl exec np-client -- nslookup example.com
vcluster connect team-nlp --namespace vcluster-team-nlp -- \
  kubectl exec np-client -- wget -qO- -T 3 "$VISION_POD_IP"
vcluster connect team-nlp --namespace vcluster-team-nlp -- \
  kubectl exec np-client -- wget -qO- -T 3 http://example.com

nslookup must return an address. Both wget commands must return no content and exit nonzero, usually by timing out. Any returned content means the CNI did not enforce the expected policy or the egress configuration was rewritten, so the shared-nodes network boundary fails acceptance. When workloads need a model repository or external API, add the narrowest destination CIDRs and ports under policies.networkPolicy.workload.egress; do not enable publicEgress wholesale. Image pulls are performed by the node runtime and are not governed by Pod egress rules.

Delete the three test Pods after validation:

vcluster connect team-nlp --namespace vcluster-team-nlp -- \
  kubectl delete pod np-client
vcluster connect team-vision --namespace vcluster-team-vision -- \
  kubectl delete pod np-web np-client

Bind each tenant to its queue on the host

A team with vCluster cluster-admin can choose any queue label, so the host must enforce tenant-to-queue binding. MicroK8s 1.36.2 supports stable ValidatingAdmissionPolicy. The policy below treats any KAI fractional annotation, full-GPU or MIG resource, explicit RuntimeClass, or NVIDIA_VISIBLE_DEVICES environment variable as GPU access. It also intercepts the ephemeral-container subresource of GPU Pods. Every RuntimeClass enters the guard so unknown NVIDIA handler aliases fail closed. If non-GPU RuntimeClasses such as Kata or gVisor are required, audit their handlers first and design exceptions for explicit safe names. The vCluster control plane uses none of these paths and does not match the policy.

vCluster 0.35.1 writes the actual vCluster name into the managed-by label of synchronized objects. The values here are team-nlp, team-vision, and team-recommender, not a constant vcluster. Save the policy and Binding as queue-guard.yaml:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: kai-vcluster-queue-guard
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods", "pods/ephemeralcontainers"]
  matchConditions:
    - name: gpu-access
      expression: >-
        (has(object.metadata.annotations) &#x26;&#x26;
         ('gpu-fraction' in object.metadata.annotations ||
          'gpu-memory' in object.metadata.annotations)) ||
        object.spec.?runtimeClassName.orValue('') != '' ||
        object.spec.containers.exists(c,
          (has(c.resources.limits) &#x26;&#x26; c.resources.limits.exists(r,
            r == 'nvidia.com/gpu' || r.startsWith('nvidia.com/mig-'))) ||
          (has(c.resources.requests) &#x26;&#x26; c.resources.requests.exists(r,
            r == 'nvidia.com/gpu' || r.startsWith('nvidia.com/mig-'))) ||
          c.env.exists(e, e.name == 'NVIDIA_VISIBLE_DEVICES')) ||
        (has(object.spec.initContainers) &#x26;&#x26; object.spec.initContainers.exists(c,
          (has(c.resources.limits) &#x26;&#x26; c.resources.limits.exists(r,
            r == 'nvidia.com/gpu' || r.startsWith('nvidia.com/mig-'))) ||
          (has(c.resources.requests) &#x26;&#x26; c.resources.requests.exists(r,
            r == 'nvidia.com/gpu' || r.startsWith('nvidia.com/mig-'))) ||
          c.env.exists(e, e.name == 'NVIDIA_VISIBLE_DEVICES'))) ||
        (has(object.spec.ephemeralContainers) &#x26;&#x26;
          object.spec.ephemeralContainers.exists(c,
            c.env.exists(e, e.name == 'NVIDIA_VISIBLE_DEVICES')))
  validations:
    - expression: >-
        has(object.metadata.labels) &#x26;&#x26;
        'vcluster.loft.sh/managed-by' in object.metadata.labels &#x26;&#x26;
        ((object.metadata.namespace == 'vcluster-team-nlp' &#x26;&#x26;
          object.metadata.labels['vcluster.loft.sh/managed-by'] == 'team-nlp') ||
         (object.metadata.namespace == 'vcluster-team-vision' &#x26;&#x26;
          object.metadata.labels['vcluster.loft.sh/managed-by'] == 'team-vision') ||
         (object.metadata.namespace == 'vcluster-team-recommender' &#x26;&#x26;
          object.metadata.labels['vcluster.loft.sh/managed-by'] == 'team-recommender'))
      message: "GPU Pods must carry the vCluster management label that matches the host namespace"
      reason: Forbidden
    - expression: >-
        has(object.metadata.labels) &#x26;&#x26;
        'kai.scheduler/queue' in object.metadata.labels &#x26;&#x26;
        ((object.metadata.namespace == 'vcluster-team-nlp' &#x26;&#x26;
          object.metadata.labels['kai.scheduler/queue'] == 'team-nlp') ||
         (object.metadata.namespace == 'vcluster-team-vision' &#x26;&#x26;
          object.metadata.labels['kai.scheduler/queue'] == 'team-vision') ||
         (object.metadata.namespace == 'vcluster-team-recommender' &#x26;&#x26;
          object.metadata.labels['kai.scheduler/queue'] == 'team-recommender'))
      message: "A GPU Pod's kai.scheduler/queue must match its vCluster"
      reason: Forbidden
    - expression: "object.spec.?schedulerName.orValue('') == 'kai-scheduler'"
      message: "GPU Pods must set schedulerName: kai-scheduler"
      reason: Forbidden
    - expression: >-
        !has(object.spec.ephemeralContainers) ||
        size(object.spec.ephemeralContainers) == 0
      message: "Ephemeral containers are forbidden in GPU Pods; create a separate debug Pod"
      reason: Forbidden
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: kai-vcluster-queue-guard
spec:
  policyName: kai-vcluster-queue-guard
  validationActions: [Deny]
  matchResources:
    namespaceSelector:
      matchExpressions:
        - key: kubernetes.io/metadata.name
          operator: In
          values:
            - vcluster-team-nlp
            - vcluster-team-vision
            - vcluster-team-recommender

Tenant identity mapping belongs in validations, not matchConditions. Missing labels, renamed vClusters, or drifted values now deny GPU access instead of making the match condition false and skipping the policy. When a vCluster or host namespace is renamed, update the managed-by value, object.metadata.namespace, and Binding values together. Below Kubernetes 1.30, provide equivalent host enforcement with an admission webhook, Kyverno, or Gatekeeper.

NRI must remain false on this pinned path, and vcluster.yaml explicitly keeps sync.toHost.resourceClaims.enabled at false. Matching nvidia.com/mig-* prevents unguarded admission of those resources; it does not imply that L40S supports MIG. Before changing the GPU, enabling ResourceClaims, or adopting KAI 0.17.0 DRA extended resources, redesign vCluster synchronization, Queue resources, GPU-access matching, and negative tests together.

The policy applies only when a Pod is created or updated. It does not remediate existing objects. Do not distribute tenant kubeconfigs until the host policy, Binding, and negative tests below all pass. At rollout, also search the three host namespaces for existing GPU Pods. ValidatingAdmissionPolicy and its Binding are host objects; restrict their modification and deletion with RBAC and retain audit logs for both operations.

Save these six negative cases as invalid-gpu-pods.yaml. They test a wrong queue, missing queue, default scheduler, missing tenant identity, explicit RuntimeClass, and GPU environment-variable bypass:

apiVersion: v1
kind: Pod
metadata:
  name: invalid-wrong-queue
  namespace: vcluster-team-nlp
  labels:
    vcluster.loft.sh/managed-by: team-nlp
    kai.scheduler/queue: team-vision
  annotations:
    gpu-fraction: "0.10"
spec:
  schedulerName: kai-scheduler
  containers:
    - name: test
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
---
apiVersion: v1
kind: Pod
metadata:
  name: invalid-missing-queue
  namespace: vcluster-team-nlp
  labels:
    vcluster.loft.sh/managed-by: team-nlp
  annotations:
    gpu-fraction: "0.10"
spec:
  schedulerName: kai-scheduler
  containers:
    - name: test
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
---
apiVersion: v1
kind: Pod
metadata:
  name: invalid-default-scheduler
  namespace: vcluster-team-nlp
  labels:
    vcluster.loft.sh/managed-by: team-nlp
    kai.scheduler/queue: team-nlp
  annotations:
    gpu-fraction: "0.10"
spec:
  containers:
    - name: test
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
---
apiVersion: v1
kind: Pod
metadata:
  name: invalid-missing-managed-by
  namespace: vcluster-team-nlp
  labels:
    kai.scheduler/queue: team-nlp
  annotations:
    gpu-fraction: "0.10"
spec:
  schedulerName: kai-scheduler
  containers:
    - name: test
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
---
apiVersion: v1
kind: Pod
metadata:
  name: invalid-runtime-bypass
  namespace: vcluster-team-nlp
  labels:
    vcluster.loft.sh/managed-by: team-nlp
    kai.scheduler/queue: team-nlp
spec:
  runtimeClassName: nvidia
  containers:
    - name: test
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
---
apiVersion: v1
kind: Pod
metadata:
  name: invalid-visible-devices-bypass
  namespace: vcluster-team-nlp
  labels:
    vcluster.loft.sh/managed-by: team-nlp
    kai.scheduler/queue: team-nlp
spec:
  containers:
    - name: test
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      env:
        - name: NVIDIA_VISIBLE_DEVICES
          value: all

Apply the policy, inspect Common Expression Language (CEL) type checking, and then run a server-side dry run on the host:

kubectl apply -f queue-guard.yaml
kubectl get validatingadmissionpolicy kai-vcluster-queue-guard -o yaml
kubectl apply --dry-run=server -f invalid-gpu-pods.yaml

The output must include status.typeChecking; expressionWarnings must be absent or []. If status has not been generated yet, wait for the API server to finish and rerun kubectl get rather than proceeding to Pod tests. Resolve every warning first. All six Pods must be denied by kai-vcluster-queue-guard; a server-side dry run leaves no objects behind. invalid-runtime-bypass sets no GPU resource or environment variable and proves that RuntimeClass alone enters the guard.

Also prove that KAI changes a container that enters its admission path without requesting a GPU to NVIDIA_VISIBLE_DEVICES=void. Save this as blocked-visible-devices.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: blocked-visible-devices
  namespace: vcluster-team-nlp
  labels:
    vcluster.loft.sh/managed-by: team-nlp
    kai.scheduler/queue: team-nlp
spec:
  schedulerName: kai-scheduler
  runtimeClassName: nvidia
  containers:
    - name: test
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      env:
        - name: NVIDIA_VISIBLE_DEVICES
          value: all

kubectl apply --dry-run=server -f blocked-visible-devices.yaml \
  -o jsonpath='{.spec.containers[0].env[?(@.name=="NVIDIA_VISIBLE_DEVICES")].value}{"\n"}'

The output must be void. Do not continue if it remains all, the request is admitted without the rewrite, or the KAI webhook does not respond.

A host dry run validates physical-Pod admission only. Submit a wrong-queue case and a GPU environment-variable bypass through a real vCluster as well. Save these as invalid-tenant-pods.yaml and do not add the vCluster management label manually:

apiVersion: v1
kind: Pod
metadata:
  name: invalid-tenant-wrong-queue
  labels:
    kai.scheduler/queue: team-vision
  annotations:
    gpu-fraction: "0.10"
spec:
  schedulerName: kai-scheduler
  containers:
    - name: test
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
---
apiVersion: v1
kind: Pod
metadata:
  name: invalid-tenant-visible-devices-bypass
  labels:
    kai.scheduler/queue: team-nlp
spec:
  containers:
    - name: test
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      env:
        - name: NVIDIA_VISIBLE_DEVICES
          value: all

After submission from the NLP vCluster, the tenant API may accept the objects first. The syncer adds managed-by: team-nlp, after which host policy denies the physical Pods for the wrong queue and missing KAI scheduler, respectively. Inspect the control-plane log and prove that no physical Pod exists:

vcluster connect team-nlp --namespace vcluster-team-nlp -- \
  kubectl apply -f invalid-tenant-pods.yaml
kubectl logs -n vcluster-team-nlp statefulset/team-nlp \
  -c syncer --since=5m
kubectl get pods -n vcluster-team-nlp -o name | \
  grep -E 'invalid-tenant-(wrong-queue|visible-devices-bypass)'
vcluster connect team-nlp --namespace vcluster-team-nlp -- \
  kubectl delete -f invalid-tenant-pods.yaml

After at least one synchronization attempt, the log must contain a kai-vcluster-queue-guard denial for each Pod. grep must have no output and exit with status 1. Retries can produce multiple records for the same name, so do not assert on the total log count. A successful tenant-side kubectl apply does not mean the host created a physical Pod.

Submit GPU test Pods for borrow and reclaim validation

The following Pod tests scheduling, borrowing, and reclaim only; it is not a production inference service. It uses the preemptible train PriorityClass, whose value is 50. Production inference should use the non-preemptible inference class, whose value is 125, and keep requests within guaranteed queue capacity. KAI 0.16.4 does not let workloads with priority values of 100 or higher borrow over-quota capacity; see the workload priority documentation. Public image tags make the reproduction convenient. Production must use scanned internal images pinned by digest. Save this as nlp-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: nlp-batch-test
  labels:
    kai.scheduler/queue: team-nlp
  annotations:
    gpu-fraction: "0.33"
spec:
  schedulerName: kai-scheduler
  priorityClassName: train
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
  containers:
    - name: cuda-test
      image: nvidia/cuda:12.4.0-base-ubuntu22.04
      command: ["bash", "-c", "nvidia-smi -L &#x26;&#x26; nvidia-smi &#x26;&#x26; sleep infinity"]
  nodeSelector:
    nvidia.com/gpu.present: "true"

Copy nlp-pod.yaml to vision-pod.yaml and recommender-pod.yaml. Rename the Pods to vision-batch-test and recommender-batch-test, update kai.scheduler/queue, and submit each manifest to its vCluster:

vcluster connect team-nlp --namespace vcluster-team-nlp -- \
  kubectl apply -f nlp-pod.yaml
vcluster connect team-vision --namespace vcluster-team-vision -- \
  kubectl apply -f vision-pod.yaml
vcluster connect team-recommender --namespace vcluster-team-recommender -- \
  kubectl apply -f recommender-pod.yaml

All three Pods must become Ready, and both nvidia-smi commands in their logs must succeed. The pinned setup contains one L40S, so all three nvidia-smi -L outputs must also show the same GPU UUID. A timeout, CrashLoopBackOff, missing GPU list, or NVML error fails acceptance:

vcluster connect team-nlp --namespace vcluster-team-nlp -- \
  kubectl wait --for=condition=Ready pod/nlp-batch-test --timeout=180s
vcluster connect team-vision --namespace vcluster-team-vision -- \
  kubectl wait --for=condition=Ready pod/vision-batch-test --timeout=180s
vcluster connect team-recommender --namespace vcluster-team-recommender -- \
  kubectl wait --for=condition=Ready pod/recommender-batch-test --timeout=180s
vcluster connect team-nlp --namespace vcluster-team-nlp -- \
  kubectl logs pod/nlp-batch-test
vcluster connect team-vision --namespace vcluster-team-vision -- \
  kubectl logs pod/vision-batch-test
vcluster connect team-recommender --namespace vcluster-team-recommender -- \
  kubectl logs pod/recommender-batch-test

On the host, confirm that both the real management label and queue label exist, and prove that the bare Pod has no physical owner reference:

kubectl get pods -n vcluster-team-nlp \
  -l vcluster.loft.sh/managed-by=team-nlp --show-labels
kubectl get pods -n vcluster-team-nlp \
  -l vcluster.loft.sh/managed-by=team-nlp,kai.scheduler/queue=team-nlp \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.ownerReferences}{"\n"}{end}'

The owner references after nlp-batch-test in the second command must be empty. If a vCluster Service or another owner appears, setOwner: false did not take effect and the bare-Pod scheduling conclusion is invalid.

Once a physical GPU Pod exists, call its host ephemeralcontainers subresource directly. This proves that admission prevents a debug container from entering a GPU Pod through a path KAI 0.16.4 does not process:

HOST_NLP_POD="$(kubectl get pods -n vcluster-team-nlp \
  -l vcluster.loft.sh/managed-by=team-nlp,kai.scheduler/queue=team-nlp \
  -o jsonpath='{.items[0].metadata.name}')"
test -n "$HOST_NLP_POD"
kubectl debug -n vcluster-team-nlp "$HOST_NLP_POD" \
  --image=nvidia/cuda:12.4.0-base-ubuntu22.04 \
  --target=cuda-test -- nvidia-smi
kubectl get pod -n vcluster-team-nlp "$HOST_NLP_POD" \
  -o jsonpath='{.spec.ephemeralContainers}{"\n"}'

kubectl debug must be denied by kai-vcluster-queue-guard with Forbidden, and the final command must print an empty array or empty value. Stop and fix the policy if an ephemeral container appears.

According to the KAI 0.16.4 GPU sharing documentation, gpu-fraction: "0.33" records a request for 33% of device memory and permits colocating Pods whose total requests do not exceed device capacity. KAI does not enforce the actual memory ceiling by default, so a Pod can still consume more memory than it requested.

gpu-fraction does not guarantee 33% of GPU cores, time slices, throughput, or latency, and it is not a performance SLA. NVIDIA's GPU Operator 26.3 time-slicing documentation likewise states that the number of shared requests does not represent proportional compute capacity.

Validate tenant views, queue reclaim, and memory boundaries

Check tenant views and Node exposure

Each vCluster must show only its team's namespaced workloads:

vcluster connect team-nlp --namespace vcluster-team-nlp -- \
  kubectl get pods -A -o wide
vcluster connect team-vision --namespace vcluster-team-vision -- \
  kubectl get pods -A -o wide
vcluster connect team-recommender --namespace vcluster-team-recommender -- \
  kubectl get pods -A -o wide

Nodes are not tenant-private objects. Each vCluster must see only GPU nodes selected by the label selector, with an empty status.images. Review every other label, taint, capacity field, and runtime field for acceptable disclosure:

vcluster connect team-nlp --namespace vcluster-team-nlp -- \
  kubectl get nodes -o yaml
vcluster connect team-vision --namespace vcluster-team-vision -- \
  kubectl get nodes -o yaml
vcluster connect team-recommender --namespace vcluster-team-recommender -- \
  kubectl get nodes -o yaml

If any sensitive Node field exceeds the approved scope, stop the rollout and adjust the patches, disable real Node synchronization, or switch to private nodes.

Check host placement and queue allocation

Inspect physical Pods, vCluster management labels, and all three queues:

kubectl get pods -A -o wide --show-labels
kubectl get pods -n kai-resource-reservation
kubectl describe queue team-nlp
kubectl describe queue team-vision
kubectl describe queue team-recommender

In NVIDIA's example, every queue shows 330m GPU requested and allocated. Do not treat 330m as a fixed expected output. Acceptance requires agreement among the actual Pod annotation, Queue status, scheduling event, and reservation Pod.

Prove borrowing and reclaim with an extra request

An extra request is required to prove that a queue borrowed idle quota. Save this preemptible Pod as nlp-burst.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: nlp-burst
  labels:
    kai.scheduler/queue: team-nlp
  annotations:
    gpu-fraction: "0.66"
spec:
  schedulerName: kai-scheduler
  priorityClassName: train
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
  containers:
    - name: nlp-burst
      image: nvidia/cuda:12.4.0-base-ubuntu22.04
      command: ["bash", "-c", "nvidia-smi -L &#x26;&#x26; nvidia-smi &#x26;&#x26; sleep infinity"]
  nodeSelector:
    nvidia.com/gpu.present: "true"

Stop the other two teams' 0.33 Pods, then submit the 0.66 Pod. Stopping other Pods does not enlarge the existing 0.33 request and is not borrowing evidence by itself.

vcluster connect team-vision --namespace vcluster-team-vision -- \
  kubectl delete -f vision-pod.yaml
vcluster connect team-recommender --namespace vcluster-team-recommender -- \
  kubectl delete -f recommender-pod.yaml
vcluster connect team-nlp --namespace vcluster-team-nlp -- \
  kubectl apply -f nlp-burst.yaml
vcluster connect team-nlp --namespace vcluster-team-nlp -- \
  kubectl wait --for=condition=Ready pod/nlp-burst --timeout=180s
vcluster connect team-nlp --namespace vcluster-team-nlp -- \
  kubectl logs pod/nlp-burst
kubectl describe queue team-nlp

The burst Pod must become Ready, with the GPU list and full nvidia-smi status output in its log. On successful borrowing, total NLP requests are about 0.99, above the 0.33 guarantee but no higher than the 1 limit. Restore Vision's 0.33 Pod next:

vcluster connect team-vision --namespace vcluster-team-vision -- \
  kubectl apply -f vision-pod.yaml
kubectl get pods -A -o wide
kubectl describe queue team-nlp
kubectl describe queue team-vision
kubectl get events -A --sort-by=.lastTimestamp

The below-guarantee Vision queue must regain 0.33 within the scheduling service-level objective (SLO) agreed by the team, and NLP must not retain 0.99 indefinitely. The 0.66 Pod is an indivisible scheduling unit; the scheduler may evict it or return it to pending. Evaluate Queue state, Pod state, and events rather than requiring a fixed victim Pod or universal number of seconds. See the fair-share documentation for reclaim rules and hierarchical exceptions.

After Vision passes, restore Recommender and repeat the checks. Recommender must also receive 0.33 within the agreed SLO. A guaranteed queue that remains pending fails the reclaim or starvation test. Finally, delete the burst Pod and restore the three-queue 0.33 baseline:

vcluster connect team-recommender --namespace vcluster-team-recommender -- \
  kubectl apply -f recommender-pod.yaml
kubectl describe queue team-nlp
kubectl describe queue team-recommender
kubectl get events -A --sort-by=.lastTimestamp
vcluster connect team-nlp --namespace vcluster-team-nlp -- \
  kubectl delete -f nlp-burst.yaml

Test memory and failure boundaries

Run a test program that increases GPU memory use in steps and record what happens after the application exceeds its agreed allocation. Default KAI configuration accounts for scheduling but does not stop a process from exceeding its gpu-fraction or gpu-memory request. Colocated workloads can therefore run out of memory (OOM) or interfere with one another.

Trusted teams that need a software memory ceiling independent of application configuration can evaluate the KAI 0.16.4 HAMi integration. kai-resource-isolator injects HAMi-core, which limits container memory by intercepting CUDA calls. Validate CUDA, image, and LD_PRELOAD compatibility before rollout. HAMi remains software isolation and does not provide a MIG hardware fault domain or security boundary.

There is no in-place “switch to MIG” option for this L40S environment. If cross-tenant memory and compute interference is unacceptable, use an exclusive L40S or move to a MIG-capable GPU. After changing hardware, redefine GPU Operator/MIG Manager behavior, Queue resources, workload requests, vCluster synchronization, and host admission. The current admission policy only preemptively matches nvidia.com/mig-*; it does not validate a MIG path.