Kueue vs Volcano vs KAI Scheduler for GPU inference and training

When a Kubernetes cluster gets its first few GPU nodes, the default scheduler is usually enough. Problems start when several teams share the cluster. Training jobs want all their pods or none. Inference deployments want guaranteed capacity at peak. Someone's idle notebook holds a GPU for days. At that point people search for "Kueue vs Volcano", and lately "Kueue vs KAI".

These three projects are often listed side by side, but they do not sit at the same layer. This guide compares them by architecture first and by features second, because the architectural difference decides most of the trade-offs.

A note on scope: I run inference on Kubernetes in my own AI Runtime Platform (vLLM and KServe paths, KEDA scaling). My GPU Admission on Kueue project is In progress, not yet published: there is no public repository and no measurements yet. So everything below comes from the projects' current documentation (links in References), not from benchmarks of mine. There are no performance numbers here because I do not have any I could stand behind.

The one-line difference

• Kueue is a job queueing and quota admission layer. It decides when a workload may start (be admitted) and leaves pod placement to the normal kube-scheduler. In Kueue's words, a Workload is "the unit of admission", and the docs say quota reservation "should not to be confused with pod scheduling" (sic).
• Volcano is a batch scheduler (a separate scheduler, schedulerName: volcano) plus its own job API. The CNCF accepted it as a batch scheduling project.
• KAI Scheduler (NVIDIA, open source, built on kube-batch) is a Kubernetes scheduler for AI workloads. Pods opt in with schedulerName: kai-scheduler and a queue label.

So Kueue works next to the default scheduler. Volcano and KAI replace it for the pods that opt in.

Kueue: admission and quota on top of kube-scheduler

Kueue's core objects are:

• ResourceFlavor: describes a class of nodes or resources (for example a GPU type), usually through node labels.
• ClusterQueue: a cluster-scoped pool with quotas per flavor and fair-sharing rules.
• LocalQueue: a namespaced handle that tenants submit to. Users never submit to a ClusterQueue directly.
• Cohort: a group of ClusterQueues that can borrow unused quota from each other.
• WorkloadPriorityClass, AdmissionCheck, Topology (for Topology Aware Scheduling), and quota for Dynamic Resource Allocation devices.

A minimal GPU setup in the current docs' API version (kueue.x-k8s.io/v1beta2; older releases used v1beta1, so check what your installed version serves with kubectl api-resources | grep kueue):

apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: gpu-a
spec:
  nodeLabels:
    nvidia.com/gpu.present: "true"
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: team-a-cq
spec:
  namespaceSelector: {}
  cohortName: gpu-pool
  resourceGroups:
  - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
    flavors:
    - name: gpu-a
      resources:
      - name: "cpu"
        nominalQuota: 32
      - name: "memory"
        nominalQuota: 256Gi
      - name: "nvidia.com/gpu"
        nominalQuota: 4
        borrowingLimit: 2
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
  namespace: team-a
  name: team-a
spec:
  clusterQueue: team-a-cq

The quota values above are placeholders. Kueue accounts quota nominally: it admits against what you declare, not against live node capacity. If you declare more GPUs than you have, pods get admitted and then sit Pending as unschedulable. The llm-d guide on sharing a GPU budget calls this out explicitly. Without Topology Aware Scheduling, Kueue checks aggregate quota only. An 8-GPU pod can be admitted when 8 GPUs are free in total but split across two nodes, and then it never schedules. The Kueue docs recommend enabling TAS for batch and ML workloads for this reason.

A workload joins a queue with a label. For a batch Job:

apiVersion: batch/v1
kind: Job
metadata:
  name: finetune-1
  namespace: team-a
  labels:
    kueue.x-k8s.io/queue-name: team-a
spec:
  suspend: true
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: train
        image: registry.example.com/train:1.0
        resources:
          limits:
            nvidia.com/gpu: "1"

Inference with Kueue. Kueue lists "mixing training and inference" as a feature: serving workloads such as Deployments or StatefulSets can be managed next to batch. The llm-d project documents a pattern where each model Deployment points at its own LocalQueue. Kueue holds every replica pod at a scheduling gate until quota is free, and models in one cohort lend idle GPUs to each other and preempt to take their floor back. The same guide lists the sharp edges: the queue-name label on a Deployment is effectively one-way, gated pods pile up if the HPA asks for more than the budget, and rollouts need slack in the budget. AWS documents a similar approach for SageMaker HyperPod on EKS: high-priority classes for inference, reserved quota for spikes, and optional borrowing.

Where Kueue fits best: you want to keep kube-scheduler (and things that depend on it, such as the cluster autoscaler's ProvisioningRequest integration), you need multi-tenant quotas with borrowing and preemption, and you run a mix of Jobs, JobSets, Kubeflow, Ray and plain pods. MultiKueue adds dispatching across clusters.

Volcano: a batch scheduler with its own job model

Volcano brings a full scheduler and a richer job abstraction. Its docs list gang scheduling, binpack, DRF, proportion/capacity scheduling with queue quotas, heterogeneous device scheduling (including GPU sharing in CUDA and MIG modes, plus NPU), network-topology-aware scheduling, NUMA-aware scheduling, online/offline colocation, multi-cluster scheduling and descheduling. It integrates with Spark, PyTorch, TensorFlow, Ray, Flink, MPI and Kubeflow.

Queues are scheduling.volcano.sh/v1beta1 objects. The fields depend on the enabled plugin: capability is the cap, deserved is used by the capacity plugin, weight by the proportion plugin, and guarantee reserves resources other queues cannot use:

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: training
spec:
  reclaimable: true
  capability:
    cpu: "64"
    memory: 512Gi
    nvidia.com/gpu: "8"
  guarantee:
    resource:
      cpu: "16"
      memory: 128Gi

Jobs use batch.volcano.sh/v1alpha1 with minAvailable for gang semantics:

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: ddp-train
spec:
  schedulerName: volcano
  queue: training
  minAvailable: 4
  maxRetry: 3
  policies:
  - event: PodEvicted
    action: RestartJob
  tasks:
  - name: worker
    replicas: 4
    template:
      spec:
        restartPolicy: OnFailure
        containers:
        - name: worker
          image: registry.example.com/ddp:1.0
          resources:
            limits:
              nvidia.com/gpu: "1"

Where Volcano fits best: large batch and HPC-style training where gang scheduling, topology and job lifecycle policies (restart on eviction, task dependencies) matter more than staying on the default scheduler. You also get one scheduler for both online and offline workloads if you use its colocation features. The cost is operating a second scheduler and learning its plugin and configuration model.

KAI Scheduler: GPU-first scheduling with hierarchical queues

KAI Scheduler describes itself as a Kubernetes scheduler that "optimizes GPU resource allocation for AI and machine learning workloads". It can run alongside other schedulers and requires the NVIDIA GPU Operator for GPU workloads. Its listed features include batch (gang) scheduling, bin packing and spread, workload priority with priority separated from preemptibility, hierarchical queues with quotas, limits and over-quota weights, DRF-based fairness and reclaim, time-based fairshare, a minimum guaranteed runtime before preemption, workload consolidation, elastic workloads, topology-aware scheduling, hierarchical PodGroups, DRA support, and GPU sharing.

A default two-level queue hierarchy is created at install. A queue uses scheduling.run.ai/v2:

apiVersion: scheduling.run.ai/v2
kind: Queue
metadata:
  name: inference
spec:
  parentQueue: default-parent-queue
  resources:
    gpu:
      quota: 4
      limit: -1
      overQuotaWeight: 1
    cpu:
      quota: -1
      limit: -1
      overQuotaWeight: 1
    memory:
      quota: -1
      limit: -1
      overQuotaWeight: 1

Pods opt in with two fields:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
  labels:
    kai.scheduler/queue: inference
spec:
  schedulerName: kai-scheduler
  containers:
  - name: main
    image: ubuntu
    args: ["sleep", "infinity"]
    resources:
      limits:
        nvidia.com/gpu: "1"

GPU sharing is chosen at install time: NonMemoryEnforced, HamiCore (memory isolation based on the HAMi project), NvFractions (fractioning with CUDA memory limit enforcement) or Disabled. The project documents integrations with KubeRay and with Grove/Dynamo for disaggregated serving.

Where KAI fits best: GPU-heavy clusters where fractional GPUs, hierarchical team quotas with over-quota sharing, and time-based fairness are the main requirements. Like Volcano, it is a scheduler you operate in addition to kube-scheduler.

Side-by-side

The table only lists what each project's docs state as of 21 October 2026. Read "not listed" as "check yourself", not as "missing".

How I would choose

These are decision rules, not results:

1. Start from the scheduler question. If replacing or adding a scheduler is off the table (managed clusters, autoscaler integration, platform team policy), Kueue is the only one of the three that keeps kube-scheduler in charge of placement.
2. Training-first, gang-heavy, topology-sensitive: Volcano and KAI both target this directly. Kueue covers it with TAS and all-or-nothing admission, so compare on the job frameworks you actually run.
3. Fractional GPUs for many small inference or notebook pods: KAI's sharing modes are the most explicit of the three. Volcano also documents GPU sharing.
4. Inference with guaranteed floors and borrowing: all three have queue quotas. With Kueue, the llm-d and AWS patterns are documented end to end, including their limits.
5. Observability and cost: whatever you pick, export queue and admission metrics next to inference metrics. Kueue ships Prometheus metrics and a visibility endpoint for pending workloads. In my runtime I already correlate routing, tokens and cost with OpenTelemetry (see cost and token observability). Queue wait time belongs in the same picture.

Pitfalls to test before production

• Declared quota versus physical capacity (Kueue): keep floors at or below real GPU counts.
• Fragmentation: enable topology-aware admission or placement for multi-GPU pods.
• Preemption of serving pods: decide which workloads may be preempted, and test it. Evicting an inference replica during peak is a user-facing incident.
• Two schedulers, one node pool: with Volcano or KAI, make sure pods that do not opt in cannot grab the GPUs you budgeted (taints/tolerations, node pools).
• Rollouts: under a full budget, a Deployment rollout can stall until a replica terminates. Leave slack or tune maxSurge.

References

• Kueue overview and features: https://kueue.sigs.k8s.io/docs/overview/
• Kueue concepts (Workload, admission, cohort, preemption, TAS): https://kueue.sigs.k8s.io/docs/concepts/
• Kueue all-or-nothing and Topology Aware Scheduling: https://kueue.sigs.k8s.io/docs/concepts/all_or_nothing/
• Kueue cluster quotas (v1beta2 examples): https://kueue.sigs.k8s.io/docs/tasks/manage/administer_cluster_quotas/
• Kueue with DRA devices: https://kueue.sigs.k8s.io/docs/tasks/run/dra/
• llm-d, sharing a GPU budget across models with Kueue: https://llm-d.ai/docs/dev/operations/autoscaling/kueue-rebalancing
• AWS, task governance for model deployment on HyperPod: https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-model-deployment-task-gov.html
• Volcano introduction: https://volcano.sh/en/docs/
• Volcano Queue: https://volcano.sh/en/docs/queue/
• Volcano example Job: https://github.com/volcano-sh/volcano/blob/master/example/job.yaml
• KAI Scheduler README: https://github.com/kai-scheduler/KAI-scheduler
• KAI quick start (queues, pod labels, GPU sharing modes): https://github.com/kai-scheduler/KAI-scheduler/blob/main/docs/quickstart/README.md
• Latest releases checked 2026-10-21: Kueue v0.20.2, Volcano v1.16.0, KAI Scheduler v0.18.6 (GitHub releases API)