KServe vs vLLM: engine, platform, or both?

"Should we use KServe or vLLM?" is a common question when teams first self-host an LLM on Kubernetes. It is a bit like asking "Deployment or nginx?". vLLM is an inference engine with an OpenAI-compatible HTTP server. KServe is a Kubernetes serving platform with an InferenceService API. For LLMs, KServe's Hugging Face runtime uses vLLM as its backend engine. In many setups the answer is "both", and the real decision is how much platform you want around the engine.

I have both paths in my AI Runtime Platform: a vLLM Helm chart and a KServe InferenceService example, with an OpenAI-compatible gateway in front that routes between backends. The repository's maturity matrix marks the vLLM Helm chart as "Supported" and the KServe examples as "Reference" (not the primary path). That status already tells you where I landed for my use case. The reasons are below. I have not run comparative benchmarks between the two paths, so this guide has no throughput or latency numbers.

What vLLM is

vLLM is the engine that loads the model and generates tokens. KServe's own runtime docs summarise its techniques: PagedAttention for KV-cache memory management, continuous batching, chunked prefill, prefix caching, speculative decoding and optimised CUDA kernels. vLLM ships an OpenAI-compatible server and exposes Prometheus metrics on /metrics (queue depth, KV-cache usage, TTFT, inter-token latency, token counters).

On Kubernetes, the simplest vLLM deployment is a normal Deployment running the vllm/vllm-openai image. You own the rest: Service, probes, model cache volume, shared memory, PodDisruptionBudget, autoscaling, rollout strategy.

What KServe is

KServe gives you the InferenceService custom resource and a controller that turns it into serving infrastructure. Relevant points from the current docs:

• Deployment modes. Standard creates plain Kubernetes Deployments, Services, Ingress or Gateway API routes and an HPA. Knative uses Knative for request-based autoscaling, scale to zero, revisions and revision-based canaries. KServe renamed the old values (RawDeployment → Standard, Serverless → Knative), and older manifests may still use the legacy names.
• The docs recommend Standard mode for generative inference (GPU control, long-running requests, streaming) and Knative mainly for predictive workloads. Standard mode does not support scale-from-zero for HTTP requests. KEDA can be added for custom-metric autoscaling.
• The Hugging Face runtime uses vLLM for supported models and falls back to the Hugging Face backend otherwise (--backend can force one). It picks a CUDA or CPU image depending on whether you request nvidia.com/gpu.
• OpenAI endpoints are prefixed with openai: /openai/v1/chat/completions, /openai/v1/completions, /openai/v1/embeddings. Set KSERVE_OPENAI_ROUTE_PREFIX to "" to remove the prefix. That matters if your clients or gateway expect /v1/....
• vLLM engine arguments can be passed through the runtime's command line.
• The docs also list model caching, KV-cache offloading, distributed multi-node serving and an AI Gateway integration for token-based rate limiting as advanced features.

A KServe InferenceService using the Hugging Face runtime, as in the KServe docs:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: huggingface-llm
  annotations:
    serving.kserve.io/deploymentMode: Standard
spec:
  predictor:
    model:
      modelFormat:
        name: huggingface
      storageUri: "hf://meta-llama/llama-3-8b-instruct"
      resources:
        requests:
          cpu: "1"
          memory: "2Gi"
          nvidia.com/gpu: 1
        limits:
          cpu: "1"
          memory: "2Gi"
          nvidia.com/gpu: 1

The resource values are copied from the docs example. Size CPU and memory for your model. An 8B model needs far more host memory than 2Gi in practice, so treat those values as placeholders.

KServe LLMInferenceService: the GenAI track

Since the v0.20 release line, KServe documents a second, LLM-specific API next to InferenceService: LLMInferenceService (serving.kserve.io/v1alpha1). The KServe docs describe it as a dual-track strategy. InferenceService stays the standard for predictive models; LLMInferenceService is purpose-built for generative workloads and is built on the llm-d architecture (vLLM as the engine plus Kubernetes-native routing). You can still serve an LLM with InferenceService, but the docs say that only covers basic single-node deployments.

What the new resource adds, per the current KServe docs:

• Workload shapes in one spec. spec.template for single-node or decode pods, spec.worker for multi-node workers (this switches the workload to a LeaderWorkerSet), spec.prefill for disaggregated prefill pods, and spec.parallelism for tensor, data and expert parallelism.
• A router. spec.router manages a Gateway, an HTTPRoute and a scheduler (the Endpoint Picker from the Gateway API Inference Extension) that routes by KV-cache and load rather than round-robin.
• Composable presets through LLMInferenceServiceConfig resources for model, workload, router and scheduler settings.

The minimal example from the docs:

apiVersion: serving.kserve.io/v1alpha1
kind: LLMInferenceService
metadata:
  name: llama-3-8b
  namespace: default
spec:
  model:
    uri: hf://meta-llama/Llama-3.1-8B-Instruct
    name: meta-llama/Llama-3.1-8B-Instruct
  replicas: 3
  template:
    containers:
      - name: main
        image: vllm/vllm-openai:latest
        resources:
          limits:
            nvidia.com/gpu: "1"
            cpu: "8"
            memory: 32Gi
  router:
    gateway: {}
    route: {}
    scheduler: {}

Empty gateway, route and scheduler blocks mean "use the managed defaults". In production pin the image tag instead of latest.

The price is more dependencies. The docs list Gateway API, the Gateway API Inference Extension (GIE) CRDs and a Gateway provider (Envoy Gateway is the default listed), plus LeaderWorkerSet for multi-node and cert-manager. The installation guide also lists Envoy AI Gateway among the required infrastructure. Order matters: the dependencies page says to install the GIE CRDs before the Gateway provider, otherwise the provider does not pick up InferencePool and has to be restarted. For autoscaling the docs point to the Workload Variant Autoscaler (WVA) and KEDA integration rather than the HPA defaults of InferenceService.

Where this leaves the comparison: with LLMInferenceService, KServe is no longer only "a platform around vLLM". It also owns inference-aware routing and multi-node orchestration, which before you had to assemble yourself. My AI Runtime Platform does not use LLMInferenceService today. Its KServe example is a v1beta1 InferenceService and is marked "Reference", so I have no operating experience with the new resource to report. The release is also moving: at the time of checking, the latest KServe release was v0.21.0 (September 2026), and the API is still v1alpha1. Check the release notes before adopting it.

The two ways I run vLLM

Path A: KServe with a custom container. In deploy/kserve/qwen-inferenceservice.yaml I skip the Hugging Face runtime and run the vLLM OpenAI image directly as the predictor container. KServe still manages the Deployment, Service and routing, and I keep vLLM's native /v1 paths:

# Requires KServe >= 0.18 installed in Standard mode.
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: qwen-vllm
  namespace: ai-runtime
  annotations:
    serving.kserve.io/deploymentMode: Standard
spec:
  predictor:
    minReplicas: 1
    containers:
      - name: kserve-container
        image: vllm/vllm-openai:v0.31.0
        args:
          - --model=Qwen/Qwen2.5-7B-Instruct
          - --served-model-name=qwen2.5-7b-instruct
          - --gpu-memory-utilization=0.90
        ports:
          - containerPort: 8000
            protocol: TCP
        resources:
          requests: { cpu: "4", memory: 24Gi, nvidia.com/gpu: "1" }
          limits:   { cpu: "8", memory: 32Gi, nvidia.com/gpu: "1" }
    nodeSelector:
      nvidia.com/gpu.present: "true"
    tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule

Path B: plain vLLM via a Helm chart. charts/vllm-runtime is the primary path in my repo. Its docs list a model-cache PVC and /dev/shm, a startup probe with a large load budget (large models take a while to load), a Recreate strategy for GPU workloads, an optional Hugging Face token secret, revision and digest annotations, PDB, security contexts, tensor-parallel and quantisation settings, a NetworkPolicy and a ServiceMonitor. Install:

helm upgrade --install qwen charts/vllm-runtime \
  -f charts/vllm-runtime/values-production.yaml \
  --set model.revision=main \
  --set model.artifactDigest=sha256:example \
  --set huggingface.existingSecret=hf-token

Recreate is deliberate. With one GPU per replica and no spare GPU, a RollingUpdate surge pod cannot schedule and the rollout stalls.

Autoscaling: queue depth, not CPU

CPU is a poor scaling signal for a GPU-bound server. My repo scales on vLLM's queue gauge with KEDA. The ScaledObject targets the Helm release's Deployment (Path B), not the KServe example. Excerpt from deploy/keda/vllm-scaledobject.yaml (marked "Reference" in the maturity matrix):

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: qwen-vllm-queue
  namespace: ai-runtime
spec:
  scaleTargetRef:
    name: qwen-vllm-runtime
  pollingInterval: 15
  cooldownPeriod: 180
  minReplicaCount: 1
  maxReplicaCount: 4
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090
        query: sum(vllm:num_requests_waiting{namespace="ai-runtime"})
        threshold: "4"
        activationThreshold: "1"

The threshold is a starting value, not a tuned result. With KServe in Knative mode you would use Knative's request-based autoscaler instead and get scale to zero. Weigh that against cold starts: model load time is the cold start.

Moving between the two

Because both paths run the same engine, switching is mostly a packaging change. Plan for these differences:

• URL paths. A gateway or client that calls /v1/chat/completions on plain vLLM must call /openai/v1/chat/completions on the KServe Hugging Face runtime, unless you clear the prefix. With a custom vLLM container in KServe (Path A), the paths stay the same.
• Model naming. The name clients send in model comes from --served-model-name on vLLM and from --model_name on the KServe runtime. Keep it stable across the switch, or the gateway's routing table breaks.
• Arguments. vLLM engine flags pass through the KServe runtime's command line, but the runtime has its own flags too (--model_id, --backend, --dtype, --max_model_len). Check which layer owns each setting.
• Object names. KServe generates the Deployment and Service names from the InferenceService. Anything that references them, such as a KEDA scaleTargetRef, a ServiceMonitor selector or a NetworkPolicy, has to follow.
• Metrics. The vllm:* metrics come from the engine either way. What changes is the scrape target and the labels your Prometheus setup attaches.
• Rollback. Keep both manifests in Git during the switch, and let the gateway shift traffic gradually instead of cutting over DNS. In my runtime that is what canary weights and fallback routing are for.

Decision guide

What neither gives you on its own: tenant identity, per-team budgets, policy on prompts and tools, provider fallback, and a single audit trail across self-hosted and external models. I put those in a gateway in front of both. See the cloud-native AI gateway and the runtime's routing, which picks a backend by health, cost and canary weights and records the routing reason. Canary rollout through Argo Rollouts is marked "Experimental" in the repo.

Operational checklist

• Pin the vLLM image tag (or digest) and the model revision; record both on the Deployment
• Startup probe long enough for model download and load; readiness separate from liveness
• Model cache on a PVC, /dev/shm sized for the engine
• GPU node pool tainted; serving pods tolerate it and select it
• Rollout strategy matches spare GPU capacity (Recreate or a surge GPU)
• Autoscale on vllm:num_requests_waiting (or KV-cache usage), not CPU
• Scrape /metrics; alert on queue growth and TTFT, not only on errors
• With KServe: decide on the deployment mode explicitly, and check the OpenAI path prefix
• With KServe LLMInferenceService: install GIE CRDs before the Gateway provider, pin the KServe chart version, and treat the v1alpha1 API as subject to change
• Manifests delivered through GitOps (see GitOps with Argo CD and Flux), so the running model is the one in Git

For the operating model around the engine (queues, priority lanes, scaling), see the LLM infrastructure runtime case.

References

• KServe Hugging Face/vLLM runtime overview: https://kserve.github.io/website/docs/model-serving/generative-inference/overview
• KServe Standard (Kubernetes Deployment) installation: https://kserve.github.io/website/docs/admin-guide/kubernetes-deployment
• KServe Knative mode installation: https://kserve.github.io/website/docs/next/admin-guide/serverless
• KServe administrator guide overview (which deployment mode for generative inference): https://kserve.github.io/website/docs/admin-guide/overview
• KServe configurations (default deployment mode): https://kserve.github.io/website/docs/admin-guide/configurations
• KServe PR #4608 (RawDeployment → Standard, Serverless → Knative): https://github.com/kserve/kserve/pull/4608
• vLLM production metrics: https://docs.vllm.ai/en/latest/usage/metrics/
• KEDA Prometheus scaler: https://keda.sh/docs/latest/scalers/prometheus/
• AI Runtime Platform (KServe example, vLLM chart, KEDA, maturity matrix): https://github.com/justrunme/ai-runtime-platform
• KServe, Understanding LLMInferenceService: https://kserve.github.io/website/docs/model-serving/generative-inference/llmisvc/llmisvc-overview
• KServe, LLMInferenceService dependencies: https://kserve.github.io/website/docs/model-serving/generative-inference/llmisvc/llmisvc-dependencies
• KServe, LLMInferenceService installation (v0.20 charts): https://kserve.github.io/website/docs/install/llmisvc-install
• KServe releases: https://github.com/kserve/kserve/releases