Autoscaling vLLM with KEDA: queue depth signals

CPU-based Horizontal Pod Autoscaler rules are a poor fit for LLM inference. A GPU pod can be saturated on KV-cache and queue depth while CPU looks calm. The search phrase that usually follows is some mix of kserve autoscaling vllm keda: use an event-driven scaler on signals the engine actually exposes.

This guide walks through that pattern with three anchors:

1. The ScaledObject checked into justrunme/ai-runtime-platform at deploy/keda/vllm-scaledobject.yaml (maturity: Reference).
2. Official KEDA Prometheus scaler fields.
3. Official vLLM production metrics (vllm:num_requests_waiting) and KServe's documented KEDA path for generative InferenceServices.

There is no public repository named justrunme/ai-runtime (GitHub returns 404 as of this check). The case study and the ScaledObject live under ai-runtime-platform. I do not invent scale-out latency or tokens-per-second numbers here; the Reference sample is a wiring example, not a load-test report.

Why queue depth

vLLM's OpenAI-compatible server exposes Prometheus metrics on /metrics. The production metrics docs include gauges such as:

• vllm:num_requests_waiting — requests waiting to be processed
• vllm:num_requests_running — requests in model execution batches
• vllm:kv_cache_usage_perc — KV-cache usage (1 means 100%)

Queue depth answers "how many requests are already waiting for capacity?". That is closer to user-visible backlog than CPU. KV-cache usage is another strong signal when memory, not admission queue, is the bottleneck. Pick the metric that matches the failure mode you care about; do not scale on four signals at once without a clear primary.

The ScaledObject in this repo (Reference)

In docs/maturity-matrix.md and docs/vllm-production.md, KEDA ScaledObject is listed as Reference ("usable examples"), while the vLLM Helm chart is Supported. That wording matters: the chart is the primary path; the ScaledObject is a sample you must validate on your Prometheus, GPU pool and model.

The committed object (comments included):

# KEDA and Prometheus must be installed before applying this resource.
# This governs the Helm release's Deployment; do not combine it with an HPA for the same target.
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: qwen-vllm-queue
  namespace: ai-runtime
spec:
  scaleTargetRef:
    name: qwen-vllm-runtime
  pollingInterval: 15
  cooldownPeriod: 180
  minReplicaCount: 1
  maxReplicaCount: 4
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090
        metricName: vllm_requests_waiting
        query: sum(vllm:num_requests_waiting{namespace="ai-runtime"})
        threshold: "4"
        activationThreshold: "1"
        ignoreNullValues: "false"

What this says, without extra claims:

• Target Deployment name: qwen-vllm-runtime in namespace ai-runtime (the Helm release's Deployment).
• Poll Prometheus every 15 seconds; cool down 180 seconds before scaling in.
• Replica band: 1–4.
• Trigger: KEDA prometheus scaler, query sum(vllm:num_requests_waiting{namespace="ai-runtime"}), threshold 4, activation threshold 1.
• ignoreNullValues: "false" — if Prometheus returns empty, the scaler errors instead of pretending the queue is fine (KEDA default for this field is true).

The file comments warn not to attach a second HPA to the same Deployment. KEDA creates and owns the HPA for the ScaledObject.

Threshold 4 is a starting value in this sample, not a tuned production result. On this setup you still need to confirm scrape labels, ServiceMonitor targets and that the metric name your vLLM build emits matches the query (colon vs underscore variants show up in dashboards; the query here uses the vllm:num_requests_waiting form from the vLLM docs).

KEDA Prometheus trigger fields that matter

From the KEDA Prometheus scaler docs (checked against the v2.17 doc set; confirm against the version you install):

Authentication (bearer, basic, TLS, cloud-managed Prometheus) is documented separately via TriggerAuthentication. The sample assumes an open in-cluster Prometheus Service DNS name from kube-prometheus-stack.

How this relates to KServe autoscaling

If you serve through KServe in Standard mode, the KServe generative-inference autoscaling docs describe setting serving.kserve.io/autoscalerClass: "keda" and driving scale from Prometheus or OpenTelemetry on LLM metrics (examples use vllm:num_requests_running). KEDA remains the engine; KServe owns the InferenceService and can create the ScaledObject for you.

Two honest boundaries from those docs and from my runtime:

• KEDA autoscaling for InferenceService is documented for Standard mode, not as a Knative substitute for every mode.
• My AI Runtime Platform marks KServe examples as Reference and the vLLM Helm chart as Supported. The ScaledObject above targets the Helm Deployment path, not an InferenceService. For engine-vs-platform trade-offs see KServe vs vLLM.

So "kserve autoscaling vllm keda" and "plain vLLM Deployment + KEDA" are sibling patterns: same metric family, different owner of the Pod template.

What this sample does not prove

• No published load-test comparing CPU HPA vs queue-based KEDA in this repository's evidence files for this ScaledObject.
• No claim that maxReplicaCount: 4 matches a particular GPU node pool.
• No multi-model or multi-tenant fairness — that belongs to admission/quota layers (see the GPU scheduling hub and Kueue vs Volcano vs KAI guide).
• Usage and cost attribution are separate (gateway / OpenTelemetry); autoscaling alone does not cap spend.

Operational checklist

• Scrape vLLM /metrics into Prometheus with stable labels (namespace, model name)
• Confirm the PromQL returns one series; fix label selectors before lowering thresholds
• Prefer queue or KV-cache metrics over CPU for generative pods
• Set minReplicaCount with cold-start (model load) in mind; scale-from-zero needs an explicit product decision
• Do not combine this ScaledObject with a second HPA on the same Deployment
• Treat the repo object as Reference: stage it, load-test it, then pin versions of KEDA, Prometheus and the vLLM image
• If you use KServe, follow Standard-mode KEDA annotations and re-check metric names on the runtime you deploy
• Wire GPU node taints, model cache PVC and rollouts as in the LLM inference on Kubernetes hub

References

• ai-runtime-platform ScaledObject: https://github.com/justrunme/ai-runtime-platform/blob/main/deploy/keda/vllm-scaledobject.yaml
• ai-runtime-platform maturity matrix: https://github.com/justrunme/ai-runtime-platform/blob/main/docs/maturity-matrix.md
• ai-runtime-platform vLLM production notes: https://github.com/justrunme/ai-runtime-platform/blob/main/docs/vllm-production.md
• KEDA Prometheus scaler: https://keda.sh/docs/2.17/scalers/prometheus/
• vLLM production metrics: https://docs.vllm.ai/en/latest/usage/metrics.html
• KServe, autoscale InferenceService with LLM metrics (KEDA): https://kserve.github.io/website/docs/model-serving/generative-inference/autoscaling
• Case: AI Runtime Platform: https://justrunme.com/cases/ai-runtime-execution-plane/