Engine vs platform (vLLM, KServe InferenceService / LLMInferenceService).
Topic hub
LLM inference on Kubernetes.
LLM inference on Kubernetes is the serving path from request to tokens: vllm kubernetes deployment choices, kserve vs vllm as engine versus platform, and kserve autoscaling vllm keda patterns on queue depth rather than CPU.
01
Coverage
What this hub covers.
Running LLM inference on Kubernetes: vLLM and KServe, LLMInferenceService, OpenAI-compatible routing, queue-based autoscaling and token budgets.
OpenAI-compatible gateway and routing.
Autoscaling on queue depth, not CPU.
GPU node pools, model cache, rollouts.
Per-tenant token budgets (Budget Guard, off by default).
02
Guides
Related guides.
KServe vs vLLM: Engine, Platform, or Both?
KServe and vLLM solve different layers of LLM serving on Kubernetes. What each does, how KServe runs vLLM, and when a plain vLLM Deployment is enough.
Autoscaling vLLM with KEDA: queue depth signals
Scale vLLM on Kubernetes with KEDA on Prometheus queue metrics: ScaledObject shape, vllm:num_requests_waiting, and how this relates to KServe autoscaling.
Inference Budget Guard: token budgets before the GPU
Per-team inference budgets on the gateway path: reserve estimated cost before chat completion, settle after, fail closed. Status from ai-runtime-platform.
OpenTelemetry LLM Observability: GenAI Conventions & vLLM
Practical OpenTelemetry for LLMs: GenAI semantic conventions, token usage metrics, vLLM Prometheus metrics and a token cost pipeline on Kubernetes.
03
Related cases
Evidence nearby.
AI Runtime Platform
AI workloads often start as API calls but become production systems that need identity, policy, scaling, telemetry, and clear ownership
Every model call: policy + trace + budget
LLM Infrastructure Runtime
LLM workloads move faster than traditional platform controls and can quickly become expensive, opaque, and hard to operate
LLM serving with operability signals
Cloud-Native AI Gateway
AI usage needs routing, policy, budget awareness, and provider resilience
One boundary: rate, budget, fallback, OTel
Cost and Token Observability
AI and cloud costs can grow quietly when usage is disconnected from teams, services, and deployment changes
Token spend visible before finance week
04
Related hubs
Nearby topics.
Available for meaningful infrastructure conversations