An AI runtime execution plane makes model calls behave like production infrastructure: governed, observable, scalable, and owned by platform contracts rather than scattered application code.
How it works
AI Runtime Platform by Andrey Lesnikov — an OpenAI-compatible execution plane that routes to vLLM and KServe on Kubernetes using health, cost, and canary policy.
Clients hit an OpenAI-compatible FastAPI gateway — one boundary for apps and agents instead of scattered provider SDKs.
Health, latency, cost, canary weights, and fallback policy select the backend; the decision is visible on every completion response.
vLLM and KServe paths run with probes, ServiceMonitors, and KEDA queue autoscaling — GPU or local CPU compose for demos.
OpenTelemetry traces the intent-to-model path; Argo CD ships the plane with SBOM, cosign, and supply-chain gates.
Evidence
How the system is framed, operated, and proven — not a slide deck summary.
AI systems often begin as direct provider API calls. In production that leaves gaps around identity, policy, MCP tool governance, scaling, telemetry, cost visibility, and auditability. Every team reinvents rate limits, fallback, and audit trails in application code.
The runtime plane combines an OpenAI-compatible gateway, MCP governance, intent proxying, KServe/vLLM serving paths, KEDA scaling, Redis state, OIDC identity, and OpenTelemetry traces. Policy and quotas sit in front of model execution; traces follow the request from client intent to provider response.
Platform owns the contracts: who can call which tools, which models are allowed, which budgets apply, and which SLOs burn when latency or error rates climb. Application teams consume a stable interface instead of wiring providers ad hoc.
A runtime plane introduces more platform responsibility than direct SDK usage. The payoff is consistent governance, reusable observability, clearer failure isolation, and a better path from prototype to production.
Useful signals include policy decision logs, token and cost metrics, provider fallback counts, KEDA scale events, and end-to-end traces that name the intent, tool, and model path.
OpenAI-compatible interfaces, MCP tool boundaries, model-serving latency, Kubernetes autoscaling, OIDC, audit trails, and cost visibility.
Related
Private AI platforms need shared governance across identity, policy, audit, cost, and SLO signals — not scattered service configs
Private AI with explicit platform contracts
AI usage needs routing, policy, budget awareness, and provider resilience
One boundary: rate, budget, fallback, OTel
LLM workloads move faster than traditional platform controls and can quickly become expensive, opaque, and hard to operate
LLM serving with operability signals