AI Runtime Platform

AI systems often begin as direct provider API calls. In production that leaves gaps around identity, policy, MCP tool governance, scaling, telemetry, cost visibility, and auditability. Every team reinvents rate limits, fallback, and audit trails in application code.

The runtime plane combines an OpenAI-compatible gateway, MCP governance, intent proxying, KServe/vLLM serving paths, KEDA scaling, Redis state, OIDC identity, and OpenTelemetry traces. Policy and quotas sit in front of model execution; traces follow the request from client intent to provider response.

Platform owns the contracts: who can call which tools, which models are allowed, which budgets apply, and which SLOs burn when latency or error rates climb. Application teams consume a stable interface instead of wiring providers ad hoc.

A runtime plane introduces more platform responsibility than direct SDK usage. The payoff is consistent governance, reusable observability, clearer failure isolation, and a better path from prototype to production.

Useful signals include policy decision logs, token and cost metrics, provider fallback counts, KEDA scale events, and end-to-end traces that name the intent, tool, and model path.

Fleet-wide Redis admission: tenant × model concurrency is a Redis-backed distributed semaphore (atomic Lua acquire, TTL leases with fencing tokens, renewal during streams), so the limit holds across gateway replicas and capacity returns after a pod dies. Redis unavailable → 503 admission_store_unavailable by default (fail_closed). Platform e2e covers two replicas sharing one limit and SIGKILL mid-stream (CI run 38121712037). GOVERNANCE_MODE defaults to enforce (PR #100): enforce without CONTROL_PLANE_URL fails readiness (503) and chat returns 503 governance_misconfigured — governance is never silently skipped; disabled must be set explicitly. Golden path: block → 403, approval → 409 then 200, Redis down → /readyz 503; suite 21.5 s on Compose (run 38060436977, docs/results.md).

v2.5.0 adds an opt-in Inference Budget Guard on the gateway chat path (INFERENCE_BUDGET_ENABLED): reserve estimated spend → execute → settle actual usage, with atomic Redis Lua reservation (shared across replicas) or an in-process lock when Redis is unset. Money is integer micro-USD. Fail-closed: 503 when the store is down, times out, or returns corrupt data (and on malformed policy); 429 when over budget — the backend is not called. On this test suite, real Redis with 8 OS processes × 25 concurrent reserves × 10 rounds held the invariant (allows/round=[142], expected 142; max reserved 994000 of 1000000 micro-USD; overspend_events=0); concurrent settle was exact in 10/10 rounds with leaked_reserved_micros=0; legitimate within-budget false_deny_count=0 (80 tests passed; docs/EVIDENCE.md, docs/results.json, docs/budget-guard.md). Documented limits on this test suite: the pre-request estimate uses the budget policy's prices while settlement uses the gateway's model prices; if Redis fails after the upstream call, that spend is lost (reservation held until TTL, then released); without Redis the budget is per gateway process.