AI gateway routing, rate limits, provider failover, and prompt policy.
Topic hub
AI infrastructure engineering.
AI infrastructure is the platform layer around model-powered systems: an AI gateway on Kubernetes that routes and governs LLM traffic, OpenTelemetry-based LLM observability for tokens, latency and cost, RAG platforms, and the policy that keeps it operable.
01
Coverage
What this hub covers.
AI gateway on Kubernetes, OpenTelemetry LLM observability for tokens and cost, RAG platforms, and policy that keeps model systems operable.
RAG knowledge platforms with citations, source freshness, and clear answer boundaries.
Kubernetes/EKS runtime patterns for AI services and platform workloads.
Token, cost, latency, and provider-health observability.
Policy-as-code and data-handling controls around AI requests.
02
Guides
Related guides.
OpenTelemetry LLM Observability: GenAI Conventions & vLLM
Practical OpenTelemetry for LLMs: GenAI semantic conventions, token usage metrics, vLLM Prometheus metrics and a token cost pipeline on Kubernetes.
KServe vs vLLM: Engine, Platform, or Both?
KServe vs vLLM on Kubernetes: what each layer does, how KServe runs vLLM, and when a plain vLLM Deployment is enough.
LLM Gateway vs MCP Gateway: boundaries and policies
LLM gateways govern model traffic; MCP gates control tool calls. Different boundaries and policies, with files from ai-runtime-platform and ASC.
03
Related cases
Evidence nearby.
Architecture Rehearsal
Teams discover architectural blast radius only after a change lands in production
Pre-deploy break risk → verified after change
TwinOps Control Plane
Industrial digital twins are often demos, not operable infrastructure with drift, reconciliation, and incident replay
Compose → detect drift → reconcile
AI Runtime Platform
AI workloads often start as API calls but become production systems that need identity, policy, scaling, telemetry, and clear ownership
Every model call: policy + trace + budget
AI Infra Control Plane
Private AI platforms need shared governance across identity, policy, audit, cost, and SLO signals — not scattered service configs
Private AI with explicit platform contracts
Cloud-Native AI Gateway
AI usage needs routing, policy, budget awareness, and provider resilience
One boundary: rate, budget, fallback, OTel
Agent Security Contract
Agent and MCP stacks expand the attack surface beyond prompt text: tool misuse, poisoned tool descriptions, argument injection, and unapproved high-impact actions run without a pre-execution gate
0 false-ALLOW · 0 false-BLOCK on this corpus
GPU Admission on Kueue
Raw ClusterQueue quota does not cover team caps, allowed GPU flavors, required FinOps labels, GPU-hour or budget ceilings, or image registry allowlists — and GPU jobs need an auditable gate before they are admitted
8 e2e scenarios on this corpus · emulated GPUs
RAG Knowledge Platform
Engineering knowledge is spread across repositories, runbooks, tickets, architecture notes, and project history
Cited answers from project evidence
LLM Infrastructure Runtime
LLM workloads move faster than traditional platform controls and can quickly become expensive, opaque, and hard to operate
LLM serving with operability signals
Cost and Token Observability
AI and cloud costs can grow quietly when usage is disconnected from teams, services, and deployment changes
Token spend visible before finance week
04
Related hubs
Nearby topics.
Available for meaningful infrastructure conversations