LLM Gateway vs MCP Gateway: different boundaries, different policies

Teams often ask for one "AI gateway" and then discover they needed two control planes. llm gateway vs mcp gateway is not a vendor bake-off. It is a boundary question: which requests are model completions, and which requests are tool calls that change the world outside the model.

This guide separates those boundaries using public files from ai-runtime-platform (OpenAI-compatible gateway + MCP execution path) and agent-security-contract (pre-execution tool-call gate), plus the MCP specification's security best practices and the OWASP MCP Security Cheat Sheet. Claims about my own repositories are scoped to those repos and their evidence files.

Two planes, one agent

An agent session usually crosses both planes:

1. LLM plane — chat/completions (or equivalent): prompts in, tokens out, routing, budgets, provider failover.
2. Tool / MCP plane — tools/call (and related MCP methods): concrete side effects through MCP servers (filesystem, HTTP, deploy, mail, …).

Policies that make sense on one plane are often meaningless on the other. A token budget does not stop a poisoned tool description. A tool allowlist does not stop an expensive model loop that never calls a tool.

What an LLM gateway owns

In ai-runtime-platform, the gateway is documented as an OpenAI-compatible front door. docs/architecture.md describes routing on health, latency and unit cost, weighted canary and failover routes, and inspectable outcomes (selected_backend, routing_reason, health_score, estimated_cost, fallback_used). Maturity: OpenAI chat/completions gateway — Supported.

docs/auth-boundary.md lists the protected surface when API keys or JWT verify are configured, including /v1/chat/completions, /v1/models, /v1/decisions/{id}, and also /mcp/*. Public routes stay narrow (/livez, /readyz, /healthz, /metrics, /v1/runtime/jwks). Production profile requires OIDC JWKS + issuer/audience, Control Plane URL and Redis.

Honest limits from the same architecture doc: cost is request attribution from token usage and configured unit rates, not cloud billing; the default health store is in-memory per replica unless REDIS_URL is set.

The product case for this layer is Cloud-Native AI Gateway. Sibling platform context: AI Runtime Platform.

What an MCP gate owns

MCP security is about who may call which tool, with which arguments, under which catalog digest, and what evidence you keep. The MCP specification's security best practices forbid token passthrough: MCP servers MUST NOT accept tokens that were not explicitly issued for them, and must not forward client tokens downstream as if the server had validated them for that audience.

The OWASP MCP Security Cheat Sheet (and the related Agent Security Cheat Sheet) push the same direction: least privilege, pinned tool catalogs, isolation, and audit — not "trust the model to refuse".

In agent-security-contract, ASC is a platform gate for agent and MCP tool calls. From README.md and policy/demo-agent.policy.yaml:

• YAML capability catalog + capability_digest (sha256:…)
• Per-agent allowlists, argument constraints, path canonicalization
• High-impact tools → REQUIRE_APPROVAL unless approved; some destructive path escapes still BLOCK
• Verdicts ALLOW / BLOCK / REQUIRE_APPROVAL (exit 0 / 1 / 2)
• HTTP sidecar (src/asc/sidecar.py): POST /v1/evaluate before execution; callers must fail closed on errors/timeouts

Evidence in docs/EVIDENCE.md is generated from results/results.json. Scope phrasing in that file: on this corpus of 66 scenarios (33 attack, 33 benign): critical false-ALLOW 0, benign false-BLOCK 0/33. That is corpus evidence, not a claim of general agent safety.

For the Kubernetes checklist that sits beside this gate, see MCP security checklist for Kubernetes and the Agent & MCP security hub.

How ai-runtime-platform splits the MCP path

docs/mcp-execution.md describes the runtime sequence:

Agent → Runtime tool call
  → Control Plane evaluate-tool (signed verdict path when enabled)
  → Runtime initialize (when sessionMode requires it) + Mcp-Session-Id
  → Runtime MCP Streamable HTTP tools/call (JSON or SSE)
  → audit-tool (session / capability / request ids)
  → filtered/truncated result to agent

Registry servers are configured via MCP_SERVER_REGISTRY (URL, transport, sessionMode, credentialRef as env:NAME only, optional capabilityDigest). Execution without the Control Plane is rejected (503 mcp_governance_required) unless MCP_ALLOW_UNGOVERNED=true for local demos. Maturity matrix: MCP Streamable HTTP (JSON) and MCP session / SSE lifecycle are Supported; MCP session HA is process-local (sticky sessions or a shared store for multi-replica).

That is an execution and governance path for tools, not a substitute for ASC's CI-oriented corpus gate — and not a substitute for LLM routing policy on /v1/chat/completions.

Side-by-side

Complementary products named in the ASC README (Promptfoo for red-team detection, LiteLLM-style proxy guardrails for request/response filtering) sit beside these planes; they do not replace a tool-capability admission gate. LiteLLM's own docs cover proxy/guardrail configuration if you use that stack: https://docs.litellm.ai/

Decision rules

1. If the risk is "which model, at what cost, with what auth" — start with an LLM gateway policy (see cloud-native AI gateway).
2. If the risk is "which tool may run, with which args, under which catalog" — start with an MCP / tool gate (Agent Security Contract, MCP path in ai-runtime-platform).
3. If you only deploy one — you still have the other plane's gap; document it rather than renaming a completion proxy an "MCP gateway".
4. Shared auth mistakes — one JWT accepted everywhere without audience checks recreates token passthrough on the MCP side and confused-deputy patterns the MCP spec warns about.

Checklist

• Separate configs for completion routing and tool capability policy
• Gateway auth covers /v1/chat/completions and does not imply MCP audience
• MCP servers validate tokens issued for them; no passthrough downstream
• Tool catalog pinned by digest; drift fails closed
• Pre-execution evaluation for tool calls (library, sidecar, or Control Plane path)
• Audit IDs join completion traces and tool calls where both exist
• Spend controls on the LLM plane; blast-radius controls on the tool plane

References

• ai-runtime-platform architecture: https://github.com/justrunme/ai-runtime-platform/blob/main/docs/architecture.md
• ai-runtime-platform auth boundary: https://github.com/justrunme/ai-runtime-platform/blob/main/docs/auth-boundary.md
• ai-runtime-platform MCP execution: https://github.com/justrunme/ai-runtime-platform/blob/main/docs/mcp-execution.md
• ai-runtime-platform maturity matrix: https://github.com/justrunme/ai-runtime-platform/blob/main/docs/maturity-matrix.md
• agent-security-contract README: https://github.com/justrunme/agent-security-contract/blob/main/README.md
• agent-security-contract demo policy: https://github.com/justrunme/agent-security-contract/blob/main/policy/demo-agent.policy.yaml
• agent-security-contract sidecar: https://github.com/justrunme/agent-security-contract/blob/main/src/asc/sidecar.py
• agent-security-contract evidence (corpus): https://github.com/justrunme/agent-security-contract/blob/main/docs/EVIDENCE.md
• MCP specification, security best practices: https://modelcontextprotocol.io/specification/draft/basic/security_best_practices
• OWASP MCP Security Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/MCP_Security_Cheat_Sheet.html
• LiteLLM docs (proxy / guardrails, as named in ASC positioning): https://docs.litellm.ai/