Inference Budget Guard: per-team token budgets before the GPU

Most llm cost optimization work starts after the invoice: scrapes, dashboards, weekly reviews. An llm token budget that matters operationally sits before the GPU call — reserve estimated spend, deny when the window is full, settle when usage returns.

This guide is based only on files in ai-runtime-platform: docs/budget-guard.md, docs/budget-guard.policy.json, docs/EVIDENCE.md, docs/budget_guard_results.json, docs/maturity-matrix.md, and the gateway modules those docs describe (app/gateway/budget_guard.py and related tests). Numbers below are copied from those evidence files on this test suite. They are not a multi-region billing guarantee.

Status in this repo

When enabled without REDIS_URL, the in-process backend is used — budgets are then per gateway process, not shared across replicas. Set REDIS_URL for a shared budget (docs/budget-guard.md).

Where it sits on the request path

From docs/budget-guard.md:

drain check
  → global admission
  → tenant / fleet concurrency admission
  → inference budget reserve   ← this guard
  → tenant attribution pre-reserve (signals)
  → Control Plane governance
  → route / execute
  → settle budget + reconcile attribution

On DENY, the gateway returns 429 (inference_budget_exceeded) or 503 (budget_store_* / budget_policy_invalid) and does not call the model backend.

It is integrated into the gateway path beside fleet concurrency admission — not a standalone sidecar.

Atomic reservation

Spend is tracked per scope (tenant:<id> or key:<tenant>:<sha256(apiKey)[:16]>; the raw key never appears in Redis key names, metric labels or error bodies) inside a fixed window. Money is integer micro-USD. Estimates and actual charges round up; the budget cap rounds down (Decimal arithmetic).

Invariant enforced atomically (Redis Lua or in-process lock):

committed + reserved + estimate ≤ budget

Redis script keys: arp:budget:{scope}:window|reservations|amounts. Window rollover resets committed_micros only; in-flight reservations carry into the new window.

Fail-closed behaviour

There is no silent allow when the guard is enabled and the store or policy fails.

Policy shape

Enable with INFERENCE_BUDGET_ENABLED=true and either INFERENCE_BUDGET_POLICY (inline JSON) or INFERENCE_BUDGET_POLICY_FILE. Example file in the repo: docs/budget-guard.policy.json (version budget-guard-1.0), with fields such as budgetUsd, windowSeconds, softWarningRatio, reservationTtlSeconds, per-tenant / per-key budgets, and a prices table (* / default required per the doc table).

INFERENCE_BUDGET_STORE_TIMEOUT_SECONDS (default 1.0) bounds the Redis exchange.

Settlement (summary)

Each reservation is settled exactly once. Highlights from docs/budget-guard.md:

• Upstream success with a usage block → actual priced usage
• Success without usage (most streams) → charged at the estimate
• Error before the model is called → released (0)
• Error after the model may have run → charged at the estimate
• Duplicate settle → no-op
• Settle after TTL expiry → actual spend still committed once (tombstone)

Actual spend above the estimate is committed as-is, so committed spend can exceed the cap by at most the estimate error of in-flight requests; later requests are then denied until the window rolls over.

Evidence on this test suite

docs/EVIDENCE.md (section Inference budget guard (this suite)), regenerated via python scripts/generate_budget_evidence.py, records for commit SHA fee974a25d4d939e2d170ffa737b74ca0de2b0d3:

• Scope: on this test suite
• Policy version: budget-guard-1.0
• Policy sha256: sha256:eff951135b69d4ff1a86da7f40775c60b3459a3462d46f4e88674d9a11e10d6c
• Scenario envelopes: 15
• Tests: 80 passed, 0 failed, 0 skipped (real Redis: True)
• False-deny count (legitimate): 0
• Concurrency overspend events: 0
• Real Redis, 8 processes: allows per round [142] (budget allows 142), max reserved 994000 of 1000000 micro-USD over 10 rounds
• Concurrent settle: exact in 10/10 rounds, leaked reserved micro-USD 0

Claim rows in the same file mark these as Tested locally; re-run in CI, with limits: one Redis — not multi-region billing; estimate may differ from actual usage; settlement can lag.

Decision envelopes are ASC-shaped (schema_version 1.0.0) with decision = ALLOW | DENY (plus verdict / acs_disposition for compatibility with agent-security-contract). Scenario ids listed in evidence include legitimate_within_budget, over_budget_denied, store_down_fail_closed, malformed_policy_fail_closed, and others — see the table in docs/EVIDENCE.md.

CI entrypoints named there: .github/workflows/ci.yaml, .github/workflows/budget-guard.yml.

Honest limits (quoted from the repo)

From docs/budget-guard.md and docs/EVIDENCE.md:

• Claims are on this test suite / this CI setup only
• Pre-request estimate can differ from actual token usage
• Settlement can lag if Redis fails after upstream success
• Estimate prices come from the budget policy; settlement prices from gateway model targets — keep them aligned
• Reservation expiry uses each gateway's wall clock (skew shifts TTL)
• Not a billing / chargeback ledger
• Do not read results as a blanket "€0 overspend" guarantee outside this suite

Operating cost and token telemetry next door: Cost and Token Observability and OpenTelemetry LLM observability. Serving context: LLM inference on Kubernetes.

Checklist

• Leave the guard off until Redis (or an accepted single-process lab) and a priced policy are ready
• Set INFERENCE_BUDGET_ENABLED=true only with a valid policy JSON/file
• Align policy prices with gateway model-target prices used at settle
• Prefer REDIS_URL whenever more than one gateway replica serves traffic
• Alert on 429 budget denies and 503 store/policy failures separately
• Treat evidence numbers as suite results; re-run scripts/generate_budget_evidence.py after policy changes
• Pair admission budgets with attribution telemetry (OTel / usage events), not instead of them

References

• budget-guard.md: https://github.com/justrunme/ai-runtime-platform/blob/main/docs/budget-guard.md
• Example policy JSON: https://github.com/justrunme/ai-runtime-platform/blob/main/docs/budget-guard.policy.json
• EVIDENCE.md: https://github.com/justrunme/ai-runtime-platform/blob/main/docs/EVIDENCE.md
• budget_guard_results.json: https://github.com/justrunme/ai-runtime-platform/blob/main/docs/budget_guard_results.json
• maturity-matrix.md: https://github.com/justrunme/ai-runtime-platform/blob/main/docs/maturity-matrix.md
• Gateway budget module (path): https://github.com/justrunme/ai-runtime-platform/blob/main/app/gateway/budget_guard.py
• Evidence generator: https://github.com/justrunme/ai-runtime-platform/blob/main/scripts/generate_budget_evidence.py