AI agent audit trail: what to log for every tool call

When an agent does something it should not have done, the first questions are always the same. Which agent was it, acting for whom? What exactly did it call, with which arguments? Was the call allowed by policy, and by which version of the policy? Did a human approve it? What did it cost? Most teams find out during the incident that their logs answer one or two of these, spread across three systems with no shared ID.

This guide defines the record I want for every agent tool call, and where each field comes from: the tool gate, the gateway, OpenTelemetry and Kubernetes. It is a design for an audit trail, not a product. Where my own repositories implement part of it, I say which part and with which limits.

Why the logs you already have are not enough

• Kubernetes audit logs record requests to the Kubernetes API server. They show that a ServiceAccount created a Job or read a Secret. They do not show that an agent called send_email through an MCP server, because that never touches the API server.
• Application logs usually record the tool call but not the decision: which policy allowed it, whether approval was required, what the policy digest was.
• LLM traces record model calls and often tool spans, but arguments and results are frequently dropped for privacy, and nobody links the span to the policy decision.
• Gateway logs record tokens and cost per request, but not which tool call the request led to.

An audit trail joins these with shared identifiers and adds the decision evidence that none of them carries on its own.

The record: one entry per tool call

The minimum set of fields, grouped by the question each group answers:

Some fields will be empty for some calls. That is fine. A missing approval on a call that did not need one is information; a missing policy digest on any call is a gap.

Use the OpenTelemetry names where they exist

The OpenTelemetry GenAI semantic conventions define an execute_tool span. Its name should be execute_tool {gen_ai.tool.name}, its kind INTERNAL, with gen_ai.operation.name set to execute_tool. Relevant attributes:

• gen_ai.tool.name (required), gen_ai.tool.type, gen_ai.tool.description
• gen_ai.tool.call.id, conditionally required: the identifier the model or provider supplied for the call
• gen_ai.agent.name and gen_ai.conversation.id, when available
• error.type when the call failed
• gen_ai.tool.call.arguments and gen_ai.tool.call.result, both Opt-In

For MCP, the conventions define client and server spans with mcp.method.name (for example tools/call), mcp.session.id, mcp.protocol.version and jsonrpc.request.id.

Two things to keep in mind. First, all of these are at Development stability, so names can still change; pin the semconv version you follow. Second, the Opt-In status of arguments and results is deliberate: the registry warns that such attributes may contain sensitive data. Do not turn them on globally to "have everything". Store a digest and a redacted copy in the audit record instead.

Using the same names in the audit record and in traces means one query language works for both, and the trace_id join is trivial.

The decision is the part most systems miss

An audit trail without the policy decision only tells you what happened. With the decision it also tells you whether it was supposed to happen. That requires a gate that runs before execution and emits evidence.

The Agent Security Contract emits an evidence envelope for every verdict, defined by a JSON schema in the repo. Its fields:

• schema_version, policy_version, policy_sha256
• scenario_id, verdict, reason
• trace_id, commit_sha, timestamp (UTC ISO-8601)
• envelope_sha256, a SHA-256 over the canonical JSON of all other fields

asc verify rejects envelopes with missing fields, inconsistent verdicts, edits that no longer match the digest, and, with --policy, a policy_sha256 that differs from the policy file. The README is careful about the limit, and so should you be: envelope_sha256 is an unkeyed integrity digest, not a signature. It catches accidental or naive edits; someone who can rewrite the file can also recompute it. Treat envelopes as trustworthy only when they come from a CI run or a pipeline you control.

The AI Runtime Platform records a similar decision on the request path. Its enforcement-evidence document lists control_plane_decision_id, policy_bundle_id, policy_digest, request_digest, runtime_version and an enforcement_outcome (executed, blocked, approval_required, upstream_error, stream states). For MCP execution, its sequence ends with an audit call carrying session, capability and request IDs. Signed decision tokens (JWS) are optional and verified against a JWKS URL and audience. The repo also states that its Redis decision store is TTL-bound operational state, not an audit log. That distinction matters: the decision store answers "what was decided recently", the audit trail must survive much longer.

Link cost to the tool call

Cost questions come up in almost every agent review: which task cost what, and was the spend expected. If the audit record carries the trace_id and the gateway records usage on the same trace, you can answer "cost per task" instead of "cost per day".

In the AI Runtime Platform, usage events carry tenant_id, request_id, decision_id, model and backend, input and output tokens, estimated cost, GPU seconds, duration and outcome, and the same fields are attached to the active OTel span. The repo calls these events attribution-grade and best-effort: they can be lost on pod crash, buffer overflow or shutdown timeout, and the docs point to an external durable sink for billing-grade delivery. That is the right framing for an audit trail too: cost in the audit record is for attribution, not invoicing.

The Cost and Token Observability case describes the operating side: tying token and cost telemetry to services, gateway requests and deployment events, so cost becomes a signal teams act on.

Arguments, results and sensitive data

The tension: auditors want exact arguments, privacy wants none. A workable rule:

1. Always store a digest of the canonical arguments and of the result. A digest proves what was requested without revealing it.
2. Store a redacted copy of arguments with known secret fields removed (tokens, passwords, keys) and free text truncated.
3. Store full arguments only for high-impact tools, in a separate store with stricter access and shorter retention, if your policy requires it.
4. Bind approvals to the argument digest. An approval for delete /workspace/tmp/report.json must not cover delete /workspace.

The OWASP Logging Cheat Sheet's general rules apply: do not log secrets or session tokens, protect logs from tampering, and keep the log pipeline from becoming an injection path. Agents make the last one more likely, because tool arguments and results are often model-generated text.

Where Kubernetes audit fits

Kubernetes audit is still useful: it covers what agents do through the Kubernetes API. Write an audit policy that records agent ServiceAccounts at Metadata level by default, RequestResponse for writes to the resources you care about, and never request bodies for Secrets:

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - RequestReceived
rules:
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets"]
  - level: RequestResponse
    userGroups: ["system:serviceaccounts:agents"]
    verbs: ["create", "update", "patch", "delete"]
  - level: Metadata
    userGroups: ["system:serviceaccounts:agents"]

Rules are matched in order, so the Secrets rule comes first. Join Kubernetes audit events to the tool-call record by ServiceAccount and time window, or better, by putting the trace_id into a request annotation where your tooling allows it.

Storage and integrity

• Append-only storage that the agent's identity cannot write to or delete from. The component that writes audit records should be different from the one that executes tools.
• Per-record digest plus a chain or batch digest, so removal of records is detectable, not only edits.
• Signatures if records are evidence for third parties. An unkeyed hash is not enough there.
• Retention defined per field group: decision evidence long, redacted arguments medium, full arguments short.
• Clock discipline: UTC timestamps from synchronized nodes; record both decision time and execution time if they differ.

An example record

Values are placeholders; the shape is what matters.

{
  "timestamp": "2026-10-22T08:00:00Z",
  "trace_id": "<trace-id>",
  "span_id": "<span-id>",
  "agent_name": "research-agent",
  "workload_identity": "system:serviceaccount:agents:research-agent",
  "on_behalf_of": "tenant:docs-team",
  "tool_name": "http_get",
  "tool_server": "mcp-docs",
  "tool_catalog_digest": "sha256:<catalog>",
  "tool_call_id": "<provider-call-id>",
  "mcp_session_id": "<session-id>",
  "arguments_digest": "sha256:<args>",
  "arguments_redacted": {"url": "https://docs.internal.example/page"},
  "verdict": "ALLOW",
  "reason": "host allowlisted",
  "policy_version": "<policy-version>",
  "policy_sha256": "sha256:<policy>",
  "approval_id": null,
  "outcome": "executed",
  "result_digest": "sha256:<result>",
  "input_tokens": null,
  "output_tokens": null,
  "commit_sha": "<commit>",
  "record_digest": "sha256:<record>"
}

Checklist

• One audit record per tool call, including blocked and approval-pending calls
• Agent, workload identity and principal recorded on every record
• Policy version and policy digest recorded with every verdict
• Approvals bound to an argument digest and recorded outside the agent
• Arguments and results stored as digests; redacted copies only; full copies only where policy requires
• trace_id, tool_call_id and MCP session/request IDs on every record
• OTel execute_tool spans emitted with a pinned semconv version; Opt-In attributes off by default
• Cost fields linked by trace, labelled attribution-grade
• Kubernetes audit policy covers agent ServiceAccounts
• Append-only storage the agent cannot modify; digests chained; signatures where third parties rely on it
• Retention defined per field group

For the risk side of the same design, see OWASP Top 10 for Agentic Applications mapped to Kubernetes controls. For the telemetry side, see OpenTelemetry for LLM observability.

References

• OpenTelemetry GenAI spans, execute tool span: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md
• OpenTelemetry semantic conventions for MCP: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/mcp.md
• OpenTelemetry GenAI attribute registry: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/registry/attributes/gen-ai.md
• OWASP Logging Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html
• OWASP AI Agent Security Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/AI_Agent_Security_Cheat_Sheet.html
• OWASP Top 10 for Agentic Applications for 2026: https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/
• Kubernetes auditing: https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/
• Agent Security Contract (evidence envelope, verify, limits): https://github.com/justrunme/agent-security-contract
• AI Runtime Platform, enforcement evidence: https://github.com/justrunme/ai-runtime-platform/blob/main/docs/enforcement-evidence.md
• AI Runtime Platform, usage events: https://github.com/justrunme/ai-runtime-platform/blob/main/docs/usage-events.md