OpenTelemetry LLM observability in practice: GenAI conventions, vLLM metrics and token cost

Many LLM dashboards start with GPU utilisation and stop there. That tells you the hardware is busy. It does not tell you which team spent the tokens, whether the fallback model is serving half your traffic, or why time-to-first-token doubled after a deploy. OpenTelemetry can answer those questions if you name things consistently. That is what the GenAI semantic conventions are for.

This guide covers three layers: the conventions (what to call spans, attributes and metrics), the model server (what vLLM exposes today), and cost (turning token counts into money without lying to yourself). I use my own AI Runtime Platform gateway as the worked example, including the places where it does not follow the conventions yet.

1. Status first: the conventions are still "Development"

The GenAI conventions have moved out of the main semantic-conventions repository into their own open-telemetry/semantic-conventions-genai repository. The old opentelemetry.io pages now say "moved". As of 21 October 2026, the metrics and attributes I use below are marked Development stability. Names can still change. Pin the convention version in your instrumentation, and expect to migrate.

2. Spans: one per model call

The span conventions define an inference span with:

• Span name {gen_ai.operation.name} {gen_ai.request.model}, for example chat qwen2.5-7b-instruct.
• gen_ai.operation.name (required), such as chat, text_completion, generate_content.
• gen_ai.provider.name, such as openai.
• gen_ai.request.model and gen_ai.response.model. These are two attributes on purpose: what you asked for and what actually answered.
• Token usage as gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. The registry also defines subsets such as gen_ai.usage.cache_read.input_tokens and gen_ai.usage.reasoning.output_tokens. The spec says input_tokens should include all input types, cached tokens included.

Prompt and response content is opt-in. The conventions say content capture must be enabled explicitly, for example via OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT. Leave it off by default on shared clusters. Prompts can carry personal data and secrets, and they make traces expensive to store.

A minimal Python span that follows the conventions:

from opentelemetry import trace

tracer = trace.get_tracer("ai-gateway")

def chat(model: str, messages: list[dict]) -> dict:
    with tracer.start_as_current_span(f"chat {model}") as span:
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.provider.name", "openai")   # OpenAI-compatible API
        span.set_attribute("gen_ai.request.model", model)
        resp = backend.chat.completions.create(model=model, messages=messages)
        span.set_attribute("gen_ai.response.model", resp.model)
        if resp.usage:
            span.set_attribute("gen_ai.usage.input_tokens", resp.usage.prompt_tokens)
            span.set_attribute("gen_ai.usage.output_tokens", resp.usage.completion_tokens)
        return resp

What my gateway does today, honestly. In app/gateway/routers/chat.py the span is named gen_ai.chat and sets gen_ai.request.model and gen_ai.operation.name. It also sets two attributes that are not in the conventions: gen_ai.usage.cost_usd and gen_ai.server.time_to_last_byte_ms. Routing decisions go under its own ai.runtime.* prefix (ai.runtime.selected_backend, ai.runtime.routing_reason), which is the right pattern. The gen_ai.*-prefixed custom attributes are not, because they squat on a namespace the spec owns. The fix on my list: rename the span to chat {model}, add gen_ai.response.model and the usage attributes, and move cost and latency extras under ai.runtime.*.

3. Metrics: separate "how much" from "how it is distributed"

The conventions split token metrics into two families. This split is the most useful thing in the spec for anyone doing cost work:

• Usage counters gen_ai.client.inference.usage.input_tokens, …usage.output_tokens, plus subsets …usage.cache_read.input_tokens, …usage.cache_write.input_tokens and …usage.reasoning.output_tokens. The spec calls these "the primary instruments for measuring token consumption" and "a proxy for cost approximation". They are monotonic counters broken down by gen_ai.token.modality.
• Per-operation histograms gen_ai.client.inference.operation.input_tokens and …operation.output_tokens. The spec says explicitly that these "should not be used for total usage or cost calculations". Use them for p95/p99 and outliers.

Latency has its own instruments: gen_ai.client.operation.duration on the client side, and on model servers gen_ai.server.request.duration, gen_ai.server.time_to_first_token and gen_ai.server.time_per_output_token. There are newer agent and tool metrics too (gen_ai.invoke_agent.duration, gen_ai.invoke_agent.tool_calls, gen_ai.execute_tool.duration). They are handy if your gateway also brokers MCP tool calls.

4. vLLM: Prometheus first

vLLM exposes metrics on /metrics of its OpenAI-compatible server. These are the ones I wire up first, as named in the vLLM production metrics docs:

In the Prometheus text format, counters get a _total suffix. Check your own /metrics output before writing queries. A pull request adding native OTLP metrics export to vLLM (#35469) was closed without being merged. So for metrics, scrape Prometheus and convert with the OpenTelemetry Collector if your backend wants OTLP.

# OpenTelemetry Collector: scrape vLLM, receive app traces/metrics over OTLP
receivers:
  otlp:
    protocols:
      grpc: {}
      http: {}
  prometheus:
    config:
      scrape_configs:
      - job_name: vllm
        scrape_interval: 15s
        kubernetes_sd_configs:
        - role: pod
          namespaces: { names: ["ai-runtime"] }
        relabel_configs:
        - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
          regex: vllm
          action: keep
processors:
  memory_limiter: { check_interval: 1s, limit_mib: 256 }
  batch: {}
exporters:
  otlphttp:
    endpoint: http://tempo.observability.svc.cluster.local:4318
  prometheusremotewrite:
    endpoint: http://prometheus.monitoring.svc:9090/api/v1/write
service:
  pipelines:
    traces:  { receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlphttp] }
    metrics: { receivers: [otlp, prometheus], processors: [memory_limiter, batch], exporters: [prometheusremotewrite] }

This is a sketch: the label selector, endpoints and exporters depend on your stack. The prometheusremotewrite exporter ships in the Collector contrib distribution, and Prometheus must have its remote-write receiver enabled. The collector in my runtime repo (deploy/observability/otel-collector.yaml) currently has only a traces pipeline (OTLP in, Tempo and debug out). vLLM metrics there are picked up by a ServiceMonitor in the Helm chart, not by the collector.

The same vllm:num_requests_waiting gauge drives scaling. My KEDA ScaledObject (deploy/keda/vllm-scaledobject.yaml) uses the query sum(vllm:num_requests_waiting{namespace="ai-runtime"}) with threshold 4. That threshold is a starting value in a reference manifest, not a tuned result.

5. Token cost without self-deception

Cost is where observability most often becomes fiction. Rules I follow:

1. Count tokens from usage counters, never from histograms. The spec says so, and summing histogram buckets gives wrong totals.
2. Price outside the instrumentation. Keep a versioned price table (model → price per input and output token) and multiply in queries or in a recording rule. Prices change, and your history should not be rewritten when they do.
3. Attribute at the gateway. Tenant, team or API key is known at the boundary, not at the model server. vLLM counts tokens per model, and the gateway knows who sent them. Put the owner on gateway metrics and keep label cardinality bounded: no raw keys, no user IDs.
4. Label estimates as estimates. My gateway emits gateway_chat_estimated_cost_usd_total and an x-ai-estimated-cost-usd response header. The name says "estimated" because it comes from a price table, not from an invoice.
5. Enforce, do not only observe. A dashboard tells you after the fact. The Inference Budget Guard in the same gateway reserves the estimated cost (tokens × price) against a per-tenant or per-key window before the completion runs, refuses over-budget requests with 429, fails closed with 503 when its store or policy is broken, and settles to actual priced usage afterwards. It is off by default. Its evidence is on its test suite with one Redis: the repo says estimates can differ from actual usage and settlement can lag.

An example recording rule for cost per model from server-side counters, assuming a price-table metric you maintain yourself (llm_price_usd_per_token, labelled by model_name and direction):

groups:
- name: llm-cost
  rules:
  - record: llm:cost_usd:rate5m
    expr: |
      sum by (model_name) (
        rate(vllm:prompt_tokens_total[5m])
          * on (model_name) group_left
            llm_price_usd_per_token{direction="input"}
      )
      +
      sum by (model_name) (
        rate(vllm:generation_tokens_total[5m])
          * on (model_name) group_left
            llm_price_usd_per_token{direction="output"}
      )

For self-hosted models the "price" is an internal rate (GPU-hour cost divided by expected tokens), and that is an assumption you should write down next to the rule. The cost and token observability case describes the full loop (meter, attribute, budget, alert or cap).

6. Connect the signals

Separate dashboards for traces, metrics and logs produce three stories per incident. The minimum glue I use:

• One request ID across gateway, model server and tool calls. Put it on spans and on structured logs.
• Exemplars on the gateway latency histogram, so a slow bucket links to a trace.
• Routing context on every span: selected backend, fallback used, canary weight. Without it, a routing change and a slow model look the same on a latency chart.

The wider pattern (collector topology, correlation, runbooks) is in the OpenTelemetry observability mesh case. The boundary that makes attribution possible is described in the cloud-native AI gateway case.

Checklist

• Inference spans named {operation} {model} with gen_ai.operation.name, gen_ai.provider.name, request and response model, and usage tokens
• Content capture off by default and enabled per environment
• Custom attributes under your own prefix, not under gen_ai.*
• Token totals from usage counters; histograms only for distributions
• vLLM /metrics scraped: queue, KV cache, TTFT, inter-token latency, tokens
• Price table versioned; cost labelled as estimated
• Owner attribution at the gateway with bounded cardinality
• Budget enforcement before execution, not only alerts after

References

• OpenTelemetry GenAI semantic conventions repository: https://github.com/open-telemetry/semantic-conventions-genai
• GenAI spans: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md
• GenAI metrics: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-metrics.md
• GenAI inference token metrics: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-token-metrics.md
• GenAI attribute registry: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/registry/attributes/gen-ai.md
• Old location notice: https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-metrics/
• OpenTelemetry blog, LLM observability introduction: https://opentelemetry.io/blog/2024/llm-observability/
• vLLM production metrics: https://docs.vllm.ai/en/latest/usage/metrics/
• vLLM PR #35469 (OTLP metrics, closed unmerged): https://github.com/vllm-project/vllm/pull/35469
• KEDA Prometheus scaler: https://keda.sh/docs/latest/scalers/prometheus/
• AI Runtime Platform (gateway code, collector, KEDA, budget guard docs): https://github.com/justrunme/ai-runtime-platform