Observability

Prometheus metrics

Every gateway replica serves metrics on OBLETH_METRICS_LISTEN (default 0.0.0.0:9091, path /metrics). In Kubernetes, set serviceMonitor.enabled: true if the Prometheus Operator is installed. In Compose, the observability profile adds Prometheus on 127.0.0.1:9090 and Grafana on port 3001 with an Obleth Gateway dashboard; set GRAFANA_ADMIN_PASSWORD in deploy/docker/.env first, or Compose refuses to start it.

Labels stay low-cardinality: no metric carries a tenant, key or model label. The series worth an alert:

Every metric and its labels are in Prometheus metrics.

Request logs

Each request, rejections included, writes one row to the ClickHouse usage ledger: time, tenant, key, model, session, request type, tokens (with cached input tokens when the backend reports them), time to first token, total time, status and cost. Request logs in the dashboard shows them within a few seconds, with filters for status, model, team, key, type and session. Select a request to see what happened in one line, its timeline, every number, the variant it named, and the Helper calls its boons made, each linked back to it.

Without tracing, the timeline comes from the log row: the wait for a slot, the time to first token, and the rest. Traced requests show each step, and a step's attributes, such as why the auto router picked the model. See Control plane for the rest of the page.

The same rows are at GET /api/v1/usage/logs on the Management API. Health probes and traffic from tenants flagged synthetic (obench runs, the dashboard's own tests) are excluded from usage and cost figures by default; pass include_internal=true to include them.

Per-request span tracing

Turn on Trace requests for a tenant (all its keys) on the tenant's page, or for one key in its panel on API keys. Or use the API:

curl -X PUT http://localhost:9180/api/v1/tenants/$TENANT_ID/tracing \
  -H "Authorization: Bearer $OBLETH_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"tracing_enabled": true}'

Use /api/v1/keys/$KEY_ID/tracing for a single key. Traced requests write their spans to the ClickHouse spans table, the log row carries has_trace: true, and GET /api/v1/usage/logs/{request_id}/spans returns them. Spans produced while ClickHouse is unreachable are discarded; unlike usage rows, they are not spilled.

OTLP export

Set OBLETH_OTEL_ENDPOINT to an OTLP/HTTP collector's base URL and every request's spans are exported to {endpoint}/v1/traces under the service name obleth. Unset, there is no tracing overhead.

# Compose .env: the observability profile runs Jaeger, UI on 127.0.0.1:16686
OBLETH_OTEL_ENDPOINT=http://jaeger:4318

In Helm, set obleth.otelEndpoint. Export runs through a background batch processor, so a slow collector does not slow requests. Each request's spans carry session.id and session.id.source (client or derived), so a conversation's turns can be grouped. See Conversations and sessions.

Per-tenant data

Prometheus answers "how is the gateway doing"; per-tenant, per-key and per-model questions go to ClickHouse. The raw usage table keeps OBLETH_USAGE_RETENTION_DAYS (default 180) of rows, and the usage_daily rollup keeps one row per day, tenant, key and model forever. Reports in the dashboard reads both, and the tables are described in ClickHouse usage.

Model health and the Slurm provisioner's state are per-model state, not time series, so they are shown in the dashboard rather than exported: see Health checks and Slurm provisioning.