Prometheus metrics

The gateway serves Prometheus text format at /metrics on the metrics listener (OBLETH_METRICS_LISTEN, default 0.0.0.0:9091). Every metric starts with obleth_. Labels are low-cardinality: no tenant, key, model or collection is a label. For per-tenant and per-model breakdowns, query the ClickHouse usage ledger.

Each gateway process exports its own values. With several replicas, sum() across targets, except where the table says max().

Metrics

Metric	Type	Labels	What it counts
`obleth_requests_total`	Counter	`admission`, `status`	Requests by admission class and status class (`2xx`, `4xx`, …)
`obleth_input_tokens_total`	Counter		Input tokens of every recorded request, boon calls included
`obleth_output_tokens_total`	Counter		Output tokens of every recorded request, boon calls included
`obleth_ttft_ms`	Histogram		Time to first token, ms; buckets 5 to 5000
`obleth_total_ms`	Histogram		Total request duration, ms; buckets 10 to 30000
`obleth_in_flight`	Gauge		Requests holding a slot in any pool of this process
`obleth_queue_depth`	Gauge		Requests waiting for a slot in any pool
`obleth_fairshare_replicas`	Gauge		Live gateway replicas from Redis heartbeats; take `max()`
`obleth_fairshare_shared_slots_mode`	Gauge	`mode`	`1` for this replica's current mode, `0` for the others
`obleth_fairshare_cluster_in_flight`	Gauge		In flight across all replicas in `shared` mode, else this replica's; take `max()`
`obleth_fairshare_held_slots`	Gauge		Slots this replica holds for sessions' next steps (session affinity); in `shared` mode they are held in Redis and not counted here
`obleth_fairshare_affinity_total`	Counter	`event`	Session affinity on this replica: `jump` (a request went ahead of an older one), `hold` (a slot was held), `redeem` (a session's next step took it), `expire` (a slot held here was freed unused; `shared` mode frees them in Redis, so take `hold` minus `redeem`)
`obleth_capacity_discovery_models`	Gauge	`state`	Models in the `discovered` capacity mode, by state
`obleth_telemetry_dropped`	Gauge		Usage rows lost since start (never reached ClickHouse or the WAL)
`obleth_cache_lookups_total`	Counter	`result`	Response-cache lookups: `hit` or `miss`
`obleth_cache_tokens_saved_total`	Counter		Tokens served from the response cache
`obleth_compression_tokens_saved_total`	Counter		Input tokens removed by the compression boon
`obleth_mcp_requests_total`	Counter	`server`, `status`	MCP gateway requests by server name and status class
`obleth_upstream_attempts_total`	Counter	`outcome`	Upstream dispatch attempts by outcome
`obleth_jwt_verify_total`	Counter	`result`	JWT bearer verification results
`obleth_jwks_refresh_total`	Counter	`issuer_index`, `result`	JWKS fetches per issuer: `ok` or `error`
`obleth_knowledge_retrievals_total`	Counter	`outcome`	Knowledge-boon retrievals by outcome
`obleth_auto_routes_total`	Counter	`model`, `reason`	`auto` requests by the model served and why; `reason` is `per_turn`, or with sticky conversations one of the reasons listed there

Label values

Label	Values
`admission`	`fast`, `queued`, `rejected`, `boon` (a helper call made by a boon; always `2xx`)
`mode`	`shared`, `split`, `fallback` (shared slots on, Redis unreachable), `local`
`state`	`discovered`, `stale` (last value kept), `fallback` (static `max_in_flight` or default)
`outcome` (upstream)	`success`, `retry`, `conn_retry`, `timeout`, `failover`, `exhausted`
`result` (JWT)	`ok`, `unknown_issuer`, `bad_alg`, `unknown_kid`, `bad_signature`, `expired`, `not_yet_valid`, `bad_audience`, `bad_subject`, `malformed`, `not_provisioned`, `provision_failed`
`outcome` (knowledge)	`hit`, `miss`, `no_query`, `no_window`, `error`
`issuer_index`	Position in `OBLETH_JWT_ISSUERS`, from 0

What the less obvious values mean

boon admission. Exclude it to count client requests only: obleth_requests_total{admission!="boon"}.
Fairshare mode. split enforces the configured limits divided by the live replica count; local is one live replica, or both OBLETH_FAIRSHARE_SHARED_SLOTS and OBLETH_FAIRSHARE_REPLICA_AWARE off. See Multiple gateway replicas.
Upstream outcomes. retry has a configured retry left on the endpoint; conn_retry is the one extra attempt on a fresh connection after a connection-level failure; failover moves to the next endpoint; exhausted means the client got 502 or 504 and was not billed. See Reliability.
JWT results. not_provisioned and provision_failed are counted in addition to ok, so rejections are the total minus ok. See Identity tokens.
JWKS errors. A sustained error rate means the gateway verifies against its last good key set and rejects tokens signed with a newly rotated key.
Knowledge outcomes. miss ran retrieval and injected nothing; no_window means the model has no context window set; error means every embedding attempt failed. See Knowledge base.
Dropped telemetry. Rows spilled to the WAL are not dropped; they replay when ClickHouse returns. Any increase is lost usage. See Observability.

Example queries

Client request rate by admission class:

sum by (admission) (rate(obleth_requests_total{admission!="boon"}[5m]))

P95 time to first token:

histogram_quantile(0.95, sum by (le) (rate(obleth_ttft_ms_bucket[5m])))

Response-cache hit rate:

sum(rate(obleth_cache_lookups_total{result="hit"}[5m]))
  / sum(rate(obleth_cache_lookups_total[5m]))

Upstream failures that reached clients:

sum(rate(obleth_upstream_attempts_total{outcome="exhausted"}[5m]))

Replicas in fairshare fallback:

sum(obleth_fairshare_shared_slots_mode{mode="fallback"}) > 0

Lost usage rows (alert on any increase):

increase(obleth_telemetry_dropped[15m]) > 0

Scrape configuration

scrape_configs:
  - job_name: obleth
    static_configs:
      - targets: ["obleth:9091"]

The Compose observability profile ships this job (5 s interval) plus Postgres, Redis, ClickHouse (9363) and HAProxy (8404). With the Prometheus Operator, set serviceMonitor.enabled: true in the Helm chart; the ServiceMonitor scrapes the metrics port every 5 s.

PreviousAuthentication

NextClickHouse usage