Troubleshooting

Enable debug logging

RUST_LOG=obleth=debug

This adds verbose output for request routing, cache hits/misses, Redis operations, and fairshare decisions.

HTTP error reference

499 appears in the request log but is never returned to anyone: it marks a request whose client disconnected before the response finished. Those requests are still settled — budgets are reconciled against the admission estimate when final usage never arrived — so a burst of 499s is a client-side cancellation pattern, not lost accounting.

Auth failures (401/403)

# 1. Confirm the key exists
curl http://localhost:9180/api/v1/keys \
  -H "Authorization: Bearer $TOKEN" | jq '.[] | select(.key_prefix == "sk_abc1")'

# 2. Confirm it's not disabled
# If "disabled": true, re-enable:
curl -X PUT http://localhost:9180/api/v1/keys/$KEY_ID/disabled \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"disabled": false}'

# 3. Check the key format: it must be the full secret (sk_...), not the prefix

Key resolution flow: the gateway hashes the full key with SHA-256 (mixing in OBLETH_API_KEY_PEPPER when set), checks its in-process moka cache, then obleth:key:{hash} in Redis. Postgres is not consulted on the hot path — Redis is populated from Postgres when the gateway starts and on every config write. If both caches miss, the request is rejected with 401.

A key you just deleted or disabled that still works on one replica is a stale moka entry — it expires within its 5-minute TTL, and the obleth:invalidate pub/sub message normally evicts it immediately. A key that authenticates nowhere after a Redis flush means Redis was never repopulated: restart the gateway, or make any config write, to re-sync it from Postgres.

404 on model

# List models
curl http://localhost:9180/api/v1/models \
  -H "Authorization: Bearer $TOKEN" | jq '.[] | {name: .model_name, enabled: .enabled}'

The model field in the request body must match a model_name in the registry exactly. If the model exists but enabled=false, re-enable it.

Fresh Helm install (empty registry)

A new Kubernetes install starts with zero models. helm install does not run model registration — you must POST /api/v1/models (or import via the dashboard) before inference traffic can succeed. See Installation — post-install steps.

Model registration rejected (400, SSRF)

Symptom: POST /api/v1/models or a dashboard import fails with a blocked-host message when api_base points at an internal Service.

By default, private/LAN addresses are allowed, so *.svc.cluster.local hostnames that resolve to RFC1918 IPs should register cleanly. Failures usually mean:

1. Strict SSRF is on (OBLETH_BLOCK_PRIVATE_NETWORKS=1) and the pod CIDR is not in OBLETH_ALLOWED_PRIVATE_CIDRS — add e.g. 10.0.0.0/8 in Helm obleth.allowedPrivateCidrs or the equivalent env var.
2. The host resolves to a blocked range — link-local / cloud metadata (169.254.x.x) is always rejected.

# Confirm current models and api_base values
curl http://localhost:9180/api/v1/models \
  -H "Authorization: Bearer $TOKEN" | jq '.[] | {name: .model_name, api_base}'

See Security — SSRF.

404 {"detail":"Not Found"} from a non-chat route

If embeddings, audio, or image requests return {"detail":"Not Found"} (note: {"detail":...} is the upstream's error shape, not obleth's {"error":...}), the model's api_base is almost certainly a full endpoint URL instead of the provider base.

obleth appends the client's request path to api_base. A route configured with api_base: https://provider.example/v1/embeddings plus a client call to /v1/embeddings yields a doubled path the upstream can't resolve.

# Inspect the route's api_base
curl http://localhost:9180/api/v1/models \
  -H "Authorization: Bearer $TOKEN" | jq '.[] | {name: .model_name, type: .model_type, api_base}'

# Fix: set api_base to the provider base ending in /v1
curl -X PUT http://localhost:9180/api/v1/models/$MODEL_ID \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{..., "api_base": "https://provider.example/v1"}'

See Multi-modal Models.

column "…" does not exist after an upgrade

The gateway applies its schema itself: the numbered files under schema/postgres/ are embedded in the binary and run in order on every boot, serialized across replicas by a Postgres advisory lock. Every statement is idempotent and additive, so re-running them is safe. A startup that reaches postgres connected + schema applied has a current schema; a migration failure is fatal and the process exits.

So a missing-column error at runtime means the schema was never applied, not that you need to patch it by hand. Check, in order:

1. The gateway actually started — look for postgres connected + schema applied in its logs, and for a migration error just before an exit.
2. OBLETH_DATABASE_URL points at the database you are inspecting.
3. The database user can run DDL (CREATE TABLE, ALTER TABLE) — a read-only or restricted role fails the migration.
4. Nothing else holds the advisory lock indefinitely (a stuck earlier boot).

Restarting the gateway re-runs the migrations. Do not reach for docker compose down -v: it destroys the volume and every tenant, key, and model in it.

429 quota exceeded

# Check tenant quota and current budget
curl http://localhost:9180/api/v1/tenants/$TENANT_ID \
  -H "Authorization: Bearer $TOKEN" | jq '{tpm: .tokens_per_minute, group: .fairshare_group}'

# Increase quota
curl -X PUT http://localhost:9180/api/v1/tenants/$TENANT_ID/quota \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"tokens_per_minute": 200000}'

The budget is a token bucket, not a fixed window: it refills continuously at tokens_per_minute / 60000 tokens per millisecond, up to a burst ceiling equal to tokens_per_minute. If a tenant is consistently hitting 429, either increase their quota or check for a runaway workload. Setting tokens_per_minute to 0 removes the per-minute cap entirely.

Requests piling up in the queue

When the cluster is at its in-flight cap, new requests wait in the fairshare queue until a slot frees up. They are admitted as soon as one does — they are not dropped — but persistent queue growth means OBLETH_GLOBAL_MAX_IN_FLIGHT is too low for the offered load. Check:

# Live stats
curl http://localhost:9180/api/v1/stats -H "Authorization: Bearer $TOKEN"
# Look at: in_flight, queued, max_in_flight

# Prometheus
curl http://localhost:9091/metrics | grep 'obleth_queue_depth\|obleth_in_flight'

Options:

1. Increase OBLETH_GLOBAL_MAX_IN_FLIGHT (only if backend can handle more concurrency)
2. Add more obleth pods
3. Review tenant weights so high-priority traffic keeps advancing while the queue is deep

Usage or cost numbers look too low

Usage, cost, and report reads exclude gateway-internal traffic by default: health probes (request_type = 'health_probe') and traffic from tenants flagged synthetic (request_type = 'benchmark'), which covers obench fixture tenants and the dashboard's own model-test and benchmark runs. Pass include_internal=true on the usage/cost endpoints — or the equivalent dashboard toggle — to see it.

Benchmark traffic is also never written into the permanent usage_daily rollup, so historical reports will not show it even with the flag set.

Replica state is frozen (Slurm)

If a model's Replicas panel shows a warning with a failure reason, its state badges are grayed out with a ?, or Settings → Slurm reads "running but failing since X", the provisioner process is alive but its reconcile ticks are failing — so every replica state shown is the last value it managed to reconcile, not live truth. The usual cause is slurmrestd being unreachable or rejecting the gateway's JWT.

After 10 minutes without a successful tick (and only while Slurm provisioning is enabled), a slurm_reconcile_held alert fires, with a recovery notice when it clears. See Slurm Provisioning.

Redis connectivity issues

obleth fails open when Redis is unavailable. If you see repeated log lines like:

WARN obleth_redis: Redis error: Connection refused
WARN obleth_proxy: budget reserve failed; failing open

Check:

docker logs obleth-redis-1
redis-cli -h $REDIS_HOST ping

When Redis reconnects, obleth resumes normal operation automatically. Check OBLETH_REDIS_URL is correct. The scheme must be redis:// (not rediss:// unless TLS is configured).

ClickHouse connectivity issues

WARN obleth_telemetry: clickhouse insert failed; spilling N records to WAL

This is non-fatal — requests continue. Check:

curl http://$CLICKHOUSE_HOST:8123/ping
# Expected: Ok.

Common causes: wrong OBLETH_CLICKHOUSE_URL, wrong credentials, ClickHouse not started. After connectivity is restored, obleth replays the spilled segments automatically, in checkpointed batches.

The spill is bounded — watch for dropped rows

Spill is capped at 256 MiB total and 1,024 segments (segments live in <OBLETH_WAL_PATH>.segments/). Past either limit, new spill is refused rather than older accounting being evicted, and the batch is dropped:

ERROR obleth_telemetry: telemetry WAL spill failed; records dropped

obleth_telemetry_dropped counts records that reached neither ClickHouse nor the WAL. Any non-zero value means usage rows are gone for good. If a ClickHouse outage is going to outlast the spill budget, restore ClickHouse (or point OBLETH_CLICKHOUSE_URL at a reachable instance) before the cap is hit.

A growing spill directory with ClickHouse healthy points at replay failing rather than insertion — look for telemetry WAL read failed or telemetry WAL checkpoint failed in the logs.

Health check

Both listeners expose a plain liveness probe that returns the literal string ok:

curl http://localhost:8080/health      # data plane
curl http://localhost:9180/api/v1/health   # Management API

It proves the process is up and serving; it does not check Redis, Postgres, or ClickHouse. Check dependency health from the logs, from the Prometheus metrics, or by connecting to each datastore directly (below).

Build identity is separate: GET /api/v1/version returns the running gateway's version and, for released images, the commit and build timestamp.