Troubleshooting

Cause: the key is unknown: a typo, the prefix instead of the full sk_… secret, a deleted key, or a different OBLETH_API_KEY_PEPPER after a restore. After a Redis flush, every key is unknown until the next rewarm.

Fix: check the key on API keys. After a flush, run Reconcile cache… in Settings › Data and backups.

Cause: the credential was verified as an identity token (JWT) and failed. The response is the same for every reason.

Fix: read the reason from obleth_jwt_verify_total{result} on :9091/metrics; Identity tokens maps each label to its cause. No obleth_jwt* metric means OBLETH_JWT_ISSUERS is unset; it is read at start-up.

Cause: the model is switched off, or the tenant's model allowlist excludes it. New models are created switched off and come on when their first health check passes.

Fix: turn the model on from its page (Check and turn on), or add it to the tenant's models.

Cause: the request's model matches no registered name or alias. A fresh install has no models.

Fix: register the model or add an alias. Claude Code and Anthropic SDKs send their own model names: set Default model for Anthropic clients in Settings › Routing, or they get not_found_error.

Cause: the model's api_base is a full endpoint URL such as …/v1/embeddings. The gateway appends the request path, so the path doubles.

Fix: set api_base to the provider's base, ending in /v1. See Images, audio and speech.

Cause: the outbound URL policy refused it. The host resolves to a link-local or metadata address, or strict mode (OBLETH_BLOCK_PRIVATE_NETWORKS=1) is on and the range is not in OBLETH_ALLOWED_PRIVATE_CIDRS.

Fix: add the cluster's range (for example 10.0.0.0/8) to the allowlist. See Security.

Cause: the tenant's tokens_per_minute bucket is empty. It refills continuously, up to one minute's worth.

Fix: raise the tenant's token rate, or look for a runaway client. 0 removes the limit. See Quotas and rate limits.

Cause: the request waited longer than OBLETH_ADMISSION_TIMEOUT_SECS (60) for a slot in the model's pool.

Fix: open the pool on the Fairshare page. If it is full, add backend capacity, or raise max_in_flight if the backend can serve more. If not, look for OBLETH_GLOBAL_MAX_IN_FLIGHT is below the sum of the enabled models' pool sizes in the start-up log and raise the ceiling. See Scaling.

Cause: Redis failed or took longer than OBLETH_REDIS_RESPONSE_TIMEOUT_MS (250) during a key lookup. Authentication never fails open. A redis_key_lookup_failed alert goes with it.

Fix: check redis-cli -a "$REDIS_PASSWORD" --no-auth-warning ping and OBLETH_REDIS_URL (scheme redis://). When Redis returns, the gateway re-pushes its configuration by itself.

Cause: Redis failed during the budget reserve and OBLETH_FAIL_OPEN=false.

Fix: restore Redis; see Reliability.

Cause: every attempt failed (502 upstream request failed) or timed out (504 upstream request timed out). A 502 recorded for a stream that ended early means the upstream broke off, went silent longer than OBLETH_UPSTREAM_READ_TIMEOUT_SECS, or closed the stream without its last event. The client got the error event upstream stream ended before the response was complete (after N s) and then an aborted response, and the request is billed for what reached the client; see When a stream is cut short. upstream_stream_error is alerted, unless the tool loop or the speculation boon ran the stream.

Fix: check the backend's logs at that time for restarts, out-of-memory kills, or idle timeouts on proxies in front of it. The seconds in the client's error count from the start of the upstream call, so a value that matches a proxy's timeout points at that proxy. Debug upstream failures on the model's page records a DNS and port check with the next failure.

Cause: the change was saved in Postgres, but the gateway could not evict the old entry from Redis, so it may still be serving.

Fix: once Redis is healthy, run Reconcile cache… in Settings › Data and backups, or POST /api/v1/resync.

Cause: since v2.0.0 Redis requires a password. NOAUTH means the gateway connected without one; WRONGPASS means it differs from Redis's.

Fix: put the same value in REDIS_PASSWORD and in OBLETH_REDIS_URL (redis://:PASSWORD@host:6379). In Compose both come from REDIS_PASSWORD in .env; in Helm from redis.password or your existingSecret. See Upgrading from v1.x.

Cause: ClickHouse is unreachable. Requests continue and usage rows spill to disk.

Fix: check curl http://$CLICKHOUSE_HOST:8123/ping, OBLETH_CLICKHOUSE_URL and the credentials; spilled rows replay by themselves. Past 256 MiB of spill, or with an unwritable path, rows are lost and obleth_telemetry_dropped rises. A spill that grows while ClickHouse is healthy means replay fails; look for telemetry WAL read failed.

Cause: the schema was never applied. Migrations run at every start and a failure stops the gateway, so a running gateway that logged postgres connected + schema applied has a current schema.

Fix: check that OBLETH_DATABASE_URL points at the database you are looking at, that its user can run DDL, and that no stuck boot holds the migration lock. Restart the gateway. Never run docker compose down -v: it deletes the volume and every tenant, key and model.

Cause: health probes and traffic from synthetic tenants (obench, the dashboard's model tests and benchmarks) are left out by default, and never enter the daily rollup.

Fix: pass include_internal=true, or use the dashboard's toggle for internal traffic.

Cause: the provisioner runs but its reconcile ticks fail, usually because slurmrestd is unreachable or refuses the JWT. States shown are the last ones it reconciled. After 10 minutes slurm_reconcile_held is alerted.

Fix: check the Slurm connection on Deployments. See Slurm provisioning.