Troubleshooting

Enable debug logging

RUST_LOG=obleth=debug

This adds verbose output for request routing, cache hits/misses, Redis operations, and fairshare decisions.

HTTP error reference

Auth failures (401/403)

# 1. Confirm the key exists
curl http://localhost:9180/api/v1/keys \
  -H "Authorization: Bearer $TOKEN" | jq '.[] | select(.key_prefix == "sk_abc1")'

# 2. Confirm it's not disabled
# If "disabled": true, re-enable:
curl -X PUT http://localhost:9180/api/v1/keys/$KEY_ID/disabled \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"disabled": false}'

# 3. Check the key format: it must be the full secret (sk_...), not the prefix

Key resolution flow: the gateway hashes the full key with SHA-256, then looks up obleth:key:{hash} in Redis. If Redis is down, it falls back to moka, then Postgres. If all three miss, the request is rejected.

404 on model

# List models
curl http://localhost:9180/api/v1/models \
  -H "Authorization: Bearer $TOKEN" | jq '.[] | {name: .model_name, enabled: .enabled}'

The model field in the request body must match a model_name in the registry exactly. If the model exists but enabled=false, re-enable it.

Fresh Helm install (empty registry)

A new Kubernetes install starts with zero models. helm install does not run model registration — you must POST /api/v1/models (or import via the dashboard) before inference traffic can succeed. See Installation — post-install steps.

Model registration rejected (400, SSRF)

Symptom: POST /api/v1/models or a dashboard import fails with a blocked-host message when api_base points at an internal Service.

By default, private/LAN addresses are allowed, so *.svc.cluster.local hostnames that resolve to RFC1918 IPs should register cleanly. Failures usually mean:

1. Strict SSRF is on (OBLETH_BLOCK_PRIVATE_NETWORKS=1) and the pod CIDR is not in OBLETH_ALLOWED_PRIVATE_CIDRS — add e.g. 10.0.0.0/8 in Helm obleth.allowedPrivateCidrs or the equivalent env var.
2. The host resolves to a blocked range — link-local / cloud metadata (169.254.x.x) is always rejected.

# Confirm current models and api_base values
curl http://localhost:9180/api/v1/models \
  -H "Authorization: Bearer $TOKEN" | jq '.[] | {name: .model_name, api_base}'

See Security — SSRF.

404 {"detail":"Not Found"} from a non-chat route

If embeddings, audio, or image requests return {"detail":"Not Found"} (note: {"detail":...} is the upstream's error shape, not obleth's {"error":...}), the model's api_base is almost certainly a full endpoint URL instead of the provider base.

obleth appends the client's request path to api_base. A route configured with api_base: https://provider.example/v1/embeddings plus a client call to /v1/embeddings yields a doubled path the upstream can't resolve.

# Inspect the route's api_base
curl http://localhost:9180/api/v1/models \
  -H "Authorization: Bearer $TOKEN" | jq '.[] | {name: .model_name, type: .model_type, api_base}'

# Fix: set api_base to the provider base ending in /v1
curl -X PUT http://localhost:9180/api/v1/models/$MODEL_ID \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{..., "api_base": "https://provider.example/v1"}'

See Multi-modal Models.

Models disappeared after a schema change

Symptom: every model read fails and logs show column "model_type" does not exist (or another missing column). In the Docker Compose stack, Postgres runs schema/postgres/0001_init.sql from docker-entrypoint-initdb.d only on a fresh volume. Editing that file does not migrate an existing database.

Apply the missing columns in place — this is non-destructive and preserves all rows:

docker exec -i obleth-postgres-1 psql -U obleth -d obleth -c \
  "ALTER TABLE models
     ADD COLUMN IF NOT EXISTS model_type text NOT NULL DEFAULT 'chat',
     ADD COLUMN IF NOT EXISTS cost_per_image double precision NOT NULL DEFAULT 0,
     ADD COLUMN IF NOT EXISTS cost_per_audio_second double precision NOT NULL DEFAULT 0,
     ADD COLUMN IF NOT EXISTS cost_per_character double precision NOT NULL DEFAULT 0;"

Alternatively, recreate the volume with docker compose down -v to re-run the init script from scratch — but that wipes all data (tenants, keys, models). Only do this in a throwaway environment.

429 quota exceeded

# Check tenant quota and current budget
curl http://localhost:9180/api/v1/tenants/$TENANT_ID \
  -H "Authorization: Bearer $TOKEN" | jq '{tpm: .tokens_per_minute, group: .fairshare_group}'

# Increase quota
curl -X PUT http://localhost:9180/api/v1/tenants/$TENANT_ID/quota \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"tokens_per_minute": 200000}'

The token budget resets every 60 seconds via the Redis Lua script. If a tenant is consistently hitting 429, either increase their quota or check if they have a runaway workload.

Requests piling up in the queue

When the cluster is at its in-flight cap, new requests wait in the fairshare queue until a slot frees up. They are admitted as soon as one does — they are not dropped — but persistent queue growth means OBLETH_GLOBAL_MAX_IN_FLIGHT is too low for the offered load. Check:

# Live stats
curl http://localhost:9180/api/v1/stats -H "Authorization: Bearer $TOKEN"
# Look at: in_flight, queued, max_in_flight

# Prometheus
curl http://localhost:9091/metrics | grep 'obleth_queue_depth\|obleth_in_flight'

Options:

1. Increase OBLETH_GLOBAL_MAX_IN_FLIGHT (only if backend can handle more concurrency)
2. Add more obleth pods
3. Review tenant weights so high-priority traffic keeps advancing while the queue is deep

Redis connectivity issues

obleth fails open when Redis is unavailable. If you see repeated log lines like:

WARN obleth_redis: Redis error: Connection refused
WARN obleth_proxy: budget reserve failed; failing open

Check:

docker logs obleth-redis-1
redis-cli -h $REDIS_HOST ping

When Redis reconnects, obleth resumes normal operation automatically. Check OBLETH_REDIS_URL is correct. The scheme must be redis:// (not rediss:// unless TLS is configured).

ClickHouse connectivity issues

WARN obleth_telemetry: clickhouse insert failed; spilling N records to WAL

This is non-fatal — requests continue. Check:

curl http://$CLICKHOUSE_HOST:8123/ping
# Expected: Ok.

Common causes: wrong OBLETH_CLICKHOUSE_URL, wrong credentials, ClickHouse not started. After connectivity is restored, obleth replays the WAL automatically.

Health check

curl http://localhost:9180/api/v1/health
# {"status": "ok", "redis": "ok", "postgres": "ok"}

If redis or postgres shows an error, check those services. clickhouse is not included in the health check (fail-open design).