Reliability

What fails open and what fails closed

OBLETH_FAIL_OPEN covers budgets only, never identity or telemetry. Fail-open keeps serving through a Redis blip at the cost of possible over-spend while it lasts; set it to false when a budget must never be exceeded. In Helm, set obleth.failOpen. The Compose file sets it to true directly, so change it in a Compose override file rather than .env. A value that is not a boolean stops the gateway at start-up.

Redis outages

Keys, models and MCP servers are resolved from an in-process cache first (entries live 5 minutes), then Redis, never Postgres. So a Redis outage does not affect keys used in the last few minutes on that replica; a key that has to come from Redis gets the fast 503 above. A command waits at most OBLETH_REDIS_RESPONSE_TIMEOUT_MS (250) and a connect OBLETH_REDIS_CONNECT_TIMEOUT_MS (2000), so a stalled Redis fails quickly instead of hanging requests.

Postgres is the source of truth. The gateway pushes every key, model and MCP server into Redis at start, on every change, every OBLETH_REDIS_REWARM_SECS (300; 0 turns the timer off) and after every reconnect, and prunes entries that no longer exist. A Redis restart, flush or eviction therefore heals on its own within one interval. To heal it at once, use Reconcile cache… in Settings › Data and backups, or POST /api/v1/resync.

Per-minute buckets, term-budget usage and the response cache live only in Redis. Losing them resets the current windows and empties the cache; keep AOF persistence on if you enforce term budgets. See Backup and restore.

ClickHouse outages

ClickHouse is never on the request path. Usage rows are batched in the background; when an insert fails, whatever OBLETH_FAIL_OPEN says, the batch is written to spill segments next to OBLETH_WAL_PATH. When ClickHouse is back, the segments replay in batches and are deleted once committed. Replay is idempotent. The gateway also starts without ClickHouse and applies its schema when it becomes reachable.

Spill is capped at 256 MiB and 1,024 segments. Past either limit, or when the path is not writable, new batches are dropped rather than older ones evicted, and obleth_telemetry_dropped increases. Alert on any increase.

Request spans are not spilled: spans produced during an outage are lost.

Postgres outages

The gateway will not start without Postgres, because it applies its migrations at boot. Once running, Postgres is off the request path: traffic continues, while configuration changes and rewarm passes fail until it returns.

Retries and failover

Each model has its own policy, under Connection on the model's page, or PUT /api/v1/models/{id}/reliability:

Connection errors, timeouts, and HTTP 408, 429, 500, 502, 503 and 504 are retried; any other 4xx is returned at once. When an endpoint's retries are spent, the request moves to the next endpoint. When all fail, the client gets 502, or 504 if the last failure was a timeout. One fairshare slot and one budget reservation cover every attempt, and a request that ends in 502 or 504 bills nothing.

Retries and failover stop once the first byte has gone to the client. A stream that then breaks ends with an error event and an aborted response, recorded as 502 and billed for what reached the client; see When a stream is cut short. Multipart requests (audio uploads) get one attempt, since their bodies cannot be replayed.

Multiple endpoints

A model can have several endpoints serving the same upstream_model, each with its own api_base, key, priority, weight and health. Add them in the model's Endpoints section or with POST /api/v1/models/{id}/endpoints. Only endpoints that are enabled and not confirmed unhealthy take traffic; a model with no eligible endpoint uses its own api_base.

Selection modes

Failover is between endpoints of one model. The gateway never swaps in a different model; to spread load across models, use auto routing.

Evictions that fail

Deleting or disabling a key, tenant, model, MCP server or alias removes it from Redis. If that eviction fails, the change is saved in Postgres but the call returns 502 naming POST /api/v1/resync, rather than reporting success while the old entry keeps serving. Once Redis is healthy, run the resync, or Reconcile cache… in Settings › Data and backups.

Timeouts, connection reuse and diagnostics+

OBLETH_UPSTREAM_READ_TIMEOUT_SECS bounds any silence from an upstream, including the wait for headers; it defaults to the request timeout, floored at 120 seconds. Idle pooled connections close after OBLETH_UPSTREAM_POOL_IDLE_SECS (15), and live ones send TCP keepalives every OBLETH_UPSTREAM_TCP_KEEPALIVE_SECS (30). A send failure on a stale pooled connection gets one extra attempt on a fresh connection after 50 ms, on top of max_retries. All timeouts are listed in Shutdown and timeouts.

Debug upstream failures, under Connection on a model's page, runs a read-only DNS and port check after a request to that model ends in 502 or 504 and records it as an upstream_diagnostics span. It never changes routing.

Each attempt increments obleth_upstream_attempts_total{outcome}, with success, retry, conn_retry, timeout, failover or exhausted. Endpoint health is probed like the model's; see Health checks.