Reliability & Fail-open

One of obleth's core design goals is that a Redis or ClickHouse outage should not take down inference traffic. This is the fail-open behavior.

This page covers obleth's resilience to datastore outages (Redis, ClickHouse, Postgres). For resilience to upstream failures — per-request timeouts, retries with backoff, and failover across multiple endpoints — see Reliability & Failover.

Two caching layers

Key resolution on the data plane reads two caches and never Postgres:

Bearer token → SHA-256 hash
  → moka (in-process, TTL=5min, cap=100k)  ← zero network latency
  → Redis (sub-ms, shared across pods)       ← shared, always fresh
  → miss on both = 401

Redis is warmed from Postgres when the gateway starts and rewritten on every config change, so Postgres stays the source of truth without ever sitting in front of a request.

The in-process moka cache means that even a total Redis outage doesn't break auth lookups for keys that were recently resolved. Keys stay valid in moka for 5 minutes after their last resolution. A key that is in neither cache is rejected — there is no Postgres fallback on the hot path.

The moka cache is kept up-to-date via Redis pub/sub invalidation: when a key is created, disabled, or has its tenant's weight changed via the Management API, a message is published to obleth:invalidate. Each pod's moka evicts the affected key on receipt.

Fail-open vs fail-closed

OBLETH_FAIL_OPEN (default: true) governs the budget checks, not identity. What a Redis failure does depends on which lookup failed:

For production deployments handling real customer traffic, fail-open is the safer default — a Redis blip doesn't cause a customer-visible outage for keys that are already resolved. Token budget enforcement is bypassed temporarily, but requests are still served.

For strict budget enforcement scenarios where over-spend must never happen, set OBLETH_FAIL_OPEN=false to reject rather than over-serve.

OBLETH_FAIL_OPEN=true

Telemetry WAL

ClickHouse is never on the request hot path. Usage records are sent to a bounded async channel (mpsc::Sender), and a background task batches them into ClickHouse every second.

If ClickHouse is unavailable:

1. The background flusher catches the error.
2. If fail_open=true, each failed batch is written as a new spill segment in a directory alongside OBLETH_WAL_PATH (default: ./obleth-telemetry.wal, segments in <path>.segments/).
3. Once ClickHouse recovers, the replayer reads the oldest segment in checkpointed batches (at most 1 MiB and 500 records at a time) and re-inserts them, with retry backoff on failure.
4. A segment is deleted once its records are committed. A pre-existing single-file WAL from an older version is read first, so upgrades lose nothing.

# Where to spill telemetry when ClickHouse is unavailable
OBLETH_WAL_PATH=/var/lib/obleth/telemetry.wal

The spill is bounded

Spill is capped at 256 MiB total across segments and 1,024 segments. New spill is refused past either limit — obleth never silently evicts older accounting to make room. When spill is refused (or the disk is full), the batch is dropped, an error is logged, and the dropped-record counter advances.

Monitor it via Prometheus: obleth_telemetry_dropped is the count of records dropped because they could be written neither to ClickHouse nor to the WAL. Any non-zero value means usage rows were lost.

What obleth does NOT protect against

• Total Postgres outage at startup: obleth won't boot if it can't connect to Postgres and run migrations. Once running, Postgres is off the hot path and its outage doesn't affect traffic.
• Redis outage of any kind: a request whose key is not already in moka is rejected 401, because identity is never assumed. Fail-open only relaxes budget enforcement.
• Redis outage with OBLETH_FAIL_OPEN=false: requests that do resolve are additionally rejected 503 at the budget check.
• A ClickHouse outage longer than the spill budget: once 256 MiB or 1,024 segments have accumulated, further batches are dropped and obleth_telemetry_dropped increments. The already-spilled records still replay.
• Disk full with fail_open=true: WAL writes fail, telemetry is dropped, and obleth_telemetry_dropped increments.
• fail_open=false with ClickHouse down: batches are not spilled at all; they are counted as dropped immediately.
• Pod restart with empty moka cache: after a restart, moka is cold, so the first request for each key must reach Redis. If Redis is also down at that moment, the request is rejected 401 — fail-open does not cover identity.
• A flushed Redis while the gateway keeps running: nothing repopulates it per-request. Keys already in moka keep working until their TTL expires, then start failing 401. Restart the gateway (it warms Redis from Postgres at boot) or make any config write to re-sync.