Datastores

How to run each one is in Install on Kubernetes; backups are in Backup and restore.

Postgres

Postgres holds everything that must be durable and consistent: tenants, API keys (hashes only, never the secret), fairshare groups, models and their endpoints, MCP servers, settings, knowledge collections and their chunks, Slurm deployments and recipes, video job records, health-check history and the audit log. The dashboard keeps its sign-in tables in the same database.

It is never read on the request path. The management API writes to it first and then updates Redis. It is needed at start: the gateway applies its migrations (schema/postgres/0001 to 0030, embedded in the binary and idempotent) before it listens. Tables are listed in Postgres schema.

Redis

Redis serves what the request path needs at low latency:

• resolved keys, models (under their name and every alias) and MCP servers;
• per-minute token buckets, committed term-budget usage and in-flight term reservations, changed by atomic Lua scripts so many replicas stay correct;
• the response cache and the compression boon's stored originals, scoped per tenant;
• the live replica set and cluster-wide fairshare slots (Multiple gateway replicas);
• the Slurm provisioner's heartbeat and last reconcile result, so every replica reports the same;
• the obleth:invalidate channel that tells every replica to drop an in-process entry.

Everything except live bucket and reservation state can be derived from Postgres, so Redis needs no backup. The gateway repopulates it at start, on every configuration write, every OBLETH_REDIS_REWARM_SECS (default 300) and after every reconnect, and prunes entries Postgres no longer has. POST /api/v1/resync does it on demand. Every key is in Redis layout.

Requirements:

• A password. The request path reads keys from Redis, so an open instance would let anyone who reaches it add keys.
• A single primary. The budget and slot scripts touch several keys in one call, so clustered Redis is not supported. For high availability, use a failover endpoint that presents one primary.
• An eviction policy that spares committed state. Under volatile-lru, which the chart's bundled Redis uses with maxmemory 400mb, only keys with a TTL are evicted: cache entries, buckets and reservations. Resolved keys, models and committed term usage have none. Once nothing evictable is left, Redis refuses writes, and the gateway treats that like any Redis failure.

ClickHouse

ClickHouse holds the usage ledger: one row per request with tenant, key, model, admission class, estimated and real tokens, queue wait, time to first token, total time, status, cache status, cost, energy and CO₂, request type and session. Rows are inserted in batches from a background task and never block a request. Two more tables sit beside it: usage_daily, a rollup per day, tenant, key and model that is kept forever while raw rows are pruned after the retention window (default 180 days), and spans, per-step traces kept 14 days. The gateway creates and migrates all three at start. Columns and queries are in ClickHouse usage.

The gateway starts without ClickHouse. While it is unreachable, rows go to a local spill file (OBLETH_WAL_PATH) and are replayed, without duplicates, when it returns. On Kubernetes the chart gives that file a writable volume.

Why three stores

Each workload breaks the others. Per-request rows at high volume would bloat Postgres tables and slow the configuration API. ClickHouse has no transactions or row updates, which configuration needs. Redis is fast and atomic but not a durable source of truth. Keeping them apart also bounds failures: a Redis outage costs live counters, a ClickHouse outage delays usage rows, and neither loses configuration.

What the request path does when a store is down+

• Redis: key lookups that miss the in-process cache return 503. Budget checks follow OBLETH_FAIL_OPEN: admitted without enforcement when true (the default), 503 when false. With several replicas, fairshare falls back to dividing the limits between them.
• ClickHouse: requests are unaffected; usage spills to disk. Usage views miss those rows until they replay.
• Postgres: requests keep being served from the caches, but configuration writes fail, and a gateway can't start.

See Reliability.