Datastores

obleth uses exactly three datastores. Each handles a fundamentally different workload; collapsing any two would sacrifice correctness or performance.

The three-store principle

Postgres — configuration source of truth

Postgres is the durable, relational backbone for everything that needs to be queryable, auditable, and strongly consistent.

Tables:

• tenants — id, name, weight, tokens_per_minute, max_in_flight, fairshare_group
• api_keys — id, tenant_id, name, key_prefix, key_hash (SHA-256, never the raw secret), disabled
• models — model registry with routing, costs, and capability flags
• fairshare_groups — group names and weights for the hierarchical algorithm
• mcp_servers — registered MCP servers with upstream URLs and credentials
• audit_log — every config write: actor, action, entity, timestamp, detail (JSONB)

Postgres is never on the request hot path. The Management API writes to Postgres first, then syncs to Redis. The data plane reads Redis only.

In production, use CloudNativePG or a managed Postgres service for HA. The schema is embedded in the binary and applied idempotently on boot (CREATE TABLE IF NOT EXISTS).

Redis — hot cache and atomic budgets

Redis provides sub-millisecond access for the two hot-path operations: key resolution and token-budget enforcement.

Key layout:

Token budgets are enforced by Lua scripts that run atomically server-side: check, refill, reserve — all in one round-trip, correct across many gateway pods.

Pub/sub invalidation: the obleth:invalidate channel lets the Management API broadcast key evictions to every gateway pod's moka cache instantly. This is how live weight changes and key disables take effect in milliseconds without a restart.

For high availability in production, use a Redis Sentinel or Redis Cluster setup. obleth treats Redis as a hot cache: any data in Redis is always derivable from Postgres, so a Redis failure doesn't cause data loss (just a cold-start period).

ClickHouse — usage and cost ledger

ClickHouse stores one row per completed request, written asynchronously and never blocking the hot path.

Schema (managed by obleth):

CREATE TABLE usage (
  request_id UUID,
  tenant_id  UUID,
  key_id     UUID,
  model      String,
  admission  LowCardinality(String),  -- 'fast' | 'queued' | 'brownout' | 'rejected'
  weight     Int64,
  input_tokens  UInt32,
  output_tokens UInt32,
  estimated_tokens UInt32,
  queue_wait_ms UInt32,
  ttft_ms    UInt32,
  total_ms   UInt32,
  status_code UInt16,
  cache_status LowCardinality(String),  -- 'hit' | 'miss' | 'off'
  ts_ms      Int64,
  ts         DateTime64(3)
) ENGINE = MergeTree()
PARTITION BY toYYYYMMDD(ts)
ORDER BY (tenant_id, ts_ms);

The Management API's usage endpoints query this table: GET /api/v1/usage, /api/v1/usage/series, /api/v1/costs.

Why not Postgres for usage? ClickHouse handles millions of inserts per second and analytical aggregations over billions of rows with orders of magnitude less I/O than Postgres. Shoving high-frequency time-series data into Postgres would bloat its tables, slow down VACUUM, and degrade the config API's performance.

Why not ClickHouse for config? ClickHouse doesn't support transactions, foreign keys, or the OLTP update patterns that config management requires (e.g. UPDATE tenants SET weight = $2 WHERE id = $1 RETURNING ...).

Write path discipline

All config mutations follow a single path with no shortcuts:

Management API (validated, authenticated)
  → 1. Write to Postgres (durable, audited)
  → 2. Sync to Redis (cache update)
  → 3. Publish invalidation (evict moka on all pods)

There is no "write directly to Redis" shortcut. This means Postgres is always the authoritative source and Redis is always derivable from it.