Datastores

obleth uses exactly three datastores. Each handles a fundamentally different workload; collapsing any two would sacrifice correctness or performance.

The three-store principle

Postgres — configuration source of truth

Postgres is the durable, relational backbone for everything that needs to be queryable, auditable, and strongly consistent.

Tables:

• tenants — id, name, weight, tokens_per_minute, max_in_flight, fairshare_group
• api_keys — id, tenant_id, name, key_prefix, key_hash (SHA-256, never the raw secret), disabled
• models — model registry with routing, costs, and capability flags
• fairshare_groups — group names and weights for the hierarchical algorithm
• model_endpoints — the endpoints behind each model route (static or provisioner-registered)
• mcp_servers — registered MCP servers with upstream URLs and credentials
• audit_log — every config write: actor, action, entity, timestamp, detail (JSONB)
• app_settings — runtime settings persisted as JSON (alerts, auto-router, boons, energy, retention)
• managed_models, model_replicas, recipes, saved_recipes — Slurm-provisioned model specs, their replicas, and launch recipes
• model_health_checks — health-probe history per model

Postgres is never on the request hot path. The Management API writes to Postgres first, then syncs to Redis. The data plane reads Redis only.

In production, use CloudNativePG or a managed Postgres service for HA. The migrations live in schema/postgres/ as numbered files, are embedded in the binary, and are applied in order on boot. Every statement is idempotent (create table if not exists, add column if not exists) and additive, so applying them repeatedly — or running an older binary against a newer schema — is safe.

Redis — hot cache and atomic budgets

Redis provides sub-millisecond access for the two hot-path operations: key resolution and token-budget enforcement.

Key layout:

The provisioner keys live in Redis rather than pod memory so the dashboard reads a consistent value no matter which gateway replica served the provisioner's poll. The heartbeat proves the process is alive; the tick key proves reconciliation is actually succeeding.

Token budgets are enforced by Lua scripts that run atomically server-side: check, refill, reserve — all in one round-trip, correct across many gateway pods.

Pub/sub invalidation: the obleth:invalidate channel lets the Management API broadcast key evictions to every gateway pod's moka cache instantly. This is how live weight changes and key disables take effect in milliseconds without a restart.

For high availability in production, use a Redis Sentinel or Redis Cluster setup. obleth treats Redis as a hot cache: everything in it is derivable from Postgres, so a Redis failure loses no durable data (only live token-bucket state). It is not, however, rebuilt lazily per request — the data plane reads moka and Redis only. Redis is repopulated from Postgres when the gateway boots and on every config write, so an emptied Redis stays empty until one of those happens.

ClickHouse — usage and cost ledger

ClickHouse stores one row per completed request, written asynchronously and never blocking the hot path.

Schema (managed by obleth):

CREATE TABLE usage (
  request_id UUID,
  tenant_id  UUID,
  key_id     UUID,
  model      String,
  admission  LowCardinality(String),  -- 'fast' | 'queued' | 'rejected'
  weight     Int64,
  input_tokens  UInt32,
  output_tokens UInt32,
  estimated_tokens UInt32,
  queue_wait_ms UInt32,
  ttft_ms    UInt32,
  total_ms   UInt32,
  status_code UInt16,
  cache_status LowCardinality(String) DEFAULT 'off',  -- 'hit' | 'miss' | 'off'
  cost_usd   Float64 DEFAULT 0,
  energy_wh  Float64 DEFAULT 0,
  energy_cost_usd Float64 DEFAULT 0,
  co2_g      Float64 DEFAULT 0,
  ts_ms      Int64,
  session_id String DEFAULT '',
  session_id_source LowCardinality(String) DEFAULT '',  -- 'client' | 'derived' | 'none'
  request_type LowCardinality(String) DEFAULT '',
  ts         DateTime64(3) MATERIALIZED fromUnixTimestamp64Milli(ts_ms),
  INDEX idx_ts_ms ts_ms TYPE minmax GRANULARITY 4
) ENGINE = MergeTree()
PARTITION BY toYYYYMMDD(ts)
ORDER BY (tenant_id, ts_ms);

The sort key leads with tenant_id, so cross-tenant time-range reads (the live request log, the usage series) cannot use the primary index; the ts_ms minmax skip index lets them skip granules outside the requested window.

Two further tables are created alongside it:

• usage_daily — a permanent SummingMergeTree rollup of one row per day × tenant × key × model, kept current by a materialized view. usage rows are pruned on a retention window (default 180 days) to bound storage; the rollup is kept forever. Benchmark traffic (request_type = 'benchmark') is excluded from the rollup.
• spans — per-request pipeline spans written when tracing_enabled is set on the tenant or key, partitioned by day with a 14-day TTL.

The Management API's usage endpoints query these tables. Aggregates: GET /api/v1/usage, /api/v1/usage/series, /api/v1/costs. Individual rows: GET /api/v1/usage/logs (powers the dashboard Request Logs live view).

Gateway-internal traffic — health probes (request_type = 'health_probe') and traffic from tenants flagged synthetic (request_type = 'benchmark') — is excluded from usage and cost reads by default. Pass include_internal=true to include it.

Why not Postgres for usage? ClickHouse handles millions of inserts per second and analytical aggregations over billions of rows with orders of magnitude less I/O than Postgres. Shoving high-frequency time-series data into Postgres would bloat its tables, slow down VACUUM, and degrade the config API's performance.

Why not ClickHouse for config? ClickHouse doesn't support transactions, foreign keys, or the OLTP update patterns that config management requires (e.g. UPDATE tenants SET weight = $2 WHERE id = $1 RETURNING ...).

Write path discipline

All config mutations follow a single path with no shortcuts:

Management API (validated, authenticated)
  → 1. Write to Postgres (durable, audited)
  → 2. Sync to Redis (cache update)
  → 3. Publish invalidation (evict moka on all pods)

There is no "write directly to Redis" shortcut. This means Postgres is always the authoritative source and Redis is always derivable from it.