Postgres Schema

All Postgres tables used by obleth, their columns, and the migration file that introduced each.

The base schema lives in schema/postgres/0001_init.sql. Incremental migrations are applied in order:

File	What it adds
`0001_init.sql`	Full initial schema (all tables and indexes)
`0002_tracing_flag.sql`	`tracing_enabled` on `tenants` and `api_keys`
`0003_guardrails_policy.sql`	`guardrails_policy` on `tenants`
`0004_saved_recipes.sql`	`saved_recipes` table (superseded by `0006`)
`0005_managed_launcher_spec.sql`	`launcher_spec` on `managed_models`
`0006_recipes.sql`	`recipes` table; drops `saved_recipes`
`0007_replica_port_and_min_replicas.sql`	`port_base` on `model_replicas`, `min_replicas` on `managed_models`
`0008_managed_provision_error.sql`	`last_provision_error` and `last_provision_error_at` on `managed_models`
`0009_replica_cancel_requested.sql`	`cancel_requested` on `model_replicas`
`0010_drop_replica_model_cascade.sql`	Drops the `model_replicas` → `models` foreign key so replica rows outlive their model and the provisioner can still cancel their Slurm jobs
`0011_endpoint_selection_session_hash.sql`	Widens the `endpoint_selection_mode` check to accept `session_hash`
`0012_model_debug_diagnostics.sql`	`debug_diagnostics` on `models`
`0013_compression_policy.sql`	`compression_policy` on `tenants`
`0014_model_energy_slots.sql`	`energy_slots_per_node` on `models`
`0015_tenant_synthetic.sql`	`synthetic` on `tenants`

Every statement is idempotent (create … if not exists / add column if not exists) and is applied automatically when the gateway boots, so all files are safe to re-run against an existing database.

Tables

`tenants`

Stores all tenants.

Column	Type	Notes
`id`	`UUID` PRIMARY KEY
`name`	`TEXT` NOT NULL UNIQUE	Human-readable name
`weight`	`BIGINT` NOT NULL DEFAULT `100`	Fairshare weight (`>= 1`)
`tokens_per_minute`	`BIGINT` NOT NULL DEFAULT `0`	TPM token-bucket refill rate (`>= 0`); `0` means unlimited
`max_in_flight`	`BIGINT`	Optional per-tenant concurrency cap; `null` = only the global limit applies
`fairshare_group`	`TEXT` NOT NULL DEFAULT `'default'`	FK → `fairshare_groups.name`
`description`	`TEXT` NOT NULL DEFAULT `''`	Operator note
`organization`	`TEXT` NOT NULL DEFAULT `''`	Optional org metadata
`contact_email`	`TEXT` NOT NULL DEFAULT `''`	Optional contact metadata
`status`	`TEXT` NOT NULL DEFAULT `'active'`	`active`, `suspended`, or `archived` — only `active` admits traffic
`tracing_enabled`	`BOOLEAN` NOT NULL DEFAULT `false`	When `true`, every request from any key in this tenant is span-traced to the ClickHouse `spans` table (added by `0002_tracing_flag.sql`)
`timezone`	`TEXT` NOT NULL DEFAULT `'UTC'`	IANA timezone for access windows
`active_from`	`TIMESTAMPTZ`	Optional activation start; `null` = no bound
`active_until`	`TIMESTAMPTZ`	Optional expiry cutoff; `null` = no bound
`weekly_windows`	`JSONB`	Recurring windows `[{day:0-6, start_min, end_min}]`; `null`/empty = any time
`budget_tokens`	`BIGINT`	Optional cumulative token cap for the term
`budget_cost_usd`	`DOUBLE PRECISION`	Optional cumulative USD cap for the term
`budget_period`	`TEXT`	`lifetime`, `monthly`, or `term`
`budget_started_at`	`TIMESTAMPTZ`	When the current term began
`allowed_models`	`JSONB`	Optional model-name allowlist; `null`/empty = all models permitted
`guardrails_policy`	`JSONB`	Per-tenant guardrails content policy (scanners + action); `null` = none (added by `0003_guardrails_policy.sql`)
`compression_policy`	`JSONB`	Per-tenant compression policy; `null` = follow the global default (added by `0013_compression_policy.sql`)
`synthetic`	`BOOLEAN` NOT NULL DEFAULT `false`	Marks a benchmark/test tenant. Its requests are recorded with `request_type = 'benchmark'` and excluded from usage and cost stats by default (added by `0015_tenant_synthetic.sql`)
`created_at`	`TIMESTAMPTZ`	Auto-set
`updated_at`	`TIMESTAMPTZ`	Auto-updated

Indexed by fairshare_group and by status (partial, where status <> 'active').

`api_keys`

Stores API key hashes. The raw key is never stored.

Column	Type	Notes
`id`	`UUID` PRIMARY KEY
`tenant_id`	`UUID` NOT NULL	FK → `tenants.id` (`ON DELETE CASCADE`)
`name`	`TEXT` NOT NULL	Display name
`description`	`TEXT` NOT NULL DEFAULT `''`	Operator note
`key_prefix`	`TEXT` NOT NULL	First 18 chars (safe to display)
`key_hash`	`TEXT` NOT NULL UNIQUE	SHA-256 of the full secret
`budget_tokens`	`BIGINT`	Optional cumulative token cap for this key alone
`budget_cost_usd`	`DOUBLE PRECISION`	Optional cumulative USD cap for this key alone
`budget_period`	`TEXT`	`lifetime`, `monthly`, or `term`
`budget_started_at`	`TIMESTAMPTZ`	When this key's current budget term began
`disabled`	`BOOLEAN` NOT NULL DEFAULT `false`
`tracing_enabled`	`BOOLEAN` NOT NULL DEFAULT `false`	When `true`, requests using this key are span-traced to the ClickHouse `spans` table; overrides the tenant default when set per-key (added by `0002_tracing_flag.sql`)
`created_at`	`TIMESTAMPTZ`
`updated_at`	`TIMESTAMPTZ`

Indexed by tenant_id (api_keys_tenant_id_idx).

`models`

Model registry: client-facing names mapped to upstream OpenAI-compatible endpoints.

Column	Type	Notes
`id`	`UUID` PRIMARY KEY
`model_name`	`TEXT` NOT NULL UNIQUE	Name used in client requests
`description`	`TEXT` NOT NULL DEFAULT `''`	Operator-facing model description
`model_type`	`TEXT` NOT NULL DEFAULT `'chat'`	Modality: `chat`, `embedding`, `audio_transcription`, `audio_speech`, or `image`
`upstream_model`	`TEXT` NOT NULL	Model name sent to upstream
`api_base`	`TEXT` NOT NULL	Per-model upstream override (provider base ending in `/v1`)
`api_key`	`TEXT`	Injected upstream credential (encrypted at rest)
`input_cost_per_token`	`FLOAT8` NOT NULL DEFAULT `0`	For billing (chat, embedding)
`output_cost_per_token`	`FLOAT8` NOT NULL DEFAULT `0`	For billing (chat)
`cost_per_image`	`FLOAT8` NOT NULL DEFAULT `0`	Per-image billing (`image`, `× n`)
`cost_per_audio_second`	`FLOAT8` NOT NULL DEFAULT `0`	Per-audio-second billing (`audio_transcription`; reserved)
`cost_per_character`	`FLOAT8` NOT NULL DEFAULT `0`	Per-character billing (`audio_speech`)
`context_window`	`BIGINT` NOT NULL DEFAULT `8192`
`admission_weight`	`BIGINT` NOT NULL DEFAULT `100`	Scales the tenant's fairshare weight for this model (`>= 1`); `100` is neutral
`max_in_flight`	`BIGINT`	Optional per-model in-flight cap; `null` = no cap
`capacity_mode`	`TEXT` NOT NULL DEFAULT `'static'`	`static` or `tuned` — how `max_in_flight` was chosen
`capacity_tuned_at`	`TIMESTAMPTZ`	When auto-tune last applied a slot count; `null` until tuned
`supports_function_calling`	`BOOLEAN` NOT NULL DEFAULT `false`
`supports_system_messages`	`BOOLEAN` NOT NULL DEFAULT `true`
`supports_response_schema`	`BOOLEAN` NOT NULL DEFAULT `false`	Required for JSON-schema requests in `auto` routing
`supports_tool_choice`	`BOOLEAN` NOT NULL DEFAULT `false`	Required for `tool_choice` requests in `auto` routing
`supports_vision`	`BOOLEAN` NOT NULL DEFAULT `false`	Native image input. When `false`, the vision boon can relay images to a describer model instead
`tags`	`JSONB` NOT NULL DEFAULT `'[]'`	Auto-router tags (fixed vocabulary)
`boons`	`JSONB` NOT NULL DEFAULT `'[]'`	Per-model gateway boons opted into (fixed vocabulary: `vision`, `structured_output`, `compression`)
`tool_servers`	`JSONB` NOT NULL DEFAULT `'[]'`	Registered MCP servers whose tools this model may use (gateway tool loop; operator-defined names)
`enabled`	`BOOLEAN` NOT NULL DEFAULT `true`
`cache_enabled`	`BOOLEAN` NOT NULL DEFAULT `false`	Response cache toggle
`cache_ttl_secs`	`BIGINT` NOT NULL DEFAULT `300`	Cache TTL in seconds
`request_timeout_secs`	`BIGINT`	Per-request upstream timeout; `null` = use `OBLETH_UPSTREAM_TIMEOUT_SECS`
`max_retries`	`BIGINT` NOT NULL DEFAULT `0`	Extra attempts per endpoint on retryable failures
`retry_backoff_ms`	`BIGINT` NOT NULL DEFAULT `200`	Base retry backoff (exponential, capped)
`endpoint_selection_mode`	`TEXT` NOT NULL DEFAULT `'failover'`	`failover`, `load_balance`, or `session_hash` across endpoints. `session_hash` was rejected by the original check constraint until `0011_endpoint_selection_session_hash.sql` widened it
`debug_diagnostics`	`BOOLEAN` NOT NULL DEFAULT `false`	When on, a terminal `502`/`504` triggers a read-only DNS-resolve and TCP-connect probe recorded as a trace span (added by `0012_model_debug_diagnostics.sql`)
`energy_slots_per_node`	`BIGINT` NOT NULL DEFAULT `0`	Concurrent requests that saturate one node (energy accounting); `0` = opted out
`health_checks_enabled`	`BOOLEAN` DEFAULT `true`	Scheduled model health checks
`health_alerts_enabled`	`BOOLEAN` DEFAULT `true`	Slack alerts for this model
`health_check_interval_secs`	`BIGINT` DEFAULT `900`	Per-model check interval
`health_failure_threshold`	`BIGINT` DEFAULT `2`	Consecutive failures before alerting
`health_maintenance_until`	`TIMESTAMPTZ`	Suppress alerts while active
`health_maintenance_note`	`TEXT`	Operator note for maintenance
`health_status`	`TEXT` DEFAULT `unknown`	Latest status
`health_consecutive_failures`	`BIGINT` DEFAULT `0`	Current failure streak
`health_alert_state`	`TEXT` DEFAULT `ok`	Alert state (`ok` or `firing`)
`health_next_check_at`	`TIMESTAMPTZ`	Next scheduled check time
`health_last_checked_at`	`TIMESTAMPTZ`	Latest check timestamp
`health_last_latency_ms`	`BIGINT`	Latest check latency
`health_last_http_status`	`BIGINT`	Latest upstream/proxy status
`health_last_message`	`TEXT`	Sanitized latest check summary
`created_at`	`TIMESTAMPTZ`
`updated_at`	`TIMESTAMPTZ`

Indexed by enabled (partial, where enabled = true) and by health_next_check_at (partial, where the model is enabled and health checks are on) so the health worker can find due models without a full scan.

`model_health_checks`

Append-only history of model health probes.

Column	Type	Notes
`id`	`BIGSERIAL` PRIMARY KEY
`model_id`	`UUID` NOT NULL	FK → `models.id` (`ON DELETE CASCADE`)
`checked_at`	`TIMESTAMPTZ`	Probe timestamp
`trigger`	`TEXT`	`scheduled`, `manual`, or `bulk`
`status`	`TEXT`	`healthy`, `degraded`, `unhealthy`, or `skipped`
`latency_ms`	`BIGINT`	End-to-end proxy latency
`http_status`	`BIGINT`	HTTP status from the proxy/upstream
`message`	`TEXT`	Sanitized summary
`response_excerpt`	`TEXT`	Sanitized body excerpt for failures

`model_endpoints`

One row per upstream cluster that a model can be routed to. A model with no rows here uses its own api_base/api_key (legacy single-upstream path); with rows, the data plane routes across the enabled, healthy ones using the model's endpoint_selection_mode. See Reliability & Failover.

Column	Type	Notes
`id`	`UUID` PRIMARY KEY
`model_id`	`UUID` NOT NULL	FK → `models.id` (`ON DELETE CASCADE`)
`name`	`TEXT` NOT NULL	Unique per model (`UNIQUE (model_id, name)`)
`api_base`	`TEXT` NOT NULL	Upstream base ending in `/v1`
`api_key`	`TEXT`	Injected upstream credential (encrypted at rest); inherits the model key when null
`priority`	`BIGINT` NOT NULL DEFAULT `100`	Lower is tried first in `failover` mode
`weight`	`BIGINT` NOT NULL DEFAULT `100`	Traffic share in `load_balance` mode
`enabled`	`BOOLEAN` NOT NULL DEFAULT `true`	Disabled endpoints are removed from rotation
`health_status`	`TEXT` NOT NULL DEFAULT `'unknown'`	Latest per-endpoint probe status
`consecutive_failures`	`BIGINT` NOT NULL DEFAULT `0`	Current failure streak
`alert_state`	`TEXT` NOT NULL DEFAULT `'ok'`	Alert state
`last_checked_at`	`TIMESTAMPTZ`	Latest probe timestamp
`last_latency_ms`	`BIGINT`	Latest probe latency
`last_http_status`	`BIGINT`	Latest probe HTTP status
`last_message`	`TEXT`	Sanitized latest probe summary
`created_at`	`TIMESTAMPTZ`
`updated_at`	`TIMESTAMPTZ`

Indexed by model_id (model_endpoints_model_idx).

`managed_models`

Optional Slurm provisioning spec, one row per model obleth hosts on a cluster. Absent rows mean the model uses static endpoints and the data plane behaves exactly as it would without the provisioner. See Slurm Provisioning.

Column	Type	Notes
`model_id`	`UUID` PRIMARY KEY	FK → `models.id` (`ON DELETE CASCADE`)
`enabled`	`BOOLEAN` NOT NULL DEFAULT `true`	Master switch; `false` drains replicas to zero and keeps the spec
`partition`	`TEXT` NOT NULL	Slurm partition
`gres`	`TEXT` NOT NULL DEFAULT `''`	Generic resource, e.g. `gpu:h100:2`
`nodes`	`BIGINT` NOT NULL DEFAULT `1`	Nodes per replica (`>= 1`)
`constraints`	`TEXT`	Slurm `--constraint`
`exclude`	`TEXT`	Nodes or features to avoid
`account`	`TEXT`	Slurm account
`qos`	`TEXT`	Quality-of-service class
`time_limit`	`TEXT`	Slurm `--time`, e.g. `12:00:00`
`cpus_per_task`	`BIGINT`	Slurm `--cpus-per-task`; `null` = partition default
`mem`	`TEXT`	Slurm `--mem`, e.g. `560G`; `null` = partition default
`image`	`TEXT` NOT NULL	Apptainer image reference
`preamble`	`TEXT` NOT NULL DEFAULT `''`	Shell lines injected before `apptainer exec`
`log_output_dir`	`TEXT` NOT NULL DEFAULT `''`	Directory for Slurm stdout/stderr; empty = Slurm default
`launch_command`	`TEXT` NOT NULL	Command run inside the container
`script_body`	`TEXT` NOT NULL DEFAULT `''`	Fully rendered job script; when set, submitted verbatim and overrides `image`/`preamble`/`launch_command`
`serving_port`	`BIGINT` NOT NULL	Base port the inference server binds (1–65535)
`health_path`	`TEXT` NOT NULL DEFAULT `'/health'`	Health probe path
`target_replicas`	`BIGINT` NOT NULL DEFAULT `2`	Replica count the reconciler submits toward (`>= 1`)
`min_replicas`	`BIGINT` NOT NULL DEFAULT `1`	Health floor: the model is healthy at or above this many healthy replicas (added by `0007_replica_port_and_min_replicas.sql`)
`max_job_failures`	`BIGINT` NOT NULL DEFAULT `0`	Stop resubmitting after this many consecutive lost replicas; `0` = unlimited
`launcher_spec`	`JSONB`	Metadata recorded by recipe-sourced deploys so the edit view can identify them; the provisioner ignores it (added by `0005_managed_launcher_spec.sql`)
`last_provision_error`	`TEXT`	The provisioner's last submit failure, surfaced in the dashboard and cleared on a successful submit (added by `0008_managed_provision_error.sql`)
`last_provision_error_at`	`TIMESTAMPTZ`	When that failure occurred
`created_at`	`TIMESTAMPTZ`
`updated_at`	`TIMESTAMPTZ`

`model_replicas`

One row per known Slurm-backed replica of a managed model. The provisioner owns these rows.

Column	Type	Notes
`id`	`UUID` PRIMARY KEY
`model_id`	`UUID` NOT NULL	The model this replica serves. Not a foreign key: `0010_drop_replica_model_cascade.sql` dropped it so replica rows outlive a deleted model and the provisioner's drain pass can still cancel their Slurm jobs
`slurm_job_id`	`TEXT` NOT NULL	Unique per model (`model_replicas_job_uniq`), which makes replica creation idempotent
`nodes`	`TEXT`	Allocated hostnames (comma-separated)
`endpoint_id`	`UUID`	FK → `model_endpoints.id` (`ON DELETE SET NULL`); the routing entry created once the replica is healthy
`state`	`TEXT` NOT NULL DEFAULT `'pending'`	`pending`, `starting`, `healthy`, `draining`, or `lost`
`port_base`	`BIGINT`	Base of the disjoint port window assigned to this replica; the job binds the first free port in `[port_base, port_base + OBLETH_PORT_SPAN)` (added by `0007_replica_port_and_min_replicas.sql`)
`cancel_requested`	`BOOLEAN` NOT NULL DEFAULT `false`	Operator-requested restart: the provisioner cancels this replica's job regardless of target, and the resubmit launches a fresh one (added by `0009_replica_cancel_requested.sql`)
`last_message`	`TEXT`	Latest provisioner status detail
`created_at`	`TIMESTAMPTZ`
`updated_at`	`TIMESTAMPTZ`

Indexed by model_id and by state.

`recipes`

Admin-authored recipe templates for the Slurm model launcher, created and edited from the dashboard. body is the raw recipe document (YAML header plus sbatch script) — the same shape as a .recipe file on disk, stored in the database so it is editable at runtime without redeploying the control plane. Added by 0006_recipes.sql, which also drops the superseded saved_recipes table from 0004.

Column	Type	Notes
`id`	`UUID` PRIMARY KEY
`name`	`TEXT` NOT NULL	Display name
`body`	`TEXT` NOT NULL DEFAULT `''`	Raw recipe document
`author`	`TEXT` NOT NULL DEFAULT `''`	Display label of who saved it
`created_at`	`TIMESTAMPTZ`
`updated_at`	`TIMESTAMPTZ`

Indexed by updated_at desc (recipes_updated_idx).

`fairshare_groups`

Scheduler groups for hierarchical mode. A default group (weight 100) is seeded on first boot.

Column	Type	Notes
`name`	`TEXT` PRIMARY KEY	Group name
`weight`	`BIGINT` NOT NULL DEFAULT `100`	Group-level weight (`>= 1`)
`created_at`	`TIMESTAMPTZ`
`updated_at`	`TIMESTAMPTZ`

`mcp_servers`

MCP server registry. obleth reverse-proxies these through its auth + audit layer.

Column	Type	Notes
`id`	`UUID` PRIMARY KEY
`name`	`TEXT` NOT NULL UNIQUE	Reached at `/mcp/{name}`
`upstream_url`	`TEXT` NOT NULL	SSRF-validated upstream base
`auth_header`	`TEXT`	Injected auth header value (encrypted at rest)
`enabled`	`BOOLEAN` DEFAULT `true`
`created_at`	`TIMESTAMPTZ`
`updated_at`	`TIMESTAMPTZ`

`audit_log`

Append-only record of all Management API mutations.

Column	Type	Notes
`id`	`BIGSERIAL` PRIMARY KEY
`ts`	`TIMESTAMPTZ` NOT NULL DEFAULT `now()`	When the action occurred
`actor`	`TEXT` NOT NULL	Who performed it (e.g. `admin`)
`action`	`TEXT` NOT NULL	e.g. `create_tenant`, `delete_key`
`entity_type`	`TEXT` NOT NULL	e.g. `tenant`, `api_key`
`entity_id`	`TEXT` NOT NULL	ID of the affected entity
`detail`	`JSONB` NOT NULL DEFAULT `'{}'`	Structured detail payload

Indexed by ts desc (audit_log_ts_idx), by actor, ts desc (audit_log_actor_idx), and by entity_type, entity_id, ts desc (audit_log_entity_idx) so filtering the trail by who acted or which entity was touched stays fast as the log grows.

`app_settings`

Key-value store for runtime-reloadable gateway settings. Holds the alerting configuration (Slack webhook + SMTP email) under key alerts, the auto_router classifier settings, the usage-retention window, the model-boon settings (boons), the Slurm provisioner connection (slurm), the energy accounting configuration (energy), and the Charo assistant settings (charo_settings). New fields are stored as JSON so they need no schema change.

Column	Type	Notes
`key`	`TEXT` PRIMARY KEY	Setting name (e.g. `alerts`, `auto_router`)
`value`	`JSONB` NOT NULL	Setting payload
`updated_at`	`TIMESTAMPTZ` NOT NULL DEFAULT `now()`	Auto-updated

Applying the schema

The gateway applies all schema files under schema/postgres/ in lexicographic order on boot — there is no per-version migration table to query. All statements are idempotent, so re-running against an existing database is safe. To inspect the live schema, read it from Postgres directly:

-- List obleth's tables
\dt

-- Describe a table
\d+ tenants

PreviousClickHouse Usage

NextRedis Layout

Getting Started

Concepts

Guides

Reference

Operations