Backup & Restore

obleth has two complementary backup strategies:

Config backup

The Settings → Config backup card in the dashboard lets you export every piece of gateway configuration to a single JSON file and restore it onto any obleth instance. Usage history (audit log, request ledger, ClickHouse data) is never included.

What is included

Provider secrets (api_key on models/endpoints, auth_header on MCP servers, alert credentials) are exported exactly as stored. On an instance with OBLETH_ENCRYPTION_KEY set that means AES-256-GCM ciphertext, and restoring onto a different instance requires the same key. On an instance without a key set, those columns are plaintext in the database and therefore plaintext in the backup file — treat the file as a credential in that case.

Exporting a backup

In the dashboard go to Settings → Config backup and click Download backup. The file is named obleth-backup-<timestamp>.json.

Via the Management API directly:

curl -H "Authorization: Bearer $TOKEN" \
  http://localhost:9180/api/v1/backup/export \
  -o obleth-backup-$(date +%Y%m%d-%H%M%S).json

Restoring a backup

From the dashboard

1. Go to Settings → Config backup and click Restore from backup…
2. Pick the .json file. The dashboard shows a preview: how many tenants, keys, models, etc. are in the file.
3. Tick the checkbox and type RESTORE to confirm.
4. A report shows how many entities were added vs updated per category.

From the API

curl -X POST http://localhost:9180/api/v1/backup/restore \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d @obleth-backup-20260101-120000.json

The response is a RestoreReport:

{
  "fairshare_groups": { "inserted": 1, "updated": 0 },
  "tenants":          { "inserted": 5, "updated": 2 },
  "api_keys":         { "inserted": 12, "updated": 0 },
  "models":           { "inserted": 8, "updated": 1 },
  "model_endpoints":  { "inserted": 10, "updated": 1 },
  "mcp_servers":      { "inserted": 2, "updated": 0 },
  "app_settings":     { "inserted": 0, "updated": 6 },
  "warnings":         []
}

Merge semantics

Restore is always a merge. Entities present in the backup are created or updated by their primary key; anything that exists only on the target instance is left untouched. Restore never deletes. The entire operation applies atomically — all or nothing.

After the write, obleth re-syncs the Redis hot caches (keys, models, MCP servers) and reloads alert settings live, so the data plane picks up the restored configuration without waiting for a refresh or a restart.

Validation before any write

Three checks run before the first row is written, and each rejects the whole restore:

1. Format and version. The file must be an obleth config backup at the version this gateway supports.
2. Encryption key. The key_check sentinel must decrypt (see below).
3. Destination policy. Every api_base on a model or endpoint and every upstream_url on an MCP server is validated against the same SSRF policy the create/update forms enforce. A backup pointing any of them at a blocked address — link-local or cloud metadata — fails with 400 naming the offending entry, because restored rows are dispatched to by the data plane without any further check. See Security — SSRF.

Encryption and key compatibility

The backup file embeds a key_check sentinel: a known plaintext encrypted with the exporting instance's OBLETH_ENCRYPTION_KEY. On restore, obleth decrypts it first. If the keys don't match the restore is rejected with a 400 before any database write. The sentinel also detects the case where the backup was made with encryption enabled but the target instance has no key configured.

A backup exported with the cipher disabled carries no sentinel. In that case the restore still refuses to proceed if the file contains enc:v1: ciphertext while the target has no key — secrets it could never decrypt.

API key pepper

If the source instance used OBLETH_API_KEY_PEPPER when hashing keys, the target must use the same pepper or those keys will not authenticate after restore. obleth cannot detect a pepper mismatch from the stored hashes alone — the backup records only whether a pepper was set. When that flag disagrees with the target and the backup contains API keys, the restore still applies and the report carries a warning that the restored keys will not authenticate until the pepper matches or the keys are rotated. Two instances that both use a pepper, but different ones, cannot be detected at all.

Audit

Both endpoints record an entry in the audit log: export_backup and restore_backup with entity counts (never the backup payload, which contains ciphertext and key hashes).

Datastore backup

Infrastructure-level backup of all three datastores. Use this for complete disaster recovery, point-in-time restore, or when you need usage history alongside config.

Postgres

Postgres is the source of truth for all configuration. Back it up like any production Postgres database.

pg_dump (simple)

pg_dump -h localhost -U obleth -d obleth -F c -f obleth-$(date +%Y%m%d).dump

Continuous WAL archiving

For production, configure WAL archiving with pgBackRest or Barman to get point-in-time recovery (PITR). With CloudNativePG:

spec:
  backup:
    barmanObjectStore:
      destinationPath: "s3://my-bucket/obleth-pg/"
      s3Credentials:
        accessKeyId:
          name: pg-backup-creds
          key: ACCESS_KEY_ID

Restore

pg_restore -h localhost -U obleth -d obleth obleth-20240101.dump

After restoring Postgres, restart obleth pods to reload the cache from the restored database.

Redis

Redis is a hot cache. All Redis data can be reconstructed from Postgres on startup (obleth warms the cache on first use). Redis backup is recommended but not strictly required for data safety.

Enable Redis persistence

In the Redis configuration (or via Docker Compose environment):

appendonly yes
appendfsync everysec

This writes an AOF (append-only file) that can be replayed on restart. For Docker Compose:

redis:
  command: redis-server --appendonly yes --appendfsync everysec

Restore

Simply restore the AOF or RDB file and start Redis. obleth will resume using the warm cache.

If Redis data is lost entirely, nothing is permanently lost — every cached value is derivable from Postgres — but it is not rebuilt per request: the data plane reads only moka and Redis, never Postgres. Redis is repopulated from Postgres when the gateway starts, and rewritten on every config write. So after an empty Redis, restart the gateway pods (or perform any configuration change) to re-sync keys, models, and MCP servers. Live token-bucket state is genuinely lost and simply starts fresh.

ClickHouse

ClickHouse holds the usage ledger. It is append-only and does not need to be consistent with real-time traffic (the WAL handles in-flight records during an outage).

Data retention

Configure a TTL on usage to automatically drop old data:

ALTER TABLE obleth.usage
MODIFY TTL toDateTime(ts_ms / 1000) + INTERVAL 90 DAY;

Backup

For managed ClickHouse (ClickHouse Cloud, Altinity), use the provider's backup feature. For self-hosted:

clickhouse-backup create obleth-backup-$(date +%Y%m%d)
clickhouse-backup upload obleth-backup-$(date +%Y%m%d) --remote-storage=s3

Using clickhouse-backup.

Restore

clickhouse-backup download obleth-backup-20240101
clickhouse-backup restore obleth-backup-20240101

After a ClickHouse outage without backup

If ClickHouse data is lost and no backup exists, usage history is gone. Current tenants, keys, and config are safe in Postgres. Billing/audit reconstruction requires replaying the WAL files from all obleth pods during the outage window.

Disaster recovery summary