Backup and restore
Copy a gateway's configuration to a file and restore it, and back up the datastores for full disaster recovery.
4 min readUpdated for v2.4.0Edit on GitHub
OBLETH_ENCRYPTION_KEYA config backup is one JSON file with the gateway's configuration. Use it to clone an instance, move staging to production, or recover settings. It holds no usage history; for that, and for everything a config backup leaves out, back up the datastores.
Download a backup
In Settings › Data and backups, select Download backup. The file is named obleth-backup-<timestamp>.json. Or use the API:
curl -H "Authorization: Bearer $OBLETH_ADMIN_TOKEN" \
http://localhost:9180/api/v1/backup/export \
-o obleth-backup-$(date +%Y%m%d-%H%M%S).json
Treat the file as a credential. Upstream API keys, MCP auth headers, upstream header values, the Slurm JWT and the dashboard's own key are exported as stored: encrypted when OBLETH_ENCRYPTION_KEY is set, plaintext when it is not. The Slack webhook and SMTP password are always exported as stored.
Restore a backup
- 1
Choose the file
In Settings › Data and backups, select Restore from backup… and pick the file. The dialog shows when it was taken, by which gateway version, and how many of each thing it holds.
- 2
Confirm
Tick the checkbox, type
RESTORE, and select Restore backup. - 3
Read the report
It lists how many of each were added and updated, and any warnings.
Through the API, post the file to POST /api/v1/backup/restore; the response is the same report as JSON, with inserted and updated counts per section and a warnings list.
Restore is a merge: everything in the file is created or updated by id, and nothing that exists only on the target is deleted. It is applied in one transaction, all or nothing. Afterwards the gateway rebuilds its Redis caches and reloads alert settings, so the data plane serves the restored configuration at once. Existing client keys keep working, because their hashes are in the file. Both export and restore are recorded in the audit log, with counts but never the payload.
What a backup contains
| Section | Contents |
|---|---|
fairshare_groups | Name and weight |
tenants | Profile, group and weight, token rate, in-flight cap, status, time zone, access window and weekly hours, budget, model allowlist |
api_keys | Prefix and hash, kind and identity fields, budget, weight, in-flight cap, per-user fairshare, disabled flag |
models | The whole route: names and aliases, upstream and api_base, key and upstream headers, type, prices, context window, capacity (including discovered-capacity settings), capabilities, cache, tags, boons, tool servers, routing, speculation, timeouts and retries, endpoint selection, health-check settings |
model_endpoints | Name, api_base, key, priority, weight, enabled flag |
mcp_servers | Name, URL, auth header, enabled flag |
app_settings | Every settings row: alerts, routing, boons, energy, Slurm, assistant, retention, and the dashboard's own key |
Each model carries its variants and lifecycle status; a model in a backup taken before v2.1.0 restores as active.
Not included: the audit log and all of ClickHouse; model health history and state; knowledge collections and documents; recipes, Slurm deployments and replicas; video job records; tenant guardrails and compression policies; the synthetic flag and tracing flags. Recreate those on the target, or restore the datastores instead.
Back up the datastores
| Datastore | Holds | Back it up with |
|---|---|---|
| Postgres | All configuration, keys, tenants, knowledge, the audit log, the dashboard's accounts | pg_dump, or continuous WAL archiving (CloudNativePG, pgBackRest, Barman) for point-in-time recovery |
| Redis | Caches rebuilt from Postgres; per-minute buckets, term-budget usage, the response cache | AOF persistence (on in the bundled Redis) |
| ClickHouse | The usage ledger | Your provider's backups, or clickhouse-backup |
pg_dump -h $PGHOST -U obleth -d obleth -F c -f obleth-$(date +%Y%m%d).dump
pg_restore -h $PGHOST -U obleth -d obleth obleth-20260101.dump
After restoring Postgres or Redis, run Reconcile cache… in Settings › Data and backups, or POST /api/v1/resync, so Redis matches Postgres. Without it, the next rewarm (OBLETH_REDIS_REWARM_SECS, 300 s) gets there.
Retention is the gateway's job, not ClickHouse's: it drops raw usage days older than OBLETH_USAGE_RETENTION_DAYS (180, or Keep request history for in Settings › Data and backups) and never prunes the usage_daily rollup. Don't add your own TTL to usage.
What you lose without one
| Lost | Result | Recovery |
|---|---|---|
| Redis | Caches empty; per-minute buckets, term-budget usage and the response cache reset | Automatic within one rewarm, or at once with a resync |
| Postgres, with a config backup | Knowledge, recipes, Slurm deployments and the other exclusions above | Restore the config backup, then recreate the rest |
| Postgres, no backup | Every tenant, key and model | None |
| ClickHouse, no backup | Usage history and billing data | Only rows still in gateway spill files replay |
| A gateway pod | Its unreplayed spill file, which is per pod | None |
Checks before a restore writes anything
Three checks run first, and any failure rejects the whole restore:
- Format and version. The file must be an obleth config backup of a version this gateway reads.
- Encryption key. The file carries a
key_checkvalue encrypted with the exporter's key. A different key, or no key on the target for an encrypted backup, is refused with400. Two instances without keys restore in plaintext; an unencrypted backup restored onto an instance with a key is encrypted on write. - Destination policy. Every outbound URL the restore would register (model and endpoint
api_base, verify URLs, MCP URLs, and the URLs inside settings such as the Slack webhook) is checked against the same policy as the create forms. See Security.
A pepper mismatch cannot be detected from the hashes. The file records whether the exporter used OBLETH_API_KEY_PEPPER; if that differs from the target, the restore applies and warns that restored keys won't authenticate until the peppers match.