Self-Hosting

Run obleth on your own infrastructure: storage scenarios for the Helm chart and how to host Postgres, Redis, and ClickHouse yourself in Docker.

obleth is built to be self-hosted. This guide covers the decisions that matter once you move past a throwaway demo: where your data lives and who manages the datastores.

obleth needs three backing stores (see Datastores):

Store	Role	Lose it and…
Postgres	Config source of truth (tenants, keys, models, audit)	you lose all configuration
ClickHouse	Usage / cost ledger	you lose usage history
Redis	Hot cache + token budgets	nothing permanent — obleth re-warms from Postgres

How you host those three stores is the single biggest self-hosting decision. The Helm chart in deploy/k8s/obleth/ supports four scenarios.

Choosing a scenario

Scenario	Who manages the data	Survives restart?	Use for
Persistent (PVC)	the chart, on PVCs	Yes	Self-hosted single cluster, no external DBs yet
Ephemeral	the chart, in memory	No — data is lost	Demos, CI, kicking the tires
External	you (managed / Docker / operator)	Managed by you	Self-host with your own managed datastores
Production	you (operator / managed)	Managed by you	Redundant production: external datastores + full hardening

The profiles differ along two independent axes — datastore durability (where state lives) and workload posture (how the stateless obleth pods are hardened and made redundant). External and Production share the same datastore story; Production layers on pre-created Secrets, pod spread, disruption budgets, and restricted security contexts.

Ready-made values files for each live in deploy/k8s/obleth/examples/.

Scenario 1 — bundled datastores with PVCs

The chart runs its own Postgres, Redis, and ClickHouse and stores their data on PersistentVolumeClaims. Data survives pod restarts, reschedules, and node drains. This is the recommended starting point when you don't yet have managed datastores.

Each datastore has a persistence block (enabled by default):

postgres:
  enabled: true
  persistence:
    enabled: true
    size: 10Gi
    storageClass: ""          # "" = cluster default StorageClass; set a name to pin one
    accessMode: ReadWriteOnce
redis:
  persistence: { enabled: true, size: 1Gi }
clickhouse:
  persistence: { enabled: true, size: 20Gi }

The datastore Deployments use the Recreate rollout strategy so a new pod never tries to attach the same ReadWriteOnce PVC while the old pod still holds it.

Requirements: a StorageClass must exist in the cluster. Check with:

kubectl get storageclass

Most managed clusters (EKS/GKE/AKS), k3s, and kind ship a default. If none is marked (default), set persistence.storageClass explicitly.

Install:

helm install obleth deploy/k8s/obleth -n obleth --create-namespace \
  -f deploy/k8s/obleth/examples/values-persistent.yaml \
  --set obleth.adminToken="$(openssl rand -hex 32)" \
  --set obleth.encryptionKey="$(openssl rand -base64 32)" \
  --set postgres.password="$(openssl rand -hex 16)" \
  --set clickhouse.password="$(openssl rand -hex 16)" \
  --set controlPlane.dashboardPassword="$(openssl rand -hex 16)" \
  --set controlPlane.dashboardSessionSecret="$(openssl rand -hex 32)"

Note:

Single-replica Deployments with a PVC are fine for self-hosting, but they are not highly available — during a node failure the datastore is down until the pod reschedules. For HA, move to the External scenario with an operator (CloudNativePG, the ClickHouse operator, Redis Sentinel/cluster).

Scenario 2 — bundled datastores without persistence (test only)

Set persistence.enabled: false and the datastores use emptyDir volumes.

Note:

All data is lost when a datastore pod restarts. That includes every tenant, API key, registered model, audit entry (Postgres) and all usage history (ClickHouse). A helm upgrade that rolls the pods, a node drain, or an OOM kill all wipe the data. Use this only for throwaway demos, CI, or a cluster with no StorageClass.

postgres:   { persistence: { enabled: false } }
redis:      { persistence: { enabled: false } }
clickhouse: { persistence: { enabled: false } }

helm install obleth deploy/k8s/obleth -n obleth --create-namespace \
  -f deploy/k8s/obleth/examples/values-ephemeral.yaml \
  --set obleth.adminToken="$(openssl rand -hex 32)" \
  --set postgres.password=devpassword \
  --set clickhouse.password=devpassword \
  --set controlPlane.dashboardPassword=devpassword \
  --set controlPlane.dashboardSessionSecret=dev-session-secret-at-least-32-chars

This profile keeps the GPU-free benchmark fixture backend enabled, so inference calls succeed without a real upstream.

Scenario 3 — external datastores

obleth runs only its stateless workloads (data plane + control plane) and connects to Postgres, Redis, and ClickHouse you operate yourself. This is the recommended production topology — stateful systems get purpose-built backups, HA, and point-in-time recovery instead of plain Deployments.

postgres:
  enabled: false
  external:
    url: "postgres://obleth:STRONGPASSWORD@my-pg-host:5432/obleth"
redis:
  enabled: false
  external:
    url: "redis://my-redis-host:6379"
clickhouse:
  enabled: false
  # obleth authenticates to ClickHouse with these even when external:
  user: obleth
  password: "STRONGPASSWORD"
  db: obleth
  external:
    url: "http://my-clickhouse-host:8123"
benchmarkBackend:
  enabled: false
obleth:
  upstreamBaseUrl: "https://my-vllm-or-aibrix-endpoint/v1"

When a store is enabled: false, its external.url is required — the chart fails to render with a clear message if it is missing, so obleth never boots half-configured.

helm install obleth deploy/k8s/obleth -n obleth --create-namespace \
  -f deploy/k8s/obleth/examples/values-external.yaml \
  --set obleth.adminToken="$(openssl rand -hex 32)" \
  --set obleth.encryptionKey="$(openssl rand -base64 32)"

Where to get those external stores:

Managed cloud — RDS/Cloud SQL (Postgres), ElastiCache/MemoryStore (Redis), ClickHouse Cloud / Altinity.
Kubernetes operators — CloudNativePG for Postgres, the ClickHouse operator, Redis Sentinel/cluster charts.
Docker — the standalone stack below.

Scenario 4 — production

Production is the External datastore topology plus full hardening of the stateless data plane. Nothing stateful runs in the chart; the difference is how the obleth pods are secured and kept redundant. The ready-made values-production.yaml wires all of this together.

1. Pre-create the Secrets

So real credentials never touch values files, git, or --set/CLI history, production references pre-created Secrets via existingSecret instead of letting the chart render its own. Use your secrets manager / SOPS / External Secrets Operator in place of raw kubectl below:

kubectl create secret generic obleth-secrets -n obleth \
  --from-literal=OBLETH_ADMIN_TOKEN="$(openssl rand -hex 32)" \
  --from-literal=OBLETH_DATABASE_URL="postgres://obleth:PASS@pg-rw.db.svc:5432/obleth" \
  --from-literal=OBLETH_CLICKHOUSE_PASSWORD="CLICKHOUSE_PASS" \
  --from-literal=OBLETH_ENCRYPTION_KEY="$(openssl rand -base64 32)" \
  --from-literal=OBLETH_API_KEY_PEPPER="$(openssl rand -hex 32)" \
  --from-literal=OBLETH_SLACK_WEBHOOK_URL=""

kubectl create secret generic obleth-control-plane -n obleth \
  --from-literal=DASHBOARD_ADMIN_EMAIL="admin@your-org.example" \
  --from-literal=DASHBOARD_PASSWORD="$(openssl rand -hex 16)" \
  --from-literal=DASHBOARD_SESSION_SECRET="$(openssl rand -hex 32)" \
  --from-literal=DATABASE_URL="postgres://obleth:PASS@pg-rw.db.svc:5432/obleth" \
  --from-literal=BETTER_AUTH_URL="https://dashboard.example.com" \
  --from-literal=OIDC_PROVIDERS=""   # JSON array to enable SSO; empty = break-glass admin only

When existingSecret is set the chart renders no Secret of its own, so the named Secret must carry every key listed above. See Helm Values — Pre-created Secrets.

2. Hardening the values turn on

obleth:
  replicas: 3
  existingSecret: obleth-secrets        # no inline secrets
  upstreamBaseUrl: "https://my-vllm-or-aibrix-endpoint/v1"

postgres:   { enabled: false, external: { url: "postgres://obleth:PASS@pg-rw.db.svc:5432/obleth" } }
redis:      { enabled: false, external: { url: "redis://redis-master.db.svc:6379" } }
clickhouse: { enabled: false, user: obleth, db: obleth, external: { url: "http://clickhouse.db.svc:8123" } }
benchmarkBackend: { enabled: false }

hpa:
  enabled: true
  minReplicas: 3
  maxReplicas: 20

affinity:
  antiAffinity: soft                    # "hard" to require distinct nodes on multi-node clusters

podDisruptionBudget:
  enabled: true
  minAvailable: 2

controlPlane:
  enabled: true
  replicas: 2
  existingSecret: obleth-control-plane

serviceMonitor:
  enabled: true                         # requires the Prometheus Operator CRDs

ingress:
  enabled: true
  className: nginx
  host: obleth.example.com
  servicePort: 8080
  tls:
    - secretName: obleth-tls
      hosts: [obleth.example.com]

On top of these, the chart applies restricted Pod Security Standard security contexts (runAsNonRoot, seccompProfile: RuntimeDefault, no privilege escalation, all capabilities dropped) to the stateless workloads by default — nothing to set. networkPolicy.enabled: true is available to fence the bundled datastore ports, but it's a no-op here since the datastores are external. See Helm Values for each knob.

3. Install

helm install obleth ./deploy/k8s/obleth -n obleth --create-namespace \
  -f deploy/k8s/obleth/examples/values-production.yaml

No --set for secrets — they all come from the pre-created Secrets.

Note:

Point the datastores at purpose-built tooling for backups, HA, and PITR: CloudNativePG or managed Postgres, the Altinity ClickHouse operator or managed ClickHouse, and an HA Redis (operator/managed). obleth treats Redis as a hot cache and re-warms from Postgres, but HA avoids cold-start latency spikes.

Hosting external datastores in Docker

If you want external datastores without a managed service or a database operator, the repo ships a dedicated Compose stack tuned to be a durable backing tier (named volumes, healthchecks, restart: unless-stopped, host-published ports): deploy/docker/datastores.compose.yml.

This is separate from the all-in-one docker-compose.yml — it runs only the three datastores so that an obleth elsewhere (a Helm install with *.enabled: false, or a bare-metal binary) can connect to them.

1. Start the stack

cd deploy/docker
cp datastores.env.example datastores.env   # then edit the passwords
docker compose --env-file datastores.env -f datastores.compose.yml up -d

datastores.env controls credentials and the bind address:

# Loopback only by default — change to a private interface IP to allow an
# off-box obleth, and protect it with a firewall / private network.
BIND_ADDR=127.0.0.1

POSTGRES_USER=obleth
POSTGRES_PASSWORD=change-me-postgres
POSTGRES_DB=obleth

CLICKHOUSE_USER=obleth
CLICKHOUSE_PASSWORD=change-me-clickhouse
CLICKHOUSE_DB=obleth

Service	Port	Notes
Postgres	5432	config source of truth
Redis	6379	AOF persistence (`appendfsync everysec`)
ClickHouse	8123 (HTTP), 9000 (native)	obleth uses the HTTP port

Data persists in the pgdata, redisdata, and chdata named volumes across docker compose down/up. (docker compose down -v deletes them.)

Note:

ClickHouse's built-in default user is restricted to localhost, so the stack provisions a network-accessible obleth user via CLICKHOUSE_USER / CLICKHOUSE_PASSWORD. Use those credentials from obleth, not default.

2. Point obleth at them

Bare-metal binary (OBLETH_* env vars). Replace DB_HOST with the Docker host's reachable address:

export OBLETH_DATABASE_URL="postgres://obleth:change-me-postgres@DB_HOST:5432/obleth"
export OBLETH_REDIS_URL="redis://DB_HOST:6379"
export OBLETH_CLICKHOUSE_URL="http://DB_HOST:8123"
export OBLETH_CLICKHOUSE_USER=obleth
export OBLETH_CLICKHOUSE_PASSWORD=change-me-clickhouse
export OBLETH_CLICKHOUSE_DB=obleth
export OBLETH_ADMIN_TOKEN="$(openssl rand -hex 32)"
./target/release/obleth-proxy

Helm (values-external.yaml):

postgres:
  enabled: false
  external: { url: "postgres://obleth:change-me-postgres@DB_HOST:5432/obleth" }
redis:
  enabled: false
  external: { url: "redis://DB_HOST:6379" }
clickhouse:
  enabled: false
  user: obleth
  password: "change-me-clickhouse"
  db: obleth
  external: { url: "http://DB_HOST:8123" }

obleth applies its embedded schema on first boot — there is no separate migration step. It connects, creates tables idempotently, and starts serving.

Note:

The datastore ports bind to 127.0.0.1 by default so they are not exposed to your network. Only widen BIND_ADDR to a private interface and put the ports behind a firewall / private network. Never expose Postgres, Redis, or ClickHouse to the public internet.

Upgrading

To apply chart or values changes to a release that already exists, use helm upgrade --install — not helm install, which fails against a live release with cannot re-use a name that is still in use.

helm upgrade --install obleth ./deploy/k8s/obleth -n obleth \
  -f my-values.yaml \
  --wait

Point the upgrade at the same chart source you installed from. The command above uses the local chart path; to track a published release instead, use the OCI chart pinned to a version:

helm upgrade --install obleth oci://ghcr.io/thediymaker/charts/obleth \
  --version 0.7.2 -n obleth \
  -f my-values.yaml \
  --wait

Note:

The two sources are not interchangeable. Editing the chart's own deploy/k8s/obleth/values.yaml only affects the local chart path — the OCI artifact carries the values published with that version. Keep per-deployment settings in your own -f values file so they apply regardless of source.

Note:

Values that flow into the obleth Secret — obleth.adminToken, the database URL, clickhouse.password, obleth.encryptionKey, obleth.apiKeyPepper — update the Secret on upgrade but do not restart running pods by themselves, so the containers keep the old values until they next restart. After changing one, force a rollout:

kubectl rollout restart deployment/obleth -n obleth
# and the control plane if its DATABASE_URL changed:
kubectl rollout restart deployment/obleth-control-plane -n obleth

After install

Helm and the binary bring services up but do not register models or create tenant keys. Continue with:

First Tenant & Key — mint a sk_... key
Installation — post-install steps
Backup & Restore — protect Postgres and ClickHouse
Production Checklist — secrets, TLS, scaling
Helm Values — full chart reference

PreviousModular Deploy

NextSlurm Provisioning

Getting Started

Concepts

Guides

Reference

Operations

Self-Hosting

Choosing a scenario

Scenario 1 — bundled datastores with PVCs

Scenario 2 — bundled datastores without persistence (test only)

Scenario 3 — external datastores

Scenario 4 — production

1. Pre-create the Secrets

2. Hardening the values turn on

3. Install

Hosting external datastores in Docker

1. Start the stack

2. Point obleth at them

Upgrading

After install