Upgrades

Overview

obleth upgrades involve three components:

1. Gateway binary (obleth-proxy, obleth-admin, etc.)
2. Control plane (Next.js dashboard)
3. Database schema (Postgres migrations run automatically on startup)

The typical upgrade order is: upgrade the gateway pods (rolling), then upgrade the control plane.

Schema migrations

obleth uses embedded migrations (in the obleth-config crate) that run automatically at startup via SQLx. Migrations are idempotent and append-only — they never drop columns or alter existing data.

Because migrations run on startup, a rolling deploy means:

1. One pod with the new schema runs alongside pods with the old schema.
2. New migrations add nullable columns or new tables — never rename or drop.
3. Old pods continue running against the updated schema (they ignore new columns they don't know about).

You do not need to run migrations manually. They run on the first pod that starts with the new version.

Rolling upgrade on Kubernetes

# Update the image tag
kubectl set image deployment/obleth \
  obleth=ghcr.io/obleth-ai/obleth-gateway:v1.2.0

# Watch the rollout
kubectl rollout status deployment/obleth

# If something goes wrong
kubectl rollout undo deployment/obleth

The default RollingUpdate strategy (1 pod at a time) ensures continuous availability. With 3 replicas:

• Pod 1 starts with new version; migrations run
• Pod 1 passes health check; Pod 2 cycles
• Pod 3 cycles

Upgrading with Helm

helm upgrade obleth ./deploy/helm/obleth \
  --set image.obleth=ghcr.io/obleth-ai/obleth-gateway:v1.2.0 \
  --namespace obleth \
  --wait

--wait blocks until the rollout completes successfully. If the new pods fail health checks, Helm reports the error and does not proceed.

Upgrading the control plane

The control plane is a separate deployment and can be upgraded independently:

kubectl set image deployment/obleth-control-plane \
  control-plane=ghcr.io/obleth-ai/obleth-control-plane:v1.2.0

kubectl rollout status deployment/obleth-control-plane

The control plane is stateless (all state is in the Management API). A brief downtime during control plane upgrade is acceptable.

Docker Compose upgrades

# Pull the new images
docker compose -f deploy/docker/docker-compose.yml pull

# Recreate containers (brief downtime)
docker compose -f deploy/docker/docker-compose.yml up -d --force-recreate obleth

Docker Compose does not support zero-downtime rolling upgrades natively. For production, use Kubernetes.

Post-upgrade verification

# Check all pods are running the new version
kubectl get pods -l app=obleth -o jsonpath='{.items[*].spec.containers[0].image}'

# Health check
curl http://localhost:9090/api/v1/health

# Verify schema version (check latest migration applied)
psql $DATABASE_URL -c "SELECT filename FROM _sqlx_migrations ORDER BY installed_on DESC LIMIT 5;"

Rollback

# Kubernetes
kubectl rollout undo deployment/obleth

# Helm
helm rollback obleth 1  # Roll back to revision 1

Note: schema migrations are not rolled back automatically. The old binary will run against the new schema. Since migrations are additive (new nullable columns only), this is safe — old code ignores unknown columns.