Upgrades

obleth releases are published in lockstep to GitHub Container Registry (GHCR). A version-matched upgrade is normally just a chart version bump — the chart pins its own matching images, so you do not set image tags by hand.

Published artifacts

The Slurm provisioner ships as its own image and is off by default (provisioner.enabled). When it is enabled, it advances with the chart like the rest — its running build is shown on Settings → Slurm, so a stale provisioner deployment is visible.

Tag convention. The chart version has no prefix (0.9.6); the images are v-prefixed (v0.9.6). Same release, two forms — don't mix them up.

Standard upgrade (recommended)

Pull the published chart and carry your existing release values (secrets, datastore URLs, registry overrides, etc.) forward so only the version changes:

helm upgrade obleth \
  oci://ghcr.io/thediymaker/charts/obleth \
  --version 0.9.6 \
  --namespace obleth-system \
  --reset-then-reuse-values \
  --wait

• --reset-then-reuse-values resets to the new chart's defaults, then re-applies only the overrides you explicitly set on the previous release. This is what you want for a version bump: chart-managed defaults like the image tag advance to the new release automatically, while your secrets and overrides are preserved.
• --wait blocks until the new pods pass their health checks and reports failure (without serving traffic from a broken pod) if they don't.

Why not plain --reuse-values? --reuse-values reuses the fully computed values from the last release and ignores the new chart's defaults. Because this chart pins image.tag to its own appVersion, a plain --reuse-values upgrade leaves the old image tag in place — you'd be running the previous binary against the new chart templates. Use --reset-then-reuse-values (Helm 3.14+) for version bumps; reserve plain --reuse-values for re-applying the same chart version. If you're on older Helm, force the tag explicitly with --reuse-values --set image.tag=v0.9.6.

Kubernetes cluster unreachable on a non-standard distro. If Helm fails with Error: UPGRADE FAILED: Kubernetes cluster unreachable: Get "http://localhost:8080/version": dial tcp [::1]:8080: connect: connection refused, it could not find a kubeconfig and fell back to the default localhost:8080 API address. Distros like k0s/k3s keep their admin kubeconfig outside the default path — prefix the same command with KUBECONFIG pointing at it:

KUBECONFIG=/var/lib/k0s/pki/admin.conf helm upgrade obleth \
  oci://ghcr.io/thediymaker/charts/obleth \
  --version 0.9.6 \
  --namespace obleth-system \
  --reset-then-reuse-values \
  --wait

(k3s commonly uses /etc/rancher/k3s/k3s.yaml.)

--set + value reuse caveat. This chart marks obleth.adminToken, datastore passwords, and dashboard secrets as required. A bare helm upgrade ... --set foo=bar (without a reuse flag or -f) resets to chart defaults and fails with obleth.adminToken is required. Always pass --reset-then-reuse-values (or your -f values-prod.yaml) when using --set on an existing release.

Preview first

helm upgrade obleth oci://ghcr.io/thediymaker/charts/obleth --version 0.9.6 \
  -n obleth-system --reset-then-reuse-values --dry-run | grep 'image:'

Confirm the rendered images are the expected …:v0.9.6 before applying.

Schema migrations

Migrations are automatic and idempotent. The numbered files under schema/postgres/ are embedded in the gateway binary and applied in order on every startup (create … if not exists, add column … if not exists), serialized across replicas by a Postgres advisory lock so concurrent boots cannot race. Changes are additive (new nullable columns / tables only), so old and new pods coexist safely during the rolling update. Nothing to run by hand.

The ClickHouse schema is applied the same way: tables and columns are created if absent on each boot, including the permanent usage_daily rollup and its materialized view.

Upgrade notes worth checking

• Gateway replicas and the concurrency ceiling. The chart now defaults to a single gateway replica (obleth.replicas: 1) with the HPA off, because globalMaxInFlight is enforced per replica. If your release pins a higher replica count or enables the HPA, divide globalMaxInFlight accordingly. See Scaling.
• OBLETH_PROVISIONER_RESTART_AFTER_FAILURES. The default is now 20 (~5 minutes of sustained probe failure at the default tick interval). Earlier deployment defaults pinned it to 3, which cancelled healthy replicas after about 45 seconds of probe flapping. If you set it to 3 yourself, remove or raise it when upgrading.

Verify

helm history obleth -n obleth-system | tail -2

kubectl -n obleth-system get deploy -o \
  'jsonpath={range .items[*]}{.metadata.name}{"\t"}{.spec.template.spec.containers[0].image}{"\n"}{end}'

kubectl -n obleth-system get pods

The gateway log should end with postgres connected + schema applied, redis connected, clickhouse connected + schema applied, and listening: proxy=…:

kubectl -n obleth-system logs deploy/obleth-obleth -c obleth --tail=25

Rollback

helm rollback obleth <previous-revision> -n obleth-system   # e.g. 18

Schema migrations are not reverted, but because they are additive the older binary simply ignores columns it does not read.

Docker Compose upgrades

# Pull the new images (OBLETH_VERSION pinned in .env or passed inline)
OBLETH_VERSION=v0.9.6 docker compose -f deploy/docker/docker-compose.yml pull

# Recreate containers (brief downtime)
OBLETH_VERSION=v0.9.6 docker compose -f deploy/docker/docker-compose.yml up -d --force-recreate

Docker Compose does not support zero-downtime rolling upgrades natively. For production, use Kubernetes.

Edge / unreleased builds

Every push to main publishes amd64 :main images. To track edge on top of the current chart:

helm upgrade obleth oci://ghcr.io/thediymaker/charts/obleth --version 0.9.6 \
  -n obleth-system --reset-then-reuse-values --set image.tag=main --wait

Switching an existing release to GHCR

If a release was originally installed with a private registry override (image.obleth, image.tag, …), value reuse keeps that registry. To move it back onto the public GHCR images, override the image block explicitly once:

helm upgrade obleth oci://ghcr.io/thediymaker/charts/obleth --version 0.9.6 \
  -n obleth-system --reset-then-reuse-values \
  --set image.obleth=ghcr.io/thediymaker/obleth-gateway/obleth \
  --set image.controlPlane=ghcr.io/thediymaker/obleth-gateway/control-plane \
  --set image.benchmarkBackend=ghcr.io/thediymaker/obleth-gateway/benchmark-backend \
  --set image.tag=v0.9.6 \
  --wait

Subsequent --reuse-values upgrades then stay on GHCR automatically.