Upgrades

What changes in v2.4.0

There is nothing to do before upgrading from v2.3.x: no new required values or Secret keys, and the schema change is additive, so old and new pods run side by side. Coming from v2.2.x, also read the v2.3.0 section below.

Applied at startup. Postgres migration 0038: a session_affinity column on models, {} for every existing model, so every model follows the gateway default, which is off.

Turn session affinity on after the rollout. Session affinity holds slots that only v2.4.0 knows how to hand over and free. Leave it off, gateway-wide and on every model, until every replica runs v2.4.0; with it off, v2.3.x and v2.4.0 replicas share slots as before.

Behaviour that needs no action:

• Derived session ids change once for conversations whose system prompt and first user message together were longer than 8 KiB. Those conversations get ids of their own instead of sharing one, so Request logs group them apart and session_hash routing spreads them across endpoints. Shorter openings hash exactly as before.
• OpenAI's prompt_cache_key is read as a session id when the request has no session_id or metadata.session_id.
• Every auto response carries x-obleth-auto-model, and obleth_auto_routes_total counts auto picks. Keeping a conversation on its model is off until you turn it on in Settings › Routing.
• The chart's obleth.fairshareSessionAffinity values are new, with enabled: false.
• Reports has a Model speed section, backed by GET /api/v1/usage/speed.

Read the release notes for everything new.

What changed in v2.3.0

Read this too when you upgrade from v2.2.x. There is nothing to do before upgrading from v2.2.x: no new required values or Secret keys, and every schema change is additive.

Applied at startup.

• Postgres migrations 0036 and 0037: rejected request fields on models, and a kind on the job table so a mesh job is only found through /v1/meshes. Video jobs written before 0037 are read as videos.
• ClickHouse columns on usage: key_name, tenant_name and streamed, added in place. Rows the older version spilled to its WAL replay with empty names and streamed = false.

Behaviour that needs no action:

• First-token time is recorded only for replies streamed to the client. A buffered reply has none (0) in the ledger, Request logs and obleth_ttft_ms, so a model mostly called without streaming shows far fewer first-token samples, and a lower median, than before. Older rows keep their values.
• Calls through /mcp/<server> appear in Request logs as mcp rows with no tokens or cost, so reports that count every usage row count them too.
• A deleted key or team keeps its name on its past requests.
• Energy per model and rejected request fields change nothing until you configure them.

What changed in v2.2.0

Read this too when you upgrade from v2.1.x. There is nothing to do before upgrading from v2.1.x: no new required values or Secret keys, and every schema change is additive, so old and new pods run side by side. Coming from v2.0.x, also read the v2.1.0 section below.

Applied at startup.

• Postgres migration 0035: reasoning effort maps and the listing switch on models, and the model_families table. Every existing model keeps passing effort through as sent and stays listed.
• ClickHouse columns on usage: reasoning_effort, reasoning_effort_requested and model_family, added in place. A call through a model family now records the family in model_family; model_variant holds real variants only.

The compressor follows image.tag. The chart's compressor.image.tag and compressor.image.pullPolicy now default to image.tag and image.pullPolicy, like every other component. A release install is unchanged. An install that sets image.tag to dev or main now runs the edge compressor too, which edge builds publish from this release on. Set compressor.image.tag to keep it on its own tag.

Behaviour that needs no action:

• A stream the upstream breaks off, goes silent on past the read timeout, or closes with neither [DONE] nor a finish reason ends with one SSE error event ("type": "obleth_gateway_error", "code": 502) and then an aborted response. OpenAI's SDKs raise it as an APIError. It used to end with finish_reason: "stop" on models with a tool boon, and with a bare abort elsewhere. The request is recorded as a 502, billed for what reached the client.
• A 200 response that carries the upstream's failure (an SSE error event, finish_reason abort or error, or a JSON error body) is recorded with the error's status, billed for what it produced and never cached. The client still gets the bytes as sent.
• A read timeout before the response headers returns 504 and is not retried. It was retried as a dropped connection, which ran the generation twice.
• auto classifies the newest user message, stepping back to the one before for a follow-up with no topic of its own, so a conversation can move between models from turn to turn.
• In shared fairshare mode, GET /api/v1/fairshare/live counts every live gateway and says so in scope; each gateway's own counts are in replica_in_flight and replica_queued. The Overview judges "Below share" per pool.
• /v1/models entries carry what a catalog needs (description, context window, prices, supports_*) and the boons that run on each model (boons, boon_details).
• /v1/verdicts takes images and model family names, and answers a request with no model from Settings › Routing.
• GET /v1/audio/voices lists a speech model's voices, and the Playground has a Speech mode. Speech from a backend that streams already streamed through the gateway.
• datastores.nodeSelector and datastores.tolerations are new and empty, which renders nothing.

Read the release notes for everything new.

What changed in v2.1.0

Read this too when you upgrade from v2.0.x. There is nothing to do before upgrading from v2.0.x: no new required values or Secret keys, and every schema change is additive, so old and new pods run side by side. Coming from v1.x, first follow Upgrading from v1.x.

Applied at startup.

• Postgres migrations 0031 to 0034: the Slurm launch history, per-user fairshare on keys, model variants and the model lifecycle. Every existing model starts active. While a rollout is in progress, an older gateway serves a staged model as active.
• ClickHouse columns on usage: cached_input_tokens, end_user, parent_request_id and model_variant, added in place. usage_daily is unchanged, and rows the older version spilled to its WAL replay with zero or empty values.

The bundled ClickHouse keeps a week of its own logs. The chart's clickhouse.systemLogRetentionDays (7) and clickhouse.logLevel (information), and Compose's clickhouse/system-logs.xml, bound the server's system log tables, which the stock image keeps forever. The ClickHouse pod restarts once; meanwhile the gateway spills usage to its write-ahead log and replays it. On that first start, ClickHouse renames each system log table to <name>_0 and starts a fresh one; the old tables keep their data until you drop them:

SELECT name, formatReadableSize(total_bytes) FROM system.tables
WHERE database = 'system' AND match(name, '_log_[0-9]+$');
DROP TABLE system.query_log_0;  -- and so on for each one listed

An external ClickHouse is not changed; give it a retention of its own.

Upgrade the chart with --reset-then-reuse-values, as below, so the values this chart adds take their defaults. If you use plain --reuse-values, the new ClickHouse values still fall back to their defaults: 7 days of system logs at level information.

Fill in the Slurm connection's Cluster defaults before launching from the library. The shipped recipes take the weight cache, setup lines and container images from them, and a launch is refused while one it needs is unset. See Launch on Slurm.

Usage reaches Request logs up to 5 seconds after a request, because each gateway now inserts usage every OBLETH_TELEMETRY_FLUSH_INTERVAL_MS (Helm obleth.telemetryFlushIntervalMs, 5000) instead of every second. Budgets are enforced in Redis and don't change.

Behaviour that needs no action:

• Models added on the dashboard, or launched from a recipe, start staged: unlisted and never picked by auto until you make them active. The Management API still creates active models unless asked.
• On /v1/messages, usage input_tokens is now the input the backend didn't serve from its prefix cache; add cache_read_input_tokens for the whole.
• Image generation and web search follow a model's Function calling switch on its page.
• A text-only model with the vision boon never receives a raw image; one that can't be described becomes a note.
• An optional recipe input left empty renders as nothing, and Slurm memory 0 is the whole node.
• Creating a model under a name that is another model's alias or variant returns 400.
• kubectl port-forward reaches the dashboard.
• Router readiness warns until a model auto may pick carries the new writing tag.

Read the release notes for everything new.

Upgrade with Helm

1. 1

   Preview the images

   helm upgrade obleth oci://ghcr.io/thediymaker/charts/obleth --version 2.4.0 \
     -n obleth --reset-then-reuse-values --dry-run | grep 'image:'
   

   Every image should end in :v2.4.0. Image tags default to the chart's version, so one chart upgrade moves every component.

2. 2

   Upgrade

   helm upgrade obleth oci://ghcr.io/thediymaker/charts/obleth \
     --version 2.4.0 \
     --namespace obleth \
     --reset-then-reuse-values \
     --wait
   

   --reset-then-reuse-values (Helm 3.14+) takes the new chart's defaults and re-applies the values you set. --wait fails the upgrade if the new pods do not become ready.

3. 3

   Check it

   helm history obleth -n obleth | tail -2
   kubectl -n obleth get pods
   kubectl -n obleth logs deploy/obleth-obleth -c obleth --tail=25
   

   The log shows postgres connected + schema applied, redis connected, clickhouse connected + schema applied (or telemetry started in spill mode) and listening: proxy=…. Settings › About shows the running version.

Migrations run by themselves at every start, under a lock so replicas cannot race, and only add tables and columns, so old and new pods run side by side. v2.4.0 adds 0038, v2.3.0 added 0036 and 0037, v2.2.0 0035, and v2.1.0 0031 to 0034; see Postgres schema.

Upgrade with Docker Compose

Update your checkout of deploy/docker/ to the release first (for example git checkout v2.4.0), so the Compose files match it, then pull and recreate:

OBLETH_VERSION=v2.4.0 docker compose -f deploy/docker/docker-compose.yml pull
OBLETH_VERSION=v2.4.0 docker compose -f deploy/docker/docker-compose.yml up -d --force-recreate

With the compressor profile, set OBLETH_COMPRESSOR_VERSION=v2.4.0 too. There is no rolling upgrade: the old gateway drains for up to 345 seconds first.

Roll back

helm rollback obleth <previous-revision> -n obleth

Migrations are not reverted, but the older gateway ignores columns it does not read. Before rolling back from v2.4.0, turn session affinity off on the gateway and on every model, and let the hold time (3 seconds by default, at most 60) pass so no slot is still held: an older gateway counts a held slot against its pool and never frees it. A v2.3.x gateway ignores per-model session affinity and keeps no conversation on its model. A v2.2.x gateway also has no mesh jobs or rejected request fields. A v2.1.x gateway has no reasoning maps or model families: it passes effort through, answers a family's name as an unknown model, and lists unlisted models. A v2.0.x gateway also has no model lifecycle, so it serves staged, deprecated and retired models as active.