Shutdown and timeouts

What happens on SIGTERM

1. 1

   Stop accepting connections

   All three listeners close: data plane (8080), Management API (9180) and metrics (9091). The log reads shutdown signal received; waiting for in-flight connections.

2. 2

   Drain in-flight requests

   Running requests and open streams get up to OBLETH_SHUTDOWN_GRACE_SECS (default 300) to finish. Then the log reads in-flight requests drained, or shutdown grace elapsed; remaining connections end at process exit. With several replicas, the replica gives back its fairshare slots once drained.

3. 3

   Flush telemetry

   Queued usage rows and spans are written to ClickHouse, or to the spill file if it is unreachable, for up to 20 seconds.

4. 4

   Flush traces and exit

   With OBLETH_OTEL_ENDPOINT set, spans are flushed to the collector for up to 10 seconds. The log ends with obleth gateway stopped.

A second SIGTERM during the drain exits at once, without draining, with exit code 130.

Keep the stop grace above the drain

The worst case is the drain plus 20 plus 10 seconds. If the orchestrator kills the process before that, queued usage rows are lost.

The Helm value is fixed in the chart template, not in values.yaml. If you raise obleth.shutdownGraceSecs (or OBLETH_SHUTDOWN_GRACE_SECS in Compose), raise the template's grace period or stop_grace_period by the same amount.

Because long streams can hold the drain open, docker compose stop, restart or up --force-recreate can take up to 345 seconds, and a Helm rolling update keeps each old pod until its requests finish.

Timeouts

A zero, blank or invalid value for the admission or Redis timeouts falls back to the default. Every variable is read at start-up, so restart the gateway after changing one. The Helm values are obleth.admissionTimeoutSecs, obleth.redisResponseTimeoutMs, obleth.redisConnectTimeoutMs, obleth.upstreamTimeoutSecs, obleth.upstreamReadTimeoutSecs and obleth.shutdownGraceSecs; Compose forwards each from .env. See Environment variables.

How the upstream timeouts interact

• For a streaming request, the request timeout is the time to the first byte; for a non-streaming request, it is the whole generation.
• When a boon that rewrites the response is active, the upstream call is made non-streaming, and the request timeout is raised to at least 120 seconds.
• The read timeout also bounds the header wait, so it cuts a slow non-streaming generation even when the request timeout is longer. The gateway logs a warning naming each model whose request_timeout_secs exceeds it; raise OBLETH_UPSTREAM_READ_TIMEOUT_SECS or lower the model's value.
• A timeout before the response starts refunds the request's budget reservation and bills nothing. A stream cut after tokens reached the client is billed for what streamed, never the max_tokens estimate.

Proxies in front of the gateway have their own limits. The Compose HAProxy edge sets timeout client and timeout server to 300 seconds; raise those, and your ingress's, along with the gateway's.

Slurm provisioner shutdown+

The provisioner never interrupts a reconcile tick, since stopping mid-tick could leave a submitted job unrecorded. A SIGTERM during a tick takes effect when the tick ends; one received between ticks exits at once. The chart gives the provisioner pod terminationGracePeriodSeconds: 60. Its calls to the Management API and slurmrestd time out after 30 seconds (5 to connect), and a failed call holds the tick rather than acting. Its replica health probe times out after OBLETH_PROVISIONER_HEALTH_TIMEOUT_SECS (5), and its warm-up call after OBLETH_PROVISIONER_WARMUP_TIMEOUT_SECS (600; 0 skips warm-up).