Scaling

Add gateway replicas

The chart runs one gateway replica by default. For redundancy, set more:

obleth:
  replicas: 3
affinity:
  antiAffinity: soft   # "hard" requires distinct nodes; "" turns it off

Every replica is stateless: keys, models and settings come from Redis and Postgres, and a change in the dashboard reaches every replica through the obleth:invalidate channel. With more than one replica guaranteed, the chart also renders a PodDisruptionBudget (minAvailable: 1), so a node drain leaves one running. Rolling updates drain each pod before it exits; see Shutdown and timeouts.

Docker Compose runs one gateway container, and docker compose up --scale obleth=N does not work: the service publishes a fixed host port and the bundled HAProxy has one backend entry. Use Kubernetes for several replicas.

How limits hold across replicas

A model's pool size (max_in_flight), the global ceiling, and every tenant and key max_in_flight are numbers for the whole fleet. How the replicas enforce them:

With both variables off, each replica enforces the full values, so N replicas admit N times each limit. That was the behaviour before v2.0.0; set obleth.fairshareSharedSlots: false and obleth.fairshareReplicaAware: false to keep it.

Replicas count each other by heartbeats in Redis, every OBLETH_FAIRSHARE_REPLICA_HEARTBEAT_SECS (5) and live for OBLETH_FAIRSHARE_REPLICA_TTL_SECS (15). A replica that crashes has its slots reclaimed once its heartbeat expires; one that shuts down cleanly gives them back after draining. Fairness is exact within a replica; across replicas, a freed slot goes to whichever claims it first. The algorithm is in Fairshare engine.

Each replica's mode is on the Fairshare page, in GET /api/v1/fairshare/live and GET /api/v1/stats, and in obleth_fairshare_shared_slots_mode{mode}. Alert on fallback.

Autoscale with the HPA

The chart ships a CPU-based HorizontalPodAutoscaler, off by default:

hpa:
  enabled: true
  minReplicas: 2
  maxReplicas: 20
  targetCPUUtilizationPercentage: 70

With shared slots on, scaling changes gateway CPU, not the concurrency reaching your backends: a new pod draws on the same slots within one heartbeat. Adding gateway replicas does not add backend capacity either. A queue caused by a full pool means the backend needs more servers, or a larger pool if it can serve more; see How fairshare works.

Size the pools

A model's pool size is how many requests the gateway sends its backend at once. Set it to what the backend can serve without queuing internally: too high moves the queue into the backend, where you cannot see it; too low grows the gateway's queue. The pool is fleet-wide, so do not divide it by the replica count.

Set it on a model's page (Capacity), with PUT /api/v1/models/{id}/capacity, from a load probe (Capacity auto-tune), or let it follow the backend's ready replicas (Pool sizes). A model without a value uses OBLETH_DEFAULT_MODEL_MAX_IN_FLIGHT (Helm obleth.defaultModelMaxInFlight, default 32).

The global ceiling

OBLETH_GLOBAL_MAX_IN_FLIGHT (Helm obleth.globalMaxInFlight, default 4096) caps requests in flight across all pools. It is a safety cap, not a fairness input: keep it above the sum of the pool sizes, or it binds first and the gateway logs a warning at start-up. For the split and fallback modes, which round each pool's share up, leave at least one slot per enabled model per replica above the sum.

PUT /api/v1/capacity changes the ceiling live, but only on the replica that answers and only until it restarts. To change it everywhere, set the variable and roll the pods. The Compose file fixes the ceiling at 4096 and the default pool at 32, so in Compose set pool sizes per model.

Datastores

• Redis. One primary handles most deployments. Redis Cluster is not supported, because the budget, cache and slot scripts touch several keys in one call. For high availability, use a Redis whose failover presents one primary at a stable address.
• ClickHouse. A single instance takes tens of thousands of rows per second. For large histories, run a replicated cluster and set the retention window in Settings › Data and backups.
• Postgres. Holds configuration, the audit log, knowledge and the dashboard's accounts; it is off the request path and rarely the bottleneck. Read replicas are not needed.