Pool sizes

Set a model's pool

Every model has its own fairshare pool: the number of requests obleth has in flight to it at once. Requests beyond that wait in the model's queue, and tenant and key weights decide who goes next. See Fairshare engine.

On a model's page, under Capacity, set Max slots to about the number of requests the model's backend serves at once without queuing, such as the sum of your servers' maximum concurrent sequences. Blank uses OBLETH_DEFAULT_MODEL_MAX_IN_FLIGHT (default 32). To measure the number instead of guessing, run Capacity auto-tune.

curl -X PUT http://localhost:9180/api/v1/models/$MODEL_ID/capacity \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"max_in_flight": 32}'

Send {"max_in_flight": null} to go back to the default. The pool size is cluster-wide: with several gateway replicas, they share the slots between them. Admission weight does not change a model's pool.

A steady queue (obleth_queue_depth) means the pool is smaller than the load.

Follow the backend

When an autoscaler adds replicas, or a node drains, a fixed number goes stale. In the discovered mode the pool follows the backend:

pool size = max(1, ceil(ready replicas × per-replica concurrency × headroom))

Each gateway replica re-reads the backend every interval and resizes the pool without cutting in-flight requests short.

1. 1

   Turn on discovery

   Set OBLETH_CAPACITY_DISCOVERY_ENABLED=true, or obleth.capacityDiscovery.enabled: true in Helm. For the kubernetes source, also list the namespaces it may read (see Turn on discovery).

2. 2

   Pick the mode

   On the model's page, under Capacity, choose Discovered, then a Capacity source: endpoints or kubernetes.

3. 3

   Say what one replica takes

   Set Per-replica concurrency, the requests one replica serves at once (for example vLLM's --max-num-seqs), and Headroom (1.0 is exactly the ready capacity). Fallback slots is the pool used while discovery has no answer. Save.

The section then shows the live count, for example "Live: 4 ready × 32 per replica = 128".

curl -X PUT http://localhost:9180/api/v1/models/$MODEL_ID/capacity-mode \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "capacity_mode": "discovered",
    "capacity_source": "kubernetes",
    "capacity_service": "my-model-serve",
    "capacity_namespace": "inference",
    "per_replica_max_in_flight": 32,
    "capacity_headroom": 1.0
  }'

A write that can't be discovered as configured (no per-replica value, a namespace outside the list, no Service and no default) is refused with 400 and a message saying what to set.

Sources

endpoints counts the model's own enabled, healthy endpoints: all of them under load balancing and sticky sessions, only the one in use under failover. A model with no endpoints counts its API base as one replica while it is healthy. An endpoint's own Max in flight wins over the model's per-replica value, and the values are added up: endpoints at 8, 2 and unset, with a per-replica value of 4, give 14. It needs nothing outside obleth.

kubernetes counts the ready endpoints of a Service, from its EndpointSlices. An endpoint counts when it is ready, serving and not terminating; nothing else is read. Which pods count is up to the Service's selector: point the model at a Service that selects only the model-server pods that take requests, not a gateway in front of them.

Choosing the Service

With the kubernetes source, Capacity lists the Services it can see in the allowed namespaces, each with its ready count, so you pick one rather than type it. If the model's default Service exists, it reads "Using <service> in <namespace> · N ready", with Change. Use default clears the choice.

The default comes from OBLETH_CAPACITY_DEFAULT_SERVICE, a template with {upstream_model} and {model_name} placeholders. A model with no namespace is looked up in each allowed namespace in order. The list comes from GET /api/v1/capacity/services.

Turn on discovery

obleth:
  capacityDiscovery:
    enabled: true
    intervalSecs: 15
    namespaces:
      - inference
    defaultService: "{upstream_model}"

The chart ships this as examples/values-capacity-discovery.yaml. Docker Compose passes the switch and the interval through from .env; the kubernetes source needs the gateway to run in a pod.

RBAC

With discovery on and at least one namespace listed, the chart renders a ServiceAccount for the gateway pods and, in each listed namespace, a Role granting only get, list and watch on endpointslices (discovery.k8s.io), with its RoleBinding. There is no ClusterRole and nothing on pods, Services or Secrets. The namespaces must exist, and whoever runs helm must be allowed to create Roles in them.

Fallback states

Each change of state is logged once. Gateway replicas can disagree for up to one interval after a change.

Observe it

GET /api/v1/capacity/discovery reports the answering replica's settings and, for each discovered model, its source, Service, ready replicas, per-replica value, derived and in-force pool size, in-flight counts, state and the reason for a stale or fallback state. The metric obleth_capacity_discovery_models{state} counts models by state. The Deployments page shows the same for watched Kubernetes models.