How fairshare works

Each model has its own pool

A pool is the number of requests the gateway lets reach a model's backend at once. Its size is the model's max in flight, or OBLETH_DEFAULT_MODEL_MAX_IN_FLIGHT (default 32) when the model has none. A model in the discovered capacity mode sizes its pool from the backend's ready replicas instead; see Pool sizes.

Pools are independent. A full pool never delays a model with free slots, and a tenant's use of one model costs it no priority on another. OBLETH_GLOBAL_MAX_IN_FLIGHT (default 4096) is a safety ceiling across all pools together; keep it above the sum of your pool sizes so the pools do the limiting.

When a pool is full, requests queue

A request that finds its model's pool full waits in that pool's queue instead of failing. When a running request finishes, the scheduler hands its slot to the waiter that is furthest below its fair share. The client sees a slower response, not an error.

A waiter whose client has disconnected is dropped from the queue and charged nothing. Requests already running are never cut short to make room.

Who gets the next slot

Slots are divided in three steps, each by weight:

1. Groups. Each pool's slots are split between the active fairshare groups by group weight.
2. Tenants. A group's share is split between its active tenants by tenant weight.
3. Keys. A tenant's share is split between its keys by key weight, so one busy key cannot take all of its tenant's slots.

Only tenants that are running or waiting on the pool count. A share is a claim when the pool is contended, not a reservation: slots a quiet group or tenant leaves unused are lent to whoever is waiting, and are returned as those borrowed requests finish.

Fairness is measured in tokens, not requests. Each request is charged its estimated token count when it is admitted, so a tenant sending long prompts uses up its share faster than one sending short ones.

A tenant or key can also carry a max in flight: a hard cap on how many of its requests run at once in each model's pool. Requests over the cap wait like any other. Set weights and caps on the tenant's page and in the key's panel; see Tenants and keys. To change a weight during an incident, see Priority boost.

The scheduling algorithm, the scores it keeps and how shares are computed are in Fairshare engine.

Limits hold across replicas

With more than one gateway replica, every limit above (pool sizes, the global ceiling, tenant and key caps) holds for the fleet as a whole. A model with 20 slots behind three gateways admits 20 requests in total, on whichever gateways they reach. The slots are counted in Redis (OBLETH_FAIRSHARE_SHARED_SLOTS, on by default). With a single replica there is no Redis call on the request path.

If Redis stops answering, each gateway falls back to its share of each limit (the configured value divided by the number of live gateways, rounded up) and returns to shared slots when Redis recovers. It never admits without a limit.

Queued requests time out

A request that waits OBLETH_ADMISSION_TIMEOUT_SECS (default 60) without getting a slot leaves the queue with a 503 and a Retry-After header:

HTTP/1.1 503 Service Unavailable
Retry-After: 5
Content-Type: application/json

{"error":{"message":"timed out waiting for model capacity","type":"obleth_gateway_error"}}

Clients should wait and retry. The OpenAI and Anthropic SDKs already retry on 503 and respect Retry-After.

Queuing covers contention. A request is refused at once only when it crosses a limit you set:

Read the Fairshare page

Fairshare opens on every model pool, fullest first, with the number of requests running and waiting, the busiest pool, and each group's use against the share its weight guarantees.

A pool is Busy above 80% and Full at 100%. A pool whose model has nowhere to send a request (no API base, no enabled endpoint, or a watched backend with no ready replica) reads No servers, and its slots are left out of the totals. Open a pool (or follow a Full item from the Overview) to see who holds its slots:

The (?) beside each tenant explains why it is waiting and who holds slots above their share; the one beside Full lists what you can do. Open a tenant to change its weight or cap it on that model, with a preview of who gives up slots. Groups & weights changes group weights, previewing every group's new share before you save.

When requests wait too long

A pool that queues most of the time is too small for the traffic it gets.

• Raise the model's max in flight, if the backend can serve more at once. Capacity auto-tune finds the backend's limit.
• Add backend replicas. Adding gateway replicas does not add backend capacity.
• Check weights: one high-weight tenant can hold most of a busy pool.
• Keep OBLETH_GLOBAL_MAX_IN_FLIGHT above the sum of your pool sizes. The gateway logs a warning at start when it is not.

SELECT tenant_id, admission, count() AS requests, avg(queue_wait_ms) AS avg_wait_ms
FROM usage
WHERE ts_ms > toUnixTimestamp(now() - INTERVAL 1 HOUR) * 1000
GROUP BY tenant_id, admission
ORDER BY requests DESC