Fairshare engine
How the admission scheduler decides which waiting request gets a model's next free slot, from the per-model pool down to the key, and how replicas share the limits.
10 min readUpdated for v2.4.0Edit on GitHub
This page covers the scheduler's internals. To set weights, groups and caps, see Tenants and keys, API keys and Priority boost.
Admission model
Every open slot is assigned deliberately.
A single scheduler owns queue order and in-flight counts. This example shows two tenants competing for an eight-slot pool: chatbot has weight 500, while api-batch has weight 50.
Admit
obleth identifies the tenant and policy before routing upstream.
Queue
When the pool is full, requests wait by tenant instead of racing.
Pick
The next open slot goes to the tenant most entitled to it.
Release
The slot returns when the model response finishes streaming.
Hierarchical mode - current runtime default
Weights split the pool between groups first.
With an eight-slot pool and active groups weighted 500:50, the weights give chatbot seven slots and the api group one. A group can borrow slots another leaves idle; here both are busy, so api-batch queues but keeps being served.
chatbot
group chatbot / weight 500
cap 7 / queued 57
api-batch
group api / weight 50
cap 1 / queued 31
8-slot pool
example split
Inside a group
Each group gets its own fairshare queue.
Once a group's share of the pool is set, the tenants inside it are balanced by how much work each has already received. A busy tenant cannot permanently crowd out its peers.
chatbot
served 8.6k
queued 28
ahead
chatbot-2
served 8.4k
queued 29
next
batch-job
served 9.1k
queued 12
waits
Fast
If capacity is open and no backlog exists, the request is admitted immediately.
Queued
If the pool is full, the request waits for a slot. Past OBLETH_ADMISSION_TIMEOUT_SECS (60 by default) it gets 503 with Retry-After.
Rejected
Only when a hard limit is crossed — the per-minute token budget (429) or a term budget (403).
Per-model pools
Every model has its own pool of slots: its max_in_flight, or OBLETH_DEFAULT_MODEL_MAX_IN_FLIGHT (default 32). A model in the discovered capacity mode sizes its pool from its ready backend replicas (Pool sizes). A request holds a slot from admission until the upstream response has finished streaming.
Fairness is worked out in each pool on its own: a full model never delays one with free slots, and heavy use of one model doesn't lower a tenant's standing on another. OBLETH_GLOBAL_MAX_IN_FLIGHT (default 4096) caps the sum across pools and plays no part in fairness; when it binds, pools are served round-robin. Keep it above the pools you expect to be busy together; the gateway warns at start when it isn't.
Unregistered model names share one unrouted pool. A pool idle for 10 minutes, or whose model is no longer routed, is dropped.
Effective weight
A tenant competes in a pool with its weight scaled by the model's admission_weight (a percentage, default 100):
effective weight = round(tenant weight × admission_weight / 100), at least 1
Every tenant in a pool is scaled alike, so their shares of it don't change. Group and key weights are not scaled.
Choosing the next request
Each tenant and key has a served counter in estimated tokens. share_score is served divided by weight: lower means further behind. When a slot frees, the scheduler picks at three levels.
Group, then tenant (OBLETH_FAIRSHARE_ALGORITHM=hierarchical, the default). The pool is split between the groups that have work by group weight, and each group's part between its active tenants by tenant weight, with largest-remainder rounding and at least one slot each when there is room. An 8-slot pool with groups of weight 500 and 50 splits 7 and 1. Among waiting tenants under their share, the scheduler takes the group with the lowest group score, then its tenant with the lowest share_score.
Tenant only (weighted). Every tenant competes directly on share_score: start-time fair queuing on token cost.
Key. Inside the chosen tenant, the key with the lowest served-per-key-weight goes next (key weight default 100), so one busy key can't starve its teammates.
A tenant's or key's max_in_flight caps its running requests in each model's pool, not across models.
Shares are ceilings, not reservations. If no waiting tenant is under its share but a slot is free, it is lent to whoever is waiting. The lending group gets its share back as those requests finish; nothing is preempted. The live state reports each group's slot_cap and borrowed.
No saved-up credit. A tenant or key that starts sending again after a quiet spell has its counter raised to the pool's virtual time, the lowest counter among those waiting, so it rejoins level with the others. Every tenant keeps advancing, so none waits forever.
One scheduler task
One async task owns every pool, queue and count, so admission decisions never race; dropping a request's permit releases its slot. Weights travel with each request from the key cache, so a change applies on the next admission after cache invalidation, without a restart. A request still waiting after OBLETH_ADMISSION_TIMEOUT_SECS (default 60) gets 503 with Retry-After: 5; one whose client left is dropped without charge. See How fairshare works.
Multiple gateway replicas
So that configured numbers hold for the fleet, replicas share them through Redis (OBLETH_FAIRSHARE_SHARED_SLOTS, default true). With more than one live, every pool size, the global ceiling and every tenant and key cap is a set of cluster-wide slots. A replica picks its next waiter as it would alone, then takes a slot in one Redis script call that checks the pool, ceiling and caps together against the configured values, never divided. Finishing gives it back in one call. A 20-slot model behind three replicas admits 20 requests wherever they land, and never a 21st. With one live replica there is no Redis call on this path.
| Held across replicas | Not shared |
|---|---|
| Pool sizes (configured, default and discovered) | Group, tenant, key and model weights, which are ratios |
| The global in-flight ceiling | Token and term budgets, already fleet-wide in Redis |
Tenant and key max_in_flight |
Fairness is exact within a replica; across replicas a freed slot goes to whichever claims it first. A crashed replica's slots stay taken until its heartbeat expires, up to OBLETH_FAIRSHARE_REPLICA_TTL_SECS (default 15).
Queues stay with the replica that holds them, so in shared mode each replica also publishes a view of its scheduler: every pool with work, with its totals and the groups, tenants and keys running or waiting there. It goes out every 500 ms, or every heartbeat if that is shorter, and lives for three intervals, so one slow publish doesn't drop a replica. The live state and its fair shares merge every live replica's view with the answering replica's own, and shares are split over the tenants with work across all of them.
Heartbeats and repair
Replicas heartbeat every OBLETH_FAIRSHARE_REPLICA_HEARTBEAT_SECS (default 5); the TTL is raised to at least twice that. Redis records each replica's holdings: an expired replica's are reclaimed by the next live heartbeat, a clean shutdown returns them after draining, and each replica re-asserts its holdings every interval to repair a lost release. A freed slot that another replica is waiting for is announced on obleth:fairshare:released, backed by a random retry every 100 to 250 ms. A replica's scheduler view is obleth:fairshare:view:<instance>, deleted on a clean shutdown. It holds at most 512 tenants and 128 keys per pool, and 2,048 and 512 across pools, the busiest first; pool and group totals are always complete. Keys are in Redis layout.
When Redis fails: the split
If a slot call fails or takes longer than OBLETH_REDIS_RESPONSE_TIMEOUT_MS (default 250), the replica divides every limit by the live replica count (OBLETH_FAIRSHARE_REPLICA_AWARE, default true): max(1, ceil(configured / replicas)), so a 32-slot pool behind three replicas is 11 on each. It retries every second and returns to shared slots once Redis answers. It never admits without a limit.
| Mode | When | Enforces |
|---|---|---|
shared | Shared slots on, several replicas, holdings reconciled | Configured values, cluster-wide |
fallback | Shared slots on, Redis unavailable or not reconciled | The split, or full values with replica-aware off |
split | Shared slots off, replica-aware on, several replicas | The split |
local | One replica, or both settings off | Configured values, locally |
With both settings false (Helm obleth.fairshareSharedSlots, obleth.fairshareReplicaAware), N replicas admit N times every limit, as before v2.0.0. Mode and replica-count changes resize pools live without cutting running requests short.
Session affinity
An agent sends the steps of a session one after another, each over the same, growing prompt. The backend keeps that prefix in its prompt cache only until other work pushes it out. When many sessions share too few slots, a step that waits behind other sessions' steps often finds its prefix gone and is prefilled again from the start, so a step that should take seconds takes minutes. With OBLETH_FAIRSHARE_SESSION_AFFINITY=true (off by default), each pool favours sessions that are still warm.
A session is the request's session id, sent by the client or derived by the gateway, under the key that sent it. Requests with no session id, and requests to a model that load balances across several endpoints (so a session would not reach the same backend twice), are scheduled as before.
- Order within a key. A session with a step running, or whose last step finished within the warm window, goes ahead of older requests from the same key. Which group, tenant and key is served next is still decided as above.
- Held slots. When a step finishes that arrived within the hold time of the session's previous step finishing, as an agent's next step does and a person's next message does not, its slot is held for the hold time. The session's next step takes it at once; if none comes, the slot is freed. A held slot counts like a running request against the pool, the global ceiling and the tenant's and key's
max_in_flight, and the step that takes it is charged like any other. - Limits. No more than a set share of a pool's slots is held at once. A queued request can be passed by later requests of its key, whether they went ahead as warm sessions or took a held slot, a set number of times. Then it goes next, and no slot is held for its key, nor a held one taken, until it has been served. A request that has waited the bypass limit, and would get a free slot (neither its tenant nor its key is at its cap), is passed by nothing: its key is served in arrival order and its pool's held slots are freed.
| Variable | Helm | Default | Meaning |
|---|---|---|---|
OBLETH_FAIRSHARE_SESSION_AFFINITY | enabled | false | Turns session affinity on |
..._WARM_SECS | warmSecs | 120 | How long after its last step a session counts as warm |
..._HOLD_MS | holdMs | 3000 | How long a slot is held, and the longest gap between steps that counts as a loop; 0 holds nothing |
..._MAX_JUMPS | maxJumps | 8 | Times a queued request may be passed by later requests of its key; also a session's most holds in a row while others wait (across replicas) |
..._MAX_BYPASS_SECS | maxBypassSecs | 30 | Wait after which nothing passes a request; keep it well under OBLETH_ADMISSION_TIMEOUT_SECS |
..._MAX_HELD_PERCENT | maxHeldPercent | 25 | Most of a pool's slots held at once, rounded up; 0 holds nothing |
Variables share the OBLETH_FAIRSHARE_SESSION_AFFINITY prefix; Helm values are under obleth.fairshareSessionAffinity. They are the gateway default; a model can set its own.
Per model
A model can turn affinity on or off for its own pool, whatever the gateway default, and set its own hold time, held share and bypass limit. Anything it leaves out uses the gateway's values, which apply even while the gateway default is off. Set it on the model page under Capacity → Session affinity, with PUT /api/v1/models/{id}/session-affinity, or with session_affinity on model create, update and import:
{ "enabled": true, "hold_ms": 5000, "max_held_percent": 50, "max_bypass_secs": 120 }
{} follows the gateway. hold_ms is 0 to 60000, max_held_percent 0 to 100 and max_bypass_secs 1 to 3600. A change applies to the model's next request; turning affinity off frees the slots its pool holds. This lets you try affinity on one agent-heavy model while every other model is scheduled as before.
A held slot that no step takes was idle for the hold time, so affinity trades some throughput for shorter agent steps. Turn it on where agent sessions compete for a pool, and compare hold with redeem in obleth_fairshare_affinity_total: when they are close, held slots are being used.
Across gateway replicas
In shared mode a held slot lives in Redis, so the session's next step can take it on whichever replica it reaches. The finished step's slot leaves its replica's holdings and is kept under an id made of the pool and the session, still counted against every limit. Redis also records when each session's step last finished, so whether a step followed straight on from the one before is judged across replicas. Every claim for a session names its hold, and a request queued behind a full pool makes one claim for its session's held slot alone, ahead of the queue. A hold nobody takes is freed by the next slot call or heartbeat after it runs out. Holds add no Redis calls: a hold replaces the release.
Queues stay on their replica, so passes are counted where the request waits. Two rules work across replicas. A replica with a request that has waited the bypass limit marks its slot claims for that pool as starving: each one frees the pool's held slots, and none is held there until such claims have stopped for a second. And Redis counts each session's holds in a row while any replica is waiting for the pool or the global ceiling; once a session has had MAX_JUMPS of them, its next slot is released instead. A release, or a hold while nobody waits, starts that count again. Rarely, a session's next step is admitted before its hold is recorded; that held slot then goes unused until the hold time runs out. Ordering within a key uses each replica's own view of which sessions are warm. Keys are in Redis layout.
Watching it
The dashboard's Fairshare page lists pools fullest first; /fairshare?pool=<model> opens one.
GET /api/v1/fairshare/live returns the same state per pool, group, tenant and key, with the replica's mode and, in shared mode, cluster_in_flight. Its scope says whose counts it holds:
cluster: every live replica, merged, in shared mode.replicas_reportingsays how many views it includes (fewer thanreplicaswhile a new replica has not published yet) andstate_age_msthe age of the oldest.replica: the answering replica alone. Always so outside shared mode, and in shared mode when the other replicas' views can't be read, with the reason inscope_reason.
replica_in_flight and replica_queued, at the top and on each pool, are always the answering replica's own.
The dashboard judges Below share inside each pool: a tenant is below its share when it has requests queued, holds fewer slots than its fair share of that pool, and another tenant in the same pool holds at least a slot more than its own. Admission never takes a slot back, so without such a tenant the wait is the pool or a cap being full. When the live state covers only one of several gateways, Overview says the counts are "on this gateway".
GET /api/v1/fairshare/history returns 2-second samples for one model or all pools, kept OBLETH_FAIRSHARE_HISTORY_SECS (default 3600). It is the answering replica's and resets on restart. Fields are in Management API, metrics in Prometheus metrics.