Priority boost

Boost a tenant from the Fairshare page

1. 1

   Open the pool

   On Fairshare, open the model pool that is queuing. Tenants waiting below their share are listed first as Next in line.

2. 2

   Open the tenant

   Choose the tenant. Its panel shows its weight, its cap on this model, and its keys.

3. 3

   Change the weight and save

   Enter the new weight. The preview shows the tenant's new fair share in each pool and names the tenants that give up slots. A tenant's weight only moves slots inside its own group.

You can also change the weight under Fairshare group and weight on the tenant's page. To move slots between groups, change group weights on Fairshare › Groups & weights, which previews every group's new share before you save.

What a weight does

Weights are ratios. In a full pool, a tenant with weight 400 gets twice the slots of a tenant with weight 200 in the same group. Weight does nothing while a pool has free slots, because every request is admitted at once. It does not change a tenant's token rate, budget or in-flight cap; see Budgets and rate limits.

There are three levels of weight:

Raising a tenant's weight raises all of its keys together. Raising one key's weight moves share toward that key inside its tenant. See How fairshare works.

Boost through the API

# A tenant
curl -X PATCH http://localhost:9180/api/v1/tenants/$TENANT_ID/weight \
  -H "Authorization: Bearer $OBLETH_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"weight": 1000}'

# A group
curl -X PATCH http://localhost:9180/api/v1/fairshare/groups/teaching/weight \
  -H "Authorization: Bearer $OBLETH_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"weight": 400}'

A key's weight is set with PUT /api/v1/keys/{id}, which replaces all of the key's fields, so send its name, budget and cap along with the new weight. See Tenants and keys.

During an incident

A batch job is filling a model that an interactive service also uses. Raise the service's weight and lower the job's:

curl -X PATCH http://localhost:9180/api/v1/tenants/$SERVICE_ID/weight \
  -H "Authorization: Bearer $OBLETH_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"weight": 1000}'

curl -X PATCH http://localhost:9180/api/v1/tenants/$BATCH_ID/weight \
  -H "Authorization: Bearer $OBLETH_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"weight": 10}'

Requests already running finish; the next freed slots go to the service. If the job should never hold more than a few slots, also give it an in-flight cap on the model from the tenant's panel. Put the weights back when the incident is over.

For work that should only ever use spare capacity, such as labeling runs or benchmarks, give the tenant weight 1. It loses every contended decision but still uses slots nobody else wants.

Finding past changes

Every weight change is in the audit log as update_weight (tenant), update_fairshare_group_weight (group) or update_key (key), with who made it. On Audit, filter by kind or search for the tenant's name. See Audit log.