Budgets and rate limits

Set limits on a tenant

1. 1

   Open the tenant

   On Tenants, open the tenant. Limits right now shows what is in force, with Change beside each line.

2. 2

   Change the sections you need

   Limits holds the token rate and in-flight cap, Access hours the weekly grid and dates, Budget the spend and token caps, and Models the model list.

3. 3

   Save

   The save bar lists what changed. The gateway applies it on the next request, on every replica.

A key's own cap and budget are in its panel on API keys. Keys have no token rate of their own; they share their tenant's.

Token rate

The token rate (tokens_per_minute) is a token bucket per tenant. The bucket holds up to one minute's worth, so an idle tenant can burst that many tokens at once, and then refills at the set rate. A request that would overdraw it is refused with 429 token budget exceeded. 0, the default, means no rate limit.

Each request is charged its estimate when admitted and corrected to the real count when it finishes, so an estimate that ran high is refunded.

curl -X PUT http://localhost:9180/api/v1/tenants/$TENANT_ID/quota \
  -H "Authorization: Bearer $OBLETH_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"tokens_per_minute": 2000000, "max_in_flight": 10}'

The quota call sets both fields at once: send "max_in_flight": null for no cap.

In-flight caps

A tenant's max in flight caps how many of its requests run at once in each model's pool. A key's max in flight does the same for one key. With several gateway replicas, the caps hold across all of them together.

Caps do not refuse requests. A request over its tenant's or key's cap waits in the queue like any other, and times out with a 503 after OBLETH_ADMISSION_TIMEOUT_SECS; see How fairshare works. Use a cap to keep a batch job from filling a model even when nobody else is waiting.

Budgets

A budget is a cumulative cap over a period: a token cap, a dollar cap (from each model's prices), or both.

A tenant over its budget gets 403 tenant term budget exhausted; a key over its own gets 403 api key term budget exhausted. The key's budget is checked first.

curl -X PATCH http://localhost:9180/api/v1/tenants/$TENANT_ID/budget \
  -H "Authorization: Bearer $OBLETH_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"budget_tokens": 50000000, "budget_cost_usd": 250, "budget_period": "monthly"}'

null clears a cap. Key budgets use the same three fields on the key; see Tenants and keys. GET /api/v1/budgets/usage lists every capped tenant and key with its use in the current period, when the period started and when it resets.

Reserved, then reconciled

Budgets are enforced in Redis, so they hold across every gateway replica.

1. At admission, the request's estimated tokens and cost are reserved against the cap. Concurrent requests count each other's reservations, so together they cannot pass the cap.
2. At completion, the reservation is released and the real usage is committed, once.
3. A reservation that never settles, for example after a crash, expires after 10 minutes.

A request that fails upstream without usage is not billed; a 502 or 504 after retries refunds its reservation. Calls the gateway makes on a request's behalf (vision, structured output, tool loops, speculative drafts, image generation) count against the same tenant and key budgets.

The gateway raises an alert when a tenant reaches 80% of a cap (term_budget_warn:<tenant_id>) and when it is used up (term_budget_exhausted:<tenant_id>), and the same per key. See Alerting.

Access hours

Access hours limit when a tenant may send traffic: a start date, an end date, weekly windows, or any mix, in the tenant's time zone. Outside them, requests are refused with 403 and a message saying why (not active yet, expired, or outside its scheduled window).

On the tenant's page, drag across the week grid. Through the API, windows are a day (0 is Sunday) and start and end minutes from local midnight:

curl -X PATCH http://localhost:9180/api/v1/tenants/$TENANT_ID/schedule \
  -H "Authorization: Bearer $OBLETH_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"timezone": "America/Phoenix", "active_until": "2026-12-20T00:00:00Z",
       "weekly_windows": [{"day": 1, "start_min": 1080, "end_min": 1440}]}'

Model lists

A model list limits a tenant to the models you name. Requests for any other model get 403 model not permitted for tenant, GET /v1/models lists only the permitted ones, and auto routing picks only among them. An empty list permits every model.

curl -X PATCH http://localhost:9180/api/v1/tenants/$TENANT_ID/allowlist \
  -H "Authorization: Bearer $OBLETH_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"allowed_models": ["llama-3.3-70b-instruct", "bge-m3"]}'

How the limits combine

A request refused by a budget or the token rate gives its slot back at once and never reaches the backend.