Token-measured Fairness

Most gateways count requests. obleth counts tokens. The difference matters because one inference request can consume anywhere from a few tokens to tens of thousands — a single long-context request can occupy a GPU slot 1000× longer than a short one.

The estimate-reserve-reconcile cycle

obleth can't know how many tokens a response will consume before it runs. It uses a three-step process:

1. Estimate

Before fairshare admission, obleth estimates the total token cost from the request body:

• Input tokens: HeuristicTokenizer counts ~4 characters per token, with 4 overhead tokens per message for role/formatting. Deterministic and fast.
• Output tokens: max_tokens (or max_completion_tokens) from the request, capped at 8192. When neither is set, the estimate is the input token count clamped to the range 512–8192, so a large prompt reserves a proportionally larger output allowance instead of a flat default.

The total (input + estimated_output) is the request's cost used for both the fairshare score and the Redis budget reservation.

2. Reserve (atomic, cross-pod)

After admission but before the upstream call, a Redis Lua script atomically:

1. Reads the tenant's token bucket: tokens + last-refill timestamp ts.
2. Refills: tokens = min(capacity, tokens + (now_ms - ts) × rate) where rate = tokens_per_minute / 60000.
3. Checks if tokens >= estimated_cost.
4. If yes: subtracts the estimate, stores the new state, returns allowed=1.
5. If no: returns allowed=0 → 429 token budget exceeded.

The same script also evaluates the tenant's cumulative term budget when one is configured, and it evaluates it first — so a term-exhausted request never reserves per-minute tokens it would have no completion path to refund. Both checks are one Redis round-trip. Correct across all pods with no application-level locking.

A tokens_per_minute of 0 means no per-minute cap: the script returns immediately without touching the bucket.

3. Reconcile

After the stream finishes, obleth reads actual token counts from the upstream's usage field and runs a second Lua script:

delta = estimated_output_tokens - actual_output_tokens
bucket_tokens = min(capacity, bucket_tokens + delta)

Positive delta (over-reserved) refunds tokens. Negative delta (under-reserved) charges more. The bucket can briefly go negative (bounded by −capacity) and is paid back over subsequent requests.

If the client disconnects before the upstream delivers final usage, the request is still settled: it is recorded with status 499 and reconciled against the admission estimate rather than being left unaccounted.

Why estimation accuracy matters less than you'd think

The estimate only affects admission ordering and budget reservation — not billing. Billing always uses reconciled actual cost. Slight estimation error just means the reservation is off, which reconciliation corrects. The heuristic is good enough for English text; for dense code or CJK content you may want a real BPE tokenizer.

Pluggable tokenizer

The Tokenizer trait in obleth-tokenizer is the seam:

pub trait Tokenizer: Send + Sync {
    fn count_text(&self, text: &str) -> u32;
    fn estimate_request(&self, body: &Value) -> CostEstimate { ... }
}

Drop in a tiktoken-rs or HuggingFace tokenizers implementation for model-accurate counting without changing anything else in the pipeline.

Token bucket parameters

A tenant with tokens_per_minute=2000000 can burst up to 2M tokens, then refills at ~33 tokens per millisecond. Setting tokens_per_minute=0 disables the per-minute cap entirely. Adjust via PUT /api/v1/tenants/{id}/quota.