Response cache

Turn it on

On the model's page, under Capacity, turn on Response cache and set how many seconds to keep answers (300 by default), then save. The change reaches every gateway replica at once.

curl -X PUT "http://localhost:9180/api/v1/models/$MODEL_ID/cache" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"cache_enabled": true, "cache_ttl_secs": 300}'

A cache_ttl_secs of 0 turns caching off for the model even with cache_enabled on.

What a hit does

A request that matches a stored answer is answered by the gateway: it takes no slot in the model's pool, reserves nothing against the tenant's budget, and never reaches the backend. The response carries x-obleth-cache: hit, and streaming responses are replayed as they were sent. The usage row records cache_status: hit with the cached token counts and their cost at the model's prices, but nothing is charged against the tenant's budgets.

A miss goes through admission to the backend as usual, and a 200 OK answer is stored with the model's lifetime.

What is never cached

• Answers larger than 512 KiB, which are passed through.
• Anything other than a 200 OK, so an error can't be replayed.
• Requests handled by the gateway tool loop, since their answer depends on live tool results.
• Requests from a tenant with output guardrails set to block or redact, so an answer stored before the policy can't bypass it.
• Anything across tenants: entries are keyed by tenant, model and request body.

Matching is exact: a request that differs by one byte misses, and there is no similarity matching. Turn the cache off, or keep the lifetime short, for models whose answers should vary from call to call.

See what it saves

The Response cache tile on Models shows the 24-hour hit rate and the tokens saved. Each request's cache_status (hit, miss or off) is in the usage ledger.

# Last 24 hours; add ?since_ms=<epoch ms> for another window
curl "http://localhost:9180/api/v1/usage/cache" \
  -H "Authorization: Bearer $TOKEN"
# {"hits": 1234, "misses": 5678, "tokens_saved": 987654}

Prometheus counts obleth_cache_lookups_total{result="hit|miss"} and obleth_cache_tokens_saved_total.