64 docs indexed

Redis Layout

All Redis key patterns, TTLs, hash structures, and Lua scripts used by obleth.

Redis is obleth's hot cache and the real-time token budget store. All Redis data is derivable from Postgres and can be rebuilt on cache miss.

Key patterns

Key patternTypeTTLContents
obleth:key:{sha256_hex}String300sJSON blob of ResolvedKey (tenant, weight, group, quotas)
obleth:model:{model_name}String300sJSON blob of model config
obleth:mcp:{mcp_name}String300sJSON blob of MCP server config
obleth:budget:{tenant_uuid}Hashslidingtokens (remaining this window), ts (window start timestamp ms)
obleth:term_usage:{tenant_uuid}Hashperiod-scopedCumulative tokens and cost used in the current term-budget period
obleth:cache:{sha256_of_prompt}Stringmodel-configured TTLCached response body

ResolvedKey structure

{
  "key_id": "661f9500-...",
  "tenant_id": "550e8400-...",
  "tenant_name": "chatbot",
  "weight": 100,
  "tokens_per_minute": 50000,
  "max_in_flight": null,
  "fairshare_group": "default",
  "group_weight": 100,
  "disabled": false
}

The hot path reads only this blob; it carries everything fairshare admission and budgeting need without a relational lookup.

Token budget hash

The obleth:budget:{tenant_uuid} key is a Redis hash with two fields:

FieldTypeMeaning
tokensintegerTokens remaining in the current 60-second window
tsintegerWindow start time (Unix milliseconds)

On each request, a Lua script atomically:

  1. Checks if 60 seconds have elapsed since ts; if so, resets tokens to tpm.
  2. Subtracts the estimated token count from tokens.
  3. Returns the new balance or -1 if exhausted.

Term usage

For tenants with a term budget, obleth:term_usage:{tenant_uuid} tracks cumulative tokens and cost for the current period (lifetime, monthly, or term). obleth reads it before admission and adds the reconciled true cost after each stream; when either configured cap is met the data plane returns 403.

Response cache

obleth:cache:{sha256} stores the cached response for a chat completion request. The cache key is a SHA-256 of the canonicalized request body (model + messages + parameters). TTL is set per-model via cache_ttl_secs.

Pub/sub invalidation

Channel: obleth:invalidate

When a config change is made via the Management API, obleth publishes a JSON invalidation message to this channel. All pods subscribed to the channel delete their moka and Redis cache entries for the affected resource.

{
  "kind": "key",
  "id": "sha256-hex-of-changed-key"
}

Cache miss behavior

On a Redis miss for any obleth:key:* or obleth:model:* key, obleth:

  1. Looks up the moka in-process cache (TTL=5min)
  2. If moka misses, queries Postgres and re-populates Redis + moka

This means a Redis restart does not require any manual action — the cache self-heals on first use.