Security

Security model for obleth: key hashing, admin token hardening, TLS architecture, and fail-open tradeoffs.

API key storage and verification

obleth never stores raw API keys. When a key is created:

  1. The gateway generates a secret in the format sk_{24 random bytes hex-encoded}.
  2. The secret is returned to the caller once and never stored.
  3. A SHA-256 hash of the secret is stored in Postgres (api_keys.key_hash) and cached in Redis (obleth:key:{hash}).

On each request, the gateway hashes the incoming Authorization: Bearer ... value and looks up the hash. A stolen database or Redis dump does not expose valid API keys.

Admin token

The Management API and control plane are protected by a single bearer token (OBLETH_ADMIN_TOKEN). This is intentionally simple — it's a service-to-service credential, not a user-facing RBAC system.

Hardening:

  • Use a randomly generated token of at least 32 characters: openssl rand -hex 32
  • Store it in a Kubernetes Secret or a secrets manager (Vault, AWS Secrets Manager)
  • Restrict access to :9090 at the network layer (firewall, Kubernetes NetworkPolicy)
  • Rotate by updating the env var and rolling the pods; the control plane config is updated simultaneously

TLS architecture

obleth itself does not terminate TLS. TLS should be terminated at the edge by HAProxy or your Ingress controller.

Internet
  |  HTTPS (TLS)
  v
HAProxy / Ingress
  |  HTTP (internal only)
  v
obleth :8080

Never expose obleth's data plane or admin port directly to the internet. They are HTTP-only.

For internal service-to-service communication (e.g., control plane to Management API within a cluster), TLS is optional and typically omitted for pods within the same Kubernetes namespace.

Sensitive fields in Postgres

Two fields in the database contain credentials:

FieldTableWhat it is
api_keymodelsUpstream API key for the inference backend
auth_headermcp_serversAuth header value for MCP server calls

These are stored as plaintext in Postgres. Mitigations:

  • Enable encryption at rest for the Postgres volume/managed service.
  • Restrict Postgres network access to obleth pods only.
  • Use column-level encryption (e.g., pgcrypto) if your compliance requirements demand it.

Key hashing implementation

Hash = SHA-256(raw_key_bytes)
Redis key = "obleth:key:" + hex(Hash)

The hash is computed in the obleth-proxy crate before any Redis lookup. The raw key value never touches the network after the initial creation response.

Fail-open security tradeoff

With OBLETH_FAIL_OPEN=true (default): if Redis is unavailable and the key is not in the moka cache, the request is served with no budget check. This means:

  • A key that was deleted in Postgres/Redis may still be served from moka until its TTL expires (5 minutes).
  • A budget that was exhausted may be bypassed while Redis is down.

With OBLETH_FAIL_OPEN=false: requests are rejected when Redis is unavailable. This provides strict budget enforcement at the cost of availability during Redis outages.

Choose based on your requirements:

  • Consumer-facing AI products: fail-open (availability > strict billing during outages)
  • Hard billing caps / compliance requirements: fail-closed

Attack surface summary

SurfaceExposureMitigations
Data plane :8080Public (via HAProxy/Ingress)API key auth, TLS at edge
Admin API :9090Internal onlyBearer token, network restriction
Metrics :9091Internal only (Prometheus)Network restriction
PostgresInternal onlyStrong password, encryption at rest
RedisInternal onlyStrong password (if enabled), network restriction
ClickHouseInternal onlyStrong password, network restriction