Security

API key storage and verification

obleth never stores raw API keys. When a key is created:

1. The gateway generates a secret in the format sk_{24 random bytes hex-encoded}.
2. The secret is returned to the caller once and never stored.
3. A SHA-256 hash of the secret is stored in Postgres (api_keys.key_hash) and cached in Redis (obleth:key:{hash}).

On each request, the gateway hashes the incoming Authorization: Bearer ... value and looks up the hash. A stolen database or Redis dump does not expose valid API keys.

Optional pepper

Set OBLETH_API_KEY_PEPPER to mix a secret server-side value into every key hash. This is defense-in-depth: even if the config database leaks, an attacker cannot pre-compute or brute-force key hashes without also knowing the pepper. Keep it secret and stable — changing it invalidates all previously issued keys. Generate with openssl rand -hex 32.

Admin token

The Management API and control plane are protected by a single bearer token (OBLETH_ADMIN_TOKEN). This is intentionally simple — it's a service-to-service credential, not a user-facing RBAC system. There is no default; the gateway and control plane refuse to start without it. The token is compared in constant time to avoid timing side-channels.

Hardening:

• Use a randomly generated token of at least 32 characters: openssl rand -hex 32
• Store it in a Kubernetes Secret or a secrets manager (Vault, AWS Secrets Manager)
• Restrict access to :9180 at the network layer (firewall, Kubernetes NetworkPolicy)
• Rotate by updating the env var and rolling the pods; the control plane config is updated simultaneously

TLS architecture

obleth itself does not terminate TLS. TLS should be terminated at the edge by HAProxy or your Ingress controller.

Internet
  |  HTTPS (TLS)
  v
HAProxy / Ingress
  |  HTTP (internal only)
  v
obleth :8080

Never expose obleth's data plane or admin port directly to the internet. They are HTTP-only.

For internal service-to-service communication (e.g., control plane to Management API within a cluster), TLS is optional and typically omitted for pods within the same Kubernetes namespace.

Sensitive fields in Postgres

Two fields in the database contain credentials:

When OBLETH_ENCRYPTION_KEY is set (base64 of 32 random bytes), these values are encrypted at rest with AES-256-GCM before being written to Postgres. Encrypted values carry an enc:v1: prefix followed by a per-value 12-byte nonce; the gateway transparently decrypts them when resolving routes. Generate a key with:

openssl rand -base64 32

If OBLETH_ENCRYPTION_KEY is unset, these columns are stored in plaintext and a warning is logged at startup. Legacy plaintext rows are still readable after you enable a key (they are re-encrypted on next write). Additional mitigations:

• Enable encryption at rest for the Postgres volume/managed service.
• Restrict Postgres network access to obleth pods only.

Upstream URL validation (SSRF)

Admin-registered upstream URLs — a model's api_base and an MCP server's upstream_url — are validated when created or updated. obleth is built for self-hosted, local-first deployments where the upstreams you register usually live on the same private network (another cluster node, a LAN VM, a model server on localhost). So by default, private/RFC1918, loopback, CGNAT (100.64.0.0/10), and IPv6 unique-local targets are allowed — these are the addresses a local operator legitimately needs to reach.

What is always blocked is the genuinely dangerous class with no legitimate local-upstream use: link-local and the cloud metadata endpoint (169.254.0.0/16, including 169.254.169.254, and fe80::/10), the unspecified address, and broadcast/documentation ranges. Hostnames are resolved first, so a public name that maps to a blocked address is still rejected. This keeps a compromised or careless admin token from turning the gateway into a server-side request forgery (SSRF) pivot at cloud metadata.

For locked-down deployments that forward to untrusted upstreams, switch to strict mode with OBLETH_BLOCK_PRIVATE_NETWORKS=1. This rejects all private/internal targets unless their exact range is allow-listed in OBLETH_ALLOWED_PRIVATE_CIDRS:

OBLETH_BLOCK_PRIVATE_NETWORKS=1
OBLETH_ALLOWED_PRIVATE_CIDRS=10.0.0.0/8,192.168.0.0/16

Rejected URLs return 400 Bad Request from the Management API.

Request-path traversal

The upstream URL is built by joining the validated api_base with the inbound request path. As defense-in-depth, the data plane rejects request paths that contain traversal sequences — literal .. segments, backslash separators, and their percent-encoded forms (%2e, %2f, %5c) — with 400 Bad Request before the upstream call is made. This keeps a crafted path from escaping the configured upstream base onto a different path on the same host. The same guard applies to MCP gateway requests.

Key hashing implementation

Hash = SHA-256(raw_key_bytes)
Redis key = "obleth:key:" + hex(Hash)

The hash is computed in the obleth-proxy crate before any Redis lookup. The raw key value never touches the network after the initial creation response.

Fail-open security tradeoff

With OBLETH_FAIL_OPEN=true (default): if Redis is unavailable and the key is not in the moka cache, the request is served with no budget check. This means:

• A key that was deleted in Postgres/Redis may still be served from moka until its TTL expires (5 minutes).
• A budget that was exhausted may be bypassed while Redis is down.

With OBLETH_FAIL_OPEN=false: requests are rejected when Redis is unavailable. This provides strict budget enforcement at the cost of availability during Redis outages.

Choose based on your requirements:

• Multi-tenant deployments where availability matters most: fail-open (availability > strict budget enforcement during outages)
• Hard budget caps / compliance requirements: fail-closed

Control plane hardening

The Next.js control plane has several built-in protections:

• Required secrets, no weak defaults. A break-glass admin email (DASHBOARD_ADMIN_EMAIL), a password (DASHBOARD_PASSWORD or DASHBOARD_PASSWORD_HASH), and DASHBOARD_SESSION_SECRET must all be supplied. The app fails closed at startup if the session secret is missing or shorter than 32 characters. Prefer a bcrypt DASHBOARD_PASSWORD_HASH over a plaintext password.
• Login rate limiting. Sign-in attempts are limited per client IP (fixed window) to slow credential-stuffing and brute-force attempts.
• Security headers. Responses set a strict Content-Security-Policy, Strict-Transport-Security (HSTS), X-Frame-Options: DENY, X-Content-Type-Options: nosniff, Referrer-Policy, and Permissions-Policy.
• Server-side admin token. OBLETH_ADMIN_TOKEN is only ever used server-side and is never exposed to the browser.

Container hardening

The published Docker images run as a non-root user. In Compose, the Postgres, Redis, and ClickHouse ports are bound to 127.0.0.1 so the datastores are not reachable from outside the host. The Compose stack and Helm chart both require secrets to be set explicitly (${VAR:?} / Helm required) rather than shipping weak defaults. The data plane and control plane images also declare container HEALTHCHECKs (the data plane probes /health; the control plane probes its login page) so orchestrators can detect and restart unhealthy containers.

Kubernetes restricted Pod Security Standard

The Helm chart applies the Kubernetes restricted Pod Security Standard to its stateless workloads (obleth data plane, control-plane, benchmark-backend) by default — runAsNonRoot, seccompProfile: RuntimeDefault, allowPrivilegeEscalation: false, and all Linux capabilities dropped. UID is not pinned because the images ship distinct non-root users (obleth 10001, control-plane 1000); runAsNonRoot enforces non-root without hardcoding a UID. The bundled datastores are excluded on purpose — their official images manage their own users (the Postgres entrypoint must start as root). See Helm Values — Security contexts.

Pre-created Secrets and NetworkPolicy

For production, the chart can reference pre-created Secrets (obleth.existingSecret / controlPlane.existingSecret) so real credentials never enter values files or --set/CLI history. An opt-in NetworkPolicy (networkPolicy.enabled: true) restricts the bundled datastore ports to the obleth pods; it requires a CNI that enforces NetworkPolicy and is a no-op for external datastores. See the Production scenario.

Attack surface summary