Security

Lock down the network

The gateway serves plain HTTP. Terminate TLS at your ingress or HAProxy, and expose only what clients need:

In Helm, networkPolicy.enabled: true enforces most of this for the bundled datastores and port 9180; it needs a CNI that enforces NetworkPolicy. In Compose, the datastores, Management API, metrics, Prometheus and Jaeger listen on 127.0.0.1; the data plane, dashboard, edge proxy, Grafana, benchmark backend and Dex listen on all interfaces. See Endpoints and ports.

API keys

A new key is sk_ and 48 hex characters. The secret is shown once and never stored: Postgres keeps its SHA-256 hash, and Redis caches it under that hash. A leaked database or Redis dump does not reveal usable keys.

Set OBLETH_API_KEY_PEPPER to mix a server-side secret into every hash, so hashes from a leaked database cannot be checked against guessed keys without it. Keep it stable: changing it invalidates every issued key. Generate it with openssl rand -hex 32.

Deleting or disabling a key evicts it from Redis and from every replica's in-process cache at once. If the Redis eviction fails, the call returns 502 instead of reporting success; see Reliability. Authentication never fails open during a Redis outage; budgets can, by choice. See Reliability.

The admin token

The Management API accepts one bearer token, OBLETH_ADMIN_TOKEN, compared in constant time. It is a service credential: people sign in to the dashboard, which uses the token server-side only. There is no default. Use at least 32 random characters (openssl rand -hex 32), keep it in a Secret, and rotate it by updating the Secret and rolling the gateway and dashboard pods.

Secrets at rest

With OBLETH_ENCRYPTION_KEY set (base64 of 32 random bytes, openssl rand -base64 32), these are encrypted with AES-256-GCM before they are written: model and endpoint api_key, MCP auth_header, upstream header values, the Slurm JWT, and the dashboard's own data-plane key. Without it they are stored in plaintext and the gateway logs a warning at start.

Encrypted values carry an enc:v1: prefix and untagged values are read as plaintext, so turning the key on later works, but old rows are encrypted only when next written: re-save them. A malformed key stops the gateway at boot.

The Management API never returns these secrets. Responses carry api_key_set or auth_header_set, alert settings report slack_webhook_set and password_set, and the Slurm settings show only the JWT's last four characters. The tenant portal gets models without upstream URLs or keys.

Upstream URL validation (SSRF)

Every outbound URL an admin registers is checked when it is saved: a model's api_base and verify_api_base, endpoint and MCP URLs, the speculation URL template, the energy Prometheus URL, the Slack webhook, the slurmrestd URL, every URL in a config restore, and at start-up each OBLETH_JWT_ISSUERS address. Hostnames are resolved, so a public name that points at a blocked address is refused too. Refusals return 400.

• Always blocked: link-local and cloud metadata addresses (169.254.0.0/16, fe80::/10, fd00:ec2::254, 100.100.100.200), the unspecified address, broadcast and documentation ranges, including IPv4 written in IPv6 forms. No allowlist entry reopens them.
• Allowed by default: private, loopback, CGNAT and IPv6 unique-local addresses, since the servers you run usually live there.
• Strict mode: OBLETH_BLOCK_PRIVATE_NETWORKS=1 blocks all private targets except the ranges in OBLETH_ALLOWED_PRIVATE_CIDRS:

OBLETH_BLOCK_PRIVATE_NETWORKS=1
OBLETH_ALLOWED_PRIVATE_CIDRS=10.0.0.0/8,172.16.0.0/12

Redirects are not followed to unchecked destinations, so register final URLs. On the data plane, request paths containing traversal sequences (.., backslashes, or their percent-encoded forms) are refused with 400 before any upstream call, on the MCP gateway too.

The dashboard

• It needs DASHBOARD_ADMIN_EMAIL, DASHBOARD_PASSWORD and a DASHBOARD_SESSION_SECRET of at least 32 characters, and refuses to start without the secret.
• Email sign-up is closed. The only local account is the break-glass admin; everyone else signs in through SSO and waits for approval. Sign-in attempts are rate-limited.
• Every page and live route checks the admin role itself. Pending accounts are refused on the auth service's admin API, and no one can deactivate or demote themselves or the last active admin.
• Pages are served with a nonce-based Content Security Policy (no unsafe-inline or unsafe-eval for scripts in production), HSTS, X-Frame-Options: DENY and X-Content-Type-Options: nosniff.
• Changes made in the dashboard are recorded with the acting user in the audit log. CSV exports neutralise spreadsheet formulas.

Containers and Kubernetes

Images run as non-root users. The chart applies the restricted Pod Security Standard to its stateless workloads by default (runAsNonRoot, RuntimeDefault seccomp, no privilege escalation, all capabilities dropped); the bundled datastores manage their own users. No secret, the Redis password included, ships with a default. See the Production checklist.