Security

API key storage and verification

obleth never stores raw API keys. When a key is created:

1. The gateway generates a secret in the format sk_{24 random bytes hex-encoded}.
2. The secret is returned to the caller once and never stored.
3. A SHA-256 hash of the secret is stored in Postgres (api_keys.key_hash) and cached in Redis (obleth:key:{hash}).

On each request, the gateway hashes the incoming Authorization: Bearer ... value and looks up the hash. A stolen database or Redis dump does not expose valid API keys.

Optional pepper

Set OBLETH_API_KEY_PEPPER to mix a secret server-side value into every key hash. This is defense-in-depth: even if the config database leaks, an attacker cannot pre-compute or brute-force key hashes without also knowing the pepper. Keep it secret and stable — changing it invalidates all previously issued keys. Generate with openssl rand -hex 32.

Admin token

The Management API and control plane are protected by a single bearer token (OBLETH_ADMIN_TOKEN). This is intentionally simple — it's a service-to-service credential, not a user-facing RBAC system. There is no default; the gateway and control plane refuse to start without it. The token is compared in constant time to avoid timing side-channels.

Hardening:

• Use a randomly generated token of at least 32 characters: openssl rand -hex 32
• Store it in a Kubernetes Secret or a secrets manager (Vault, AWS Secrets Manager)
• Restrict access to :9180 at the network layer (firewall, Kubernetes NetworkPolicy)
• Rotate by updating the env var and rolling the pods; the control plane config is updated simultaneously

TLS architecture

obleth itself does not terminate TLS. TLS should be terminated at the edge by HAProxy or your Ingress controller.

Internet
  |  HTTPS (TLS)
  v
HAProxy / Ingress
  |  HTTP (internal only)
  v
obleth :8080

Never expose obleth's data plane or admin port directly to the internet. They are HTTP-only.

For internal service-to-service communication (e.g., control plane to Management API within a cluster), TLS is optional and typically omitted for pods within the same Kubernetes namespace.

Sensitive fields in Postgres

Two fields in the database contain credentials:

When OBLETH_ENCRYPTION_KEY is set (base64 of 32 random bytes), these values are encrypted at rest with AES-256-GCM before being written to Postgres. Encrypted values carry an enc:v1: prefix followed by a per-value 12-byte nonce; the gateway transparently decrypts them when resolving routes. Generate a key with:

openssl rand -base64 32

If OBLETH_ENCRYPTION_KEY is unset, the cipher is disabled: these columns are stored in plaintext, and a warning is logged at startup. Nothing else changes — the gateway starts and serves normally — so treat setting the key as a deliberate production step, not something the gateway will force on you.

Decryption is prefix-driven: a value carrying enc:v1: is decrypted, and an untagged value is passed through as-is. That is what makes migration painless (legacy plaintext rows keep working once you enable a key, and are re-encrypted on their next write), and it is also why enabling a key does not retroactively protect rows you never write again — rotate or re-save those secrets.

Additional mitigations:

• Enable encryption at rest for the Postgres volume/managed service.
• Restrict Postgres network access to obleth pods only.

A wrong or malformed key is not silently tolerated: an enc:v1: value that fails to decrypt is an error, and a key that is present but not valid base64 of 32 bytes fails the process at boot.

Upstream URL validation (SSRF)

Admin-registered upstream URLs — a model's api_base and an MCP server's upstream_url — are validated when created or updated. obleth is built for self-hosted, local-first deployments where the upstreams you register usually live on the same private network (another cluster node, a LAN VM, a model server on localhost). So by default, private/RFC1918, loopback, CGNAT (100.64.0.0/10), and IPv6 unique-local targets are allowed — these are the addresses a local operator legitimately needs to reach.

What is always blocked is the genuinely dangerous class with no legitimate local-upstream use: link-local and the cloud metadata endpoints (169.254.0.0/16, including 169.254.169.254; fe80::/10; and AWS's IPv6 IMDS address fd00:ec2::254), the unspecified address, and broadcast/documentation ranges. Hostnames are resolved first, so a public name that maps to a blocked address is still rejected. This keeps a compromised or careless admin token from turning the gateway into a server-side request forgery (SSRF) pivot at cloud metadata.

This class is checked before the allowlist, so a broad OBLETH_ALLOWED_PRIVATE_CIDRS entry (say 0.0.0.0/0 or 169.254.0.0/16) cannot reopen it.

The same policy covers every admin-supplied destination the gateway will later call — not just a model's api_base and an MCP server's upstream_url, but also the energy Prometheus URL, the Slack alert webhook, and the slurmrestd URL. It is also applied to an entire configuration restore before any row is written. Redirects are not followed to unvalidated destinations: configure final upstream URLs directly.

For locked-down deployments that forward to untrusted upstreams, switch to strict mode with OBLETH_BLOCK_PRIVATE_NETWORKS=1. This rejects all private/internal targets unless their exact range is allow-listed in OBLETH_ALLOWED_PRIVATE_CIDRS:

OBLETH_BLOCK_PRIVATE_NETWORKS=1
OBLETH_ALLOWED_PRIVATE_CIDRS=10.0.0.0/8,192.168.0.0/16

Rejected URLs return 400 Bad Request from the Management API.

Request-path traversal

The upstream URL is built by joining the validated api_base with the inbound request path. As defense-in-depth, the data plane rejects request paths that contain traversal sequences — literal .. segments, backslash separators, and their percent-encoded forms (%2e, %2f, %5c) — with 400 Bad Request before the upstream call is made. This keeps a crafted path from escaping the configured upstream base onto a different path on the same host. The same guard applies to MCP gateway requests.

Key hashing implementation

Hash = SHA-256(raw_key_bytes)
Redis key = "obleth:key:" + hex(Hash)

The hash is computed in the obleth-proxy crate before any Redis lookup. The raw key value never touches the network after the initial creation response.

Fail-open security tradeoff

OBLETH_FAIL_OPEN governs the budget checks, not identity. A request whose key is in neither the moka cache nor Redis is always rejected 401 — the gateway never assumes an identity it could not resolve.

With OBLETH_FAIL_OPEN=true (default), a Redis failure at the budget step is logged, alerted, and the request proceeds unmetered. This means:

• A key that was deleted or disabled may still be served from moka until its TTL expires (5 minutes), if the invalidation message did not reach that pod.
• A budget that was exhausted may be bypassed while Redis is down.

With OBLETH_FAIL_OPEN=false, those requests are rejected with 503 instead. This provides strict budget enforcement at the cost of availability during Redis outages.

Choose based on your requirements:

• Multi-tenant deployments where availability matters most: fail-open (availability > strict budget enforcement during outages)
• Hard budget caps / compliance requirements: fail-closed

Control plane hardening

The Next.js control plane has several built-in protections:

• Required secrets, no weak defaults. A break-glass admin email (DASHBOARD_ADMIN_EMAIL), a password (DASHBOARD_PASSWORD), and DASHBOARD_SESSION_SECRET must all be supplied. The app fails closed at startup if the session secret is missing or shorter than 32 characters. The seeded password is stored hashed by the auth layer; rotate it after first sign-in and prefer SSO for day-to-day access.
• Login rate limiting. Sign-in attempts are limited per client IP (fixed window) to slow credential-stuffing and brute-force attempts.
• Nonce-based Content Security Policy. Every dashboard document is served with a per-request CSP carrying a fresh 128-bit nonce. Scripts run only by nonce (script-src 'self' 'nonce-…' 'strict-dynamic'); there is no 'unsafe-inline' and no 'unsafe-eval' in production, so an injected inline script does not execute. Development builds add 'unsafe-eval' for React Refresh and source maps only. The policy also sets default-src 'self', frame-ancestors 'none', object-src 'none', base-uri 'self', form-action 'self', and connect-src 'self'. Styles remain on 'unsafe-inline': the Next.js font loader, Recharts, and the shadcn primitives write style attributes a nonce cannot cover.
• Other security headers. Responses also set Strict-Transport-Security (HSTS), X-Frame-Options: DENY, X-Content-Type-Options: nosniff, Referrer-Policy, and Permissions-Policy.
• Server-side admin token. OBLETH_ADMIN_TOKEN is only ever used server-side and is never exposed to the browser.

Container hardening

The published Docker images run as a non-root user. In Compose, every port except the data plane, the dashboard, and the edge proxy is published on 127.0.0.1 only — Postgres, Redis, and ClickHouse, and also the Management API (9180), the metrics endpoint (9091), and the bundled Prometheus and Jaeger. They are reachable from the host, not from the network. The Compose stack and Helm chart both require secrets to be set explicitly (${VAR:?} / Helm required) rather than shipping weak defaults. The data plane and control plane images also declare container HEALTHCHECKs (the data plane probes /health; the control plane probes its login page) so orchestrators can detect and restart unhealthy containers.

Kubernetes restricted Pod Security Standard

The Helm chart applies the Kubernetes restricted Pod Security Standard to its stateless workloads (obleth data plane, control-plane, benchmark-backend) by default — runAsNonRoot, seccompProfile: RuntimeDefault, allowPrivilegeEscalation: false, and all Linux capabilities dropped. UID is not pinned because the images ship distinct non-root users (obleth 10001, control-plane 1000); runAsNonRoot enforces non-root without hardcoding a UID. The bundled datastores are excluded on purpose — their official images manage their own users (the Postgres entrypoint must start as root). See Helm Values — Security contexts.

Pre-created Secrets and NetworkPolicy

For production, the chart can reference pre-created Secrets (obleth.existingSecret / controlPlane.existingSecret) so real credentials never enter values files or --set/CLI history. An opt-in NetworkPolicy (networkPolicy.enabled: true) restricts the bundled datastore ports to the obleth pods; it requires a CNI that enforces NetworkPolicy and is a no-op for external datastores. See the Production scenario.

Outbound alert transport

Alert delivery is an outbound connection from the gateway, and its confidentiality depends on how you configure it:

• Slack uses the HTTPS incoming-webhook URL you supply, validated against the destination policy above when saved.
• Email uses STARTTLS when the starttls setting is on (the default). With it off, the gateway opens a plain, unencrypted SMTP connection — the alert body and any configured SMTP username and password cross the network in the clear. Only disable STARTTLS for a relay on a trusted network path.

The Slack webhook URL and the SMTP password are write-only over the API and the dashboard: reads report only whether each is set.

Attack surface summary