Production Checklist

This checklist covers the minimum steps to harden an obleth deployment for production traffic.

Datastores

• Postgres: Use a managed service (CloudNativePG, RDS, Cloud SQL) or HA setup. Do not use the bundled Docker Compose Postgres in production.
• Redis: Use Redis Sentinel or Redis Cluster for HA. obleth treats Redis as a hot cache — all data is derivable from Postgres — so a Redis failover is recoverable without data loss.
• ClickHouse: Use a replicated cluster or a managed ClickHouse service. The WAL provides short-term durability for single-node outages.
• All connection URLs set via environment variables or Kubernetes Secrets, not hardcoded.
• On Kubernetes: point the chart at pre-created Secrets (obleth.existingSecret / controlPlane.existingSecret) so credentials never enter values files or --set/CLI history. See Helm Values — Pre-created Secrets.

Secrets

• OBLETH_ADMIN_TOKEN: strong random token (≥32 chars). No default — the gateway will not start without it. Use a Kubernetes Secret or Vault. Never commit to source control.
• OBLETH_ENCRYPTION_KEY: set to a base64 32-byte key (openssl rand -base64 32) so model api_key and MCP auth_header are AES-256-GCM encrypted at rest. Without it they are stored in plaintext.
• OBLETH_API_KEY_PEPPER: (optional, recommended) set a stable secret to strengthen API-key hashes against a DB leak.
• DASHBOARD_SESSION_SECRET: required, ≥32 characters. The dashboard fails closed if missing or too short.
• DASHBOARD_ADMIN_EMAIL (valid email) / DASHBOARD_PASSWORD (≥8 chars): required to seed the break-glass admin. (DASHBOARD_USERNAME is no longer used.)
• DATABASE_URL: required for the control plane — the auth tables (users, sessions) live here. Point it at the same database the gateway uses.
• TRUSTED_ORIGINS: set when the dashboard is reached by a LAN IP or an alternate hostname rather than the exact BETTER_AUTH_URL — comma-separated origins. Without it those logins are rejected as invalid-origin. * trusts every origin and belongs on trusted private networks only.
• ClickHouse password: required (no default in Compose/Helm).
• Postgres password: required (no default in Compose/Helm).
• If running the Compose observability profile: set GRAFANA_ADMIN_PASSWORD (anonymous Grafana access is disabled and Compose errors out without it).
• If running the Compose edge profile: change HAPROXY_STATS_USER / HAPROXY_STATS_PASSWORD from the dev examples — they protect the HAProxy /stats page.
• Confirm the real .env is gitignored — only .env.example (labeled dev examples) should be committed.

TLS and network

• TLS terminated at HAProxy or your Ingress controller — the data plane is HTTP-only.
• obleth data plane (:8080) reachable only from HAProxy/Ingress, not from the internet.
• Admin port (:9180) not publicly accessible — restrict to internal network or VPN. The admin token is the only auth and is compared in constant time.
• Metrics port (:9091) accessible only to your Prometheus scraper.
• On Docker Compose: the stack publishes the Management API (9180), metrics (9091), Postgres (5432), Redis (6379), ClickHouse (8123/9000), Prometheus (9090), and Jaeger (16686/4317/4318) on 127.0.0.1 only. The data plane (8088), dashboard (3002), and the edge HAProxy (80) are published on all interfaces — put a TLS-terminating proxy or firewall in front of those. Verify no override widens the loopback binds.
• SSRF policy: default (local-first) mode allows private/LAN api_base targets; link-local and cloud metadata are always blocked. For strict mode (OBLETH_BLOCK_PRIVATE_NETWORKS=1), set OBLETH_ALLOWED_PRIVATE_CIDRS to trusted internal ranges (e.g. 10.0.0.0/8) so cluster Services can be registered.
• Control plane served over HTTPS; verify the built-in security headers (CSP, HSTS, X-Frame-Options) and login rate limiting are active.

Kubernetes workload hardening

These are toggles in the Helm chart, defaulted sensibly and tuned for the Production profile.

• Restricted Pod Security Standard. The stateless workloads (obleth, control-plane, benchmark-backend) run with runAsNonRoot, seccompProfile: RuntimeDefault, allowPrivilegeEscalation: false, and all capabilities dropped. On by default — verify they aren't overridden.
• Spread across nodes. affinity.antiAffinity: soft (or hard on multi-node clusters) so a single node failure can't take the whole data plane down.
• PodDisruptionBudget. podDisruptionBudget.enabled: true keeps a minimum number of obleth pods available during node drains and rolling upgrades (rendered only when replicas > 1).
• NetworkPolicy (bundled datastores only). If you run the bundled Postgres/Redis/ClickHouse, set networkPolicy.enabled: true to restrict their ports to obleth pods. Requires a CNI that enforces NetworkPolicy; a no-op for external datastores.

Capacity planning

• Set OBLETH_GLOBAL_MAX_IN_FLIGHT to match real inference backend concurrency (start conservative: 64, increase based on queue depth).
• Set per-model max_in_flight for routes that share a backend; use static mode with a manual cap for cloud APIs.
• For self-hosted chat/embedding models, run capacity auto-tune in a quiet window before go-live (avoid probing busy production models — the probe sends real upstream load).
• Decide the obleth replica count deliberately. The chart defaults to obleth.replicas: 1 because each pod enforces OBLETH_GLOBAL_MAX_IN_FLIGHT on its own — with one pod the configured ceiling is the ceiling that reaches your upstream. For redundancy, raise replicas and divide globalMaxInFlight by it (values-production.yaml uses 3 × 85).
• Leave hpa.enabled=false when the replica count is load-bearing — an autoscaled replica count moves the aggregate concurrency limit with no config change. To autoscale anyway, divide globalMaxInFlight by maxReplicas and accept that the upstream is under-subscribed below full scale-out.
• Remember: each obleth pod has its own independent concurrency budget. 3 pods × 64 = 192 total slots.
• Size OBLETH_GLOBAL_MAX_IN_FLIGHT to the upstream's real concurrency so requests queue briefly rather than overwhelming the backend.

Reliability

• Decide on OBLETH_FAIL_OPEN: true (keep serving under Redis failure) or false (strict budget enforcement).
• Set OBLETH_WAL_PATH to a persistent volume path (not /tmp). The WAL must survive pod restarts.
• Verify the WAL volume has adequate disk space for your traffic volume.

Monitoring

• Prometheus scraping :9091/metrics.
• Alert on obleth_queue_depth > threshold (admission saturation).
• Alert on obleth_telemetry_dropped > 0 (WAL pressure).
• Alert on obleth_requests_total{status="5xx"} spike (upstream failures).
• Enable serviceMonitor.enabled=true if using Prometheus Operator.
• (Optional) Set OBLETH_OTEL_ENDPOINT for distributed tracing.

Initial configuration

• Create your tenant hierarchy and fairshare groups before sending production traffic.
• Register all models (POST /api/v1/models or dashboard import) — a fresh Helm install has none. Use api_base ending in /v1 and the bare upstream_model name.
• Mint tenant API keys — there is no shared proxy key; each client needs its own sk_... secret.
• Set conservative tokens_per_minute quotas initially; increase based on observed usage.
• Verify the audit log is recording changes: GET /api/v1/audit.

Pre-launch testing

• Run the benchmark harness against staging to verify fairshare behavior.
• Run obench score against staging for a graded readiness scorecard, and keep the run as the baseline later runs are diffed against.
• Run the chaos harness against staging to verify fail-open and WAL replay.
• Verify ClickHouse rows appear after test requests.
• Smoke test the dashboard against the production Management API.