64 docs indexed

Production Checklist

Everything to verify before deploying obleth to production: datastores, secrets, TLS, capacity, monitoring, and backup.

This checklist covers the minimum steps to harden an obleth deployment for production traffic.

Datastores

  • Postgres: Use a managed service (CloudNativePG, RDS, Cloud SQL) or HA setup. Do not use the bundled Docker Compose Postgres in production.
  • Redis: Use Redis Sentinel or Redis Cluster for HA. obleth treats Redis as a hot cache — all data is derivable from Postgres — so a Redis failover is recoverable without data loss.
  • ClickHouse: Use a replicated cluster or a managed ClickHouse service. The WAL provides short-term durability for single-node outages.
  • All connection URLs set via environment variables or Kubernetes Secrets, not hardcoded.
  • On Kubernetes: point the chart at pre-created Secrets (obleth.existingSecret / controlPlane.existingSecret) so credentials never enter values files or --set/CLI history. See Helm Values — Pre-created Secrets.

Secrets

  • OBLETH_ADMIN_TOKEN: strong random token (≥32 chars). No default — the gateway will not start without it. Use a Kubernetes Secret or Vault. Never commit to source control.
  • OBLETH_ENCRYPTION_KEY: set to a base64 32-byte key (openssl rand -base64 32) so model api_key and MCP auth_header are AES-256-GCM encrypted at rest. Without it they are stored in plaintext.
  • OBLETH_API_KEY_PEPPER: (optional, recommended) set a stable secret to strengthen API-key hashes against a DB leak.
  • DASHBOARD_SESSION_SECRET: required, ≥32 characters. The dashboard fails closed if missing or too short.
  • DASHBOARD_ADMIN_EMAIL (valid email) / DASHBOARD_PASSWORD (≥8 chars): required to seed the break-glass admin. Prefer a bcrypt DASHBOARD_PASSWORD_HASH over a plaintext password. (DASHBOARD_USERNAME is no longer used.)
  • DATABASE_URL: required for the control plane — the auth tables (users, sessions) live here. Point it at the same database the gateway uses.
  • ClickHouse password: required (no default in Compose/Helm).
  • Postgres password: required (no default in Compose/Helm).
  • If running the Compose observability profile: set GRAFANA_ADMIN_PASSWORD (anonymous Grafana access is disabled and Compose errors out without it).
  • If running the Compose edge profile: change HAPROXY_STATS_USER / HAPROXY_STATS_PASSWORD from the dev examples — they protect the HAProxy /stats page.
  • Confirm the real .env is gitignored — only .env.example (labeled dev examples) should be committed.

TLS and network

  • TLS terminated at HAProxy or your Ingress controller — the data plane is HTTP-only.
  • obleth data plane (:8080) reachable only from HAProxy/Ingress, not from the internet.
  • Admin port (:9180) not publicly accessible — restrict to internal network or VPN. The admin token is the only auth and is compared in constant time.
  • Metrics port (:9091) accessible only to your Prometheus scraper.
  • SSRF policy: default (local-first) mode allows private/LAN api_base targets; link-local and cloud metadata are always blocked. For strict mode (OBLETH_BLOCK_PRIVATE_NETWORKS=1), set OBLETH_ALLOWED_PRIVATE_CIDRS to trusted internal ranges (e.g. 10.0.0.0/8) so cluster Services can be registered.
  • Control plane served over HTTPS; verify the built-in security headers (CSP, HSTS, X-Frame-Options) and login rate limiting are active.

Kubernetes workload hardening

These are toggles in the Helm chart, defaulted sensibly and tuned for the Production profile.

  • Restricted Pod Security Standard. The stateless workloads (obleth, control-plane, benchmark-backend) run with runAsNonRoot, seccompProfile: RuntimeDefault, allowPrivilegeEscalation: false, and all capabilities dropped. On by default — verify they aren't overridden.
  • Spread across nodes. affinity.antiAffinity: soft (or hard on multi-node clusters) so a single node failure can't take the whole data plane down.
  • PodDisruptionBudget. podDisruptionBudget.enabled: true keeps a minimum number of obleth pods available during node drains and rolling upgrades (rendered only when replicas > 1).
  • NetworkPolicy (bundled datastores only). If you run the bundled Postgres/Redis/ClickHouse, set networkPolicy.enabled: true to restrict their ports to obleth pods. Requires a CNI that enforces NetworkPolicy; a no-op for external datastores.

Capacity planning

  • Set OBLETH_GLOBAL_MAX_IN_FLIGHT to match real inference backend concurrency (start conservative: 64, increase based on queue depth).
  • Set per-model max_in_flight for routes that share a backend; use static mode with a manual cap for cloud APIs.
  • For self-hosted chat/embedding models, run capacity auto-tune in a quiet window before go-live (avoid probing busy production models — the probe sends real upstream load).
  • Enable HPA in Helm (hpa.enabled=true); set maxReplicas based on traffic patterns.
  • Remember: each obleth pod has its own independent concurrency budget. 3 pods × 64 = 192 total slots.
  • Size OBLETH_GLOBAL_MAX_IN_FLIGHT to the upstream's real concurrency so requests queue briefly rather than overwhelming the backend.

Reliability

  • Decide on OBLETH_FAIL_OPEN: true (keep serving under Redis failure) or false (strict budget enforcement).
  • Set OBLETH_WAL_PATH to a persistent volume path (not /tmp). The WAL must survive pod restarts.
  • Verify the WAL volume has adequate disk space for your traffic volume.

Monitoring

  • Prometheus scraping :9091/metrics.
  • Alert on obleth_queue_depth > threshold (admission saturation).
  • Alert on obleth_telemetry_dropped > 0 (WAL pressure).
  • Alert on obleth_requests_total{status="5xx"} spike (upstream failures).
  • Enable serviceMonitor.enabled=true if using Prometheus Operator.
  • (Optional) Set OBLETH_OTEL_ENDPOINT for distributed tracing.

Initial configuration

  • Create your tenant hierarchy and fairshare groups before sending production traffic.
  • Register all models (POST /api/v1/models or dashboard import) — a fresh Helm install has none. Use api_base ending in /v1 and the bare upstream_model name.
  • Mint tenant API keys — there is no shared proxy key; each client needs its own sk_... secret.
  • Set conservative tokens_per_minute quotas initially; increase based on observed usage.
  • Verify the audit log is recording changes: GET /api/v1/audit.

Pre-launch testing

  • Run the benchmark harness against staging to verify fairshare behavior.
  • Run the chaos harness against staging to verify fail-open and WAL replay.
  • Verify ClickHouse rows appear after test requests.
  • Smoke test the dashboard against the production Management API.