Operate Chat
Health checks, Prometheus metrics, alerts, backups and the limits that keep one replica from running away.
2 min readUpdated for Chat 0.5.2Edit on GitHub
Health checks
| Path | Answers when |
|---|---|
/healthz | The process is up. Use it for liveness |
/readyz | The process is up and the database answers. Use it for readiness and startup |
Every replica is the same. Postgres holds sessions, limits, running responses, approvals and the document queue, so a replica can stop at any time and another takes over. Stop, sign-out and a graceful shutdown save the partial output of running responses.
Metrics
/metrics is Prometheus text: HTTP requests by route and outcome, active streams, time to first token, generation outcomes and duration, and reported token counters. Labels are bounded: no usernames, conversation IDs, prompt text or arbitrary model names. Scrape it from your internal network only; deploy/prometheus.yaml is a sample scrape configuration. Set OBLETH_CHAT_METRICS_TOKEN and /metrics answers only requests with Authorization: Bearer <token>; others get 401.
Alerts
deploy/prometheus-alerts.yaml has sample rules. OblethChatBackupStale expects backup status files (below); drop it where nothing writes them, or it fires two hours after install.
Backups
Back up Chat's Postgres database: it holds everything except files in your bucket and workspace files, which are on Kubernetes volumes that Chat deletes 7 days after their last use. With a bucket configured, it holds Knowledge documents and, from v0.4.2, chat attachments and generated images, so back it up as well (see Where files are kept).
To have Chat report your backups, point OBLETH_CHAT_BACKUP_STATUS_DIR at a directory where your backup job writes two files after each success:
| File | Contents |
|---|---|
lastSuccess | Unix seconds of the last successful backup |
bytes | Its size in bytes |
Chat reads them at each scrape and exports obleth_chat_backup_last_success_timestamp_seconds and obleth_chat_backup_bytes. A missing or non-integer file leaves that gauge out.
Capacity
| Setting | Default | Meaning |
|---|---|---|
OBLETH_CHAT_MAX_STREAMS | 128 | Responses one replica streams at once (1 to 4096) |
OBLETH_CHAT_DB_CONNECTIONS | 20 | Postgres connections per replica (2 to 200) |
No database transaction stays open while Chat waits for tokens, so a replica with 128 streams needs far fewer than 128 connections. Per-person limits are in Models, groups and limits.
Upgrades
Chat applies new database migrations at startup. Take a backup before you upgrade, then roll out the new image. Responses running on a replica that stops are saved and marked interrupted; people can regenerate them.
What the tests cover
The repository's suites cover local auth and revocation, last-admin concurrency, OIDC against a local RSA issuer, group and model access, locked defaults, concurrent leases and quotas, reconnect persistence, gateway identity and streaming, thinking, cancellation, safe Markdown, private files and imports, and Knowledge. Run them with:
npm test
cargo check --all-targets
DATABASE_URL=postgresql://… TEST_DATABASE_URL=postgresql://… cargo test --all-targets
Use a disposable Postgres server; the Rust tests create isolated databases. scripts/run-load-fixture.py runs a load smoke test against a deterministic local gateway.