Chaos Testing

obleth is designed to keep serving requests when Redis or ClickHouse blink. To verify this, drive a sustained load with obench and pause those services while the run is active.

What chaos mode verifies

Drive a sustained load

Start a long-running benchmark so there is live traffic to disrupt. The heavy profile runs for 600 seconds at concurrency 64, which leaves plenty of room to pause and resume services mid-run:

obench --target demo --profile heavy --all

While that run is active, pause and unpause services in a second terminal (see below). obench's verdict still fails if tenants stop making progress or the client error rate exceeds the threshold, so a clean PASS after a pause/resume cycle is the proof that obleth failed open and recovered.

Pause and resume services

Pause Redis and watch obleth fall back to its in-process key cache and fail-open budgeting:

cd deploy/docker            # so Compose auto-loads .env
docker compose pause redis
docker logs -f obleth-obleth-1
docker compose unpause redis

Pause ClickHouse and confirm usage records spill to the write-ahead log:

docker compose pause clickhouse
docker exec obleth-obleth-1 ls -lh /tmp/obleth-telemetry.wal
docker compose unpause clickhouse

The WAL path comes from OBLETH_WAL_PATH, which the Compose stack sets to /tmp/obleth-telemetry.wal — the container's working directory is not writable by the non-root runtime user, so the default relative path would drop records instead of spilling them. In production point it at a persistent volume.

Replace docker with podman if you are using Podman Compose.

Verify WAL replay

After ClickHouse resumes, check that buffered records were inserted:

docker exec -it obleth-clickhouse-1 clickhouse-client \
  --user "$CLICKHOUSE_USER" --password "$CLICKHOUSE_PASSWORD" \
  --query "SELECT count() FROM obleth.usage WHERE ts_ms > $(date -d '10 minutes ago' +%s)000"

The count should include records created during the outage, and obleth logs should report WAL replay. CLICKHOUSE_USER / CLICKHOUSE_PASSWORD are the values from deploy/docker/.env — the image restricts the built-in default user to localhost, so the stack provisions the obleth user instead.

For a repeatable, graded version of the same idea, obench score's resilience section injects a real backend fault through the demo backend's POST /control endpoint and measures detection and recovery times. See Benchmark Suite — the deployment scorecard.