64 docs indexed
Verify obleth's fail-open behavior and WAL replay by pausing Redis and ClickHouse while obench drives load.
obleth is designed to keep serving requests when Redis or ClickHouse blink. To verify this, drive a sustained load with obench and pause those services while the run is active.
| Scenario | Expected behavior |
|---|---|
| ClickHouse pause | Usage records spill to the local WAL; client requests continue |
| ClickHouse resume | WAL records replay into ClickHouse |
| Redis pause | Auth uses the in-process key cache; budget checks fail open; requests continue |
| Redis resume | Normal Redis-backed auth and budgeting resume |
Start a long-running benchmark so there is live traffic to disrupt. The heavy profile runs for 600 seconds at concurrency 64, which leaves plenty of room to pause and resume services mid-run:
obench --target demo --profile heavy --all
While that run is active, pause and unpause services in a second terminal (see below). obench's verdict still fails if tenants stop making progress or the client error rate exceeds the threshold, so a clean PASS after a pause/resume cycle is the proof that obleth failed open and recovered.
Pause Redis and watch obleth fall back to its in-process key cache and fail-open budgeting:
docker compose -f deploy/docker/docker-compose.yml pause redis
docker logs -f obleth-obleth-1
docker compose -f deploy/docker/docker-compose.yml unpause redis
Pause ClickHouse and confirm usage records spill to the write-ahead log:
docker compose -f deploy/docker/docker-compose.yml pause clickhouse
docker exec obleth-obleth-1 ls -lh /tmp/obleth-telemetry.wal
docker compose -f deploy/docker/docker-compose.yml unpause clickhouse
Replace docker with podman if you are using Podman Compose.
After ClickHouse resumes, check that buffered records were inserted:
docker exec -it obleth-clickhouse-1 clickhouse-client \
--user obleth --password obleth \
--query "SELECT count() FROM obleth.usage WHERE ts_ms > $(date -d '10 minutes ago' +%s)000"
The count should include records created during the outage, and obleth logs should report WAL replay.