Benchmarks and chaos testing

obench seeds a set of models and tenants, drives load through the gateway, and ends with a verdict. It ships in the gateway repository.

Build and run

cargo build --release --manifest-path obench/Cargo.toml
./obench/target/release/obench --target demo --profile smoke --all

Pass both --target and --profile to run headless; run obench with neither for an interactive wizard that walks through the target, the models and what will be created, and shows a cost warning before a live run. It connects to the local Compose stack by default:

BENCHMARK_API_BASE is the address the gateway uses to reach the benchmark backend for the models a demo run seeds. Results go to BENCH_OUT_DIR (default /tmp/obleth-bench), and the run prints links to the Fairshare and Reports pages, built from UI_BASE.

Run against a Helm install

A demo run needs the admin and proxy URLs on the machine running obench, so forward the gateway and the benchmark backend from a release named obleth:

1. 1

   Forward the ports

   kubectl -n obleth port-forward svc/obleth-obleth 8080:8080 9180:9180 &
   kubectl -n obleth port-forward svc/obleth-benchmark-backend 8081:8081 &
   

   A port-forward reaches the pods directly, so it works with networkPolicy.enabled.

2. 2

   Point obench at them

   export ADMIN_TOKEN=<your obleth.adminToken>
   export PROXY_BASE=http://localhost:8080
   export BENCHMARK_API_BASE=http://obleth-benchmark-backend:8081
   export UI_BASE=https://dashboard.example.org   # for the printed links
   obench --target demo --profile smoke --all
   

   BENCHMARK_API_BASE is the in-cluster Service the gateway calls; without it, the seeded models point at the Compose name benchmark-backend and seeding fails with could not resolve host. --backend-base keeps its default, the forwarded port 8081.

Every forwarded request goes through the Kubernetes API server, so high-concurrency runs (extreme, auto and the capacity ramps of obench score) measure the port-forward as much as the gateway. For those numbers, run obench in a pod in the cluster whose localhost ports relay to the same Services, for example through a socat sidecar, because demo refuses any other host.

Targets and profiles

For headless live runs, obench reads live.config.json (or --config <path>):

{
  "proxy_url": "https://gateway.example.com",
  "models": ["my-model-a", "my-model-b"],
  "keys": [
    { "label": "tenant-a", "weight": 100, "secret": "${OBENCH_KEY_A}" },
    { "label": "tenant-b", "weight": 200, "secret": "${OBENCH_KEY_B}" }
  ]
}

Each key is one tenant; list two or more for real contention. A ${VAR} that is not set is an error, never an empty string.

Override any of it with --conc, --duration-s, --output-tokens, --input-tokens, --stream, --capacity (each seeded model's pool size) and --max-error-rate. Use --all for the whole fleet or --model <name> for one.

A run passes when the gateway stayed up and the error rate stayed under the profile's limit (0 for smoke, up to 0.05 for heavy), with no two 10-second windows in a row without a completion. On demo, obench also samples the scheduler and fails the run if a waiting tenant was never served, a pool or cap was exceeded, or a pool sat with free slots and a queue for more than a tenth of the run.

Score a deployment

obench score runs a set of sections, grades each from 0 to 100, and combines them into a weighted system score.

obench score --target demo --all
obench score --target live --all --max-conc 128

A section that is skipped or fails hands its weight to the others. Each run writes scorecard.json and scorecard.md, plus a dated copy under scorecards/; the next run for the same target diffs against the newest copy (or --baseline <path>), and a regression caps that section's score. --fail-under <n> exits non-zero below n or on any regression, for CI. --quick skips the capacity ramps, and --skip and --only take section names.

Compression savings (obench compression)

obench compression --model <name> --api-key sk_… sends fixed payloads (logs, JSON, code, prose, repeated context) three ways, with boons off, by default, and with x-obleth-boons: lossy, and reports the savings the gateway reports in x-obleth-compression. It changes the global boon settings for the run and always puts them back. See Boons.

Inject faults

The benchmark backend takes faults at runtime, per model or for all of them:

curl -X POST http://localhost:8081/control \
  -H "Content-Type: application/json" \
  -d '{"model": "turbo", "mode": "fail"}'

model is a substring of the model name, or *. fail answers 500, slow makes responses ten times slower, stall holds each response for 60 seconds (a stream sends its first chunk, then ends without finishing), and ok (or recover) clears the fault; * with ok clears every fault. The scorecard's resilience section uses this, runs first, and clears its fault even on Ctrl-C. --backend-base sets where obench finds the backend (default http://localhost:8081).

Pause the datastores

To check the gateway's behaviour under datastore failures, drive a long run and pause services from a second terminal:

obench --target demo --profile heavy --all

cd deploy/docker
docker compose pause redis        # then: docker compose unpause redis
docker compose pause clickhouse   # then: docker compose unpause clickhouse

Keep a Redis pause under 5 minutes, or obench's own keys fall out of the cache and the error rate climbs. A PASS after the pauses means the gateway rode them out. What each failure does is in Reliability.