# The soak `soak.mjs` runs one PDS, or several side by side, under a steady mixed load for as long as it is left, and checks as it goes that nothing answered is lost. Where the benchmark (`../README.md`) measures short, fresh rounds, the soak is for what only hours show: memory over time, the data directory's growth, what a crash in the middle of it costs, and whether every write still reads back. It needs Node, the version `.node-version` names, and nothing else. ## What it does Each server gets `ACCOUNTS` accounts (with an invite code made with its admin's password where one is given), then `RATE` operations a second, the same operations in the same order on each from `SEED`: | share | operations | | --: | --- | | 56% | writes: posts, likes, follows, a profile, deletes, `applyWrites` of five posts, a post with an image blob | | 34% | reads: `getRecord`, `listRecords`, `getLatestCommit`, `describeRepo`, `getRepo` | | 10% | `refreshSession`, `describeServer` | It keeps what each account holds as its writes were answered (a follow or like replacing the account's others of the same subject, as `spec/VALIDATION.md`, "Duplicates", says), and once a minute reads one account back whole, nothing else done to it meanwhile: a record answered and gone is **lost**, one with another CID a **mismatch**. A write whose answer never came, as when the server was killed, is settled by the next read back. A firehose subscriber on each server holds every commit answered to come through within two minutes (**missing commits**), and subscribes again from its last `seq` after a kill or a stall. Every five minutes it writes each operation's count, p50, p99 and outcomes, and the firehose's lag, to `events-.jsonl`, with each loss, mismatch, outage and 5xx as it happens, and its totals to `summary-.json`. ## Running it | Variable | Is | Default | | --- | --- | --- | | `TARGET_URL`, `TARGET_NAME`, `TARGET_DOMAIN`, `TARGET_ADMIN_PASSWORD` | one server: its base URL, a name for its files, the domain its handles end in (no dot), and its admin's password for invite codes | | | `SIDES` | several, in place of `TARGET_*`: a JSON list of `{name, url, domain, admin}` | | | `ACCOUNTS`, `RATE`, `SEED`, `RUN` | accounts on each, operations a second on each, the seed, and the handles' prefix (`-001`) | `50`, `6`, `1`, `soak1` | | `OUT` | where its files go; a `STOP` file there pauses it | `/out` | | `DURATION_MS` | how long the load runs; unset, until it is stopped | unset | | `W_MS`, `DATA_DIR` | with `DURATION_MS`: the server's backfill window and its data directory, for the settled check (below) | | | `SETTLE_ONLY` | the settled check alone, on the accounts of an earlier run in `OUT` | unset | It carries on after a restart of its own from `state-.json`: the accounts, signed in again, and what they hold, read from the server. Against a server of this machine from `harness/dev.mjs`, with the rate limits off so that one address may send it all: ```sh cd harness && PDS_RATE_LIMITS_ENABLED=false node dev.mjs --port 2594 OUT=/tmp/soak TARGET_URL=http://127.0.0.1:2594 TARGET_DOMAIN=test \ ACCOUNTS=20 RATE=20 node ../bench/soak/soak.mjs ``` Against a server on the internet, keep `RATE` under its `global` rate limit for one address (`spec/LIMITS.md`: 3000 in 5 minutes, so 10 a second) with room for the read-backs. ## The settled check With `DURATION_MS`, `W_MS` and `DATA_DIR`, the load stops after `DURATION_MS`, the soak waits `2 W`, writes once to each account, and answers, as one line of JSON, the data directory's size against `spec/STORE.md`'s settled bound: three times the accounts' live states (`getRepo` whole and the blobs each holds), plus 128 KiB an account and 1 MiB. `loadEndMiB` is the size when the load stopped, `leastMiB` the least in the 10 seconds after the writes, and `pass` whether that is within the bound. It exits 1 if not. The server must be started with `PDS_REPO_BACKFILL_LIMIT_MS` as `W_MS`, short, so that `2 W` is minutes. Across a restart: run with `DURATION_MS` and no `DATA_DIR` (the load alone), stop the server, wait `2 W`, start it again on the same data (`dev.mjs --keep`), and run with `SETTLE_ONLY=1` and `DATA_DIR`: each account's first use after the start is then that write. ## Results `RESULTS.md` holds the runs, each run's files in `data//`, and every table in it is the output of `analyze.mjs` for them: ```sh node bench/soak/analyze.mjs bench/soak/data/ [--split ]... [--from ] [--to ] ``` It reads what a run's directory holds: the soak's `summary-*.json` and `events-*.jsonl`; a sampler's `samples.jsonl` and `kills.jsonl`, where one ran beside it (each minute, each server's RSS, CPU and data directory, the free space, and Monolith's fsync histogram and the disk's flush counters); and Monolith's log cut to its `request`, `start`, `shutdown` and warning lines as `server-log.jsonl`, each gzipped or not. It answers the soak's totals and times, the kills, memory and data over time, the syncs against the disk's flushes, each method's times on the server, `createRecord`'s by period with the minute after each start apart, what a slow write held up, and the settled check. `--split` cuts the time into periods, as before and after a change. A run is added by putting its files in a directory of `data/`, a description and its command in `RESULTS.md`, and the command's output under it.