The soak
soak.mjs runs one PDS, or several side by side, under a steady mixed load
for as long as it is left, and checks as it goes that nothing answered is
lost. Where the benchmark (../README.md) measures short, fresh rounds, the
soak is for what only hours show: memory over time, the data directory's
growth, what a crash in the middle of it costs, and whether every write still
reads back. It needs Node, the version .node-version names, and nothing
else.
What it does
Each server gets ACCOUNTS accounts (with an invite code made with its
admin's password where one is given), then RATE operations a second, the
same operations in the same order on each from SEED:
| share | operations |
|---|---|
| 56% | writes: posts, likes, follows, a profile, deletes, applyWrites of five posts, a post with an image blob |
| 34% | reads: getRecord, listRecords, getLatestCommit, describeRepo, getRepo |
| 10% | refreshSession, describeServer |
It keeps what each account holds as its writes were answered (a follow or
like replacing the account's others of the same subject, as
spec/VALIDATION.md, "Duplicates", says), and once a minute reads one
account back whole, nothing else done to it meanwhile: a record answered and
gone is lost, one with another CID a mismatch. A write whose answer
never came, as when the server was killed, is settled by the next read back.
A firehose subscriber on each server holds every commit answered to come
through within two minutes (missing commits), and subscribes again from
its last seq after a kill or a stall.
Every five minutes it writes each operation's count, p50, p99 and outcomes,
and the firehose's lag, to events-<name>.jsonl, with each loss, mismatch,
outage and 5xx as it happens, and its totals to summary-<name>.json.
Running it
| Variable | Is | Default |
|---|---|---|
TARGET_URL, TARGET_NAME, TARGET_DOMAIN, TARGET_ADMIN_PASSWORD | one server: its base URL, a name for its files, the domain its handles end in (no dot), and its admin's password for invite codes | |
SIDES | several, in place of TARGET_*: a JSON list of {name, url, domain, admin} | |
ACCOUNTS, RATE, SEED, RUN | accounts on each, operations a second on each, the seed, and the handles' prefix (<RUN>-001) | 50, 6, 1, soak1 |
OUT | where its files go; a STOP file there pauses it | /out |
DURATION_MS | how long the load runs; unset, until it is stopped | unset |
W_MS, DATA_DIR | with DURATION_MS: the server's backfill window and its data directory, for the settled check (below) | |
SETTLE_ONLY | the settled check alone, on the accounts of an earlier run in OUT | unset |
It carries on after a restart of its own from state-<name>.json: the
accounts, signed in again, and what they hold, read from the server.
Against a server of this machine from harness/dev.mjs, with the rate
limits off so that one address may send it all:
cd harness && PDS_RATE_LIMITS_ENABLED=false node dev.mjs --port 2594
OUT=/tmp/soak TARGET_URL=http://127.0.0.1:2594 TARGET_DOMAIN=test \
ACCOUNTS=20 RATE=20 node ../bench/soak/soak.mjs
Against a server on the internet, keep RATE under its global rate limit
for one address (spec/LIMITS.md: 3000 in 5 minutes, so 10 a second) with
room for the read-backs.
The settled check
With DURATION_MS, W_MS and DATA_DIR, the load stops after
DURATION_MS, the soak waits 2 W, writes once to each account, and
answers, as one line of JSON, the data directory's size against
spec/STORE.md's settled bound: three times the accounts' live states
(getRepo whole and the blobs each holds), plus 128 KiB an account and
1 MiB. loadEndMiB is the size when the load stopped, leastMiB the least
in the 10 seconds after the writes, and pass whether that is within the
bound. It exits 1 if not.
The server must be started with PDS_REPO_BACKFILL_LIMIT_MS as W_MS,
short, so that 2 W is minutes. Across a restart: run with DURATION_MS and
no DATA_DIR (the load alone), stop the server, wait 2 W, start it again
on the same data (dev.mjs --keep), and run with SETTLE_ONLY=1 and
DATA_DIR: each account's first use after the start is then that write.
Results
RESULTS.md holds the runs, each run's files in data/<run>/, and every
table in it is the output of analyze.mjs for them:
node bench/soak/analyze.mjs bench/soak/data/<run> [--split <ISO time>]... [--from <ISO>] [--to <ISO>]
It reads what a run's directory holds: the soak's summary-*.json and
events-*.jsonl; a sampler's samples.jsonl and kills.jsonl, where one
ran beside it (each minute, each server's RSS, CPU and data directory, the
free space, and Monolith's fsync histogram and the disk's flush counters);
and Monolith's log cut to its request, start, shutdown and warning
lines as server-log.jsonl, each gzipped or not. It answers the soak's
totals and times, the kills, memory and data over time, the syncs against
the disk's flushes, each method's times on the server, createRecord's by
period with the minute after each start apart, what a slow write held up,
and the settled check. --split cuts the time into periods, as before and
after a change.
A run is added by putting its files in a directory of data/, a
description and its command in RESULTS.md, and the command's output
under it.