~ knotless Search/
↑↓ move↵ open⇧↵ all results
0
0
105 lines · 5.3 KB Raw History

The soak

soak.mjs runs one PDS, or several side by side, under a steady mixed load for as long as it is left, and checks as it goes that nothing answered is lost. Where the benchmark (../README.md) measures short, fresh rounds, the soak is for what only hours show: memory over time, the data directory's growth, what a crash in the middle of it costs, and whether every write still reads back. It needs Node, the version .node-version names, and nothing else.

What it does

Each server gets ACCOUNTS accounts (with an invite code made with its admin's password where one is given), then RATE operations a second, the same operations in the same order on each from SEED:

shareoperations
56%writes: posts, likes, follows, a profile, deletes, applyWrites of five posts, a post with an image blob
34%reads: getRecord, listRecords, getLatestCommit, describeRepo, getRepo
10%refreshSession, describeServer

It keeps what each account holds as its writes were answered (a follow or like replacing the account's others of the same subject, as spec/VALIDATION.md, "Duplicates", says), and once a minute reads one account back whole, nothing else done to it meanwhile: a record answered and gone is lost, one with another CID a mismatch. A write whose answer never came, as when the server was killed, is settled by the next read back. A firehose subscriber on each server holds every commit answered to come through within two minutes (missing commits), and subscribes again from its last seq after a kill or a stall.

Every five minutes it writes each operation's count, p50, p99 and outcomes, and the firehose's lag, to events-<name>.jsonl, with each loss, mismatch, outage and 5xx as it happens, and its totals to summary-<name>.json.

Running it

VariableIsDefault
TARGET_URL, TARGET_NAME, TARGET_DOMAIN, TARGET_ADMIN_PASSWORDone server: its base URL, a name for its files, the domain its handles end in (no dot), and its admin's password for invite codes
SIDESseveral, in place of TARGET_*: a JSON list of {name, url, domain, admin}
ACCOUNTS, RATE, SEED, RUNaccounts on each, operations a second on each, the seed, and the handles' prefix (<RUN>-001)50, 6, 1, soak1
OUTwhere its files go; a STOP file there pauses it/out
DURATION_MShow long the load runs; unset, until it is stoppedunset
W_MS, DATA_DIRwith DURATION_MS: the server's backfill window and its data directory, for the settled check (below)
SETTLE_ONLYthe settled check alone, on the accounts of an earlier run in OUTunset

It carries on after a restart of its own from state-<name>.json: the accounts, signed in again, and what they hold, read from the server.

Against a server of this machine from harness/dev.mjs, with the rate limits off so that one address may send it all:

cd harness && PDS_RATE_LIMITS_ENABLED=false node dev.mjs --port 2594
OUT=/tmp/soak TARGET_URL=http://127.0.0.1:2594 TARGET_DOMAIN=test \
  ACCOUNTS=20 RATE=20 node ../bench/soak/soak.mjs

Against a server on the internet, keep RATE under its global rate limit for one address (spec/LIMITS.md: 3000 in 5 minutes, so 10 a second) with room for the read-backs.

The settled check

With DURATION_MS, W_MS and DATA_DIR, the load stops after DURATION_MS, the soak waits 2 W, writes once to each account, and answers, as one line of JSON, the data directory's size against spec/STORE.md's settled bound: three times the accounts' live states (getRepo whole and the blobs each holds), plus 128 KiB an account and 1 MiB. loadEndMiB is the size when the load stopped, leastMiB the least in the 10 seconds after the writes, and pass whether that is within the bound. It exits 1 if not.

The server must be started with PDS_REPO_BACKFILL_LIMIT_MS as W_MS, short, so that 2 W is minutes. Across a restart: run with DURATION_MS and no DATA_DIR (the load alone), stop the server, wait 2 W, start it again on the same data (dev.mjs --keep), and run with SETTLE_ONLY=1 and DATA_DIR: each account's first use after the start is then that write.

Results

RESULTS.md holds the runs, each run's files in data/<run>/, and every table in it is the output of analyze.mjs for them:

node bench/soak/analyze.mjs bench/soak/data/<run> [--split <ISO time>]... [--from <ISO>] [--to <ISO>]

It reads what a run's directory holds: the soak's summary-*.json and events-*.jsonl; a sampler's samples.jsonl and kills.jsonl, where one ran beside it (each minute, each server's RSS, CPU and data directory, the free space, and Monolith's fsync histogram and the disk's flush counters); and Monolith's log cut to its request, start, shutdown and warning lines as server-log.jsonl, each gzipped or not. It answers the soak's totals and times, the kills, memory and data over time, the syncs against the disk's flushes, each method's times on the server, createRecord's by period with the minute after each start apart, what a slow write held up, and the settled check. --split cuts the time into periods, as before and after a change.

A run is added by putting its files in a directory of data/, a description and its command in RESULTS.md, and the command's output under it.