# Soak results Each run's files are in `data//`, and every table below is `analyze.mjs`'s output for them, as the command above it gives it: run it again to check them. The files: `summary-*.json` and `events-*.jsonl.gz`, the soak's; where the machine was sampled, `samples.jsonl.gz` (each minute: each server's RSS, CPU and data directory, the free space, and from the second run Monolith's fsync histogram and the disk's counters) and `kills.jsonl`; and `server-log.jsonl.gz`, Monolith's log (`PDS_LOG_DIR`) cut to its `request`, `start`, `shutdown` and warning lines. The soak's own times ("The soak's times") are from a driver capped at half a CPU, serving both servers: compare them between servers, and take a server's own from its log. ## What they show - No write answered was lost and no commit answered missed the firehose, on either server, across nine kills. Night 1's `lost` and `mismatches` are the soak's own, from before it knew that a follow or like replaces the account's others of the same subject ("nine hours"). - At rest Monolith holds its live state, 1.08 times it; the reference five times it ("under load and at rest"), and again on `08a1db4`, 1.08 against 4.66 ("under load and at rest, on 08a1db4"). Under a day's window and hours of writes Monolith held about twice the reference's ("nine hours"). - On the network volume a slow sync held up every write under way, with Monolith's slow syncs and slow writes following the volume's flush time (r 0.91 and 0.90); on the server's own disk neither did, and `createRecord`'s p99 fell from 162 ms to 17 ms ("the volume, then the server's own disk"). - In the minute after a start, a crash's or a clean one's, writes took up to one or two seconds ("createRecord by period"). - Across a restart, `1707f04` did not settle, and `3218378` does ("settled across a restart"). ## The machine A Hetzner Cloud server in Falkenstein (`fsn1`): 8 vCPUs, 15 GiB, other services running on it throughout. Each PDS in a container capped at 1 CPU and 1 GiB, the soak in one capped at 0.5 CPU and 512 MiB, all on one Docker network, the soak calling the servers directly. Both PDSes make their DIDs in a PLC directory of their own on the same network, and tell no relay of themselves. The reference is `@atproto/pds` `0.0.0-spaces-alpha-20261029171913`. A kill is SIGKILL to the container's main process, its restart policy starting it again. ## 2026-11-13: nine hours side by side Monolith `1707f04` and the reference, each with its data on a Hetzner Volume (network block storage); 50 accounts and 6 operations a second on each, from 07:08 to 16:14 UTC; four kills of each, in turn, 40 to 80 minutes apart. The soak did not yet know that a follow or like replaces the account's others of the same subject, as both servers do in the same commit (`spec/VALIDATION.md`, "Duplicates"): its `lost` and `mismatch` here are those records, which the firehose showed each server deleting, not records lost. ```sh node bench/soak/analyze.mjs bench/soak/data/night1 --to 2026-11-13T16:14 ``` ### Totals (summary-*.json) | | monolith | reference | | --- | --: | --: | | from, to | 07:08–16:13 | 07:08–16:13 | | accounts, records | 50, 110,175 | 50, 109,868 | | operations | 209,823 | 209,710 | | 5xx | 0 | 0 | | rate limited (429) | 0 | 0 | | lost | 4,660 | 4,510 | | mismatches | 213 | 200 | | commits not on the firehose | 0 | 0 | | outages, their time | 3, 9.4 s | 4, 31.1 s | ### The soak's times, by five-minute window (events-*.jsonl) Each window has its own p50 and p99 per operation; the median of them, and the worst p99. Times are the soak's, from request to answer. | operation | monolith p50 | monolith p99 | monolith worst p99 | reference p50 | reference p99 | reference worst p99 | | --- | --: | --: | --: | --: | --: | --: | | post | 9.9 | 289.3 | 1797.4 | 72.9 | 650.7 | 4059.9 | | like | 12.9 | 230.3 | 2598.5 | 75.4 | 649.9 | 4373.4 | | applyWrites | 13.7 | 307.1 | 2646.7 | 89.2 | 710.5 | 4484.2 | | delete | 11.1 | 226.7 | 1928.3 | 75.8 | 628.0 | 3866.6 | | getRecord | 2.5 | 70.4 | 722.2 | 12.2 | 353.2 | 3544.1 | | listRecords | 3.8 | 84.1 | 1812.6 | 18.0 | 403.7 | 2348.6 | | getLatestCommit | 2.1 | 100.2 | 2536.2 | 11.4 | 407.5 | 2950.8 | | getRepo | 17.6 | 43.1 | 2844.7 | 103.0 | 516.2 | 5539.1 | #### Events by kind | kind | monolith | reference | | --- | --: | --: | | down | 3 | 4 | | found | 0 | 1 | | lost | 4,660 | 4,510 | | mismatch | 213 | 200 | | ready | 1 | 1 | | resumed | 1 | 1 | | seq jump | 5 | 0 | | up | 3 | 4 | ### Kills (kills.jsonl) | container | kills | serving again after | | --- | --: | --: | | spec-pds-pds-1 | 4 | 0.7–0.8 s | | opensocial-next-pds-1 | 4 | 0.6–0.9 s | ### Memory and data over time (samples.jsonl), MiB | container | RSS first, most, last | data first, most, last | | --- | --: | --: | | spec-pds-pds-1 | 24, 196, 195 | 19, 1,601, 1,597 | | opensocial-next-pds-1 | 191, 290, 284 | 37, 880, 880 | | hour (last sample) | spec-pds-pds-1 RSS | spec-pds-pds-1 data | opensocial-next-pds-1 RSS | opensocial-next-pds-1 data | free GiB | | --- | --: | --: | --: | --: | --: | | 07:59 | 55 | 127 | 275 | 114 | 13 | | 08:59 | 82 | 266 | 256 | 203 | 13 | | 09:58 | 110 | 438 | 282 | 294 | 13 | | 10:59 | 115 | 588 | 289 | 386 | 12 | | 11:59 | 149 | 806 | 275 | 476 | 12 | | 12:59 | 141 | 957 | 284 | 570 | 12 | | 13:59 | 162 | 1,184 | 284 | 665 | 12 | | 14:59 | 176 | 1,414 | 290 | 759 | 11 | | 15:59 | 180 | 1,560 | 283 | 857 | 11 | | 16:13 | 195 | 1,597 | 284 | 880 | 11 | ### Monolith's requests, server side (server-log.jsonl) Its `request` lines' `ms`. Starts in the log: 5. | method | n | p50 ms | p99 ms | p99.9 ms | worst ms | | --- | --: | --: | --: | --: | --: | | repo.createRecord | 84,272 | 8.2 | 256.8 | 781.3 | 2811.5 | | repo.listRecords | 29,766 | 1.9 | 9.1 | 58.5 | 1066.7 | | repo.getRecord | 29,262 | 0.4 | 2.3 | 50.4 | 295.6 | | server.describeServer | 19,583 | 0.2 | 0.5 | 1.1 | 13.6 | | repo.deleteRecord | 11,747 | 7.6 | 233.6 | 815.8 | 2116.2 | | sync.getLatestCommit | 9,851 | 0.2 | 0.5 | 28.8 | 417.0 | | repo.applyWrites | 7,869 | 10.7 | 271.4 | 1068.4 | 2097.2 | | repo.putRecord | 5,885 | 6.9 | 310.6 | 727.3 | 910.6 | | repo.describeRepo | 4,040 | 0.7 | 24.9 | 40.5 | 111.8 | | repo.uploadBlob | 3,889 | 2.2 | 9.2 | 166.3 | 1095.0 | | server.refreshSession | 2,029 | 4.4 | 286.3 | 1223.1 | 2301.9 | | sync.getRepo | 1,870 | 11.8 | 292.2 | 543.4 | 758.4 | #### createRecord by period, and the 60 s after each start apart | | n | p50 ms | p99 ms | p99.9 ms | worst ms | | --- | --: | --: | --: | --: | --: | | start–16:14, but after a start | 83,545 | 8.2 | 245.9 | 753.9 | 2811.5 | | start–16:14, after a start | 727 | 10.8 | 1265.2 | 1787.4 | 1787.4 | #### What a slow write (over 200 ms) held up Requests under way during each slow write, but it, slow being over 100 ms. | under way | fast | slow | | --- | --: | --: | | writes to other accounts | 590 | 1,932 | | writes to the same account | 7 | 52 | | reads | 1,321 | 36 | Slow writes: 1,563. ## 2026-11-13: the volume, then the server's own disk The same accounts on both, from 16:27 to 18:40 UTC. At 17:01 Monolith's data moved from the volume to the server's own disk (the reference's stayed on the volume), and the soak started again; at 18:11 Monolith `3218378` was deployed; at 17:54 Monolith was killed. The totals are from the soak's start at 17:01. Its `mismatch` are `getRecord`s answered `429` as it read every account back after its start, which it now takes as no answer; its `found`, writes a kill cut short that had landed. ```sh node bench/soak/analyze.mjs bench/soak/data/day-local-disk --split 2026-11-13T17:01:30 --split 2026-11-13T18:11 ``` ### Totals (summary-*.json) | | monolith | reference | | --- | --: | --: | | from, to | 17:01–18:37 | 17:01–18:37 | | accounts, records | 50, 133,930 | 50, 133,136 | | operations | 39,279 | 38,854 | | 5xx | 0 | 0 | | rate limited (429) | 300 | 381 | | lost | 0 | 0 | | mismatches | 53 | 0 | | commits not on the firehose | 0 | 0 | | outages, their time | 1, 3.5 s | 0, 0.0 s | ### The soak's times, by five-minute window (events-*.jsonl) Each window has its own p50 and p99 per operation; the median of them, and the worst p99. Times are the soak's, from request to answer. | operation | monolith p50 | monolith p99 | monolith worst p99 | reference p50 | reference p99 | reference worst p99 | | --- | --: | --: | --: | --: | --: | --: | | post | 7.0 | 201.5 | 1280.3 | 74.0 | 510.6 | 1847.7 | | like | 12.3 | 200.0 | 2184.6 | 76.8 | 543.3 | 1406.6 | | applyWrites | 10.9 | 195.7 | 4572.6 | 92.2 | 778.6 | 2182.8 | | delete | 8.5 | 190.9 | 1456.1 | 77.1 | 594.0 | 2790.5 | | getRecord | 2.5 | 190.4 | 1388.2 | 12.2 | 312.4 | 867.9 | | listRecords | 3.9 | 190.3 | 1732.1 | 21.1 | 402.2 | 1358.8 | | getLatestCommit | 2.1 | 110.4 | 519.1 | 11.2 | 373.1 | 917.3 | | getRepo | 34.2 | 75.1 | 1258.4 | 299.2 | 998.7 | 5205.9 | #### Events by kind | kind | monolith | reference | | --- | --: | --: | | down | 1 | 0 | | found | 25 | 0 | | mismatch | 93 | 14 | | ready | 2 | 2 | | resumed | 2 | 2 | | seq jump | 5 | 0 | | up | 1 | 0 | ### Kills (kills.jsonl) | container | kills | serving again after | | --- | --: | --: | | spec-pds-pds-1 | 1 | 0.6–0.6 s | ### Memory and data over time (samples.jsonl), MiB | container | RSS first, most, last | data first, most, last | | --- | --: | --: | | spec-pds-pds-1 | 27, 196, 196 | 1,587, 2,138, 2,138 | | opensocial-next-pds-1 | 220, 292, 292 | 880, 1,063, 1,063 | | hour (last sample) | spec-pds-pds-1 RSS | spec-pds-pds-1 data | opensocial-next-pds-1 RSS | opensocial-next-pds-1 data | free GiB | | --- | --: | --: | --: | --: | --: | | 16:59 | 158 | 1,730 | 287 | 929 | 11 | | 17:59 | 166 | 1,951 | 291 | 1,015 | 11 | | 18:40 | 196 | 2,138 | 292 | 1,063 | 11 | ### Monolith's syncs (samples.jsonl: its fsync histogram, the disk's counters) | period | syncs | over 25 ms | over 100 ms | slow syncs ~ disk flush time (r, a sample apart) | | --- | --: | --: | --: | --: | | start–17:01 | 7,085 | 260 | 86 | 0.91 | | 17:01–18:11 | 14,337 | 2 | 0 | – | | 18:11–end | 6,400 | 1 | 0 | – | ### Monolith's requests, server side (server-log.jsonl) Its `request` lines' `ms`. Starts in the log: 3. | method | n | p50 ms | p99 ms | p99.9 ms | worst ms | | --- | --: | --: | --: | --: | --: | | repo.createRecord | 20,450 | 6.6 | 56.0 | 447.9 | 2225.3 | | repo.listRecords | 11,591 | 2.4 | 58.7 | 203.0 | 965.4 | | repo.getRecord | 6,983 | 0.4 | 1.1 | 232.0 | 397.5 | | server.describeServer | 4,856 | 0.2 | 0.5 | 1.2 | 3.7 | | repo.deleteRecord | 2,883 | 5.8 | 121.8 | 857.6 | 1175.5 | | sync.getLatestCommit | 2,293 | 0.2 | 0.6 | 207.2 | 333.3 | | repo.applyWrites | 1,918 | 8.7 | 55.6 | 674.3 | 742.8 | | repo.putRecord | 1,390 | 4.7 | 90.4 | 787.7 | 803.8 | | repo.describeRepo | 1,030 | 0.9 | 28.1 | 626.9 | 994.8 | | repo.uploadBlob | 1,004 | 2.2 | 6.4 | 919.1 | 1148.0 | | server.refreshSession | 583 | 2.0 | 70.0 | 608.0 | 608.0 | #### createRecord by period, and the 60 s after each start apart | | n | p50 ms | p99 ms | p99.9 ms | worst ms | | --- | --: | --: | --: | --: | --: | | start–17:01, but after a start | 5,164 | 8.9 | 161.9 | 430.3 | 1198.1 | | 17:01–18:11, but after a start | 10,461 | 5.5 | 16.7 | 39.1 | 49.8 | | 17:01–18:11, after a start | 169 | 7.5 | 973.9 | 986.4 | 986.4 | | 18:11–end, but after a start | 4,519 | 5.7 | 18.4 | 35.3 | 45.0 | | 18:11–end, after a start | 137 | 7.9 | 2003.1 | 2225.3 | 2225.3 | #### What a slow write (over 200 ms) held up Requests under way during each slow write, but it, slow being over 100 ms. | under way | fast | slow | | --- | --: | --: | | writes to other accounts | 46 | 167 | | writes to the same account | 2 | 7 | | reads | 75 | 46 | Slow writes: 110. #### Slow writes against the disk, a sample apart | period | samples | r: slow writes ~ disk flush time | r: slow syncs ~ disk flush time | | --- | --: | --: | --: | | start–17:01 | 32 | 0.90 | 0.91 | | 17:01–18:11 | 67 | 0.06 | – | | 18:11–end | 29 | -0.09 | – | ## 2026-11-13: under load and at rest Both fresh, each its data on the server's own disk, a backfill window `W` of 2 minutes and the rate limits off; 20 accounts and 10 operations a second on each for 10 minutes, seed 7, Monolith `1707f04`; then the settled check (`README.md`). ```sh node bench/soak/analyze.mjs bench/soak/data/at-rest ``` ### Totals (summary-*.json) | | monolith | reference | | --- | --: | --: | | from, to | 17:43–17:54 | 17:43–17:54 | | accounts, records | 20, 3,338 | 20, 3,312 | | operations | 6,209 | 6,178 | | 5xx | 0 | 0 | | rate limited (429) | 0 | 0 | | lost | 0 | 0 | | mismatches | 0 | 0 | | commits not on the firehose | 0 | 0 | | outages, their time | 0, 0.0 s | 0, 0.0 s | ### The soak's times, by five-minute window (events-*.jsonl) Each window has its own p50 and p99 per operation; the median of them, and the worst p99. Times are the soak's, from request to answer. | operation | monolith p50 | monolith p99 | monolith worst p99 | reference p50 | reference p99 | reference worst p99 | | --- | --: | --: | --: | --: | --: | --: | | post | 5.8 | 12.1 | 12.1 | 35.2 | 61.9 | 64.1 | | like | 6.2 | 13.5 | 13.5 | 36.5 | 268.4 | 268.4 | | applyWrites | 7.9 | 12.9 | 12.9 | 43.9 | 251.7 | 251.7 | | delete | 5.9 | 11.8 | 11.8 | 35.9 | 75.9 | 75.9 | | getRecord | 2.4 | 5.7 | 8.9 | 12.1 | 20.5 | 20.5 | | listRecords | 3.0 | 9.9 | 9.9 | 15.3 | 41.1 | 41.1 | | getLatestCommit | 2.2 | 4.6 | 4.6 | 11.7 | 19.8 | 19.8 | | getRepo | 4.7 | 6.6 | 6.6 | 28.9 | 87.0 | 87.0 | #### Events by kind | kind | monolith | reference | | --- | --: | --: | | ready | 1 | 1 | | settled | 1 | 1 | | settling | 1 | 1 | #### The settled check | | monolith, 17:58 | reference, 17:58 | | --- | --: | --: | | live state: repositories + blobs | 7.1 MiB (1 + 6.1) | 6.8 MiB (1 + 5.8) | | when the load stopped | 25.6 MiB | 33.9 MiB | | after 2 W with no write | 16.6 MiB | 33.9 MiB | | at rest, after a write to each | 7.7 MiB, 1.08 × live | 33.9 MiB, 4.99 × live | | the settled bound | 24.9 MiB: within | 23.9 MiB: over | ## 2026-11-13: settled, on a laptop `harness/dev.mjs` on an Apple laptop, `W` of 1 minute, Monolith `1707f04`: 5 accounts and 30 operations a second for 10 minutes, then the settled check, the server running throughout. ```sh node bench/soak/analyze.mjs bench/soak/data/settle-laptop ``` ### Totals (summary-*.json) | | local | | --- | --: | | from, to | 16:31–16:41 | | accounts, records | 5, 9,397 | | operations | 18,540 | | 5xx | 0 | | rate limited (429) | 0 | | lost | 0 | | mismatches | 0 | | commits not on the firehose | 0 | | outages, their time | 0, 0.0 s | ### The soak's times, by five-minute window (events-*.jsonl) Each window has its own p50 and p99 per operation; the median of them, and the worst p99. Times are the soak's, from request to answer. | operation | local p50 | local p99 | local worst p99 | | --- | --: | --: | --: | | post | 6.6 | 14.7 | 17.9 | | like | 7.2 | 18.0 | 18.0 | | applyWrites | 7.2 | 16.4 | 16.4 | | delete | 6.8 | 17.6 | 17.6 | | getRecord | 0.7 | 2.5 | 2.5 | | listRecords | 1.0 | 4.3 | 4.5 | | getLatestCommit | 0.7 | 3.3 | 3.3 | | getRepo | 3.5 | 33.6 | 33.6 | #### Events by kind | kind | local | | --- | --: | | ready | 1 | | settled | 1 | | settling | 1 | #### The settled check | | local, 16:43 | | --- | --: | | live state: repositories + blobs | 22.9 MiB (2.9 + 20) | | when the load stopped | – | | after 2 W with no write | 44.8 MiB | | at rest, after a write to each | 24.1 MiB, 1.05 × live | | the settled bound | 70.2 MiB: within | ## 2026-11-13: settled across a restart, on a laptop `harness/dev.mjs` on an Apple laptop, `W` of 2 minutes: 5 accounts and 30 operations a second for 4 minutes, the server stopped, `2 W` with it down, started again on the same data, then the settled check with `SETTLE_ONLY`. At 16:58 Monolith `1707f04`; at 17:15, after `2 W` and a start, `3218378`. 45 accounts more, made by the soak at 16:53, each have one write. ```sh node bench/soak/analyze.mjs bench/soak/data/settle-laptop-restart ``` ### Totals (summary-*.json) | | local | | --- | --: | | from, to | 16:44–16:48 | | accounts, records | 5, 3,714 | | operations | 7,408 | | 5xx | 0 | | rate limited (429) | 0 | | lost | 0 | | mismatches | 0 | | commits not on the firehose | 0 | | outages, their time | 1, 5.1 s | ### The soak's times, by five-minute window (events-*.jsonl) Each window has its own p50 and p99 per operation; the median of them, and the worst p99. Times are the soak's, from request to answer. | operation | local p50 | local p99 | local worst p99 | | --- | --: | --: | --: | | post | 6.7 | 12.0 | 12.0 | | like | 6.9 | 12.0 | 12.0 | | applyWrites | 7.0 | 16.8 | 16.8 | | delete | 6.7 | 12.4 | 12.4 | | getRecord | 0.7 | 2.0 | 2.0 | | listRecords | 0.9 | 3.7 | 3.7 | | getLatestCommit | 0.7 | 2.0 | 2.0 | | getRepo | 2.2 | 26.0 | 26.0 | #### Events by kind | kind | local | | --- | --: | | createAccount refused | 1 | | down | 3 | | ready | 4 | | resumed | 3 | | settled | 2 | | settling | 3 | | up | 3 | #### The settled check | | local, 16:58 | local, 17:15 | | --- | --: | --: | | live state: repositories + blobs | 9 MiB (1.2 + 7.9) | 9 MiB (1.2 + 7.9) | | when the load stopped | – | – | | after 2 W with no write | 37.4 MiB | 37.6 MiB | | at rest, after a write to each | 37.5 MiB, 4.17 × live | 9.7 MiB, 1.08 × live | | the settled bound | 34.3 MiB: over | 34.3 MiB: within | ## 2026-11-14: under load and at rest, on 08a1db4 The run of "under load and at rest" again, as `run.sh at-rest` in the deployment's repository makes it: both fresh, each its data on the server's own disk, a backfill window `W` of 2 minutes and the rate limits off; 20 accounts and 10 operations a second on each for 10 minutes, seed 7; then the settled check. Monolith `08a1db4`, the image deployed on the machine, which has `3218378` and its log capped at 4 MiB; the server was not started again between the load and the check. ```sh node bench/soak/analyze.mjs bench/soak/data/at-rest-08a1db4 ``` ### Totals (summary-*.json) | | monolith | reference | | --- | --: | --: | | from, to | 04:34–04:44 | 04:34–04:44 | | accounts, records | 20, 3,337 | 20, 3,287 | | operations | 6,209 | 6,186 | | 5xx | 0 | 0 | | rate limited (429) | 0 | 0 | | lost | 0 | 0 | | mismatches | 0 | 0 | | commits not on the firehose | 0 | 0 | | outages, their time | 0, 0.0 s | 0, 0.0 s | ### The soak's times, by five-minute window (events-*.jsonl) Each window has its own p50 and p99 per operation; the median of them, and the worst p99. Times are the soak's, from request to answer. | operation | monolith p50 | monolith p99 | monolith worst p99 | reference p50 | reference p99 | reference worst p99 | | --- | --: | --: | --: | --: | --: | --: | | post | 6.0 | 12.6 | 12.6 | 33.1 | 51.3 | 51.3 | | like | 6.2 | 11.5 | 11.5 | 33.9 | 64.8 | 64.8 | | applyWrites | 7.4 | 15.3 | 15.3 | 43.8 | 110.8 | 110.8 | | delete | 6.0 | 11.4 | 11.4 | 34.2 | 85.0 | 101.9 | | getRecord | 2.4 | 6.6 | 20.0 | 12.3 | 35.1 | 35.1 | | listRecords | 2.9 | 6.6 | 6.6 | 15.1 | 39.2 | 39.2 | | getLatestCommit | 2.2 | 5.1 | 5.1 | 11.7 | 22.7 | 22.7 | | getRepo | 4.3 | 14.3 | 14.3 | 27.0 | 168.0 | 168.0 | #### Events by kind | kind | monolith | reference | | --- | --: | --: | | ready | 1 | 1 | | settled | 1 | 1 | | settling | 1 | 1 | #### The settled check | | monolith, 04:48 | reference, 04:48 | | --- | --: | --: | | live state: repositories + blobs | 7.1 MiB (1 + 6.1) | 7.4 MiB (1 + 6.3) | | when the load stopped | 25.1 MiB | 34.5 MiB | | after 2 W with no write | 16.4 MiB | 34.5 MiB | | at rest, after a write to each | 7.7 MiB, 1.08 × live | 34.5 MiB, 4.66 × live | | the settled bound | 24.9 MiB: within | 25.6 MiB: over |