Soak results
Each run's files are in data/<run>/, and every table below is
analyze.mjs's output for them, as the command above it gives it: run it
again to check them. The files: summary-*.json and events-*.jsonl.gz,
the soak's; where the machine was sampled, samples.jsonl.gz (each minute:
each server's RSS, CPU and data directory, the free space, and from the
second run Monolith's fsync histogram and the disk's counters) and
kills.jsonl; and server-log.jsonl.gz, Monolith's log (PDS_LOG_DIR)
cut to its request, start, shutdown and warning lines.
The soak's own times ("The soak's times") are from a driver capped at half
a CPU, serving both servers: compare them between servers, and take a
server's own from its log.
What they show
- No write answered was lost and no commit answered missed the firehose, on
either server, across nine kills. Night 1's
lost and mismatches are
the soak's own, from before it knew that a follow or like replaces the
account's others of the same subject ("nine hours").
- At rest Monolith holds its live state, 1.08 times it; the reference five
times it ("under load and at rest"), and again on
08a1db4, 1.08 against
4.66 ("under load and at rest, on 08a1db4"). Under a day's window and
hours of writes Monolith held about twice the reference's ("nine hours").
- On the network volume a slow sync held up every write under way, with
Monolith's slow syncs and slow writes following the volume's flush time
(r 0.91 and 0.90); on the server's own disk neither did, and
createRecord's p99 fell from 162 ms to 17 ms ("the volume, then the
server's own disk").
- In the minute after a start, a crash's or a clean one's, writes took up
to one or two seconds ("createRecord by period").
- Across a restart,
1707f04 did not settle, and 3218378 does ("settled
across a restart").
The machine
A Hetzner Cloud server in Falkenstein (fsn1): 8 vCPUs, 15 GiB, other
services running on it throughout. Each PDS in a container capped at 1 CPU
and 1 GiB, the soak in one capped at 0.5 CPU and 512 MiB, all on one Docker
network, the soak calling the servers directly. Both PDSes make their DIDs in
a PLC directory of their own on the same network, and tell no relay of
themselves. The reference is @atproto/pds 0.0.0-spaces-alpha-20261029171913.
A kill is SIGKILL to the container's main process, its restart policy
starting it again.
2026-11-13: nine hours side by side
Monolith 1707f04 and the reference, each with its data on a Hetzner Volume
(network block storage); 50 accounts and 6 operations a second on each,
from 07:08 to 16:14 UTC; four kills of each, in turn, 40 to 80 minutes apart.
The soak did not yet know that a follow or like replaces the account's
others of the same subject, as both servers do in the same commit
(spec/VALIDATION.md, "Duplicates"): its lost and mismatch here are
those records, which the firehose showed each server deleting, not records
lost.
node bench/soak/analyze.mjs bench/soak/data/night1 --to 2026-11-13T16:14
Totals (summary-*.json)
| monolith | reference |
| from, to | 07:08–16:13 | 07:08–16:13 |
| accounts, records | 50, 110,175 | 50, 109,868 |
| operations | 209,823 | 209,710 |
| 5xx | 0 | 0 |
| rate limited (429) | 0 | 0 |
| lost | 4,660 | 4,510 |
| mismatches | 213 | 200 |
| commits not on the firehose | 0 | 0 |
| outages, their time | 3, 9.4 s | 4, 31.1 s |
The soak's times, by five-minute window (events-*.jsonl)
Each window has its own p50 and p99 per operation; the median of them, and the worst p99. Times are the soak's, from request to answer.
| operation | monolith p50 | monolith p99 | monolith worst p99 | reference p50 | reference p99 | reference worst p99 |
| post | 9.9 | 289.3 | 1797.4 | 72.9 | 650.7 | 4059.9 |
| like | 12.9 | 230.3 | 2598.5 | 75.4 | 649.9 | 4373.4 |
| applyWrites | 13.7 | 307.1 | 2646.7 | 89.2 | 710.5 | 4484.2 |
| delete | 11.1 | 226.7 | 1928.3 | 75.8 | 628.0 | 3866.6 |
| getRecord | 2.5 | 70.4 | 722.2 | 12.2 | 353.2 | 3544.1 |
| listRecords | 3.8 | 84.1 | 1812.6 | 18.0 | 403.7 | 2348.6 |
| getLatestCommit | 2.1 | 100.2 | 2536.2 | 11.4 | 407.5 | 2950.8 |
| getRepo | 17.6 | 43.1 | 2844.7 | 103.0 | 516.2 | 5539.1 |
Events by kind
| kind | monolith | reference |
| down | 3 | 4 |
| found | 0 | 1 |
| lost | 4,660 | 4,510 |
| mismatch | 213 | 200 |
| ready | 1 | 1 |
| resumed | 1 | 1 |
| seq jump | 5 | 0 |
| up | 3 | 4 |
Kills (kills.jsonl)
| container | kills | serving again after |
| spec-pds-pds-1 | 4 | 0.7–0.8 s |
| opensocial-next-pds-1 | 4 | 0.6–0.9 s |
Memory and data over time (samples.jsonl), MiB
| container | RSS first, most, last | data first, most, last |
| spec-pds-pds-1 | 24, 196, 195 | 19, 1,601, 1,597 |
| opensocial-next-pds-1 | 191, 290, 284 | 37, 880, 880 |
| hour (last sample) | spec-pds-pds-1 RSS | spec-pds-pds-1 data | opensocial-next-pds-1 RSS | opensocial-next-pds-1 data | free GiB |
| 07:59 | 55 | 127 | 275 | 114 | 13 |
| 08:59 | 82 | 266 | 256 | 203 | 13 |
| 09:58 | 110 | 438 | 282 | 294 | 13 |
| 10:59 | 115 | 588 | 289 | 386 | 12 |
| 11:59 | 149 | 806 | 275 | 476 | 12 |
| 12:59 | 141 | 957 | 284 | 570 | 12 |
| 13:59 | 162 | 1,184 | 284 | 665 | 12 |
| 14:59 | 176 | 1,414 | 290 | 759 | 11 |
| 15:59 | 180 | 1,560 | 283 | 857 | 11 |
| 16:13 | 195 | 1,597 | 284 | 880 | 11 |
Monolith's requests, server side (server-log.jsonl)
Its request lines' ms. Starts in the log: 5.
| method | n | p50 ms | p99 ms | p99.9 ms | worst ms |
| repo.createRecord | 84,272 | 8.2 | 256.8 | 781.3 | 2811.5 |
| repo.listRecords | 29,766 | 1.9 | 9.1 | 58.5 | 1066.7 |
| repo.getRecord | 29,262 | 0.4 | 2.3 | 50.4 | 295.6 |
| server.describeServer | 19,583 | 0.2 | 0.5 | 1.1 | 13.6 |
| repo.deleteRecord | 11,747 | 7.6 | 233.6 | 815.8 | 2116.2 |
| sync.getLatestCommit | 9,851 | 0.2 | 0.5 | 28.8 | 417.0 |
| repo.applyWrites | 7,869 | 10.7 | 271.4 | 1068.4 | 2097.2 |
| repo.putRecord | 5,885 | 6.9 | 310.6 | 727.3 | 910.6 |
| repo.describeRepo | 4,040 | 0.7 | 24.9 | 40.5 | 111.8 |
| repo.uploadBlob | 3,889 | 2.2 | 9.2 | 166.3 | 1095.0 |
| server.refreshSession | 2,029 | 4.4 | 286.3 | 1223.1 | 2301.9 |
| sync.getRepo | 1,870 | 11.8 | 292.2 | 543.4 | 758.4 |
createRecord by period, and the 60 s after each start apart
| n | p50 ms | p99 ms | p99.9 ms | worst ms |
| start–16:14, but after a start | 83,545 | 8.2 | 245.9 | 753.9 | 2811.5 |
| start–16:14, after a start | 727 | 10.8 | 1265.2 | 1787.4 | 1787.4 |
What a slow write (over 200 ms) held up
Requests under way during each slow write, but it, slow being over 100 ms.
| under way | fast | slow |
| writes to other accounts | 590 | 1,932 |
| writes to the same account | 7 | 52 |
| reads | 1,321 | 36 |
Slow writes: 1,563.
2026-11-13: the volume, then the server's own disk
The same accounts on both, from 16:27 to 18:40 UTC. At 17:01 Monolith's data
moved from the volume to the server's own disk (the reference's stayed on
the volume), and the soak started again; at 18:11 Monolith 3218378 was
deployed; at 17:54 Monolith was killed. The totals are from the soak's
start at 17:01. Its mismatch are getRecords answered 429 as it read
every account back after its start, which it now takes as no answer; its
found, writes a kill cut short that had landed.
node bench/soak/analyze.mjs bench/soak/data/day-local-disk --split 2026-11-13T17:01:30 --split 2026-11-13T18:11
Totals (summary-*.json)
| monolith | reference |
| from, to | 17:01–18:37 | 17:01–18:37 |
| accounts, records | 50, 133,930 | 50, 133,136 |
| operations | 39,279 | 38,854 |
| 5xx | 0 | 0 |
| rate limited (429) | 300 | 381 |
| lost | 0 | 0 |
| mismatches | 53 | 0 |
| commits not on the firehose | 0 | 0 |
| outages, their time | 1, 3.5 s | 0, 0.0 s |
The soak's times, by five-minute window (events-*.jsonl)
Each window has its own p50 and p99 per operation; the median of them, and the worst p99. Times are the soak's, from request to answer.
| operation | monolith p50 | monolith p99 | monolith worst p99 | reference p50 | reference p99 | reference worst p99 |
| post | 7.0 | 201.5 | 1280.3 | 74.0 | 510.6 | 1847.7 |
| like | 12.3 | 200.0 | 2184.6 | 76.8 | 543.3 | 1406.6 |
| applyWrites | 10.9 | 195.7 | 4572.6 | 92.2 | 778.6 | 2182.8 |
| delete | 8.5 | 190.9 | 1456.1 | 77.1 | 594.0 | 2790.5 |
| getRecord | 2.5 | 190.4 | 1388.2 | 12.2 | 312.4 | 867.9 |
| listRecords | 3.9 | 190.3 | 1732.1 | 21.1 | 402.2 | 1358.8 |
| getLatestCommit | 2.1 | 110.4 | 519.1 | 11.2 | 373.1 | 917.3 |
| getRepo | 34.2 | 75.1 | 1258.4 | 299.2 | 998.7 | 5205.9 |
Events by kind
| kind | monolith | reference |
| down | 1 | 0 |
| found | 25 | 0 |
| mismatch | 93 | 14 |
| ready | 2 | 2 |
| resumed | 2 | 2 |
| seq jump | 5 | 0 |
| up | 1 | 0 |
Kills (kills.jsonl)
| container | kills | serving again after |
| spec-pds-pds-1 | 1 | 0.6–0.6 s |
Memory and data over time (samples.jsonl), MiB
| container | RSS first, most, last | data first, most, last |
| spec-pds-pds-1 | 27, 196, 196 | 1,587, 2,138, 2,138 |
| opensocial-next-pds-1 | 220, 292, 292 | 880, 1,063, 1,063 |
| hour (last sample) | spec-pds-pds-1 RSS | spec-pds-pds-1 data | opensocial-next-pds-1 RSS | opensocial-next-pds-1 data | free GiB |
| 16:59 | 158 | 1,730 | 287 | 929 | 11 |
| 17:59 | 166 | 1,951 | 291 | 1,015 | 11 |
| 18:40 | 196 | 2,138 | 292 | 1,063 | 11 |
Monolith's syncs (samples.jsonl: its fsync histogram, the disk's counters)
| period | syncs | over 25 ms | over 100 ms | slow syncs ~ disk flush time (r, a sample apart) |
| start–17:01 | 7,085 | 260 | 86 | 0.91 |
| 17:01–18:11 | 14,337 | 2 | 0 | – |
| 18:11–end | 6,400 | 1 | 0 | – |
Monolith's requests, server side (server-log.jsonl)
Its request lines' ms. Starts in the log: 3.
| method | n | p50 ms | p99 ms | p99.9 ms | worst ms |
| repo.createRecord | 20,450 | 6.6 | 56.0 | 447.9 | 2225.3 |
| repo.listRecords | 11,591 | 2.4 | 58.7 | 203.0 | 965.4 |
| repo.getRecord | 6,983 | 0.4 | 1.1 | 232.0 | 397.5 |
| server.describeServer | 4,856 | 0.2 | 0.5 | 1.2 | 3.7 |
| repo.deleteRecord | 2,883 | 5.8 | 121.8 | 857.6 | 1175.5 |
| sync.getLatestCommit | 2,293 | 0.2 | 0.6 | 207.2 | 333.3 |
| repo.applyWrites | 1,918 | 8.7 | 55.6 | 674.3 | 742.8 |
| repo.putRecord | 1,390 | 4.7 | 90.4 | 787.7 | 803.8 |
| repo.describeRepo | 1,030 | 0.9 | 28.1 | 626.9 | 994.8 |
| repo.uploadBlob | 1,004 | 2.2 | 6.4 | 919.1 | 1148.0 |
| server.refreshSession | 583 | 2.0 | 70.0 | 608.0 | 608.0 |
createRecord by period, and the 60 s after each start apart
| n | p50 ms | p99 ms | p99.9 ms | worst ms |
| start–17:01, but after a start | 5,164 | 8.9 | 161.9 | 430.3 | 1198.1 |
| 17:01–18:11, but after a start | 10,461 | 5.5 | 16.7 | 39.1 | 49.8 |
| 17:01–18:11, after a start | 169 | 7.5 | 973.9 | 986.4 | 986.4 |
| 18:11–end, but after a start | 4,519 | 5.7 | 18.4 | 35.3 | 45.0 |
| 18:11–end, after a start | 137 | 7.9 | 2003.1 | 2225.3 | 2225.3 |
What a slow write (over 200 ms) held up
Requests under way during each slow write, but it, slow being over 100 ms.
| under way | fast | slow |
| writes to other accounts | 46 | 167 |
| writes to the same account | 2 | 7 |
| reads | 75 | 46 |
Slow writes: 110.
Slow writes against the disk, a sample apart
| period | samples | r: slow writes ~ disk flush time | r: slow syncs ~ disk flush time |
| start–17:01 | 32 | 0.90 | 0.91 |
| 17:01–18:11 | 67 | 0.06 | – |
| 18:11–end | 29 | -0.09 | – |
2026-11-13: under load and at rest
Both fresh, each its data on the server's own disk, a backfill window W
of 2 minutes and the rate limits off; 20 accounts and 10 operations a second
on each for 10 minutes, seed 7, Monolith 1707f04; then the settled check
(README.md).
node bench/soak/analyze.mjs bench/soak/data/at-rest
Totals (summary-*.json)
| monolith | reference |
| from, to | 17:43–17:54 | 17:43–17:54 |
| accounts, records | 20, 3,338 | 20, 3,312 |
| operations | 6,209 | 6,178 |
| 5xx | 0 | 0 |
| rate limited (429) | 0 | 0 |
| lost | 0 | 0 |
| mismatches | 0 | 0 |
| commits not on the firehose | 0 | 0 |
| outages, their time | 0, 0.0 s | 0, 0.0 s |
The soak's times, by five-minute window (events-*.jsonl)
Each window has its own p50 and p99 per operation; the median of them, and the worst p99. Times are the soak's, from request to answer.
| operation | monolith p50 | monolith p99 | monolith worst p99 | reference p50 | reference p99 | reference worst p99 |
| post | 5.8 | 12.1 | 12.1 | 35.2 | 61.9 | 64.1 |
| like | 6.2 | 13.5 | 13.5 | 36.5 | 268.4 | 268.4 |
| applyWrites | 7.9 | 12.9 | 12.9 | 43.9 | 251.7 | 251.7 |
| delete | 5.9 | 11.8 | 11.8 | 35.9 | 75.9 | 75.9 |
| getRecord | 2.4 | 5.7 | 8.9 | 12.1 | 20.5 | 20.5 |
| listRecords | 3.0 | 9.9 | 9.9 | 15.3 | 41.1 | 41.1 |
| getLatestCommit | 2.2 | 4.6 | 4.6 | 11.7 | 19.8 | 19.8 |
| getRepo | 4.7 | 6.6 | 6.6 | 28.9 | 87.0 | 87.0 |
Events by kind
| kind | monolith | reference |
| ready | 1 | 1 |
| settled | 1 | 1 |
| settling | 1 | 1 |
The settled check
| monolith, 17:58 | reference, 17:58 |
| live state: repositories + blobs | 7.1 MiB (1 + 6.1) | 6.8 MiB (1 + 5.8) |
| when the load stopped | 25.6 MiB | 33.9 MiB |
| after 2 W with no write | 16.6 MiB | 33.9 MiB |
| at rest, after a write to each | 7.7 MiB, 1.08 × live | 33.9 MiB, 4.99 × live |
| the settled bound | 24.9 MiB: within | 23.9 MiB: over |
2026-11-13: settled, on a laptop
harness/dev.mjs on an Apple laptop, W of 1 minute, Monolith 1707f04:
5 accounts and 30 operations a second for 10 minutes, then the settled
check, the server running throughout.
node bench/soak/analyze.mjs bench/soak/data/settle-laptop
Totals (summary-*.json)
| local |
| from, to | 16:31–16:41 |
| accounts, records | 5, 9,397 |
| operations | 18,540 |
| 5xx | 0 |
| rate limited (429) | 0 |
| lost | 0 |
| mismatches | 0 |
| commits not on the firehose | 0 |
| outages, their time | 0, 0.0 s |
The soak's times, by five-minute window (events-*.jsonl)
Each window has its own p50 and p99 per operation; the median of them, and the worst p99. Times are the soak's, from request to answer.
| operation | local p50 | local p99 | local worst p99 |
| post | 6.6 | 14.7 | 17.9 |
| like | 7.2 | 18.0 | 18.0 |
| applyWrites | 7.2 | 16.4 | 16.4 |
| delete | 6.8 | 17.6 | 17.6 |
| getRecord | 0.7 | 2.5 | 2.5 |
| listRecords | 1.0 | 4.3 | 4.5 |
| getLatestCommit | 0.7 | 3.3 | 3.3 |
| getRepo | 3.5 | 33.6 | 33.6 |
Events by kind
| kind | local |
| ready | 1 |
| settled | 1 |
| settling | 1 |
The settled check
| local, 16:43 |
| live state: repositories + blobs | 22.9 MiB (2.9 + 20) |
| when the load stopped | – |
| after 2 W with no write | 44.8 MiB |
| at rest, after a write to each | 24.1 MiB, 1.05 × live |
| the settled bound | 70.2 MiB: within |
2026-11-13: settled across a restart, on a laptop
harness/dev.mjs on an Apple laptop, W of 2 minutes: 5 accounts and 30
operations a second for 4 minutes, the server stopped, 2 W with it down,
started again on the same data, then the settled check with SETTLE_ONLY.
At 16:58 Monolith 1707f04; at 17:15, after 2 W and a start, 3218378.
45 accounts more, made by the soak at 16:53, each have one write.
node bench/soak/analyze.mjs bench/soak/data/settle-laptop-restart
Totals (summary-*.json)
| local |
| from, to | 16:44–16:48 |
| accounts, records | 5, 3,714 |
| operations | 7,408 |
| 5xx | 0 |
| rate limited (429) | 0 |
| lost | 0 |
| mismatches | 0 |
| commits not on the firehose | 0 |
| outages, their time | 1, 5.1 s |
The soak's times, by five-minute window (events-*.jsonl)
Each window has its own p50 and p99 per operation; the median of them, and the worst p99. Times are the soak's, from request to answer.
| operation | local p50 | local p99 | local worst p99 |
| post | 6.7 | 12.0 | 12.0 |
| like | 6.9 | 12.0 | 12.0 |
| applyWrites | 7.0 | 16.8 | 16.8 |
| delete | 6.7 | 12.4 | 12.4 |
| getRecord | 0.7 | 2.0 | 2.0 |
| listRecords | 0.9 | 3.7 | 3.7 |
| getLatestCommit | 0.7 | 2.0 | 2.0 |
| getRepo | 2.2 | 26.0 | 26.0 |
Events by kind
| kind | local |
| createAccount refused | 1 |
| down | 3 |
| ready | 4 |
| resumed | 3 |
| settled | 2 |
| settling | 3 |
| up | 3 |
The settled check
| local, 16:58 | local, 17:15 |
| live state: repositories + blobs | 9 MiB (1.2 + 7.9) | 9 MiB (1.2 + 7.9) |
| when the load stopped | – | – |
| after 2 W with no write | 37.4 MiB | 37.6 MiB |
| at rest, after a write to each | 37.5 MiB, 4.17 × live | 9.7 MiB, 1.08 × live |
| the settled bound | 34.3 MiB: over | 34.3 MiB: within |
2026-11-14: under load and at rest, on 08a1db4
The run of "under load and at rest" again, as run.sh at-rest in the
deployment's repository makes it: both fresh, each its data on the server's
own disk, a backfill window W of 2 minutes and the rate limits off; 20
accounts and 10 operations a second on each for 10 minutes, seed 7; then the
settled check. Monolith 08a1db4, the image deployed on the machine, which
has 3218378 and its log capped at 4 MiB; the server was not started again
between the load and the check.
node bench/soak/analyze.mjs bench/soak/data/at-rest-08a1db4
Totals (summary-*.json)
| monolith | reference |
| from, to | 04:34–04:44 | 04:34–04:44 |
| accounts, records | 20, 3,337 | 20, 3,287 |
| operations | 6,209 | 6,186 |
| 5xx | 0 | 0 |
| rate limited (429) | 0 | 0 |
| lost | 0 | 0 |
| mismatches | 0 | 0 |
| commits not on the firehose | 0 | 0 |
| outages, their time | 0, 0.0 s | 0, 0.0 s |
The soak's times, by five-minute window (events-*.jsonl)
Each window has its own p50 and p99 per operation; the median of them, and the worst p99. Times are the soak's, from request to answer.
| operation | monolith p50 | monolith p99 | monolith worst p99 | reference p50 | reference p99 | reference worst p99 |
| post | 6.0 | 12.6 | 12.6 | 33.1 | 51.3 | 51.3 |
| like | 6.2 | 11.5 | 11.5 | 33.9 | 64.8 | 64.8 |
| applyWrites | 7.4 | 15.3 | 15.3 | 43.8 | 110.8 | 110.8 |
| delete | 6.0 | 11.4 | 11.4 | 34.2 | 85.0 | 101.9 |
| getRecord | 2.4 | 6.6 | 20.0 | 12.3 | 35.1 | 35.1 |
| listRecords | 2.9 | 6.6 | 6.6 | 15.1 | 39.2 | 39.2 |
| getLatestCommit | 2.2 | 5.1 | 5.1 | 11.7 | 22.7 | 22.7 |
| getRepo | 4.3 | 14.3 | 14.3 | 27.0 | 168.0 | 168.0 |
Events by kind
| kind | monolith | reference |
| ready | 1 | 1 |
| settled | 1 | 1 |
| settling | 1 | 1 |
The settled check
| monolith, 04:48 | reference, 04:48 |
| live state: repositories + blobs | 7.1 MiB (1 + 6.1) | 7.4 MiB (1 + 6.3) |
| when the load stopped | 25.1 MiB | 34.5 MiB |
| after 2 W with no write | 16.4 MiB | 34.5 MiB |
| at rest, after a write to each | 7.7 MiB, 1.08 × live | 34.5 MiB, 4.66 × live |
| the settled bound | 24.9 MiB: within | 25.6 MiB: over |