~ knotless Search/
↑↓ move↵ open⇧↵ all results
0
0
537 lines · 19.4 KB Raw History

Soak results

Each run's files are in data/<run>/, and every table below is analyze.mjs's output for them, as the command above it gives it: run it again to check them. The files: summary-*.json and events-*.jsonl.gz, the soak's; where the machine was sampled, samples.jsonl.gz (each minute: each server's RSS, CPU and data directory, the free space, and from the second run Monolith's fsync histogram and the disk's counters) and kills.jsonl; and server-log.jsonl.gz, Monolith's log (PDS_LOG_DIR) cut to its request, start, shutdown and warning lines.

The soak's own times ("The soak's times") are from a driver capped at half a CPU, serving both servers: compare them between servers, and take a server's own from its log.

What they show

  • No write answered was lost and no commit answered missed the firehose, on either server, across nine kills. Night 1's lost and mismatches are the soak's own, from before it knew that a follow or like replaces the account's others of the same subject ("nine hours").
  • At rest Monolith holds its live state, 1.08 times it; the reference five times it ("under load and at rest"), and again on 08a1db4, 1.08 against 4.66 ("under load and at rest, on 08a1db4"). Under a day's window and hours of writes Monolith held about twice the reference's ("nine hours").
  • On the network volume a slow sync held up every write under way, with Monolith's slow syncs and slow writes following the volume's flush time (r 0.91 and 0.90); on the server's own disk neither did, and createRecord's p99 fell from 162 ms to 17 ms ("the volume, then the server's own disk").
  • In the minute after a start, a crash's or a clean one's, writes took up to one or two seconds ("createRecord by period").
  • Across a restart, 1707f04 did not settle, and 3218378 does ("settled across a restart").

The machine

A Hetzner Cloud server in Falkenstein (fsn1): 8 vCPUs, 15 GiB, other services running on it throughout. Each PDS in a container capped at 1 CPU and 1 GiB, the soak in one capped at 0.5 CPU and 512 MiB, all on one Docker network, the soak calling the servers directly. Both PDSes make their DIDs in a PLC directory of their own on the same network, and tell no relay of themselves. The reference is @atproto/pds 0.0.0-spaces-alpha-20261029171913. A kill is SIGKILL to the container's main process, its restart policy starting it again.

2026-11-13: nine hours side by side

Monolith 1707f04 and the reference, each with its data on a Hetzner Volume (network block storage); 50 accounts and 6 operations a second on each, from 07:08 to 16:14 UTC; four kills of each, in turn, 40 to 80 minutes apart.

The soak did not yet know that a follow or like replaces the account's others of the same subject, as both servers do in the same commit (spec/VALIDATION.md, "Duplicates"): its lost and mismatch here are those records, which the firehose showed each server deleting, not records lost.

node bench/soak/analyze.mjs bench/soak/data/night1 --to 2026-11-13T16:14

Totals (summary-*.json)

monolithreference
from, to07:08–16:1307:08–16:13
accounts, records50, 110,17550, 109,868
operations209,823209,710
5xx00
rate limited (429)00
lost4,6604,510
mismatches213200
commits not on the firehose00
outages, their time3, 9.4 s4, 31.1 s

The soak's times, by five-minute window (events-*.jsonl)

Each window has its own p50 and p99 per operation; the median of them, and the worst p99. Times are the soak's, from request to answer.

operationmonolith p50monolith p99monolith worst p99reference p50reference p99reference worst p99
post9.9289.31797.472.9650.74059.9
like12.9230.32598.575.4649.94373.4
applyWrites13.7307.12646.789.2710.54484.2
delete11.1226.71928.375.8628.03866.6
getRecord2.570.4722.212.2353.23544.1
listRecords3.884.11812.618.0403.72348.6
getLatestCommit2.1100.22536.211.4407.52950.8
getRepo17.643.12844.7103.0516.25539.1

Events by kind

kindmonolithreference
down34
found01
lost4,6604,510
mismatch213200
ready11
resumed11
seq jump50
up34

Kills (kills.jsonl)

containerkillsserving again after
spec-pds-pds-140.7–0.8 s
opensocial-next-pds-140.6–0.9 s

Memory and data over time (samples.jsonl), MiB

containerRSS first, most, lastdata first, most, last
spec-pds-pds-124, 196, 19519, 1,601, 1,597
opensocial-next-pds-1191, 290, 28437, 880, 880
hour (last sample)spec-pds-pds-1 RSSspec-pds-pds-1 dataopensocial-next-pds-1 RSSopensocial-next-pds-1 datafree GiB
07:595512727511413
08:598226625620313
09:5811043828229413
10:5911558828938612
11:5914980627547612
12:5914195728457012
13:591621,18428466512
14:591761,41429075911
15:591801,56028385711
16:131951,59728488011

Monolith's requests, server side (server-log.jsonl)

Its request lines' ms. Starts in the log: 5.

methodnp50 msp99 msp99.9 msworst ms
repo.createRecord84,2728.2256.8781.32811.5
repo.listRecords29,7661.99.158.51066.7
repo.getRecord29,2620.42.350.4295.6
server.describeServer19,5830.20.51.113.6
repo.deleteRecord11,7477.6233.6815.82116.2
sync.getLatestCommit9,8510.20.528.8417.0
repo.applyWrites7,86910.7271.41068.42097.2
repo.putRecord5,8856.9310.6727.3910.6
repo.describeRepo4,0400.724.940.5111.8
repo.uploadBlob3,8892.29.2166.31095.0
server.refreshSession2,0294.4286.31223.12301.9
sync.getRepo1,87011.8292.2543.4758.4

createRecord by period, and the 60 s after each start apart

np50 msp99 msp99.9 msworst ms
start–16:14, but after a start83,5458.2245.9753.92811.5
start–16:14, after a start72710.81265.21787.41787.4

What a slow write (over 200 ms) held up

Requests under way during each slow write, but it, slow being over 100 ms.

under wayfastslow
writes to other accounts5901,932
writes to the same account752
reads1,32136

Slow writes: 1,563.

2026-11-13: the volume, then the server's own disk

The same accounts on both, from 16:27 to 18:40 UTC. At 17:01 Monolith's data moved from the volume to the server's own disk (the reference's stayed on the volume), and the soak started again; at 18:11 Monolith 3218378 was deployed; at 17:54 Monolith was killed. The totals are from the soak's start at 17:01. Its mismatch are getRecords answered 429 as it read every account back after its start, which it now takes as no answer; its found, writes a kill cut short that had landed.

node bench/soak/analyze.mjs bench/soak/data/day-local-disk --split 2026-11-13T17:01:30 --split 2026-11-13T18:11

Totals (summary-*.json)

monolithreference
from, to17:01–18:3717:01–18:37
accounts, records50, 133,93050, 133,136
operations39,27938,854
5xx00
rate limited (429)300381
lost00
mismatches530
commits not on the firehose00
outages, their time1, 3.5 s0, 0.0 s

The soak's times, by five-minute window (events-*.jsonl)

Each window has its own p50 and p99 per operation; the median of them, and the worst p99. Times are the soak's, from request to answer.

operationmonolith p50monolith p99monolith worst p99reference p50reference p99reference worst p99
post7.0201.51280.374.0510.61847.7
like12.3200.02184.676.8543.31406.6
applyWrites10.9195.74572.692.2778.62182.8
delete8.5190.91456.177.1594.02790.5
getRecord2.5190.41388.212.2312.4867.9
listRecords3.9190.31732.121.1402.21358.8
getLatestCommit2.1110.4519.111.2373.1917.3
getRepo34.275.11258.4299.2998.75205.9

Events by kind

kindmonolithreference
down10
found250
mismatch9314
ready22
resumed22
seq jump50
up10

Kills (kills.jsonl)

containerkillsserving again after
spec-pds-pds-110.6–0.6 s

Memory and data over time (samples.jsonl), MiB

containerRSS first, most, lastdata first, most, last
spec-pds-pds-127, 196, 1961,587, 2,138, 2,138
opensocial-next-pds-1220, 292, 292880, 1,063, 1,063
hour (last sample)spec-pds-pds-1 RSSspec-pds-pds-1 dataopensocial-next-pds-1 RSSopensocial-next-pds-1 datafree GiB
16:591581,73028792911
17:591661,9512911,01511
18:401962,1382921,06311

Monolith's syncs (samples.jsonl: its fsync histogram, the disk's counters)

periodsyncsover 25 msover 100 msslow syncs ~ disk flush time (r, a sample apart)
start–17:017,085260860.91
17:01–18:1114,33720–
18:11–end6,40010–

Monolith's requests, server side (server-log.jsonl)

Its request lines' ms. Starts in the log: 3.

methodnp50 msp99 msp99.9 msworst ms
repo.createRecord20,4506.656.0447.92225.3
repo.listRecords11,5912.458.7203.0965.4
repo.getRecord6,9830.41.1232.0397.5
server.describeServer4,8560.20.51.23.7
repo.deleteRecord2,8835.8121.8857.61175.5
sync.getLatestCommit2,2930.20.6207.2333.3
repo.applyWrites1,9188.755.6674.3742.8
repo.putRecord1,3904.790.4787.7803.8
repo.describeRepo1,0300.928.1626.9994.8
repo.uploadBlob1,0042.26.4919.11148.0
server.refreshSession5832.070.0608.0608.0

createRecord by period, and the 60 s after each start apart

np50 msp99 msp99.9 msworst ms
start–17:01, but after a start5,1648.9161.9430.31198.1
17:01–18:11, but after a start10,4615.516.739.149.8
17:01–18:11, after a start1697.5973.9986.4986.4
18:11–end, but after a start4,5195.718.435.345.0
18:11–end, after a start1377.92003.12225.32225.3

What a slow write (over 200 ms) held up

Requests under way during each slow write, but it, slow being over 100 ms.

under wayfastslow
writes to other accounts46167
writes to the same account27
reads7546

Slow writes: 110.

Slow writes against the disk, a sample apart

periodsamplesr: slow writes ~ disk flush timer: slow syncs ~ disk flush time
start–17:01320.900.91
17:01–18:11670.06–
18:11–end29-0.09–

2026-11-13: under load and at rest

Both fresh, each its data on the server's own disk, a backfill window W of 2 minutes and the rate limits off; 20 accounts and 10 operations a second on each for 10 minutes, seed 7, Monolith 1707f04; then the settled check (README.md).

node bench/soak/analyze.mjs bench/soak/data/at-rest

Totals (summary-*.json)

monolithreference
from, to17:43–17:5417:43–17:54
accounts, records20, 3,33820, 3,312
operations6,2096,178
5xx00
rate limited (429)00
lost00
mismatches00
commits not on the firehose00
outages, their time0, 0.0 s0, 0.0 s

The soak's times, by five-minute window (events-*.jsonl)

Each window has its own p50 and p99 per operation; the median of them, and the worst p99. Times are the soak's, from request to answer.

operationmonolith p50monolith p99monolith worst p99reference p50reference p99reference worst p99
post5.812.112.135.261.964.1
like6.213.513.536.5268.4268.4
applyWrites7.912.912.943.9251.7251.7
delete5.911.811.835.975.975.9
getRecord2.45.78.912.120.520.5
listRecords3.09.99.915.341.141.1
getLatestCommit2.24.64.611.719.819.8
getRepo4.76.66.628.987.087.0

Events by kind

kindmonolithreference
ready11
settled11
settling11

The settled check

monolith, 17:58reference, 17:58
live state: repositories + blobs7.1 MiB (1 + 6.1)6.8 MiB (1 + 5.8)
when the load stopped25.6 MiB33.9 MiB
after 2 W with no write16.6 MiB33.9 MiB
at rest, after a write to each7.7 MiB, 1.08 × live33.9 MiB, 4.99 × live
the settled bound24.9 MiB: within23.9 MiB: over

2026-11-13: settled, on a laptop

harness/dev.mjs on an Apple laptop, W of 1 minute, Monolith 1707f04: 5 accounts and 30 operations a second for 10 minutes, then the settled check, the server running throughout.

node bench/soak/analyze.mjs bench/soak/data/settle-laptop

Totals (summary-*.json)

local
from, to16:31–16:41
accounts, records5, 9,397
operations18,540
5xx0
rate limited (429)0
lost0
mismatches0
commits not on the firehose0
outages, their time0, 0.0 s

The soak's times, by five-minute window (events-*.jsonl)

Each window has its own p50 and p99 per operation; the median of them, and the worst p99. Times are the soak's, from request to answer.

operationlocal p50local p99local worst p99
post6.614.717.9
like7.218.018.0
applyWrites7.216.416.4
delete6.817.617.6
getRecord0.72.52.5
listRecords1.04.34.5
getLatestCommit0.73.33.3
getRepo3.533.633.6

Events by kind

kindlocal
ready1
settled1
settling1

The settled check

local, 16:43
live state: repositories + blobs22.9 MiB (2.9 + 20)
when the load stopped–
after 2 W with no write44.8 MiB
at rest, after a write to each24.1 MiB, 1.05 × live
the settled bound70.2 MiB: within

2026-11-13: settled across a restart, on a laptop

harness/dev.mjs on an Apple laptop, W of 2 minutes: 5 accounts and 30 operations a second for 4 minutes, the server stopped, 2 W with it down, started again on the same data, then the settled check with SETTLE_ONLY. At 16:58 Monolith 1707f04; at 17:15, after 2 W and a start, 3218378. 45 accounts more, made by the soak at 16:53, each have one write.

node bench/soak/analyze.mjs bench/soak/data/settle-laptop-restart

Totals (summary-*.json)

local
from, to16:44–16:48
accounts, records5, 3,714
operations7,408
5xx0
rate limited (429)0
lost0
mismatches0
commits not on the firehose0
outages, their time1, 5.1 s

The soak's times, by five-minute window (events-*.jsonl)

Each window has its own p50 and p99 per operation; the median of them, and the worst p99. Times are the soak's, from request to answer.

operationlocal p50local p99local worst p99
post6.712.012.0
like6.912.012.0
applyWrites7.016.816.8
delete6.712.412.4
getRecord0.72.02.0
listRecords0.93.73.7
getLatestCommit0.72.02.0
getRepo2.226.026.0

Events by kind

kindlocal
createAccount refused1
down3
ready4
resumed3
settled2
settling3
up3

The settled check

local, 16:58local, 17:15
live state: repositories + blobs9 MiB (1.2 + 7.9)9 MiB (1.2 + 7.9)
when the load stopped––
after 2 W with no write37.4 MiB37.6 MiB
at rest, after a write to each37.5 MiB, 4.17 × live9.7 MiB, 1.08 × live
the settled bound34.3 MiB: over34.3 MiB: within

2026-11-14: under load and at rest, on 08a1db4

The run of "under load and at rest" again, as run.sh at-rest in the deployment's repository makes it: both fresh, each its data on the server's own disk, a backfill window W of 2 minutes and the rate limits off; 20 accounts and 10 operations a second on each for 10 minutes, seed 7; then the settled check. Monolith 08a1db4, the image deployed on the machine, which has 3218378 and its log capped at 4 MiB; the server was not started again between the load and the check.

node bench/soak/analyze.mjs bench/soak/data/at-rest-08a1db4

Totals (summary-*.json)

monolithreference
from, to04:34–04:4404:34–04:44
accounts, records20, 3,33720, 3,287
operations6,2096,186
5xx00
rate limited (429)00
lost00
mismatches00
commits not on the firehose00
outages, their time0, 0.0 s0, 0.0 s

The soak's times, by five-minute window (events-*.jsonl)

Each window has its own p50 and p99 per operation; the median of them, and the worst p99. Times are the soak's, from request to answer.

operationmonolith p50monolith p99monolith worst p99reference p50reference p99reference worst p99
post6.012.612.633.151.351.3
like6.211.511.533.964.864.8
applyWrites7.415.315.343.8110.8110.8
delete6.011.411.434.285.0101.9
getRecord2.46.620.012.335.135.1
listRecords2.96.66.615.139.239.2
getLatestCommit2.25.15.111.722.722.7
getRepo4.314.314.327.0168.0168.0

Events by kind

kindmonolithreference
ready11
settled11
settling11

The settled check

monolith, 04:48reference, 04:48
live state: repositories + blobs7.1 MiB (1 + 6.1)7.4 MiB (1 + 6.3)
when the load stopped25.1 MiB34.5 MiB
after 2 W with no write16.4 MiB34.5 MiB
at rest, after a write to each7.7 MiB, 1.08 × live34.5 MiB, 4.66 × live
the settled bound24.9 MiB: within25.6 MiB: over