~ knotless Search/
↑↓ move↵ open⇧↵ all results
0
0
961 lines · 74.9 KB Raw History

The store benchmark: results

just store-bench (README.md), one section a machine. A machine's numbers compare only with its own, and the sections are of different commits: the runner's is the later.

On the runner, where the server is deployed:

  • Reads: an account's table is read faster than RocksDB, fjall or SQLite read the same data, in its cache or not, on one thread; on sixteen it is ahead of RocksDB and SQLite, and behind fjall for what is not in the cache. An account touched holds no more than with RocksDB.
  • Writes, each synced: the server's logs are ahead of every SQLite layout at 4 and 16 writers and level with them at 64; a quarter behind RocksDB with its log files written over and its groups grown, to 64 writers, and less than half of it at 256. fjall as it comes is far behind, for want of a shared sync.
  • What the server's store still gives away is in its frames, JSON with hex where bytes would do (a sixth of the rate of writes once the disk is not the limit, a third of a replay's), and in where the firehose's events are kept: one log for all replays them 37 times as fast as a log for each account put in order.

The runner, Linux

The store-bench workflow's run on runner.knotless.dev, in the image rust:1.98, asked for with nothing else running there.

  • Linux 6.8.0-146-generic x86_64, 12 cores, load 1.97; its files on the run's overlay over the runner's disk. Every sync is fdatasync (sync_data), RocksDB's and fjall's too.
  • cargo 1.98.1; node v24.22.1; the repository at eb08b99.

Writes

1500 commits from 16 writers over 40 accounts, each on disk (fsync) before it is answered.

layoutper sp50 msp99 mslog syncsms per sync
a log for each account (the server's)15118.428.114448.76
one log for all159211.016.81884.98
SQLite for each account62819.977.0
one SQLite for all10311.11240.4
one RocksDB for all, a commit a batch, its log synced77116.859.22268.32
one fjall for all, a commit a batch, its journal synced114116.6520.7

1500 commits from 4 writers over 40 accounts, each on disk (fsync) before it is answered.

layoutper sp50 msp99 mslog syncsms per sync
a log for each account (the server's)4966.723.614977.67
one log for all5945.516.57503.34
SQLite for each account20318.736.6
one SQLite for all1108.4626.3
one RocksDB for all, a commit a batch, its log synced24516.432.77318.25
one fjall for all, a commit a batch, its journal synced11833.268.3
  • A log has room after its frames, so that a sync has no new size of the file to make durable: 8 ms a sync here where a log that grows takes 19. A log for each account does 1511 and 496 commits a second, twice what it did before it had room (734 and 204) and more, and 2.4 times SQLite for each account; one log for all does 1592 and 594. RocksDB as it comes does 771 and 245.

Writes of realistic commits, by the number of writers

1500 realistic commits from 4 writers over 40 accounts, each on disk (fsync) before it is answered.

layoutper sp50 msp99 mslog syncsms per synczeros written a commit, bytes
a log for each account, JSON frames (the server's)4946.723.114997.72
a log for each account, binary frames5186.422.714977.37
a log for each account, binary frames, each written at its offset21418.134.1149717.91
a log for each account, the same, allocated ahead by 64 KiB with fallocate21218.239.8149718.11
a log for each account, the same, allocated ahead by 64 KiB of zeros4706.927.715008.216712
a log for each account, the same, allocated ahead by 16 KiB of zeros29911.929.0149312.806136
one log for all, JSON frames5935.414.17503.36
one log for all, binary frames6085.416.77503.28
one log for all, binary frames, each written at its offset24116.121.27498.28
one log for all, the same, allocated ahead by 16 MiB with fallocate23716.422.07508.40
one log for all, the same, allocated ahead by 16 MiB of zeros7325.27.87502.7210168
one log for all, binary frames, each written at its offset, its waiting commits answered in turn (as a Wal's were)29215.823.96268.14
one log for all, binary frames, each written at its offset, a sync first letting the commits begun be written23516.426.17488.51
SQLite for each account18919.744.7
one SQLite for all10434.4132.1
one SQLite for all, one writer, every commit waiting in one transaction22016.839.7
one RocksDB for all, a commit a batch, its log synced23816.533.67468.33
one RocksDB for all, the same, its write groups let grow to 8 MiB23916.432.87518.23
one RocksDB for all, a commit a batch left unsynced, those that wait sharing one sync of its log, a sync first letting the commits begun be written3488.419.85178.19
one RocksDB for all, a commit a batch, its log synced, its log files written over6535.616.87452.94
one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over6695.520.87162.96
RocksDB for each account, a commit a batch, its log synced21318.436.6150017.86
one fjall for all, a commit a batch, its journal synced11733.361.9
one fjall for all, a commit a batch left unsynced, those that wait sharing one sync of its journal13926.198.413667.86
one fjall for all, the same, a sync first letting the commits begun be written3848.419.84618.29
fjall for each account, a commit a batch, its journal synced21418.437.6

one RocksDB for all, a commit a batch, its log synced, its log files written over: of the 9.6 MiB the commits wrote to its log, 0.01 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.

one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over: of the 9.6 MiB the commits wrote to its log, 0.01 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.

RocksDB for each account, a commit a batch, its log synced: 40 databases, made by 64 threads in 0.4 s; the process holds 18 MiB more with them open and 29 MiB more after the commits; each is 7 files, and all are 13 MiB on disk.

fjall for each account, a commit a batch, its journal synced: 40 databases, made by 64 threads in 1.2 s; the process holds 1 MiB more with them open and 10 MiB more after the commits.

1500 realistic commits from 16 writers over 40 accounts, each on disk (fsync) before it is answered.

layoutper sp50 msp99 mslog syncsms per synczeros written a commit, bytes
a log for each account, JSON frames (the server's)129911.432.2143610.22
a log for each account, binary frames123311.434.5145010.78
a log for each account, binary frames, each written at its offset74618.439.2142818.00
a log for each account, the same, allocated ahead by 64 KiB with fallocate72919.039.3143718.67
a log for each account, the same, allocated ahead by 64 KiB of zeros126711.434.7144310.656633
a log for each account, the same, allocated ahead by 16 KiB of zeros101313.838.3143413.046156
one log for all, JSON frames137511.620.11885.77
one log for all, binary frames17137.025.31874.66
one log for all, binary frames, each written at its offset91016.923.21888.74
one log for all, the same, allocated ahead by 16 MiB with fallocate93316.324.91888.52
one log for all, the same, allocated ahead by 16 MiB of zeros26315.69.71873.0110168
one log for all, binary frames, each written at its offset, its waiting commits answered in turn (as a Wal's were)34240.9283.75198.40
one log for all, binary frames, each written at its offset, a sync first letting the commits begun be written114616.421.81498.62
SQLite for each account58423.074.5
one SQLite for all116114.7413.7
one SQLite for all, one writer, every commit waiting in one transaction80718.246.2
one RocksDB for all, a commit a batch, its log synced76917.351.32248.48
one RocksDB for all, the same, its write groups let grow to 8 MiB78316.951.52268.25
one RocksDB for all, a commit a batch left unsynced, those that wait sharing one sync of its log, a sync first letting the commits begun be written17898.813.4948.28
one RocksDB for all, a commit a batch, its log synced, its log files written over16866.233.12263.68
one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over17306.034.02323.45
RocksDB for each account, a commit a batch, its log synced69119.357.6150018.88
one fjall for all, a commit a batch, its journal synced123112.7385.8
one fjall for all, a commit a batch left unsynced, those that wait sharing one sync of its journal28834.1223.86777.64
one fjall for all, the same, a sync first letting the commits begun be written17508.916.5958.15
fjall for each account, a commit a batch, its journal synced69218.857.4

one RocksDB for all, a commit a batch, its log synced, its log files written over: of the 9.6 MiB the commits wrote to its log, 0.05 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.

one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over: of the 9.6 MiB the commits wrote to its log, 0.00 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.

RocksDB for each account, a commit a batch, its log synced: 40 databases, made by 64 threads in 0.4 s; the process holds 13 MiB more with them open and 23 MiB more after the commits; each is 7 files, and all are 13 MiB on disk.

fjall for each account, a commit a batch, its journal synced: 40 databases, made by 64 threads in 1.2 s; the process holds 1 MiB more with them open and 2 MiB more after the commits.

2048 realistic commits from 64 writers over 2000 accounts, each on disk (fsync) before it is answered.

layoutper sp50 msp99 mslog syncsms per synczeros written a commit, bytes
a log for each account, JSON frames (the server's)252227.248.0204724.37
a log for each account, binary frames247627.542.8204624.64
a log for each account, binary frames, each written at its offset262823.745.7204823.44
a log for each account, the same, allocated ahead by 64 KiB with fallocate223625.185.1204627.62
a log for each account, the same, allocated ahead by 64 KiB of zeros67829.217.320469.0658202
a log for each account, the same, allocated ahead by 16 KiB of zeros454212.734.8204513.4216376
one log for all, JSON frames314719.947.56010.58
one log for all, binary frames437116.520.5637.37
one log for all, binary frames, each written at its offset370716.830.1648.48
one log for all, the same, allocated ahead by 16 MiB with fallocate379616.719.4648.29
one log for all, the same, allocated ahead by 16 MiB of zeros103176.26.7642.967449
one log for all, binary frames, each written at its offset, its waiting commits answered in turn (as a Wal's were)429148.2609.85608.44
one log for all, binary frames, each written at its offset, a sync first letting the commits begun be written393616.922.7608.44
SQLite for each account249521.573.5
one SQLite for all114545.72168.1
one SQLite for all, one writer, every commit waiting in one transaction205727.273.1
one RocksDB for all, a commit a batch, its log synced230527.144.9958.81
one RocksDB for all, the same, its write groups let grow to 8 MiB324318.436.5668.87
one RocksDB for all, a commit a batch left unsynced, those that wait sharing one sync of its log, a sync first letting the commits begun be written370516.924.1618.77
one RocksDB for all, a commit a batch, its log synced, its log files written over313219.453.1956.38
one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over333719.938.9658.80
RocksDB for each account, a commit a batch, its log synced31586.81158.62048195.24
one fjall for all, a commit a batch, its journal synced119522.31545.7
one fjall for all, a commit a batch left unsynced, those that wait sharing one sync of its journal64049.1555.13778.43
one fjall for all, the same, a sync first letting the commits begun be written361917.436.9667.93

one RocksDB for all, a commit a batch, its log synced, its log files written over: of the 13.2 MiB the commits wrote to its log, 0.01 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.

one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over: of the 13.1 MiB the commits wrote to its log, 0.28 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.

RocksDB for each account, a commit a batch, its log synced: 2000 databases, made by 64 threads in 11.6 s; the process holds 659 MiB more with them open and 686 MiB more after the commits; each is 7 files, and all are 184 MiB on disk.

4096 realistic commits from 256 writers over 2000 accounts, each on disk (fsync) before it is answered.

layoutper sp50 msp99 mslog syncsms per synczeros written a commit, bytes
a log for each account, JSON frames (the server's)331377.2164.5406169.39
a log for each account, binary frames346774.3152.7407065.93
a log for each account, binary frames, each written at its offset800829.762.9405228.85
a log for each account, the same, allocated ahead by 64 KiB with fallocate710027.7135.9405431.96
a log for each account, the same, allocated ahead by 64 KiB of zeros810727.470.8405328.9329094
a log for each account, the same, allocated ahead by 16 KiB of zeros460848.0136.9406349.6010383
one log for all, JSON frames557749.257.73321.90
one log for all, binary frames842129.342.23314.52
one log for all, binary frames, each written at its offset808729.243.03215.24
one log for all, the same, allocated ahead by 16 MiB with fallocate796429.958.03215.50
one log for all, the same, allocated ahead by 16 MiB of zeros229298.551.4324.437448
one log for all, binary frames, each written at its offset, its waiting commits answered in turn (as a Wal's were)345719.73401.514208.29
one log for all, binary frames, each written at its offset, a sync first letting the commits begun be written1343719.021.9319.07
SQLite for each account417237.4326.0
one SQLite for all1112116.37102.8
one SQLite for all, one writer, every commit waiting in one transaction306283.9162.9
one RocksDB for all, a commit a batch, its log synced243997.0197.41878.37
one RocksDB for all, the same, its write groups let grow to 8 MiB905124.852.0389.74
one RocksDB for all, a commit a batch left unsynced, those that wait sharing one sync of its log, a sync first letting the commits begun be written1055621.448.93211.30
one RocksDB for all, a commit a batch, its log synced, its log files written over294984.2176.91896.81
one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over858523.767.13611.07
one fjall for all, a commit a batch, its journal synced1201990.14066.1
one fjall for all, a commit a batch left unsynced, those that wait sharing one sync of its journal223762.7236.31969.23
one fjall for all, the same, a sync first letting the commits begun be written694536.266.94711.42

one RocksDB for all, a commit a batch, its log synced, its log files written over: of the 26.3 MiB the commits wrote to its log, 0.01 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.

one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over: of the 26.3 MiB the commits wrote to its log, 5.87 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.

  • Over 40 accounts each log is written many times, and the server's (JSON frames) does 494 commits a second from 4 writers and 1299 from 16: level with this benchmark's own log given its room before the timed commits (470 and 1267), and 2.3 and 1.8 times what it did when a log grew under every sync (212 and 714, the repository at 0a88af0).
  • Over 2000 accounts each log is written twice in the run, and room gains nothing: a log's first commit since it was opened grows the file. 2522 from 64 writers and 3313 from 256, where the benchmark's own log, its room made beforehand, does 6782 and 8107. The rows of a log for each account at 256 writers move by half from run to run (at 0a88af0, 5948 for the server's and 15183 for the benchmark's own), so only rows of one run compare.
  • SQLite for each account: 189, 584, 2495 and 4172. One SQLite whose writer takes every waiting commit into one transaction: 220, 807, 2057 and 3062. The server's logs are ahead of both at 4 and 16 writers, level at 64, and between them at 256.
  • RocksDB as it comes: 238, 769, 2305 and 2439. With its write groups let grow and its log files written over, which is its room: 669, 1730, 3337 and 8585, ahead of the server's logs at every count: by a third to 64 writers, and 2.6 times at 256.
  • One log for all with room and its waiters woken together does the most of any layout: 732, 2631, 10317 and 22929.
  • fallocate alone gains nothing (212, 729, 2236, 7100): the room must be written.
  • fjall as it comes syncs each commit under one lock: 117 to 123 a second whatever the writers. With commits left unsynced and one shared sync of its journal: 384, 1750, 3619 and 6945.

Writes with no sync: not durable

Realistic commits from 16 writers over 40 accounts, none synced: not durable. A commit is answered once it is written, to the system's cache, and making it is no part of its time.

layoutcommitsper sp50 µsp99 µs
a log for each account, JSON frames (the server's)2000004676772971
a log for each account, binary frames2000002732469293
a log for each account, binary frames, each written at its offset20000027637514214
a log for each account, the same, allocated ahead by 64 KiB with fallocate20000024911614219
a log for each account, the same, allocated ahead by 64 KiB of zeros2000002486109237
a log for each account, the same, allocated ahead by 16 KiB of zeros20000024372819232
one log for all, JSON frames200000301634751788
one log for all, binary frames200000567381731005
one log for all, binary frames, each written at its offset200000451282961151
one log for all, the same, allocated ahead by 16 MiB with fallocate200000483542771074
one log for all, the same, allocated ahead by 16 MiB of zeros20000068523120599
SQLite for each account40000209961793118
one SQLite for all400003098244188273
one SQLite for all, one writer, every commit waiting in one transaction40000753418075382
one RocksDB for all, a commit a batch40000369203761326
one RocksDB for all, the same, its write groups let grow to 8 MiB40000365063771353
RocksDB for each account, a commit a batch400004898411012
one fjall for all, a commit a batch40000146026692443
fjall for each account, a commit a batch40000103184659082

RocksDB for each account, a commit a batch: 40 databases, made by 64 threads in 0.4 s; the process holds 12 MiB more with them open and 165 MiB more after the commits; each is 17 files, and all are 341 MiB on disk.

fjall for each account, a commit a batch: 40 databases, made by 64 threads in 1.1 s; the process holds 1 MiB more with them open and 71 MiB more after the commits.

Realistic commits from 256 writers over 2000 accounts, none synced: not durable. A commit is answered once it is written, to the system's cache, and making it is no part of its time.

layoutcommitsper sp50 µsp99 µs
a log for each account, JSON frames (the server's)2000004020072174011
a log for each account, binary frames200000218478133331
a log for each account, binary frames, each written at its offset20000025234515180
a log for each account, the same, allocated ahead by 64 KiB with fallocate200000228859152150
a log for each account, the same, allocated ahead by 64 KiB of zeros200000205439134169
a log for each account, the same, allocated ahead by 16 KiB of zeros200000222680134649
one log for all, JSON frames20000027028897242280
one log for all, binary frames20000052595508920252
one log for all, binary frames, each written at its offset20000044908511125504
one log for all, the same, allocated ahead by 16 MiB with fallocate20000047870485023729
one log for all, the same, allocated ahead by 16 MiB of zeros20000055356303029366
SQLite for each account4000024991161195504
one SQLite for all4000038181982573538
one SQLite for all, one writer, every commit waiting in one transaction4000080662806061864
one RocksDB for all, a commit a batch4000038538601722224
one RocksDB for all, the same, its write groups let grow to 8 MiB40000329083151123553
one fjall for all, a commit a batch400001037835532819
  • With no sync the processor is what is left, and binary frames are what count: a log for each account does 273 thousand commits a second with them and 47 thousand with JSON. RocksDB does 37 thousand, SQLite for each account 21 thousand, fjall 15 thousand. One log for all is held to 30 to 57 thousand by its one lock.

A log replayed

log MiBframesreplay msMiB per s
64659967962
51252777537954

Realistic commits, the same in both logs, as many as make that many MiB of the server's frames.

framescommitslog MiBbytes a commitreplay msMiB per scommits per s
JSON frames (the server's)561064119626696784739
binary frames5610325948201556274252
JSON frames (the server's)449195121195253296284424
binary frames4491925559431631558274805
  • Binary frames are half the bytes of a commit (5944 against 11952) and replay 3.2 times as many commits a second.

The firehose replayed

The events of 200000 realistic commits over 2000 accounts. A whole replay is the faster of two, so of files the system's cache holds, if it has the memory to; a first event is timed after those, from each of 200 cursors in the middle half, the files open.

fromMiB on diskwhole replay msevents per sMiB of events per sfirst event from a cursor, p50 msp99 mswith an index entry each frame, p50 msp99 ms
one log for all, binary frames, an index entry each 64 frames, each frame's check SHA-256 (the server's)11347192781208220.1170.2270.0100.014
the same, each frame's check CRC32C1134191104925631010.0320.0630.0080.010
the same, each frame's check XXH3-641134185107835931870.0310.0620.0080.010
the same, each frame's check BLAKE3113410131974835840.1620.3170.0120.015
the same, no frame's check computed1134140142635442160.0240.0480.0070.011
SQLite, a table of events by seq78331563474218760.0100.013
RocksDB, a family of events by seq, an iterator, its blocks: Snappy43348940938112100.0160.032
RocksDB, a family of events by seq, an iterator, its blocks: no compression600197101487230000.0110.026
fjall, a keyspace of events by seq, an iterator, its blocks: LZ4 below the second level60427872004721280.0070.048
fjall, a keyspace of events by seq, an iterator, its blocks: no compression60428071434821110.0070.052
a log for each account, JSON frames (the server's), put in order2280689928990866875.6496875.649
  • From one log for all, a replay is 9.6 times the server's layout with SHA-256 as each frame's check and 37 times with XXH3-64 (1.08 million events a second against 29 thousand), level with RocksDB uncompressed (1.01 million) and ahead of fjall (0.72 million) and SQLite (0.63 million).
  • The first event from a cursor comes in 8 µs with an index entry a frame, against 10 for SQLite, 11 for RocksDB, 7 for fjall, and 6.9 s for the server's layout, which has every account's log to read first.

Reads

2000 accounts of 500 realistic records each. Cold is a process that has read nothing, its accounts open, the store's files read through just before, so in the system's cache if it has the memory to keep them; warm is the same reads again, everything they touch held by the process.

storea record by path, cold: p50 µsp99 µswarm: p50 µsp99 µson 16 threads, cold: per swarm: per s
a table for each account, as the server reads one, at most 4096 of their files open8.513.21.62.6216912971516
SQLite for each account15.223.311.015.3108730264165
one SQLite for all15.223.28.112.260625384328
one RocksDB for all, its blocks compressed (Snappy)17.428.65.07.3172904459304
one RocksDB for all, its blocks as they are12.424.75.07.3179728686914
one fjall for all, its blocks compressed below the second level (LZ4)10.616.94.76.5262260424714
one fjall for all, its blocks as they are10.519.64.76.3234770422109
storea listing of 50 records, cold: p50 µsp99 µswarm: p50 µsp99 µsfirst read of an account not yet open: p50 µsp99 µs
a table for each account, as the server reads one, at most 4096 of their files open16824534402341
SQLite for each account53863476139408490
one SQLite for all220309861642252
one RocksDB for all, its blocks compressed (Snappy)275569951072842
one RocksDB for all, its blocks as they are189407941062233
one fjall for all, its blocks compressed below the second level (LZ4)2122701071171221
one fjall for all, its blocks as they are2112871071171834
storeresident before any is open, MiBwith 2000 open, a record read from each, MiBKiB an accountafter 20000 more records, twice, MiBMiB on diskKiB an account
a table for each account, as the server reads one, at most 4096 of their files open12.334.911.6153.4504258
SQLite for each account12.0372.7184.7427.3870445
one SQLite for all12.148.718.8211.7841430
one RocksDB for all, its blocks compressed (Snappy)12.136.912.7154.6339174
one RocksDB for all, its blocks as they are12.236.712.5154.6498255
one fjall for all, its blocks compressed below the second level (LZ4)12.339.513.9155.1500256
one fjall for all, its blocks as they are12.139.414.0155.0500256

The tables read other ways. By pds_store::table::Table: as the server reads one (as above), its block cache holding at most 4096 of the tables' files open, the server's share of the file descriptors this process may have, the least recently read closed to make room and opened again when next read; and with every file held open, whatever the share. By this benchmark's reader, which holds every file open, keeps the blocks it reads in a map for each table where Table has one cache for all, has no bloom filter, and computes the check named over each block it reads: XXH3-64 is the one these files hold, and is compared with it; each other is compared with nothing. And by that reader from the layout before, a table of the version before: the same entries in blocks of 16 KiB, each block's last key whole in the index, its check SHA-256, which that file holds and is compared with, the file opened for each block read. The first read of an account is its table opened and a record read.

read byblockscheckfilea record by path, cold: p50 µsp99 µsa listing of 50, cold: p50 µsp99 µsfirst read of an account: p50 µsp99 µson 16 threads, cold: per swarm: per s
Table, as the server's4 KiBXXH3-64at most 4096 held open8.513.21682452341216912971516
Table, every file open4 KiBXXH3-64all held open8.413.01662082330211341955273
this benchmark's4 KiBSHA-256all held open11.017.222326629372082753494151
this benchmark's4 KiBCRC32Call held open8.013.816320126442252491023677
this benchmark's4 KiBXXH3-64all held open8.013.816320526431984153067272
this benchmark's4 KiBBLAKE3all held open12.318.724829131552214401001043
this benchmark's, the layout before16 KiB, last keys wholeSHA-256opened for each block25.860.13744416378231138520635
SQLite for each account4 KiB pagesnoneheld open15.223.3538634408490108730264165
one SQLite for all4 KiB pagesnoneheld open15.223.2220309225260625384328
one RocksDB for all, its blocks compressed (Snappy)4 KiB before SnappyXXH3held open17.428.62755692842172904459304
one RocksDB for all, its blocks as they are4 KiBXXH3held open12.424.71894072233179728686914
one fjall for all, its blocks compressed below the second level (LZ4)4 KiB before LZ4XXH3held open10.616.92122701221262260424714
one fjall for all, its blocks as they are4 KiBXXH3held open10.519.62112871834234770422109
  • The share of file descriptors is 4096 here, so every one of the 2000 tables' files stays open, and a table's check is XXH3-64. A record not in the cache takes 8.5 µs: RocksDB 12.4, fjall 10.5, SQLite 15.2. A listing of 50 takes 168: RocksDB 189, fjall 211, one SQLite 220. On 16 threads, 217 thousand records a second: RocksDB 180, fjall 235, SQLite 61 to 109.
  • From its cache: 1.6 µs a record (RocksDB 5.0, fjall 4.7, SQLite 8.1 to 11.0), a listing in 34 (76 to 107), and on 16 threads 972 thousand records a second (RocksDB 459 to 687, fjall 422, one SQLite 384).
  • The first read of an account, its table opened: 23 µs, level with one RocksDB or SQLite for all (22), which have nothing to open, and behind fjall (18). SQLite for each account takes 408.
  • An account touched holds 11.6 KiB with tables, 12.5 with RocksDB, 14.0 with fjall, 18.8 with one SQLite and 185 with SQLite for each account.
  • What each change was worth is in the last table: the layout before, 25.8 µs a record and 374 a listing; this benchmark's reader with SHA-256 as the check, 11.0 and 223.

A record read, a stage at a time

From a later store-bench run on the runner, at c9cf725, load 2.09; the other parts above are from the run at eb08b99.

40 accounts of 100 realistic records each, a table for each account read as the server reads one (Table through one BlockCache), after one pass over every record: every block a read touches is in the cache. A record's block is 301 bytes on average; its getRecord body 497 bytes. One at a time: 20000 records drawn from all, on one thread; on 16 threads, 5000 each.

stageadds to the one beforeone at a time: p50 µsp99 µson 16 threads: per s
1the path's CID, from the account's table0.410.9310566971
2the record's block, by its CID0.621.036847931
3the block decoded, as the server decodes it (pds_repo::data::decode)1.212.163747430
4the record as getRecord answers it: {uri, cid, value} as JSON, value from pds_repo::data::to_json, serialized3.727.981188006

Over HTTP, from the server (target/release/pds): 40 accounts of 100 of the same records each, written with applyWrites and folded into their tables, after one pass over every record. One at a time: 20000 requests on one kept-alive connection from Node's http client; at once, 16 connections of 2000 requests each.

stageisone at a time: p50 µsp99 µs16 at once: per s
5aa request answered with no record, GET /xrpc/_health: HTTP, the server's router and the client95.8144.922134
5bstage 4 over HTTP, com.atproto.repo.getRecord: that, and the server's own work around the store179.4288.020074
  • The store's part of a read is small: stages 1 to 4, the table, the decode and the rendering, are 3.7 µs at p50, rendering 2.5 of them.
  • The server's own work around the store is most of a getRecord past HTTP: 179.4 less 95.8 less 3.7, some 80 µs at p50, in the query read, the call routed to its method, the account found and its store locked, the takedowns looked at and the answer's headers.

Size and compression

200 accounts of 500 realistic commits each, a record a commit. The logs hold every commit whole, its event and the tree nodes later commits took out too; the tables and SQLite hold what the commits leave, the records and the tree, and no event.

kept asMiBKiB an accountbytes a record
a log for each account before compaction, JSON frames (the server's)1139.9583611952
a log for each account before compaction, binary frames566.929025944
a table for each account50.4258529
the tables, each block compressed (zstd 3)28.4145298
the tables, each block compressed with a dictionary (zstd 3)24.6126258
SQLite for each account86.3442905
one SQLite for all83.8429878
SQLite for each account, every record rewritten 3 times94.4483989
the same, vacuumed79.3406832
one SQLite for all, every record rewritten 3 times88.9455932
the same, vacuumed75.8388795
SQLite for each account, rewritten and vacuumed, its blocks in a table with a rowid63.0322660
one RocksDB for all, its blocks: Snappy34.6177363
the same, every record rewritten 3 times49.9255523
the same, compacted whole34.5177362
one RocksDB for all, its blocks: no compression49.8255522
the same, every record rewritten 3 times67.2344704
the same, compacted whole50.1257526
one RocksDB for all, its blocks: zstd29.8153313
the same, every record rewritten 3 times43.2221453
the same, compacted whole29.7152312
one fjall for all, its blocks: LZ4 below the second level49.9256523
the same, every record rewritten 3 times60.1308630
the same, compacted whole35.2180369
one fjall for all, its blocks: no compression49.9256523
the same, every record rewritten 3 times60.1308630
the same, compacted whole49.9256524

What RocksDB wrote to its files for what it was given, keys and values as they are, through the load and the rewrites, and then with the whole compaction too: to its log, to the files it writes from memory, and to those its compactions write.

RocksDB's blocksgiven MiBto its log MiBfrom memory MiBby compactions MiBwritten for each byte givenwith the whole compaction MiBfor each byte given
Snappy9979971261141.2412711.27
no compression9979971571711.3313741.38
zstd997997113981.2112371.24

The tables' data blocks, 50 MiB in 13583 blocks, each ended at 4 KiB as the server's writer ends them, compressed one at a time on one thread; the dictionary is 65536 bytes, trained on each 8th block and counted once above. A record is read by its path through two blocks, each read from its file (in the system's cache), its check computed, and searched: the median of 20000.

blocksstored bytes of each 100compressing MiB per sdecompressing MiB per sa record read µs
as they are1003.9
compressed (zstd 3)5617963612.0
compressed with a dictionary (zstd 3)481648509.8
  • A table for each account, RocksDB and fjall uncompressed hold the same data in the same bytes to one part in a hundred (258, 255 and 256 KiB an account); SQLite's best is 322. With zstd the tables are 145 KiB an account, 126 with a dictionary, and RocksDB 153.
  • A compressed block costs a record read 6 to 8 µs more.
  • A log before compaction is 11 times its table with binary frames and 23 times with JSON. RocksDB wrote 1.21 to 1.33 bytes for each byte it was given, a floor for it: this database never fills a level above its last.

The service log by the number of accounts

accountsemail and passwordservice log MiBthat log replayed at a start msstart on the checkpoint msstart bound msidle RSS MiBcompaction mspeak RSS compacting MiBRSS bound MiB
10000no7.11122830916.829333.6103.9
10000yes8.6912631617.231438.3103.8
30000no21.41933230617.075666.3181.7
30000yes25.82172931017.289182.8181.9
100000no71.44993030818.02419174.3455.3
100000yes85.95453331618.12632224.9455.3
  • A start on the checkpoint, and the memory it leaves, do not follow the accounts. The compaction here is of a log that holds every account, which this part writes and no server is left with: its time and memory follow that log.

Apple M1 Pro, macOS

bench/store/run.sh, and for the last part just store-bench 10000 30000 100000 500000.

  • Darwin 25.5.0 arm64, 8 cores, 32 GiB; APFS on the internal SSD (/dev/disk3s5). Every sync is F_FULLFSYNC, a flush through the drive's cache, RocksDB's and fjall's too.
  • cargo 1.99.1; node v24.22.1; the repository at 41ca6ca; at b571f21 for the reads and the sizes, and at 66cde9b for the last part.
  • The machine was not alone: its load was 3.6 to 8 through the run, and 23 as the last part began.

Writes

1500 commits from 16 writers over 40 accounts, each on disk (F_FULLFSYNC) before it is answered.

layoutper sp50 msp99 mslog syncsms per sync
a log for each account (the server's)51228.575.3146020.02
one log for all19168.612.41744.40
SQLite for each account25158.0163.2
one SQLite for all19958.0442.6
one RocksDB for all, a commit a batch, its log synced14039.530.22264.57
one fjall for all, a commit a batch, its journal synced20959.1319.5
SQLite for each account, fsync without F_FULLFSYNC36343.911.4
one SQLite for all, fsync without F_FULLFSYNC62591.87.7

1500 commits from 4 writers over 40 accounts, each on disk (F_FULLFSYNC) before it is answered.

layoutper sp50 msp99 mslog syncsms per sync
a log for each account (the server's)27114.225.0150013.08
one log for all4359.115.27444.61
SQLite for each account23916.032.1
one SQLite for all20114.094.9
one RocksDB for all, a commit a batch, its log synced4588.916.77034.56
one fjall for all, a commit a batch, its journal synced20617.094.1
SQLite for each account, fsync without F_FULLFSYNC62290.61.3
one SQLite for all, fsync without F_FULLFSYNC62010.54.2
  • F_FULLFSYNC flushes the whole drive, so syncs of different files queue behind each other here, and whatever shares a sync wins: the opposite of the runner, where the server is deployed.

Writes of realistic commits, by the number of writers

1500 realistic commits from 4 writers over 40 accounts, each on disk (F_FULLFSYNC) before it is answered.

layoutper sp50 msp99 mslog syncsms per synczeros written a commit, bytes
a log for each account, JSON frames (the server's)26814.525.2149913.38
a log for each account, binary frames26914.525.3150013.17
a log for each account, binary frames, each written at its offset26414.625.9149913.45
a log for each account, the same, allocated ahead by 64 KiB of zeros33810.028.9149811.226712
a log for each account, the same, allocated ahead by 16 KiB of zeros30411.728.2149911.986116
one log for all, JSON frames4139.515.77404.88
one log for all, binary frames4548.813.37174.58
one log for all, binary frames, each written at its offset4439.312.97244.65
one log for all, the same, allocated ahead by 16 MiB of zeros4568.813.47214.5410168
one log for all, binary frames, each written at its offset, its waiting commits answered in turn (as a Wal's were)4978.622.46454.62
one log for all, binary frames, each written at its offset, a sync first letting the commits begun be written5647.612.25664.64
SQLite for each account23316.831.9
one SQLite for all20118.092.0
one SQLite for all, one writer, every commit waiting in one transaction3849.625.1
one RocksDB for all, a commit a batch, its log synced4339.219.07434.56
one RocksDB for all, the same, its write groups let grow to 8 MiB4379.017.87404.53
one RocksDB for all, a commit a batch left unsynced, those that wait sharing one sync of its log, a sync first letting the commits begun be written6835.012.54564.70
one RocksDB for all, a commit a batch, its log synced, its log files written over4029.421.47494.90
one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over4149.323.07404.81
RocksDB for each account, a commit a batch, its log synced27014.526.6149813.30
one fjall for all, a commit a batch, its journal synced20119.069.1
one fjall for all, a commit a batch left unsynced, those that wait sharing one sync of its journal22814.758.911775.57
one fjall for all, the same, a sync first letting the commits begun be written6005.412.84695.17
fjall for each account, a commit a batch, its journal synced33310.325.9

one RocksDB for all, a commit a batch, its log synced, its log files written over: of the 9.6 MiB the commits wrote to its log, 0.01 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.

one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over: of the 9.6 MiB the commits wrote to its log, 0.02 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.

RocksDB for each account, a commit a batch, its log synced: 40 databases, made by 64 threads in 1.5 s; the process holds 18 MiB more with them open and 22 MiB more after the commits; each is 7 files, and all are 13 MiB on disk.

fjall for each account, a commit a batch, its journal synced: 40 databases, made by 64 threads in 4.1 s; the process holds 2 MiB more with them open and 3 MiB more after the commits.

1500 realistic commits from 16 writers over 40 accounts, each on disk (F_FULLFSYNC) before it is answered.

layoutper sp50 msp99 mslog syncsms per synczeros written a commit, bytes
a log for each account, JSON frames (the server's)53027.172.3146620.34
a log for each account, binary frames51228.375.9146120.45
a log for each account, binary frames, each written at its offset56225.362.0145118.72
a log for each account, the same, allocated ahead by 64 KiB of zeros70519.065.1142918.256752
a log for each account, the same, allocated ahead by 16 KiB of zeros59124.179.0143419.666106
one log for all, JSON frames18319.511.31674.72
one log for all, binary frames17999.512.71694.81
one log for all, binary frames, each written at its offset18869.418.61584.87
one log for all, the same, allocated ahead by 16 MiB of zeros20618.115.61724.1710168
one log for all, binary frames, each written at its offset, its waiting commits answered in turn (as a Wal's were)81916.975.43864.68
one log for all, binary frames, each written at its offset, a sync first letting the commits begun be written28935.111.01034.67
SQLite for each account25358.5149.9
one SQLite for all19863.9341.6
one SQLite for all, one writer, every commit waiting in one transaction130110.127.3
one RocksDB for all, a commit a batch, its log synced138310.029.52254.69
one RocksDB for all, the same, its write groups let grow to 8 MiB130710.132.12294.86
one RocksDB for all, a commit a batch left unsynced, those that wait sharing one sync of its log, a sync first letting the commits begun be written30955.09.5964.60
one RocksDB for all, a commit a batch, its log synced, its log files written over13819.035.92194.75
one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over13539.236.02254.81
RocksDB for each account, a commit a batch, its log synced50527.785.4149419.27
one fjall for all, a commit a batch, its journal synced20366.6263.5
one fjall for all, a commit a batch left unsynced, those that wait sharing one sync of its journal64720.081.24804.78
one fjall for all, the same, a sync first letting the commits begun be written23275.416.91234.85
fjall for each account, a commit a batch, its journal synced74718.961.2

one RocksDB for all, a commit a batch, its log synced, its log files written over: of the 9.6 MiB the commits wrote to its log, 0.00 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.

one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over: of the 9.6 MiB the commits wrote to its log, 0.05 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.

RocksDB for each account, a commit a batch, its log synced: 40 databases, made by 64 threads in 1.5 s; the process holds 17 MiB more with them open and 18 MiB more after the commits; each is 7 files, and all are 13 MiB on disk.

fjall for each account, a commit a batch, its journal synced: 40 databases, made by 64 threads in 4.1 s; the process holds 2 MiB more with them open and 3 MiB more after the commits.

2048 realistic commits from 64 writers over 2000 accounts, each on disk (F_FULLFSYNC) before it is answered.

layoutper sp50 msp99 mslog syncsms per synczeros written a commit, bytes
a log for each account, JSON frames (the server's)64883.5283.3204759.69
a log for each account, binary frames59590.8319.9204564.15
a log for each account, binary frames, each written at its offset59791.0302.1204863.87
a log for each account, the same, allocated ahead by 64 KiB of zeros80456.4336.0204776.1958202
a log for each account, the same, allocated ahead by 16 KiB of zeros77456.4373.7204470.9816354
one log for all, JSON frames498012.329.2735.41
one log for all, binary frames64159.718.9654.69
one log for all, binary frames, each written at its offset59299.724.2714.66
one log for all, the same, allocated ahead by 16 MiB of zeros74608.812.0594.437449
one log for all, binary frames, each written at its offset, its waiting commits answered in turn (as a Wal's were)109556.9233.34024.56
one log for all, binary frames, each written at its offset, a sync first letting the commits begun be written98585.914.2344.89
SQLite for each account277217.0475.0
one SQLite for all199310.81252.7
one SQLite for all, one writer, every commit waiting in one transaction261620.469.5
one RocksDB for all, a commit a batch, its log synced444214.524.0944.66
one RocksDB for all, the same, its write groups let grow to 8 MiB588810.119.7674.86
one RocksDB for all, a commit a batch left unsynced, those that wait sharing one sync of its log, a sync first letting the commits begun be written91416.113.5374.93
one RocksDB for all, a commit a batch, its log synced, its log files written over327415.739.0946.45
one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over433613.531.7657.01
RocksDB for each account, a commit a batch, its log synced71179.6234.7204657.09
one fjall for all, a commit a batch, its journal synced202305.81057.4
one fjall for all, a commit a batch left unsynced, those that wait sharing one sync of its journal149531.0172.62864.75
one fjall for all, the same, a sync first letting the commits begun be written82006.117.8434.80

one RocksDB for all, a commit a batch, its log synced, its log files written over: of the 13.1 MiB the commits wrote to its log, 0.02 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.

one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over: of the 13.1 MiB the commits wrote to its log, 0.14 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.

RocksDB for each account, a commit a batch, its log synced: 2000 databases, made by 64 threads in 77.7 s; the process holds 699 MiB more with them open and 761 MiB more after the commits; each is 7 files, and all are 195 MiB on disk.

4096 realistic commits from 256 writers over 2000 accounts, each on disk (F_FULLFSYNC) before it is answered.

layoutper sp50 msp99 mslog syncsms per synczeros written a commit, bytes
a log for each account, JSON frames (the server's)875250.7802.74093151.62
a log for each account, binary frames614349.91317.04073224.19
a log for each account, binary frames, each written at its offset672316.61162.04082203.73
a log for each account, the same, allocated ahead by 64 KiB of zeros906160.31922.14036242.0229094
a log for each account, the same, allocated ahead by 16 KiB of zeros939196.91041.94052210.4710335
one log for all, JSON frames811826.887.1905.37
one log for all, binary frames933625.472.3775.35
one log for all, binary frames, each written at its offset1269116.556.2465.74
one log for all, the same, allocated ahead by 16 MiB of zeros1793112.426.2326.537448
one log for all, binary frames, each written at its offset, its waiting commits answered in turn (as a Wal's were)1179161.7926.86904.93
one log for all, binary frames, each written at its offset, a sync first letting the commits begun be written1789613.525.4266.47
SQLite for each account285803.62444.5
one SQLite for all1961166.84865.0
one SQLite for all, one writer, every commit waiting in one transaction476756.765.4
one RocksDB for all, a commit a batch, its log synced446753.8107.91874.64
one RocksDB for all, the same, its write groups let grow to 8 MiB1718313.227.5385.43
one RocksDB for all, a commit a batch left unsynced, those that wait sharing one sync of its log, a sync first letting the commits begun be written1739514.423.2315.91
one RocksDB for all, a commit a batch, its log synced, its log files written over331170.3157.41876.35
one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over924124.058.13511.34
one fjall for all, a commit a batch, its journal synced2031149.94424.8
one fjall for all, a commit a batch left unsynced, those that wait sharing one sync of its journal771128.775.7846.21
one fjall for all, the same, a sync first letting the commits begun be written1371619.133.5347.39

one RocksDB for all, a commit a batch, its log synced, its log files written over: of the 26.3 MiB the commits wrote to its log, 0.00 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.

one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over: of the 26.3 MiB the commits wrote to its log, 4.09 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.

  • A log for each account is held to 268 to 875 commits a second however many write. One log for all, one RocksDB for all and one SQLite with a batching writer all pass it, from 1.4 times at 4 writers to 5 to 20 times at 256.
  • RocksDB loses to one log for all with its own group commit at 4 to 64 writers and is level with it at 256 once its groups grow. Its log files written over gain nothing here.
  • A sync that first lets the commits begun be written is worth 1.2 to 1.5 times to the benchmark's own log at 4 to 64 writers, and as much to RocksDB and fjall.

Writes with no sync: not durable

Realistic commits from 16 writers over 40 accounts, none synced: not durable. A commit is answered once it is written, to the system's cache, and making it is no part of its time.

layoutcommitsper sp50 µsp99 µs
a log for each account, JSON frames (the server's)20000058403544988
a log for each account, binary frames20000016706716964
a log for each account, binary frames, each written at its offset20000016542717980
a log for each account, the same, allocated ahead by 64 KiB of zeros20000042039710399
a log for each account, the same, allocated ahead by 16 KiB of zeros20000021046123528
one log for all, JSON frames200000531821901548
one log for all, binary frames200000841961021233
one log for all, binary frames, each written at its offset200000856791071201
one log for all, the same, allocated ahead by 16 MiB of zeros20000012181871814
SQLite for each account40000230672526211
one SQLite for all4000092188159176
one SQLite for all, one writer, every commit waiting in one transaction400001140310413512
one RocksDB for all, a commit a batch4000063837185947
one RocksDB for all, the same, its write groups let grow to 8 MiB40000599601851246
RocksDB for each account, a commit a batch40000774414514298
one fjall for all, a commit a batch4000034425243552
fjall for each account, a commit a batch4000056404373822

RocksDB for each account, a commit a batch: 40 databases, made by 64 threads in 1.6 s; the process holds 17 MiB more with them open and 22 MiB more after the commits; each is 17 files, and all are 344 MiB on disk.

fjall for each account, a commit a batch: 40 databases, made by 64 threads in 4.1 s; the process holds 2 MiB more with them open and 3 MiB more after the commits.

Realistic commits from 256 writers over 2000 accounts, none synced: not durable. A commit is answered once it is written, to the system's cache, and making it is no part of its time.

layoutcommitsper sp50 µsp99 µs
a log for each account, JSON frames (the server's)2000001002045466695
a log for each account, binary frames200000726951668213
a log for each account, binary frames, each written at its offset200000456653668896
a log for each account, the same, allocated ahead by 64 KiB of zeros200000490537922429
a log for each account, the same, allocated ahead by 16 KiB of zeros2000003100661026499
one log for all, JSON frames20000047434316431546
one log for all, binary frames20000083212202720479
one log for all, binary frames, each written at its offset20000084593178919922
one log for all, the same, allocated ahead by 16 MiB of zeros200000112607139114785
SQLite for each account40000122582194178226
one SQLite for all400001110968845461
one SQLite for all, one writer, every commit waiting in one transaction40000106162425836465
one RocksDB for all, a commit a batch4000071357287613977
one RocksDB for all, the same, its write groups let grow to 8 MiB4000072196306214536
one fjall for all, a commit a batch40000418722252260

A log replayed

log MiBframesreplay msMiB per s
646599591089
512527774741080

Realistic commits, the same in both logs, as many as make that many MiB of the server's frames.

framescommitslog MiBbytes a commitreplay msMiB per scommits per s
JSON frames (the server's)5610641196259108995460
binary frames5610325948191679295964
JSON frames (the server's)4491951211952475107994641
binary frames4491925559431541649290845

The firehose replayed

The events of 200000 realistic commits over 2000 accounts. A whole replay is the faster of two, so of files the system's cache holds, if it has the memory to; a first event is timed after those, from each of 200 cursors in the middle half, the files open.

fromMiB on diskwhole replay msevents per sMiB of events per sfirst event from a cursor, p50 msp99 mswith an index entry each frame, p50 msp99 ms
one log for all, binary frames, an index entry each 64 frames, each frame's check SHA-256 (the server's)11347052838048390.1110.2370.0100.023
the same, each frame's check CRC32C1134192104437830870.0340.0730.0080.016
the same, each frame's check XXH3-641134153130306338520.0270.0530.0080.035
the same, each frame's check BLAKE3113416391219893610.2630.5170.0140.016
the same, no frame's check computed1134121165554348930.0230.0520.0070.013
SQLite, a table of events by seq783164122157636110.0050.008
RocksDB, a family of events by seq, an iterator, its blocks: Snappy43323086789825650.0090.041
RocksDB, a family of events by seq, an iterator, its blocks: no compression600142141089141700.0070.044
fjall, a keyspace of events by seq, an iterator, its blocks: LZ4 below the second level60420299189529320.0060.071
fjall, a keyspace of events by seq, an iterator, its blocks: no compression60420896161628420.0060.067
a log for each account, JSON frames (the server's), put in order2280719927782827188.5117188.511

Reads

2000 accounts of 500 realistic records each. Cold is a process that has read nothing, its accounts open, the store's files read through just before, so in the system's cache if it has the memory to keep them; warm is the same reads again, everything they touch held by the process.

storea record by path, cold: p50 µsp99 µswarm: p50 µsp99 µson 16 threads, cold: per swarm: per s
a table for each account, as the server reads one27.663.82.25.9984381799876
SQLite for each account12.931.68.916.099132307460
one SQLite for all11.080.66.713.191848402958
one RocksDB for all, its blocks compressed (Snappy)11.022.05.312.2401988833919
one RocksDB for all, its blocks as they are9.133.35.813.4326502830714
one fjall for all, its blocks compressed below the second level (LZ4)8.939.76.114.5341002595908
one fjall for all, its blocks as they are8.947.46.012.9386496659354
storea listing of 50 records, cold: p50 µsp99 µswarm: p50 µsp99 µsfirst read of an account not yet open: p50 µsp99 µs
a table for each account, as the server reads one20244640673572
SQLite for each account450762671173951453
one SQLite for all1402328424614148
one RocksDB for all, its blocks compressed (Snappy)165274891361237
one RocksDB for all, its blocks as they are136229941451019
one fjall for all, its blocks compressed below the second level (LZ4)172276128178923
one fjall for all, its blocks as they are172268129197925
storeresident before any is open, MiBwith 2000 open, a record read from each, MiBKiB an accountafter 20000 more records, twice, MiBMiB on diskKiB an account
a table for each account, as the server reads one5.932.913.8169.5504258
SQLite for each account5.9363.3183.0406.6870445
one SQLite for all5.849.222.2245.8841430
one RocksDB for all, its blocks compressed (Snappy)5.833.814.3157.3339174
one RocksDB for all, its blocks as they are5.933.314.0157.3498255
one fjall for all, its blocks compressed below the second level (LZ4)5.938.916.9181.1500256
one fjall for all, its blocks as they are5.938.916.9181.7500256

The tables read other ways. By pds_store::table::Table: as the server reads one (as above), its block cache holding at most 64 of the tables' files open, the least recently read closed to make room and opened again when next read; and with every file held open, which the server does not do. By this benchmark's reader, which holds every file open, keeps the blocks it reads in a map for each table where Table has one cache for all, has no bloom filter, and computes the check named over each block it reads (compared with what the file holds for SHA-256 alone). And by that reader from the layout before: the same entries in blocks of 16 KiB, each block's last key whole in the index, the file opened for each block read. The first read of an account is its table opened and a record read.

read byblockscheckfilea record by path, cold: p50 µsp99 µsa listing of 50, cold: p50 µsp99 µsfirst read of an account: p50 µsp99 µson 16 threads, cold: per swarm: per s
Table, as the server's4 KiBSHA-256at most 64 held open27.663.82024463572984381799876
Table, not as the server's4 KiBSHA-256all held open10.426.720243635724216791537859
this benchmark's4 KiBSHA-256all held open10.319.118729432615330922048806
this benchmark's4 KiBCRC32Call held open6.515.411424827546255971861822
this benchmark's4 KiBXXH3-64all held open6.214.311018129637391292052960
this benchmark's4 KiBBLAKE3all held open17.827.632845641764016632150774
this benchmark's, the layout before16 KiB, last keys wholeSHA-256opened for each block33.788.2472679801301751951138128
SQLite for each account4 KiB pagesnoneheld open12.931.6450762395145399132307460
one SQLite for all4 KiB pagesnoneheld open11.080.61402321414891848402958
one RocksDB for all, its blocks compressed (Snappy)4 KiB before SnappyXXH3held open11.022.01652741237401988833919
one RocksDB for all, its blocks as they are4 KiBXXH3held open9.133.31362291019326502830714
one fjall for all, its blocks compressed below the second level (LZ4)4 KiB before LZ4XXH3held open8.939.7172276923341002595908
one fjall for all, its blocks as they are4 KiBXXH3held open8.947.4172268925386496659354
  • The order is the runner's, but for a record not in the cache: 27.6 µs with 64 files open at most against 10.4 with all open, as opening a file again takes about 20 µs here. From the cache a record takes 2.2 µs and 16 threads read 1.8 million a second (RocksDB 5.3 to 5.8, and 831 thousand).

A record read, a stage at a time

target/release/store-bench ladder 40 100 and node ladder.mjs 40 100 in bench/store, alone of the parts, at 0a88af0 with this part's files added; cargo 1.100.0, node v24.22.1. The machine was far from alone: its load was 120 to 170 through the run, a Docker VM taking two cores and builds the rest, and the stage over HTTP, which waits on the scheduler for every request, is several times what it was minutes before (a _health p50 of 86 µs then, 384 µs here): these figures say that the part runs, not what the stages cost.

40 accounts of 100 realistic records each, a table for each account read as the server reads one (Table through one BlockCache), after one pass over every record: every block a read touches is in the cache. A record's block is 301 bytes on average; its getRecord body 497 bytes. One at a time: 20000 records drawn from all, on one thread; on 16 threads, 5000 each.

stageadds to the one beforeone at a time: p50 µsp99 µson 16 threads: per s
1the path's CID, from the account's table0.581.382201981
2the record's block, by its CID1.256.00717841
3the block decoded, as the server decodes it (pds_repo::data::decode)2.047.29826944
4the record as getRecord answers it: {uri, cid, value} as JSON, value from pds_repo::data::to_json, serialized4.92152.75258587

Over HTTP, from the server (target/release/pds): 40 accounts of 100 of the same records each, written with applyWrites and folded into their tables, after one pass over every record. One at a time: 20000 requests on one kept-alive connection from Node's http client; at once, 16 connections of 2000 requests each.

stageisone at a time: p50 µsp99 µs16 at once: per s
5aa request answered with no record, GET /xrpc/_health: HTTP, the server's router and the client383.922856.33353
5bstage 4 over HTTP, com.atproto.repo.getRecord: that, and the server's own work around the store398.623835.53209
  • In the process, a record's read is 5 µs to its body, of which the table's two keys are 1.3 and rendering the JSON 2.9: rendering is the dearest stage, more than the store and the decoding together.
  • Over HTTP under this load, getRecord less _health is 15 µs, which less stage 4 leaves 10 µs of the server's own work around the store; with the scheduler's waits in both rows, that is noise of the same size.

Size and compression

200 accounts of 500 realistic commits each, a record a commit. The logs hold every commit whole, its event and the tree nodes later commits took out too; the tables and SQLite hold what the commits leave, the records and the tree, and no event.

kept asMiBKiB an accountbytes a record
a log for each account before compaction, JSON frames (the server's)1139.9583611952
a log for each account before compaction, binary frames566.929025944
a table for each account50.4258529
the tables, each block compressed (zstd 3)28.4145298
the tables, each block compressed with a dictionary (zstd 3)24.6126258
SQLite for each account86.3442905
one SQLite for all83.8429878
SQLite for each account, every record rewritten 3 times94.4483989
the same, vacuumed79.3406832
one SQLite for all, every record rewritten 3 times88.9455932
the same, vacuumed75.8388795
SQLite for each account, rewritten and vacuumed, its blocks in a table with a rowid63.0322660
one RocksDB for all, its blocks: Snappy34.6177363
the same, every record rewritten 3 times42.1216442
the same, compacted whole34.5177362
one RocksDB for all, its blocks: no compression49.8255522
the same, every record rewritten 3 times59.6305625
the same, compacted whole50.2257526
one RocksDB for all, its blocks: zstd29.8153313
the same, every record rewritten 3 times36.8188385
the same, compacted whole29.8152312
one fjall for all, its blocks: LZ4 below the second level49.9256523
the same, every record rewritten 3 times60.3309632
the same, compacted whole35.2180369
one fjall for all, its blocks: no compression49.9256523
the same, every record rewritten 3 times60.3309632
the same, compacted whole49.9256524

What RocksDB wrote to its files for what it was given, keys and values as they are, through the load and the rewrites, and then with the whole compaction too: to its log, to the files it writes from memory, and to those its compactions write.

RocksDB's blocksgiven MiBto its log MiBfrom memory MiBby compactions MiBwritten for each byte givenwith the whole compaction MiBfor each byte given
Snappy9969961331141.2512771.28
no compression9969961681711.3413851.39
zstd996996119981.2212421.25

The tables' data blocks, 50 MiB in 13583 blocks, each ended at 4 KiB as the server's writer ends them, compressed one at a time on one thread; the dictionary is 65536 bytes, trained on each 8th block and counted once above. A record is read by its path through two blocks, each read from its file (in the system's cache), its check computed, and searched: the median of 20000.

blocksstored bytes of each 100compressing MiB per sdecompressing MiB per sa record read µs
as they are1008.2
compressed (zstd 3)5617253816.2
compressed with a dictionary (zstd 3)4813663813.3

The service log by the number of accounts

accountsemail and passwordservice log MiBthat log replayed at a start msstart on the checkpoint msstart bound msidle RSS MiBcompaction mspeak RSS compacting MiBRSS bound MiB
10000no7.1652229618.915335.3105.6
10000yes8.6702035018.916240.3105.6
30000no21.41492230319.143369.5183.7
30000yes25.81822429919.151784.5183.8
100000no71.44532629719.71501175.0457.2
100000yes85.94912530519.91619225.3457.2
500000no357.647643229523.87833750.32019.7
500000yes430.925973629624.487531000.22019.7
  • A start on the checkpoint is 20 to 36 ms from 10000 accounts to 500000, and leaves 19 to 24 MiB resident: neither follows the accounts.
  • Replaying a log goes at 75 to 170 MiB a second with the accounts' entries noted, so the service log is compacted at 8 MiB, whatever its checkpoint's size.
  • The compaction timed is of a log that holds every account, which this part writes and no server is left with: what it holds follows that log, about 2 KiB an account in it.