chadtmiller.com/monolith
An atproto PDS
The store benchmark: results
just store-bench (README.md), one section a machine. A
machine's numbers compare only with its own, and the sections are of
different commits: the runner's is the later.
On the runner, where the server is deployed:
- Reads: an account's table is read faster than RocksDB, fjall or SQLite read the same data, in its cache or not, on one thread; on sixteen it is ahead of RocksDB and SQLite, and behind fjall for what is not in the cache. An account touched holds no more than with RocksDB.
- Writes, each synced: the server's logs are ahead of every SQLite layout at 4 and 16 writers and level with them at 64; a quarter behind RocksDB with its log files written over and its groups grown, to 64 writers, and less than half of it at 256. fjall as it comes is far behind, for want of a shared sync.
- What the server's store still gives away is in its frames, JSON with hex where bytes would do (a sixth of the rate of writes once the disk is not the limit, a third of a replay's), and in where the firehose's events are kept: one log for all replays them 37 times as fast as a log for each account put in order.
The runner, Linux
The store-bench workflow's run on runner.knotless.dev, in the image
rust:1.98, asked for with nothing else running there.
- Linux 6.8.0-146-generic x86_64, 12 cores, load 1.97; its files on the
run's overlay over the runner's disk. Every sync is
fdatasync(sync_data), RocksDB's and fjall's too. - cargo 1.98.1; node v24.22.1; the repository at
eb08b99.
Writes
1500 commits from 16 writers over 40 accounts, each on disk (fsync) before it is answered.
| layout | per s | p50 ms | p99 ms | log syncs | ms per sync |
|---|---|---|---|---|---|
| a log for each account (the server's) | 1511 | 8.4 | 28.1 | 1444 | 8.76 |
| one log for all | 1592 | 11.0 | 16.8 | 188 | 4.98 |
| SQLite for each account | 628 | 19.9 | 77.0 | ||
| one SQLite for all | 103 | 11.1 | 1240.4 | ||
| one RocksDB for all, a commit a batch, its log synced | 771 | 16.8 | 59.2 | 226 | 8.32 |
| one fjall for all, a commit a batch, its journal synced | 114 | 116.6 | 520.7 |
1500 commits from 4 writers over 40 accounts, each on disk (fsync) before it is answered.
| layout | per s | p50 ms | p99 ms | log syncs | ms per sync |
|---|---|---|---|---|---|
| a log for each account (the server's) | 496 | 6.7 | 23.6 | 1497 | 7.67 |
| one log for all | 594 | 5.5 | 16.5 | 750 | 3.34 |
| SQLite for each account | 203 | 18.7 | 36.6 | ||
| one SQLite for all | 110 | 8.4 | 626.3 | ||
| one RocksDB for all, a commit a batch, its log synced | 245 | 16.4 | 32.7 | 731 | 8.25 |
| one fjall for all, a commit a batch, its journal synced | 118 | 33.2 | 68.3 |
- A log has room after its frames, so that a sync has no new size of the file to make durable: 8 ms a sync here where a log that grows takes 19. A log for each account does 1511 and 496 commits a second, twice what it did before it had room (734 and 204) and more, and 2.4 times SQLite for each account; one log for all does 1592 and 594. RocksDB as it comes does 771 and 245.
Writes of realistic commits, by the number of writers
1500 realistic commits from 4 writers over 40 accounts, each on disk (fsync) before it is answered.
| layout | per s | p50 ms | p99 ms | log syncs | ms per sync | zeros written a commit, bytes |
|---|---|---|---|---|---|---|
| a log for each account, JSON frames (the server's) | 494 | 6.7 | 23.1 | 1499 | 7.72 | |
| a log for each account, binary frames | 518 | 6.4 | 22.7 | 1497 | 7.37 | |
| a log for each account, binary frames, each written at its offset | 214 | 18.1 | 34.1 | 1497 | 17.91 | |
| a log for each account, the same, allocated ahead by 64 KiB with fallocate | 212 | 18.2 | 39.8 | 1497 | 18.11 | |
| a log for each account, the same, allocated ahead by 64 KiB of zeros | 470 | 6.9 | 27.7 | 1500 | 8.21 | 6712 |
| a log for each account, the same, allocated ahead by 16 KiB of zeros | 299 | 11.9 | 29.0 | 1493 | 12.80 | 6136 |
| one log for all, JSON frames | 593 | 5.4 | 14.1 | 750 | 3.36 | |
| one log for all, binary frames | 608 | 5.4 | 16.7 | 750 | 3.28 | |
| one log for all, binary frames, each written at its offset | 241 | 16.1 | 21.2 | 749 | 8.28 | |
| one log for all, the same, allocated ahead by 16 MiB with fallocate | 237 | 16.4 | 22.0 | 750 | 8.40 | |
| one log for all, the same, allocated ahead by 16 MiB of zeros | 732 | 5.2 | 7.8 | 750 | 2.72 | 10168 |
one log for all, binary frames, each written at its offset, its waiting commits answered in turn (as a Wal's were) | 292 | 15.8 | 23.9 | 626 | 8.14 | |
| one log for all, binary frames, each written at its offset, a sync first letting the commits begun be written | 235 | 16.4 | 26.1 | 748 | 8.51 | |
| SQLite for each account | 189 | 19.7 | 44.7 | |||
| one SQLite for all | 104 | 34.4 | 132.1 | |||
| one SQLite for all, one writer, every commit waiting in one transaction | 220 | 16.8 | 39.7 | |||
| one RocksDB for all, a commit a batch, its log synced | 238 | 16.5 | 33.6 | 746 | 8.33 | |
| one RocksDB for all, the same, its write groups let grow to 8 MiB | 239 | 16.4 | 32.8 | 751 | 8.23 | |
| one RocksDB for all, a commit a batch left unsynced, those that wait sharing one sync of its log, a sync first letting the commits begun be written | 348 | 8.4 | 19.8 | 517 | 8.19 | |
| one RocksDB for all, a commit a batch, its log synced, its log files written over | 653 | 5.6 | 16.8 | 745 | 2.94 | |
| one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over | 669 | 5.5 | 20.8 | 716 | 2.96 | |
| RocksDB for each account, a commit a batch, its log synced | 213 | 18.4 | 36.6 | 1500 | 17.86 | |
| one fjall for all, a commit a batch, its journal synced | 117 | 33.3 | 61.9 | |||
| one fjall for all, a commit a batch left unsynced, those that wait sharing one sync of its journal | 139 | 26.1 | 98.4 | 1366 | 7.86 | |
| one fjall for all, the same, a sync first letting the commits begun be written | 384 | 8.4 | 19.8 | 461 | 8.29 | |
| fjall for each account, a commit a batch, its journal synced | 214 | 18.4 | 37.6 |
one RocksDB for all, a commit a batch, its log synced, its log files written over: of the 9.6 MiB the commits wrote to its log, 0.01 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.
one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over: of the 9.6 MiB the commits wrote to its log, 0.01 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.
RocksDB for each account, a commit a batch, its log synced: 40 databases, made by 64 threads in 0.4 s; the process holds 18 MiB more with them open and 29 MiB more after the commits; each is 7 files, and all are 13 MiB on disk.
fjall for each account, a commit a batch, its journal synced: 40 databases, made by 64 threads in 1.2 s; the process holds 1 MiB more with them open and 10 MiB more after the commits.
1500 realistic commits from 16 writers over 40 accounts, each on disk (fsync) before it is answered.
| layout | per s | p50 ms | p99 ms | log syncs | ms per sync | zeros written a commit, bytes |
|---|---|---|---|---|---|---|
| a log for each account, JSON frames (the server's) | 1299 | 11.4 | 32.2 | 1436 | 10.22 | |
| a log for each account, binary frames | 1233 | 11.4 | 34.5 | 1450 | 10.78 | |
| a log for each account, binary frames, each written at its offset | 746 | 18.4 | 39.2 | 1428 | 18.00 | |
| a log for each account, the same, allocated ahead by 64 KiB with fallocate | 729 | 19.0 | 39.3 | 1437 | 18.67 | |
| a log for each account, the same, allocated ahead by 64 KiB of zeros | 1267 | 11.4 | 34.7 | 1443 | 10.65 | 6633 |
| a log for each account, the same, allocated ahead by 16 KiB of zeros | 1013 | 13.8 | 38.3 | 1434 | 13.04 | 6156 |
| one log for all, JSON frames | 1375 | 11.6 | 20.1 | 188 | 5.77 | |
| one log for all, binary frames | 1713 | 7.0 | 25.3 | 187 | 4.66 | |
| one log for all, binary frames, each written at its offset | 910 | 16.9 | 23.2 | 188 | 8.74 | |
| one log for all, the same, allocated ahead by 16 MiB with fallocate | 933 | 16.3 | 24.9 | 188 | 8.52 | |
| one log for all, the same, allocated ahead by 16 MiB of zeros | 2631 | 5.6 | 9.7 | 187 | 3.01 | 10168 |
one log for all, binary frames, each written at its offset, its waiting commits answered in turn (as a Wal's were) | 342 | 40.9 | 283.7 | 519 | 8.40 | |
| one log for all, binary frames, each written at its offset, a sync first letting the commits begun be written | 1146 | 16.4 | 21.8 | 149 | 8.62 | |
| SQLite for each account | 584 | 23.0 | 74.5 | |||
| one SQLite for all | 116 | 114.7 | 413.7 | |||
| one SQLite for all, one writer, every commit waiting in one transaction | 807 | 18.2 | 46.2 | |||
| one RocksDB for all, a commit a batch, its log synced | 769 | 17.3 | 51.3 | 224 | 8.48 | |
| one RocksDB for all, the same, its write groups let grow to 8 MiB | 783 | 16.9 | 51.5 | 226 | 8.25 | |
| one RocksDB for all, a commit a batch left unsynced, those that wait sharing one sync of its log, a sync first letting the commits begun be written | 1789 | 8.8 | 13.4 | 94 | 8.28 | |
| one RocksDB for all, a commit a batch, its log synced, its log files written over | 1686 | 6.2 | 33.1 | 226 | 3.68 | |
| one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over | 1730 | 6.0 | 34.0 | 232 | 3.45 | |
| RocksDB for each account, a commit a batch, its log synced | 691 | 19.3 | 57.6 | 1500 | 18.88 | |
| one fjall for all, a commit a batch, its journal synced | 123 | 112.7 | 385.8 | |||
| one fjall for all, a commit a batch left unsynced, those that wait sharing one sync of its journal | 288 | 34.1 | 223.8 | 677 | 7.64 | |
| one fjall for all, the same, a sync first letting the commits begun be written | 1750 | 8.9 | 16.5 | 95 | 8.15 | |
| fjall for each account, a commit a batch, its journal synced | 692 | 18.8 | 57.4 |
one RocksDB for all, a commit a batch, its log synced, its log files written over: of the 9.6 MiB the commits wrote to its log, 0.05 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.
one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over: of the 9.6 MiB the commits wrote to its log, 0.00 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.
RocksDB for each account, a commit a batch, its log synced: 40 databases, made by 64 threads in 0.4 s; the process holds 13 MiB more with them open and 23 MiB more after the commits; each is 7 files, and all are 13 MiB on disk.
fjall for each account, a commit a batch, its journal synced: 40 databases, made by 64 threads in 1.2 s; the process holds 1 MiB more with them open and 2 MiB more after the commits.
2048 realistic commits from 64 writers over 2000 accounts, each on disk (fsync) before it is answered.
| layout | per s | p50 ms | p99 ms | log syncs | ms per sync | zeros written a commit, bytes |
|---|---|---|---|---|---|---|
| a log for each account, JSON frames (the server's) | 2522 | 27.2 | 48.0 | 2047 | 24.37 | |
| a log for each account, binary frames | 2476 | 27.5 | 42.8 | 2046 | 24.64 | |
| a log for each account, binary frames, each written at its offset | 2628 | 23.7 | 45.7 | 2048 | 23.44 | |
| a log for each account, the same, allocated ahead by 64 KiB with fallocate | 2236 | 25.1 | 85.1 | 2046 | 27.62 | |
| a log for each account, the same, allocated ahead by 64 KiB of zeros | 6782 | 9.2 | 17.3 | 2046 | 9.06 | 58202 |
| a log for each account, the same, allocated ahead by 16 KiB of zeros | 4542 | 12.7 | 34.8 | 2045 | 13.42 | 16376 |
| one log for all, JSON frames | 3147 | 19.9 | 47.5 | 60 | 10.58 | |
| one log for all, binary frames | 4371 | 16.5 | 20.5 | 63 | 7.37 | |
| one log for all, binary frames, each written at its offset | 3707 | 16.8 | 30.1 | 64 | 8.48 | |
| one log for all, the same, allocated ahead by 16 MiB with fallocate | 3796 | 16.7 | 19.4 | 64 | 8.29 | |
| one log for all, the same, allocated ahead by 16 MiB of zeros | 10317 | 6.2 | 6.7 | 64 | 2.96 | 7449 |
one log for all, binary frames, each written at its offset, its waiting commits answered in turn (as a Wal's were) | 429 | 148.2 | 609.8 | 560 | 8.44 | |
| one log for all, binary frames, each written at its offset, a sync first letting the commits begun be written | 3936 | 16.9 | 22.7 | 60 | 8.44 | |
| SQLite for each account | 2495 | 21.5 | 73.5 | |||
| one SQLite for all | 114 | 545.7 | 2168.1 | |||
| one SQLite for all, one writer, every commit waiting in one transaction | 2057 | 27.2 | 73.1 | |||
| one RocksDB for all, a commit a batch, its log synced | 2305 | 27.1 | 44.9 | 95 | 8.81 | |
| one RocksDB for all, the same, its write groups let grow to 8 MiB | 3243 | 18.4 | 36.5 | 66 | 8.87 | |
| one RocksDB for all, a commit a batch left unsynced, those that wait sharing one sync of its log, a sync first letting the commits begun be written | 3705 | 16.9 | 24.1 | 61 | 8.77 | |
| one RocksDB for all, a commit a batch, its log synced, its log files written over | 3132 | 19.4 | 53.1 | 95 | 6.38 | |
| one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over | 3337 | 19.9 | 38.9 | 65 | 8.80 | |
| RocksDB for each account, a commit a batch, its log synced | 315 | 86.8 | 1158.6 | 2048 | 195.24 | |
| one fjall for all, a commit a batch, its journal synced | 119 | 522.3 | 1545.7 | |||
| one fjall for all, a commit a batch left unsynced, those that wait sharing one sync of its journal | 640 | 49.1 | 555.1 | 377 | 8.43 | |
| one fjall for all, the same, a sync first letting the commits begun be written | 3619 | 17.4 | 36.9 | 66 | 7.93 |
one RocksDB for all, a commit a batch, its log synced, its log files written over: of the 13.2 MiB the commits wrote to its log, 0.01 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.
one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over: of the 13.1 MiB the commits wrote to its log, 0.28 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.
RocksDB for each account, a commit a batch, its log synced: 2000 databases, made by 64 threads in 11.6 s; the process holds 659 MiB more with them open and 686 MiB more after the commits; each is 7 files, and all are 184 MiB on disk.
4096 realistic commits from 256 writers over 2000 accounts, each on disk (fsync) before it is answered.
| layout | per s | p50 ms | p99 ms | log syncs | ms per sync | zeros written a commit, bytes |
|---|---|---|---|---|---|---|
| a log for each account, JSON frames (the server's) | 3313 | 77.2 | 164.5 | 4061 | 69.39 | |
| a log for each account, binary frames | 3467 | 74.3 | 152.7 | 4070 | 65.93 | |
| a log for each account, binary frames, each written at its offset | 8008 | 29.7 | 62.9 | 4052 | 28.85 | |
| a log for each account, the same, allocated ahead by 64 KiB with fallocate | 7100 | 27.7 | 135.9 | 4054 | 31.96 | |
| a log for each account, the same, allocated ahead by 64 KiB of zeros | 8107 | 27.4 | 70.8 | 4053 | 28.93 | 29094 |
| a log for each account, the same, allocated ahead by 16 KiB of zeros | 4608 | 48.0 | 136.9 | 4063 | 49.60 | 10383 |
| one log for all, JSON frames | 5577 | 49.2 | 57.7 | 33 | 21.90 | |
| one log for all, binary frames | 8421 | 29.3 | 42.2 | 33 | 14.52 | |
| one log for all, binary frames, each written at its offset | 8087 | 29.2 | 43.0 | 32 | 15.24 | |
| one log for all, the same, allocated ahead by 16 MiB with fallocate | 7964 | 29.9 | 58.0 | 32 | 15.50 | |
| one log for all, the same, allocated ahead by 16 MiB of zeros | 22929 | 8.5 | 51.4 | 32 | 4.43 | 7448 |
one log for all, binary frames, each written at its offset, its waiting commits answered in turn (as a Wal's were) | 345 | 719.7 | 3401.5 | 1420 | 8.29 | |
| one log for all, binary frames, each written at its offset, a sync first letting the commits begun be written | 13437 | 19.0 | 21.9 | 31 | 9.07 | |
| SQLite for each account | 4172 | 37.4 | 326.0 | |||
| one SQLite for all | 111 | 2116.3 | 7102.8 | |||
| one SQLite for all, one writer, every commit waiting in one transaction | 3062 | 83.9 | 162.9 | |||
| one RocksDB for all, a commit a batch, its log synced | 2439 | 97.0 | 197.4 | 187 | 8.37 | |
| one RocksDB for all, the same, its write groups let grow to 8 MiB | 9051 | 24.8 | 52.0 | 38 | 9.74 | |
| one RocksDB for all, a commit a batch left unsynced, those that wait sharing one sync of its log, a sync first letting the commits begun be written | 10556 | 21.4 | 48.9 | 32 | 11.30 | |
| one RocksDB for all, a commit a batch, its log synced, its log files written over | 2949 | 84.2 | 176.9 | 189 | 6.81 | |
| one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over | 8585 | 23.7 | 67.1 | 36 | 11.07 | |
| one fjall for all, a commit a batch, its journal synced | 120 | 1990.1 | 4066.1 | |||
| one fjall for all, a commit a batch left unsynced, those that wait sharing one sync of its journal | 2237 | 62.7 | 236.3 | 196 | 9.23 | |
| one fjall for all, the same, a sync first letting the commits begun be written | 6945 | 36.2 | 66.9 | 47 | 11.42 |
one RocksDB for all, a commit a batch, its log synced, its log files written over: of the 26.3 MiB the commits wrote to its log, 0.01 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.
one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over: of the 26.3 MiB the commits wrote to its log, 5.87 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.
- Over 40 accounts each log is written many times, and the server's
(JSON frames) does 494 commits a second from 4 writers and 1299 from 16:
level with this benchmark's own log given its room before the timed
commits (470 and 1267), and 2.3 and 1.8 times what it did when a log
grew under every sync (212 and 714, the repository at
0a88af0). - Over 2000 accounts each log is written twice in the run, and room gains
nothing: a log's first commit since it was opened grows the file. 2522
from 64 writers and 3313 from 256, where the benchmark's own log, its
room made beforehand, does 6782 and 8107. The rows of a log for each
account at 256 writers move by half from run to run (at
0a88af0, 5948 for the server's and 15183 for the benchmark's own), so only rows of one run compare. - SQLite for each account: 189, 584, 2495 and 4172. One SQLite whose writer takes every waiting commit into one transaction: 220, 807, 2057 and 3062. The server's logs are ahead of both at 4 and 16 writers, level at 64, and between them at 256.
- RocksDB as it comes: 238, 769, 2305 and 2439. With its write groups let grow and its log files written over, which is its room: 669, 1730, 3337 and 8585, ahead of the server's logs at every count: by a third to 64 writers, and 2.6 times at 256.
- One log for all with room and its waiters woken together does the most of any layout: 732, 2631, 10317 and 22929.
fallocatealone gains nothing (212, 729, 2236, 7100): the room must be written.- fjall as it comes syncs each commit under one lock: 117 to 123 a second whatever the writers. With commits left unsynced and one shared sync of its journal: 384, 1750, 3619 and 6945.
Writes with no sync: not durable
Realistic commits from 16 writers over 40 accounts, none synced: not durable. A commit is answered once it is written, to the system's cache, and making it is no part of its time.
| layout | commits | per s | p50 µs | p99 µs |
|---|---|---|---|---|
| a log for each account, JSON frames (the server's) | 200000 | 46767 | 72 | 971 |
| a log for each account, binary frames | 200000 | 273246 | 9 | 293 |
| a log for each account, binary frames, each written at its offset | 200000 | 276375 | 14 | 214 |
| a log for each account, the same, allocated ahead by 64 KiB with fallocate | 200000 | 249116 | 14 | 219 |
| a log for each account, the same, allocated ahead by 64 KiB of zeros | 200000 | 248610 | 9 | 237 |
| a log for each account, the same, allocated ahead by 16 KiB of zeros | 200000 | 243728 | 19 | 232 |
| one log for all, JSON frames | 200000 | 30163 | 475 | 1788 |
| one log for all, binary frames | 200000 | 56738 | 173 | 1005 |
| one log for all, binary frames, each written at its offset | 200000 | 45128 | 296 | 1151 |
| one log for all, the same, allocated ahead by 16 MiB with fallocate | 200000 | 48354 | 277 | 1074 |
| one log for all, the same, allocated ahead by 16 MiB of zeros | 200000 | 68523 | 120 | 599 |
| SQLite for each account | 40000 | 20996 | 179 | 3118 |
| one SQLite for all | 40000 | 3098 | 244 | 188273 |
| one SQLite for all, one writer, every commit waiting in one transaction | 40000 | 7534 | 1807 | 5382 |
| one RocksDB for all, a commit a batch | 40000 | 36920 | 376 | 1326 |
| one RocksDB for all, the same, its write groups let grow to 8 MiB | 40000 | 36506 | 377 | 1353 |
| RocksDB for each account, a commit a batch | 40000 | 4898 | 41 | 1012 |
| one fjall for all, a commit a batch | 40000 | 14602 | 669 | 2443 |
| fjall for each account, a commit a batch | 40000 | 10318 | 46 | 59082 |
RocksDB for each account, a commit a batch: 40 databases, made by 64 threads in 0.4 s; the process holds 12 MiB more with them open and 165 MiB more after the commits; each is 17 files, and all are 341 MiB on disk.
fjall for each account, a commit a batch: 40 databases, made by 64 threads in 1.1 s; the process holds 1 MiB more with them open and 71 MiB more after the commits.
Realistic commits from 256 writers over 2000 accounts, none synced: not durable. A commit is answered once it is written, to the system's cache, and making it is no part of its time.
| layout | commits | per s | p50 µs | p99 µs |
|---|---|---|---|---|
| a log for each account, JSON frames (the server's) | 200000 | 40200 | 72 | 174011 |
| a log for each account, binary frames | 200000 | 218478 | 13 | 3331 |
| a log for each account, binary frames, each written at its offset | 200000 | 252345 | 15 | 180 |
| a log for each account, the same, allocated ahead by 64 KiB with fallocate | 200000 | 228859 | 15 | 2150 |
| a log for each account, the same, allocated ahead by 64 KiB of zeros | 200000 | 205439 | 13 | 4169 |
| a log for each account, the same, allocated ahead by 16 KiB of zeros | 200000 | 222680 | 13 | 4649 |
| one log for all, JSON frames | 200000 | 27028 | 8972 | 42280 |
| one log for all, binary frames | 200000 | 52595 | 5089 | 20252 |
| one log for all, binary frames, each written at its offset | 200000 | 44908 | 5111 | 25504 |
| one log for all, the same, allocated ahead by 16 MiB with fallocate | 200000 | 47870 | 4850 | 23729 |
| one log for all, the same, allocated ahead by 16 MiB of zeros | 200000 | 55356 | 3030 | 29366 |
| SQLite for each account | 40000 | 24991 | 161 | 195504 |
| one SQLite for all | 40000 | 3818 | 198 | 2573538 |
| one SQLite for all, one writer, every commit waiting in one transaction | 40000 | 8066 | 28060 | 61864 |
| one RocksDB for all, a commit a batch | 40000 | 38538 | 6017 | 22224 |
| one RocksDB for all, the same, its write groups let grow to 8 MiB | 40000 | 32908 | 3151 | 123553 |
| one fjall for all, a commit a batch | 40000 | 10378 | 35 | 532819 |
- With no sync the processor is what is left, and binary frames are what count: a log for each account does 273 thousand commits a second with them and 47 thousand with JSON. RocksDB does 37 thousand, SQLite for each account 21 thousand, fjall 15 thousand. One log for all is held to 30 to 57 thousand by its one lock.
A log replayed
| log MiB | frames | replay ms | MiB per s |
|---|---|---|---|
| 64 | 6599 | 67 | 962 |
| 512 | 52777 | 537 | 954 |
Realistic commits, the same in both logs, as many as make that many MiB of the server's frames.
| frames | commits | log MiB | bytes a commit | replay ms | MiB per s | commits per s |
|---|---|---|---|---|---|---|
| JSON frames (the server's) | 5610 | 64 | 11962 | 66 | 967 | 84739 |
| binary frames | 5610 | 32 | 5948 | 20 | 1556 | 274252 |
| JSON frames (the server's) | 44919 | 512 | 11952 | 532 | 962 | 84424 |
| binary frames | 44919 | 255 | 5943 | 163 | 1558 | 274805 |
- Binary frames are half the bytes of a commit (5944 against 11952) and replay 3.2 times as many commits a second.
The firehose replayed
The events of 200000 realistic commits over 2000 accounts. A whole replay is the faster of two, so of files the system's cache holds, if it has the memory to; a first event is timed after those, from each of 200 cursors in the middle half, the files open.
| from | MiB on disk | whole replay ms | events per s | MiB of events per s | first event from a cursor, p50 ms | p99 ms | with an index entry each frame, p50 ms | p99 ms |
|---|---|---|---|---|---|---|---|---|
| one log for all, binary frames, an index entry each 64 frames, each frame's check SHA-256 (the server's) | 1134 | 719 | 278120 | 822 | 0.117 | 0.227 | 0.010 | 0.014 |
| the same, each frame's check CRC32C | 1134 | 191 | 1049256 | 3101 | 0.032 | 0.063 | 0.008 | 0.010 |
| the same, each frame's check XXH3-64 | 1134 | 185 | 1078359 | 3187 | 0.031 | 0.062 | 0.008 | 0.010 |
| the same, each frame's check BLAKE3 | 1134 | 1013 | 197483 | 584 | 0.162 | 0.317 | 0.012 | 0.015 |
| the same, no frame's check computed | 1134 | 140 | 1426354 | 4216 | 0.024 | 0.048 | 0.007 | 0.011 |
| SQLite, a table of events by seq | 783 | 315 | 634742 | 1876 | 0.010 | 0.013 | ||
| RocksDB, a family of events by seq, an iterator, its blocks: Snappy | 433 | 489 | 409381 | 1210 | 0.016 | 0.032 | ||
| RocksDB, a family of events by seq, an iterator, its blocks: no compression | 600 | 197 | 1014872 | 3000 | 0.011 | 0.026 | ||
| fjall, a keyspace of events by seq, an iterator, its blocks: LZ4 below the second level | 604 | 278 | 720047 | 2128 | 0.007 | 0.048 | ||
| fjall, a keyspace of events by seq, an iterator, its blocks: no compression | 604 | 280 | 714348 | 2111 | 0.007 | 0.052 | ||
| a log for each account, JSON frames (the server's), put in order | 2280 | 6899 | 28990 | 86 | 6875.649 | 6875.649 |
- From one log for all, a replay is 9.6 times the server's layout with SHA-256 as each frame's check and 37 times with XXH3-64 (1.08 million events a second against 29 thousand), level with RocksDB uncompressed (1.01 million) and ahead of fjall (0.72 million) and SQLite (0.63 million).
- The first event from a cursor comes in 8 µs with an index entry a frame, against 10 for SQLite, 11 for RocksDB, 7 for fjall, and 6.9 s for the server's layout, which has every account's log to read first.
Reads
2000 accounts of 500 realistic records each. Cold is a process that has read nothing, its accounts open, the store's files read through just before, so in the system's cache if it has the memory to keep them; warm is the same reads again, everything they touch held by the process.
| store | a record by path, cold: p50 µs | p99 µs | warm: p50 µs | p99 µs | on 16 threads, cold: per s | warm: per s |
|---|---|---|---|---|---|---|
| a table for each account, as the server reads one, at most 4096 of their files open | 8.5 | 13.2 | 1.6 | 2.6 | 216912 | 971516 |
| SQLite for each account | 15.2 | 23.3 | 11.0 | 15.3 | 108730 | 264165 |
| one SQLite for all | 15.2 | 23.2 | 8.1 | 12.2 | 60625 | 384328 |
| one RocksDB for all, its blocks compressed (Snappy) | 17.4 | 28.6 | 5.0 | 7.3 | 172904 | 459304 |
| one RocksDB for all, its blocks as they are | 12.4 | 24.7 | 5.0 | 7.3 | 179728 | 686914 |
| one fjall for all, its blocks compressed below the second level (LZ4) | 10.6 | 16.9 | 4.7 | 6.5 | 262260 | 424714 |
| one fjall for all, its blocks as they are | 10.5 | 19.6 | 4.7 | 6.3 | 234770 | 422109 |
| store | a listing of 50 records, cold: p50 µs | p99 µs | warm: p50 µs | p99 µs | first read of an account not yet open: p50 µs | p99 µs |
|---|---|---|---|---|---|---|
| a table for each account, as the server reads one, at most 4096 of their files open | 168 | 245 | 34 | 40 | 23 | 41 |
| SQLite for each account | 538 | 634 | 76 | 139 | 408 | 490 |
| one SQLite for all | 220 | 309 | 86 | 164 | 22 | 52 |
| one RocksDB for all, its blocks compressed (Snappy) | 275 | 569 | 95 | 107 | 28 | 42 |
| one RocksDB for all, its blocks as they are | 189 | 407 | 94 | 106 | 22 | 33 |
| one fjall for all, its blocks compressed below the second level (LZ4) | 212 | 270 | 107 | 117 | 12 | 21 |
| one fjall for all, its blocks as they are | 211 | 287 | 107 | 117 | 18 | 34 |
| store | resident before any is open, MiB | with 2000 open, a record read from each, MiB | KiB an account | after 20000 more records, twice, MiB | MiB on disk | KiB an account |
|---|---|---|---|---|---|---|
| a table for each account, as the server reads one, at most 4096 of their files open | 12.3 | 34.9 | 11.6 | 153.4 | 504 | 258 |
| SQLite for each account | 12.0 | 372.7 | 184.7 | 427.3 | 870 | 445 |
| one SQLite for all | 12.1 | 48.7 | 18.8 | 211.7 | 841 | 430 |
| one RocksDB for all, its blocks compressed (Snappy) | 12.1 | 36.9 | 12.7 | 154.6 | 339 | 174 |
| one RocksDB for all, its blocks as they are | 12.2 | 36.7 | 12.5 | 154.6 | 498 | 255 |
| one fjall for all, its blocks compressed below the second level (LZ4) | 12.3 | 39.5 | 13.9 | 155.1 | 500 | 256 |
| one fjall for all, its blocks as they are | 12.1 | 39.4 | 14.0 | 155.0 | 500 | 256 |
The tables read other ways. By pds_store::table::Table: as the server reads one (as above), its block cache holding at most 4096 of the tables' files open, the server's share of the file descriptors this process may have, the least recently read closed to make room and opened again when next read; and with every file held open, whatever the share. By this benchmark's reader, which holds every file open, keeps the blocks it reads in a map for each table where Table has one cache for all, has no bloom filter, and computes the check named over each block it reads: XXH3-64 is the one these files hold, and is compared with it; each other is compared with nothing. And by that reader from the layout before, a table of the version before: the same entries in blocks of 16 KiB, each block's last key whole in the index, its check SHA-256, which that file holds and is compared with, the file opened for each block read. The first read of an account is its table opened and a record read.
| read by | blocks | check | file | a record by path, cold: p50 µs | p99 µs | a listing of 50, cold: p50 µs | p99 µs | first read of an account: p50 µs | p99 µs | on 16 threads, cold: per s | warm: per s |
|---|---|---|---|---|---|---|---|---|---|---|---|
Table, as the server's | 4 KiB | XXH3-64 | at most 4096 held open | 8.5 | 13.2 | 168 | 245 | 23 | 41 | 216912 | 971516 |
Table, every file open | 4 KiB | XXH3-64 | all held open | 8.4 | 13.0 | 166 | 208 | 23 | 30 | 211341 | 955273 |
| this benchmark's | 4 KiB | SHA-256 | all held open | 11.0 | 17.2 | 223 | 266 | 29 | 37 | 208275 | 3494151 |
| this benchmark's | 4 KiB | CRC32C | all held open | 8.0 | 13.8 | 163 | 201 | 26 | 44 | 225249 | 1023677 |
| this benchmark's | 4 KiB | XXH3-64 | all held open | 8.0 | 13.8 | 163 | 205 | 26 | 43 | 198415 | 3067272 |
| this benchmark's | 4 KiB | BLAKE3 | all held open | 12.3 | 18.7 | 248 | 291 | 31 | 55 | 221440 | 1001043 |
| this benchmark's, the layout before | 16 KiB, last keys whole | SHA-256 | opened for each block | 25.8 | 60.1 | 374 | 441 | 63 | 78 | 231138 | 520635 |
| SQLite for each account | 4 KiB pages | none | held open | 15.2 | 23.3 | 538 | 634 | 408 | 490 | 108730 | 264165 |
| one SQLite for all | 4 KiB pages | none | held open | 15.2 | 23.2 | 220 | 309 | 22 | 52 | 60625 | 384328 |
| one RocksDB for all, its blocks compressed (Snappy) | 4 KiB before Snappy | XXH3 | held open | 17.4 | 28.6 | 275 | 569 | 28 | 42 | 172904 | 459304 |
| one RocksDB for all, its blocks as they are | 4 KiB | XXH3 | held open | 12.4 | 24.7 | 189 | 407 | 22 | 33 | 179728 | 686914 |
| one fjall for all, its blocks compressed below the second level (LZ4) | 4 KiB before LZ4 | XXH3 | held open | 10.6 | 16.9 | 212 | 270 | 12 | 21 | 262260 | 424714 |
| one fjall for all, its blocks as they are | 4 KiB | XXH3 | held open | 10.5 | 19.6 | 211 | 287 | 18 | 34 | 234770 | 422109 |
- The share of file descriptors is 4096 here, so every one of the 2000 tables' files stays open, and a table's check is XXH3-64. A record not in the cache takes 8.5 µs: RocksDB 12.4, fjall 10.5, SQLite 15.2. A listing of 50 takes 168: RocksDB 189, fjall 211, one SQLite 220. On 16 threads, 217 thousand records a second: RocksDB 180, fjall 235, SQLite 61 to 109.
- From its cache: 1.6 µs a record (RocksDB 5.0, fjall 4.7, SQLite 8.1 to 11.0), a listing in 34 (76 to 107), and on 16 threads 972 thousand records a second (RocksDB 459 to 687, fjall 422, one SQLite 384).
- The first read of an account, its table opened: 23 µs, level with one RocksDB or SQLite for all (22), which have nothing to open, and behind fjall (18). SQLite for each account takes 408.
- An account touched holds 11.6 KiB with tables, 12.5 with RocksDB, 14.0 with fjall, 18.8 with one SQLite and 185 with SQLite for each account.
- What each change was worth is in the last table: the layout before, 25.8 µs a record and 374 a listing; this benchmark's reader with SHA-256 as the check, 11.0 and 223.
A record read, a stage at a time
From a later store-bench run on the runner, at c9cf725, load 2.09; the
other parts above are from the run at eb08b99.
40 accounts of 100 realistic records each, a table for each account read as the server reads one (Table through one BlockCache), after one pass over every record: every block a read touches is in the cache. A record's block is 301 bytes on average; its getRecord body 497 bytes. One at a time: 20000 records drawn from all, on one thread; on 16 threads, 5000 each.
| stage | adds to the one before | one at a time: p50 µs | p99 µs | on 16 threads: per s |
|---|---|---|---|---|
| 1 | the path's CID, from the account's table | 0.41 | 0.93 | 10566971 |
| 2 | the record's block, by its CID | 0.62 | 1.03 | 6847931 |
| 3 | the block decoded, as the server decodes it (pds_repo::data::decode) | 1.21 | 2.16 | 3747430 |
| 4 | the record as getRecord answers it: {uri, cid, value} as JSON, value from pds_repo::data::to_json, serialized | 3.72 | 7.98 | 1188006 |
Over HTTP, from the server (target/release/pds): 40 accounts of 100 of the same records each, written with applyWrites and folded into their tables, after one pass over every record. One at a time: 20000 requests on one kept-alive connection from Node's http client; at once, 16 connections of 2000 requests each.
| stage | is | one at a time: p50 µs | p99 µs | 16 at once: per s |
|---|---|---|---|---|
| 5a | a request answered with no record, GET /xrpc/_health: HTTP, the server's router and the client | 95.8 | 144.9 | 22134 |
| 5b | stage 4 over HTTP, com.atproto.repo.getRecord: that, and the server's own work around the store | 179.4 | 288.0 | 20074 |
- The store's part of a read is small: stages 1 to 4, the table, the decode and the rendering, are 3.7 µs at p50, rendering 2.5 of them.
- The server's own work around the store is most of a
getRecordpast HTTP: 179.4 less 95.8 less 3.7, some 80 µs at p50, in the query read, the call routed to its method, the account found and its store locked, the takedowns looked at and the answer's headers.
Size and compression
200 accounts of 500 realistic commits each, a record a commit. The logs hold every commit whole, its event and the tree nodes later commits took out too; the tables and SQLite hold what the commits leave, the records and the tree, and no event.
| kept as | MiB | KiB an account | bytes a record |
|---|---|---|---|
| a log for each account before compaction, JSON frames (the server's) | 1139.9 | 5836 | 11952 |
| a log for each account before compaction, binary frames | 566.9 | 2902 | 5944 |
| a table for each account | 50.4 | 258 | 529 |
| the tables, each block compressed (zstd 3) | 28.4 | 145 | 298 |
| the tables, each block compressed with a dictionary (zstd 3) | 24.6 | 126 | 258 |
| SQLite for each account | 86.3 | 442 | 905 |
| one SQLite for all | 83.8 | 429 | 878 |
| SQLite for each account, every record rewritten 3 times | 94.4 | 483 | 989 |
| the same, vacuumed | 79.3 | 406 | 832 |
| one SQLite for all, every record rewritten 3 times | 88.9 | 455 | 932 |
| the same, vacuumed | 75.8 | 388 | 795 |
| SQLite for each account, rewritten and vacuumed, its blocks in a table with a rowid | 63.0 | 322 | 660 |
| one RocksDB for all, its blocks: Snappy | 34.6 | 177 | 363 |
| the same, every record rewritten 3 times | 49.9 | 255 | 523 |
| the same, compacted whole | 34.5 | 177 | 362 |
| one RocksDB for all, its blocks: no compression | 49.8 | 255 | 522 |
| the same, every record rewritten 3 times | 67.2 | 344 | 704 |
| the same, compacted whole | 50.1 | 257 | 526 |
| one RocksDB for all, its blocks: zstd | 29.8 | 153 | 313 |
| the same, every record rewritten 3 times | 43.2 | 221 | 453 |
| the same, compacted whole | 29.7 | 152 | 312 |
| one fjall for all, its blocks: LZ4 below the second level | 49.9 | 256 | 523 |
| the same, every record rewritten 3 times | 60.1 | 308 | 630 |
| the same, compacted whole | 35.2 | 180 | 369 |
| one fjall for all, its blocks: no compression | 49.9 | 256 | 523 |
| the same, every record rewritten 3 times | 60.1 | 308 | 630 |
| the same, compacted whole | 49.9 | 256 | 524 |
What RocksDB wrote to its files for what it was given, keys and values as they are, through the load and the rewrites, and then with the whole compaction too: to its log, to the files it writes from memory, and to those its compactions write.
| RocksDB's blocks | given MiB | to its log MiB | from memory MiB | by compactions MiB | written for each byte given | with the whole compaction MiB | for each byte given |
|---|---|---|---|---|---|---|---|
| Snappy | 997 | 997 | 126 | 114 | 1.24 | 1271 | 1.27 |
| no compression | 997 | 997 | 157 | 171 | 1.33 | 1374 | 1.38 |
| zstd | 997 | 997 | 113 | 98 | 1.21 | 1237 | 1.24 |
The tables' data blocks, 50 MiB in 13583 blocks, each ended at 4 KiB as the server's writer ends them, compressed one at a time on one thread; the dictionary is 65536 bytes, trained on each 8th block and counted once above. A record is read by its path through two blocks, each read from its file (in the system's cache), its check computed, and searched: the median of 20000.
| blocks | stored bytes of each 100 | compressing MiB per s | decompressing MiB per s | a record read µs |
|---|---|---|---|---|
| as they are | 100 | 3.9 | ||
| compressed (zstd 3) | 56 | 179 | 636 | 12.0 |
| compressed with a dictionary (zstd 3) | 48 | 164 | 850 | 9.8 |
- A table for each account, RocksDB and fjall uncompressed hold the same data in the same bytes to one part in a hundred (258, 255 and 256 KiB an account); SQLite's best is 322. With zstd the tables are 145 KiB an account, 126 with a dictionary, and RocksDB 153.
- A compressed block costs a record read 6 to 8 µs more.
- A log before compaction is 11 times its table with binary frames and 23 times with JSON. RocksDB wrote 1.21 to 1.33 bytes for each byte it was given, a floor for it: this database never fills a level above its last.
The service log by the number of accounts
| accounts | email and password | service log MiB | that log replayed at a start ms | start on the checkpoint ms | start bound ms | idle RSS MiB | compaction ms | peak RSS compacting MiB | RSS bound MiB |
|---|---|---|---|---|---|---|---|---|---|
| 10000 | no | 7.1 | 112 | 28 | 309 | 16.8 | 293 | 33.6 | 103.9 |
| 10000 | yes | 8.6 | 91 | 26 | 316 | 17.2 | 314 | 38.3 | 103.8 |
| 30000 | no | 21.4 | 193 | 32 | 306 | 17.0 | 756 | 66.3 | 181.7 |
| 30000 | yes | 25.8 | 217 | 29 | 310 | 17.2 | 891 | 82.8 | 181.9 |
| 100000 | no | 71.4 | 499 | 30 | 308 | 18.0 | 2419 | 174.3 | 455.3 |
| 100000 | yes | 85.9 | 545 | 33 | 316 | 18.1 | 2632 | 224.9 | 455.3 |
- A start on the checkpoint, and the memory it leaves, do not follow the accounts. The compaction here is of a log that holds every account, which this part writes and no server is left with: its time and memory follow that log.
Apple M1 Pro, macOS
bench/store/run.sh, and for the last part just store-bench 10000 30000 100000 500000.
- Darwin 25.5.0 arm64, 8 cores, 32 GiB; APFS on the internal SSD
(
/dev/disk3s5). Every sync isF_FULLFSYNC, a flush through the drive's cache, RocksDB's and fjall's too. - cargo 1.99.1; node v24.22.1; the repository at
41ca6ca; atb571f21for the reads and the sizes, and at66cde9bfor the last part. - The machine was not alone: its load was 3.6 to 8 through the run, and 23 as the last part began.
Writes
1500 commits from 16 writers over 40 accounts, each on disk (F_FULLFSYNC) before it is answered.
| layout | per s | p50 ms | p99 ms | log syncs | ms per sync |
|---|---|---|---|---|---|
| a log for each account (the server's) | 512 | 28.5 | 75.3 | 1460 | 20.02 |
| one log for all | 1916 | 8.6 | 12.4 | 174 | 4.40 |
| SQLite for each account | 251 | 58.0 | 163.2 | ||
| one SQLite for all | 199 | 58.0 | 442.6 | ||
| one RocksDB for all, a commit a batch, its log synced | 1403 | 9.5 | 30.2 | 226 | 4.57 |
| one fjall for all, a commit a batch, its journal synced | 209 | 59.1 | 319.5 | ||
| SQLite for each account, fsync without F_FULLFSYNC | 3634 | 3.9 | 11.4 | ||
| one SQLite for all, fsync without F_FULLFSYNC | 6259 | 1.8 | 7.7 |
1500 commits from 4 writers over 40 accounts, each on disk (F_FULLFSYNC) before it is answered.
| layout | per s | p50 ms | p99 ms | log syncs | ms per sync |
|---|---|---|---|---|---|
| a log for each account (the server's) | 271 | 14.2 | 25.0 | 1500 | 13.08 |
| one log for all | 435 | 9.1 | 15.2 | 744 | 4.61 |
| SQLite for each account | 239 | 16.0 | 32.1 | ||
| one SQLite for all | 201 | 14.0 | 94.9 | ||
| one RocksDB for all, a commit a batch, its log synced | 458 | 8.9 | 16.7 | 703 | 4.56 |
| one fjall for all, a commit a batch, its journal synced | 206 | 17.0 | 94.1 | ||
| SQLite for each account, fsync without F_FULLFSYNC | 6229 | 0.6 | 1.3 | ||
| one SQLite for all, fsync without F_FULLFSYNC | 6201 | 0.5 | 4.2 |
F_FULLFSYNCflushes the whole drive, so syncs of different files queue behind each other here, and whatever shares a sync wins: the opposite of the runner, where the server is deployed.
Writes of realistic commits, by the number of writers
1500 realistic commits from 4 writers over 40 accounts, each on disk (F_FULLFSYNC) before it is answered.
| layout | per s | p50 ms | p99 ms | log syncs | ms per sync | zeros written a commit, bytes |
|---|---|---|---|---|---|---|
| a log for each account, JSON frames (the server's) | 268 | 14.5 | 25.2 | 1499 | 13.38 | |
| a log for each account, binary frames | 269 | 14.5 | 25.3 | 1500 | 13.17 | |
| a log for each account, binary frames, each written at its offset | 264 | 14.6 | 25.9 | 1499 | 13.45 | |
| a log for each account, the same, allocated ahead by 64 KiB of zeros | 338 | 10.0 | 28.9 | 1498 | 11.22 | 6712 |
| a log for each account, the same, allocated ahead by 16 KiB of zeros | 304 | 11.7 | 28.2 | 1499 | 11.98 | 6116 |
| one log for all, JSON frames | 413 | 9.5 | 15.7 | 740 | 4.88 | |
| one log for all, binary frames | 454 | 8.8 | 13.3 | 717 | 4.58 | |
| one log for all, binary frames, each written at its offset | 443 | 9.3 | 12.9 | 724 | 4.65 | |
| one log for all, the same, allocated ahead by 16 MiB of zeros | 456 | 8.8 | 13.4 | 721 | 4.54 | 10168 |
one log for all, binary frames, each written at its offset, its waiting commits answered in turn (as a Wal's were) | 497 | 8.6 | 22.4 | 645 | 4.62 | |
| one log for all, binary frames, each written at its offset, a sync first letting the commits begun be written | 564 | 7.6 | 12.2 | 566 | 4.64 | |
| SQLite for each account | 233 | 16.8 | 31.9 | |||
| one SQLite for all | 201 | 18.0 | 92.0 | |||
| one SQLite for all, one writer, every commit waiting in one transaction | 384 | 9.6 | 25.1 | |||
| one RocksDB for all, a commit a batch, its log synced | 433 | 9.2 | 19.0 | 743 | 4.56 | |
| one RocksDB for all, the same, its write groups let grow to 8 MiB | 437 | 9.0 | 17.8 | 740 | 4.53 | |
| one RocksDB for all, a commit a batch left unsynced, those that wait sharing one sync of its log, a sync first letting the commits begun be written | 683 | 5.0 | 12.5 | 456 | 4.70 | |
| one RocksDB for all, a commit a batch, its log synced, its log files written over | 402 | 9.4 | 21.4 | 749 | 4.90 | |
| one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over | 414 | 9.3 | 23.0 | 740 | 4.81 | |
| RocksDB for each account, a commit a batch, its log synced | 270 | 14.5 | 26.6 | 1498 | 13.30 | |
| one fjall for all, a commit a batch, its journal synced | 201 | 19.0 | 69.1 | |||
| one fjall for all, a commit a batch left unsynced, those that wait sharing one sync of its journal | 228 | 14.7 | 58.9 | 1177 | 5.57 | |
| one fjall for all, the same, a sync first letting the commits begun be written | 600 | 5.4 | 12.8 | 469 | 5.17 | |
| fjall for each account, a commit a batch, its journal synced | 333 | 10.3 | 25.9 |
one RocksDB for all, a commit a batch, its log synced, its log files written over: of the 9.6 MiB the commits wrote to its log, 0.01 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.
one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over: of the 9.6 MiB the commits wrote to its log, 0.02 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.
RocksDB for each account, a commit a batch, its log synced: 40 databases, made by 64 threads in 1.5 s; the process holds 18 MiB more with them open and 22 MiB more after the commits; each is 7 files, and all are 13 MiB on disk.
fjall for each account, a commit a batch, its journal synced: 40 databases, made by 64 threads in 4.1 s; the process holds 2 MiB more with them open and 3 MiB more after the commits.
1500 realistic commits from 16 writers over 40 accounts, each on disk (F_FULLFSYNC) before it is answered.
| layout | per s | p50 ms | p99 ms | log syncs | ms per sync | zeros written a commit, bytes |
|---|---|---|---|---|---|---|
| a log for each account, JSON frames (the server's) | 530 | 27.1 | 72.3 | 1466 | 20.34 | |
| a log for each account, binary frames | 512 | 28.3 | 75.9 | 1461 | 20.45 | |
| a log for each account, binary frames, each written at its offset | 562 | 25.3 | 62.0 | 1451 | 18.72 | |
| a log for each account, the same, allocated ahead by 64 KiB of zeros | 705 | 19.0 | 65.1 | 1429 | 18.25 | 6752 |
| a log for each account, the same, allocated ahead by 16 KiB of zeros | 591 | 24.1 | 79.0 | 1434 | 19.66 | 6106 |
| one log for all, JSON frames | 1831 | 9.5 | 11.3 | 167 | 4.72 | |
| one log for all, binary frames | 1799 | 9.5 | 12.7 | 169 | 4.81 | |
| one log for all, binary frames, each written at its offset | 1886 | 9.4 | 18.6 | 158 | 4.87 | |
| one log for all, the same, allocated ahead by 16 MiB of zeros | 2061 | 8.1 | 15.6 | 172 | 4.17 | 10168 |
one log for all, binary frames, each written at its offset, its waiting commits answered in turn (as a Wal's were) | 819 | 16.9 | 75.4 | 386 | 4.68 | |
| one log for all, binary frames, each written at its offset, a sync first letting the commits begun be written | 2893 | 5.1 | 11.0 | 103 | 4.67 | |
| SQLite for each account | 253 | 58.5 | 149.9 | |||
| one SQLite for all | 198 | 63.9 | 341.6 | |||
| one SQLite for all, one writer, every commit waiting in one transaction | 1301 | 10.1 | 27.3 | |||
| one RocksDB for all, a commit a batch, its log synced | 1383 | 10.0 | 29.5 | 225 | 4.69 | |
| one RocksDB for all, the same, its write groups let grow to 8 MiB | 1307 | 10.1 | 32.1 | 229 | 4.86 | |
| one RocksDB for all, a commit a batch left unsynced, those that wait sharing one sync of its log, a sync first letting the commits begun be written | 3095 | 5.0 | 9.5 | 96 | 4.60 | |
| one RocksDB for all, a commit a batch, its log synced, its log files written over | 1381 | 9.0 | 35.9 | 219 | 4.75 | |
| one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over | 1353 | 9.2 | 36.0 | 225 | 4.81 | |
| RocksDB for each account, a commit a batch, its log synced | 505 | 27.7 | 85.4 | 1494 | 19.27 | |
| one fjall for all, a commit a batch, its journal synced | 203 | 66.6 | 263.5 | |||
| one fjall for all, a commit a batch left unsynced, those that wait sharing one sync of its journal | 647 | 20.0 | 81.2 | 480 | 4.78 | |
| one fjall for all, the same, a sync first letting the commits begun be written | 2327 | 5.4 | 16.9 | 123 | 4.85 | |
| fjall for each account, a commit a batch, its journal synced | 747 | 18.9 | 61.2 |
one RocksDB for all, a commit a batch, its log synced, its log files written over: of the 9.6 MiB the commits wrote to its log, 0.00 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.
one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over: of the 9.6 MiB the commits wrote to its log, 0.05 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.
RocksDB for each account, a commit a batch, its log synced: 40 databases, made by 64 threads in 1.5 s; the process holds 17 MiB more with them open and 18 MiB more after the commits; each is 7 files, and all are 13 MiB on disk.
fjall for each account, a commit a batch, its journal synced: 40 databases, made by 64 threads in 4.1 s; the process holds 2 MiB more with them open and 3 MiB more after the commits.
2048 realistic commits from 64 writers over 2000 accounts, each on disk (F_FULLFSYNC) before it is answered.
| layout | per s | p50 ms | p99 ms | log syncs | ms per sync | zeros written a commit, bytes |
|---|---|---|---|---|---|---|
| a log for each account, JSON frames (the server's) | 648 | 83.5 | 283.3 | 2047 | 59.69 | |
| a log for each account, binary frames | 595 | 90.8 | 319.9 | 2045 | 64.15 | |
| a log for each account, binary frames, each written at its offset | 597 | 91.0 | 302.1 | 2048 | 63.87 | |
| a log for each account, the same, allocated ahead by 64 KiB of zeros | 804 | 56.4 | 336.0 | 2047 | 76.19 | 58202 |
| a log for each account, the same, allocated ahead by 16 KiB of zeros | 774 | 56.4 | 373.7 | 2044 | 70.98 | 16354 |
| one log for all, JSON frames | 4980 | 12.3 | 29.2 | 73 | 5.41 | |
| one log for all, binary frames | 6415 | 9.7 | 18.9 | 65 | 4.69 | |
| one log for all, binary frames, each written at its offset | 5929 | 9.7 | 24.2 | 71 | 4.66 | |
| one log for all, the same, allocated ahead by 16 MiB of zeros | 7460 | 8.8 | 12.0 | 59 | 4.43 | 7449 |
one log for all, binary frames, each written at its offset, its waiting commits answered in turn (as a Wal's were) | 1095 | 56.9 | 233.3 | 402 | 4.56 | |
| one log for all, binary frames, each written at its offset, a sync first letting the commits begun be written | 9858 | 5.9 | 14.2 | 34 | 4.89 | |
| SQLite for each account | 277 | 217.0 | 475.0 | |||
| one SQLite for all | 199 | 310.8 | 1252.7 | |||
| one SQLite for all, one writer, every commit waiting in one transaction | 2616 | 20.4 | 69.5 | |||
| one RocksDB for all, a commit a batch, its log synced | 4442 | 14.5 | 24.0 | 94 | 4.66 | |
| one RocksDB for all, the same, its write groups let grow to 8 MiB | 5888 | 10.1 | 19.7 | 67 | 4.86 | |
| one RocksDB for all, a commit a batch left unsynced, those that wait sharing one sync of its log, a sync first letting the commits begun be written | 9141 | 6.1 | 13.5 | 37 | 4.93 | |
| one RocksDB for all, a commit a batch, its log synced, its log files written over | 3274 | 15.7 | 39.0 | 94 | 6.45 | |
| one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over | 4336 | 13.5 | 31.7 | 65 | 7.01 | |
| RocksDB for each account, a commit a batch, its log synced | 711 | 79.6 | 234.7 | 2046 | 57.09 | |
| one fjall for all, a commit a batch, its journal synced | 202 | 305.8 | 1057.4 | |||
| one fjall for all, a commit a batch left unsynced, those that wait sharing one sync of its journal | 1495 | 31.0 | 172.6 | 286 | 4.75 | |
| one fjall for all, the same, a sync first letting the commits begun be written | 8200 | 6.1 | 17.8 | 43 | 4.80 |
one RocksDB for all, a commit a batch, its log synced, its log files written over: of the 13.1 MiB the commits wrote to its log, 0.02 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.
one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over: of the 13.1 MiB the commits wrote to its log, 0.14 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.
RocksDB for each account, a commit a batch, its log synced: 2000 databases, made by 64 threads in 77.7 s; the process holds 699 MiB more with them open and 761 MiB more after the commits; each is 7 files, and all are 195 MiB on disk.
4096 realistic commits from 256 writers over 2000 accounts, each on disk (F_FULLFSYNC) before it is answered.
| layout | per s | p50 ms | p99 ms | log syncs | ms per sync | zeros written a commit, bytes |
|---|---|---|---|---|---|---|
| a log for each account, JSON frames (the server's) | 875 | 250.7 | 802.7 | 4093 | 151.62 | |
| a log for each account, binary frames | 614 | 349.9 | 1317.0 | 4073 | 224.19 | |
| a log for each account, binary frames, each written at its offset | 672 | 316.6 | 1162.0 | 4082 | 203.73 | |
| a log for each account, the same, allocated ahead by 64 KiB of zeros | 906 | 160.3 | 1922.1 | 4036 | 242.02 | 29094 |
| a log for each account, the same, allocated ahead by 16 KiB of zeros | 939 | 196.9 | 1041.9 | 4052 | 210.47 | 10335 |
| one log for all, JSON frames | 8118 | 26.8 | 87.1 | 90 | 5.37 | |
| one log for all, binary frames | 9336 | 25.4 | 72.3 | 77 | 5.35 | |
| one log for all, binary frames, each written at its offset | 12691 | 16.5 | 56.2 | 46 | 5.74 | |
| one log for all, the same, allocated ahead by 16 MiB of zeros | 17931 | 12.4 | 26.2 | 32 | 6.53 | 7448 |
one log for all, binary frames, each written at its offset, its waiting commits answered in turn (as a Wal's were) | 1179 | 161.7 | 926.8 | 690 | 4.93 | |
| one log for all, binary frames, each written at its offset, a sync first letting the commits begun be written | 17896 | 13.5 | 25.4 | 26 | 6.47 | |
| SQLite for each account | 285 | 803.6 | 2444.5 | |||
| one SQLite for all | 196 | 1166.8 | 4865.0 | |||
| one SQLite for all, one writer, every commit waiting in one transaction | 4767 | 56.7 | 65.4 | |||
| one RocksDB for all, a commit a batch, its log synced | 4467 | 53.8 | 107.9 | 187 | 4.64 | |
| one RocksDB for all, the same, its write groups let grow to 8 MiB | 17183 | 13.2 | 27.5 | 38 | 5.43 | |
| one RocksDB for all, a commit a batch left unsynced, those that wait sharing one sync of its log, a sync first letting the commits begun be written | 17395 | 14.4 | 23.2 | 31 | 5.91 | |
| one RocksDB for all, a commit a batch, its log synced, its log files written over | 3311 | 70.3 | 157.4 | 187 | 6.35 | |
| one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over | 9241 | 24.0 | 58.1 | 35 | 11.34 | |
| one fjall for all, a commit a batch, its journal synced | 203 | 1149.9 | 4424.8 | |||
| one fjall for all, a commit a batch left unsynced, those that wait sharing one sync of its journal | 7711 | 28.7 | 75.7 | 84 | 6.21 | |
| one fjall for all, the same, a sync first letting the commits begun be written | 13716 | 19.1 | 33.5 | 34 | 7.39 |
one RocksDB for all, a commit a batch, its log synced, its log files written over: of the 26.3 MiB the commits wrote to its log, 0.00 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.
one RocksDB for all, the same, its write groups let grow to 8 MiB, its log files written over: of the 26.3 MiB the commits wrote to its log, 4.09 MiB went past what its log files held before them, 0 of those files new; the rest went over bytes written before.
- A log for each account is held to 268 to 875 commits a second however many write. One log for all, one RocksDB for all and one SQLite with a batching writer all pass it, from 1.4 times at 4 writers to 5 to 20 times at 256.
- RocksDB loses to one log for all with its own group commit at 4 to 64 writers and is level with it at 256 once its groups grow. Its log files written over gain nothing here.
- A sync that first lets the commits begun be written is worth 1.2 to 1.5 times to the benchmark's own log at 4 to 64 writers, and as much to RocksDB and fjall.
Writes with no sync: not durable
Realistic commits from 16 writers over 40 accounts, none synced: not durable. A commit is answered once it is written, to the system's cache, and making it is no part of its time.
| layout | commits | per s | p50 µs | p99 µs |
|---|---|---|---|---|
| a log for each account, JSON frames (the server's) | 200000 | 58403 | 54 | 4988 |
| a log for each account, binary frames | 200000 | 167067 | 16 | 964 |
| a log for each account, binary frames, each written at its offset | 200000 | 165427 | 17 | 980 |
| a log for each account, the same, allocated ahead by 64 KiB of zeros | 200000 | 420397 | 10 | 399 |
| a log for each account, the same, allocated ahead by 16 KiB of zeros | 200000 | 210461 | 23 | 528 |
| one log for all, JSON frames | 200000 | 53182 | 190 | 1548 |
| one log for all, binary frames | 200000 | 84196 | 102 | 1233 |
| one log for all, binary frames, each written at its offset | 200000 | 85679 | 107 | 1201 |
| one log for all, the same, allocated ahead by 16 MiB of zeros | 200000 | 121818 | 71 | 814 |
| SQLite for each account | 40000 | 23067 | 252 | 6211 |
| one SQLite for all | 40000 | 9218 | 81 | 59176 |
| one SQLite for all, one writer, every commit waiting in one transaction | 40000 | 11403 | 1041 | 3512 |
| one RocksDB for all, a commit a batch | 40000 | 63837 | 185 | 947 |
| one RocksDB for all, the same, its write groups let grow to 8 MiB | 40000 | 59960 | 185 | 1246 |
| RocksDB for each account, a commit a batch | 40000 | 7744 | 145 | 14298 |
| one fjall for all, a commit a batch | 40000 | 34425 | 24 | 3552 |
| fjall for each account, a commit a batch | 40000 | 5640 | 43 | 73822 |
RocksDB for each account, a commit a batch: 40 databases, made by 64 threads in 1.6 s; the process holds 17 MiB more with them open and 22 MiB more after the commits; each is 17 files, and all are 344 MiB on disk.
fjall for each account, a commit a batch: 40 databases, made by 64 threads in 4.1 s; the process holds 2 MiB more with them open and 3 MiB more after the commits.
Realistic commits from 256 writers over 2000 accounts, none synced: not durable. A commit is answered once it is written, to the system's cache, and making it is no part of its time.
| layout | commits | per s | p50 µs | p99 µs |
|---|---|---|---|---|
| a log for each account, JSON frames (the server's) | 200000 | 100204 | 54 | 66695 |
| a log for each account, binary frames | 200000 | 72695 | 16 | 68213 |
| a log for each account, binary frames, each written at its offset | 200000 | 45665 | 36 | 68896 |
| a log for each account, the same, allocated ahead by 64 KiB of zeros | 200000 | 490537 | 9 | 22429 |
| a log for each account, the same, allocated ahead by 16 KiB of zeros | 200000 | 310066 | 10 | 26499 |
| one log for all, JSON frames | 200000 | 47434 | 3164 | 31546 |
| one log for all, binary frames | 200000 | 83212 | 2027 | 20479 |
| one log for all, binary frames, each written at its offset | 200000 | 84593 | 1789 | 19922 |
| one log for all, the same, allocated ahead by 16 MiB of zeros | 200000 | 112607 | 1391 | 14785 |
| SQLite for each account | 40000 | 12258 | 2194 | 178226 |
| one SQLite for all | 40000 | 11109 | 68 | 845461 |
| one SQLite for all, one writer, every commit waiting in one transaction | 40000 | 10616 | 24258 | 36465 |
| one RocksDB for all, a commit a batch | 40000 | 71357 | 2876 | 13977 |
| one RocksDB for all, the same, its write groups let grow to 8 MiB | 40000 | 72196 | 3062 | 14536 |
| one fjall for all, a commit a batch | 40000 | 41872 | 22 | 52260 |
A log replayed
| log MiB | frames | replay ms | MiB per s |
|---|---|---|---|
| 64 | 6599 | 59 | 1089 |
| 512 | 52777 | 474 | 1080 |
Realistic commits, the same in both logs, as many as make that many MiB of the server's frames.
| frames | commits | log MiB | bytes a commit | replay ms | MiB per s | commits per s |
|---|---|---|---|---|---|---|
| JSON frames (the server's) | 5610 | 64 | 11962 | 59 | 1089 | 95460 |
| binary frames | 5610 | 32 | 5948 | 19 | 1679 | 295964 |
| JSON frames (the server's) | 44919 | 512 | 11952 | 475 | 1079 | 94641 |
| binary frames | 44919 | 255 | 5943 | 154 | 1649 | 290845 |
The firehose replayed
The events of 200000 realistic commits over 2000 accounts. A whole replay is the faster of two, so of files the system's cache holds, if it has the memory to; a first event is timed after those, from each of 200 cursors in the middle half, the files open.
| from | MiB on disk | whole replay ms | events per s | MiB of events per s | first event from a cursor, p50 ms | p99 ms | with an index entry each frame, p50 ms | p99 ms |
|---|---|---|---|---|---|---|---|---|
| one log for all, binary frames, an index entry each 64 frames, each frame's check SHA-256 (the server's) | 1134 | 705 | 283804 | 839 | 0.111 | 0.237 | 0.010 | 0.023 |
| the same, each frame's check CRC32C | 1134 | 192 | 1044378 | 3087 | 0.034 | 0.073 | 0.008 | 0.016 |
| the same, each frame's check XXH3-64 | 1134 | 153 | 1303063 | 3852 | 0.027 | 0.053 | 0.008 | 0.035 |
| the same, each frame's check BLAKE3 | 1134 | 1639 | 121989 | 361 | 0.263 | 0.517 | 0.014 | 0.016 |
| the same, no frame's check computed | 1134 | 121 | 1655543 | 4893 | 0.023 | 0.052 | 0.007 | 0.013 |
| SQLite, a table of events by seq | 783 | 164 | 1221576 | 3611 | 0.005 | 0.008 | ||
| RocksDB, a family of events by seq, an iterator, its blocks: Snappy | 433 | 230 | 867898 | 2565 | 0.009 | 0.041 | ||
| RocksDB, a family of events by seq, an iterator, its blocks: no compression | 600 | 142 | 1410891 | 4170 | 0.007 | 0.044 | ||
| fjall, a keyspace of events by seq, an iterator, its blocks: LZ4 below the second level | 604 | 202 | 991895 | 2932 | 0.006 | 0.071 | ||
| fjall, a keyspace of events by seq, an iterator, its blocks: no compression | 604 | 208 | 961616 | 2842 | 0.006 | 0.067 | ||
| a log for each account, JSON frames (the server's), put in order | 2280 | 7199 | 27782 | 82 | 7188.511 | 7188.511 |
Reads
2000 accounts of 500 realistic records each. Cold is a process that has read nothing, its accounts open, the store's files read through just before, so in the system's cache if it has the memory to keep them; warm is the same reads again, everything they touch held by the process.
| store | a record by path, cold: p50 µs | p99 µs | warm: p50 µs | p99 µs | on 16 threads, cold: per s | warm: per s |
|---|---|---|---|---|---|---|
| a table for each account, as the server reads one | 27.6 | 63.8 | 2.2 | 5.9 | 98438 | 1799876 |
| SQLite for each account | 12.9 | 31.6 | 8.9 | 16.0 | 99132 | 307460 |
| one SQLite for all | 11.0 | 80.6 | 6.7 | 13.1 | 91848 | 402958 |
| one RocksDB for all, its blocks compressed (Snappy) | 11.0 | 22.0 | 5.3 | 12.2 | 401988 | 833919 |
| one RocksDB for all, its blocks as they are | 9.1 | 33.3 | 5.8 | 13.4 | 326502 | 830714 |
| one fjall for all, its blocks compressed below the second level (LZ4) | 8.9 | 39.7 | 6.1 | 14.5 | 341002 | 595908 |
| one fjall for all, its blocks as they are | 8.9 | 47.4 | 6.0 | 12.9 | 386496 | 659354 |
| store | a listing of 50 records, cold: p50 µs | p99 µs | warm: p50 µs | p99 µs | first read of an account not yet open: p50 µs | p99 µs |
|---|---|---|---|---|---|---|
| a table for each account, as the server reads one | 202 | 446 | 40 | 67 | 35 | 72 |
| SQLite for each account | 450 | 762 | 67 | 117 | 395 | 1453 |
| one SQLite for all | 140 | 232 | 84 | 246 | 14 | 148 |
| one RocksDB for all, its blocks compressed (Snappy) | 165 | 274 | 89 | 136 | 12 | 37 |
| one RocksDB for all, its blocks as they are | 136 | 229 | 94 | 145 | 10 | 19 |
| one fjall for all, its blocks compressed below the second level (LZ4) | 172 | 276 | 128 | 178 | 9 | 23 |
| one fjall for all, its blocks as they are | 172 | 268 | 129 | 197 | 9 | 25 |
| store | resident before any is open, MiB | with 2000 open, a record read from each, MiB | KiB an account | after 20000 more records, twice, MiB | MiB on disk | KiB an account |
|---|---|---|---|---|---|---|
| a table for each account, as the server reads one | 5.9 | 32.9 | 13.8 | 169.5 | 504 | 258 |
| SQLite for each account | 5.9 | 363.3 | 183.0 | 406.6 | 870 | 445 |
| one SQLite for all | 5.8 | 49.2 | 22.2 | 245.8 | 841 | 430 |
| one RocksDB for all, its blocks compressed (Snappy) | 5.8 | 33.8 | 14.3 | 157.3 | 339 | 174 |
| one RocksDB for all, its blocks as they are | 5.9 | 33.3 | 14.0 | 157.3 | 498 | 255 |
| one fjall for all, its blocks compressed below the second level (LZ4) | 5.9 | 38.9 | 16.9 | 181.1 | 500 | 256 |
| one fjall for all, its blocks as they are | 5.9 | 38.9 | 16.9 | 181.7 | 500 | 256 |
The tables read other ways. By pds_store::table::Table: as the server reads one (as above), its block cache holding at most 64 of the tables' files open, the least recently read closed to make room and opened again when next read; and with every file held open, which the server does not do. By this benchmark's reader, which holds every file open, keeps the blocks it reads in a map for each table where Table has one cache for all, has no bloom filter, and computes the check named over each block it reads (compared with what the file holds for SHA-256 alone). And by that reader from the layout before: the same entries in blocks of 16 KiB, each block's last key whole in the index, the file opened for each block read. The first read of an account is its table opened and a record read.
| read by | blocks | check | file | a record by path, cold: p50 µs | p99 µs | a listing of 50, cold: p50 µs | p99 µs | first read of an account: p50 µs | p99 µs | on 16 threads, cold: per s | warm: per s |
|---|---|---|---|---|---|---|---|---|---|---|---|
Table, as the server's | 4 KiB | SHA-256 | at most 64 held open | 27.6 | 63.8 | 202 | 446 | 35 | 72 | 98438 | 1799876 |
Table, not as the server's | 4 KiB | SHA-256 | all held open | 10.4 | 26.7 | 202 | 436 | 35 | 72 | 421679 | 1537859 |
| this benchmark's | 4 KiB | SHA-256 | all held open | 10.3 | 19.1 | 187 | 294 | 32 | 61 | 533092 | 2048806 |
| this benchmark's | 4 KiB | CRC32C | all held open | 6.5 | 15.4 | 114 | 248 | 27 | 54 | 625597 | 1861822 |
| this benchmark's | 4 KiB | XXH3-64 | all held open | 6.2 | 14.3 | 110 | 181 | 29 | 63 | 739129 | 2052960 |
| this benchmark's | 4 KiB | BLAKE3 | all held open | 17.8 | 27.6 | 328 | 456 | 41 | 76 | 401663 | 2150774 |
| this benchmark's, the layout before | 16 KiB, last keys whole | SHA-256 | opened for each block | 33.7 | 88.2 | 472 | 679 | 80 | 130 | 175195 | 1138128 |
| SQLite for each account | 4 KiB pages | none | held open | 12.9 | 31.6 | 450 | 762 | 395 | 1453 | 99132 | 307460 |
| one SQLite for all | 4 KiB pages | none | held open | 11.0 | 80.6 | 140 | 232 | 14 | 148 | 91848 | 402958 |
| one RocksDB for all, its blocks compressed (Snappy) | 4 KiB before Snappy | XXH3 | held open | 11.0 | 22.0 | 165 | 274 | 12 | 37 | 401988 | 833919 |
| one RocksDB for all, its blocks as they are | 4 KiB | XXH3 | held open | 9.1 | 33.3 | 136 | 229 | 10 | 19 | 326502 | 830714 |
| one fjall for all, its blocks compressed below the second level (LZ4) | 4 KiB before LZ4 | XXH3 | held open | 8.9 | 39.7 | 172 | 276 | 9 | 23 | 341002 | 595908 |
| one fjall for all, its blocks as they are | 4 KiB | XXH3 | held open | 8.9 | 47.4 | 172 | 268 | 9 | 25 | 386496 | 659354 |
- The order is the runner's, but for a record not in the cache: 27.6 µs with 64 files open at most against 10.4 with all open, as opening a file again takes about 20 µs here. From the cache a record takes 2.2 µs and 16 threads read 1.8 million a second (RocksDB 5.3 to 5.8, and 831 thousand).
A record read, a stage at a time
target/release/store-bench ladder 40 100 and node ladder.mjs 40 100 in
bench/store, alone of the parts, at 0a88af0 with this part's files
added; cargo 1.100.0, node v24.22.1. The machine was far from alone: its
load was 120 to 170 through the run, a Docker VM taking two cores and
builds the rest, and the stage over HTTP, which waits on the scheduler
for every request, is several times what it was minutes before (a
_health p50 of 86 µs then, 384 µs here): these figures say that the
part runs, not what the stages cost.
40 accounts of 100 realistic records each, a table for each account read as the server reads one (Table through one BlockCache), after one pass over every record: every block a read touches is in the cache. A record's block is 301 bytes on average; its getRecord body 497 bytes. One at a time: 20000 records drawn from all, on one thread; on 16 threads, 5000 each.
| stage | adds to the one before | one at a time: p50 µs | p99 µs | on 16 threads: per s |
|---|---|---|---|---|
| 1 | the path's CID, from the account's table | 0.58 | 1.38 | 2201981 |
| 2 | the record's block, by its CID | 1.25 | 6.00 | 717841 |
| 3 | the block decoded, as the server decodes it (pds_repo::data::decode) | 2.04 | 7.29 | 826944 |
| 4 | the record as getRecord answers it: {uri, cid, value} as JSON, value from pds_repo::data::to_json, serialized | 4.92 | 152.75 | 258587 |
Over HTTP, from the server (target/release/pds): 40 accounts of 100 of the same records each, written with applyWrites and folded into their tables, after one pass over every record. One at a time: 20000 requests on one kept-alive connection from Node's http client; at once, 16 connections of 2000 requests each.
| stage | is | one at a time: p50 µs | p99 µs | 16 at once: per s |
|---|---|---|---|---|
| 5a | a request answered with no record, GET /xrpc/_health: HTTP, the server's router and the client | 383.9 | 22856.3 | 3353 |
| 5b | stage 4 over HTTP, com.atproto.repo.getRecord: that, and the server's own work around the store | 398.6 | 23835.5 | 3209 |
- In the process, a record's read is 5 µs to its body, of which the table's two keys are 1.3 and rendering the JSON 2.9: rendering is the dearest stage, more than the store and the decoding together.
- Over HTTP under this load,
getRecordless_healthis 15 µs, which less stage 4 leaves 10 µs of the server's own work around the store; with the scheduler's waits in both rows, that is noise of the same size.
Size and compression
200 accounts of 500 realistic commits each, a record a commit. The logs hold every commit whole, its event and the tree nodes later commits took out too; the tables and SQLite hold what the commits leave, the records and the tree, and no event.
| kept as | MiB | KiB an account | bytes a record |
|---|---|---|---|
| a log for each account before compaction, JSON frames (the server's) | 1139.9 | 5836 | 11952 |
| a log for each account before compaction, binary frames | 566.9 | 2902 | 5944 |
| a table for each account | 50.4 | 258 | 529 |
| the tables, each block compressed (zstd 3) | 28.4 | 145 | 298 |
| the tables, each block compressed with a dictionary (zstd 3) | 24.6 | 126 | 258 |
| SQLite for each account | 86.3 | 442 | 905 |
| one SQLite for all | 83.8 | 429 | 878 |
| SQLite for each account, every record rewritten 3 times | 94.4 | 483 | 989 |
| the same, vacuumed | 79.3 | 406 | 832 |
| one SQLite for all, every record rewritten 3 times | 88.9 | 455 | 932 |
| the same, vacuumed | 75.8 | 388 | 795 |
| SQLite for each account, rewritten and vacuumed, its blocks in a table with a rowid | 63.0 | 322 | 660 |
| one RocksDB for all, its blocks: Snappy | 34.6 | 177 | 363 |
| the same, every record rewritten 3 times | 42.1 | 216 | 442 |
| the same, compacted whole | 34.5 | 177 | 362 |
| one RocksDB for all, its blocks: no compression | 49.8 | 255 | 522 |
| the same, every record rewritten 3 times | 59.6 | 305 | 625 |
| the same, compacted whole | 50.2 | 257 | 526 |
| one RocksDB for all, its blocks: zstd | 29.8 | 153 | 313 |
| the same, every record rewritten 3 times | 36.8 | 188 | 385 |
| the same, compacted whole | 29.8 | 152 | 312 |
| one fjall for all, its blocks: LZ4 below the second level | 49.9 | 256 | 523 |
| the same, every record rewritten 3 times | 60.3 | 309 | 632 |
| the same, compacted whole | 35.2 | 180 | 369 |
| one fjall for all, its blocks: no compression | 49.9 | 256 | 523 |
| the same, every record rewritten 3 times | 60.3 | 309 | 632 |
| the same, compacted whole | 49.9 | 256 | 524 |
What RocksDB wrote to its files for what it was given, keys and values as they are, through the load and the rewrites, and then with the whole compaction too: to its log, to the files it writes from memory, and to those its compactions write.
| RocksDB's blocks | given MiB | to its log MiB | from memory MiB | by compactions MiB | written for each byte given | with the whole compaction MiB | for each byte given |
|---|---|---|---|---|---|---|---|
| Snappy | 996 | 996 | 133 | 114 | 1.25 | 1277 | 1.28 |
| no compression | 996 | 996 | 168 | 171 | 1.34 | 1385 | 1.39 |
| zstd | 996 | 996 | 119 | 98 | 1.22 | 1242 | 1.25 |
The tables' data blocks, 50 MiB in 13583 blocks, each ended at 4 KiB as the server's writer ends them, compressed one at a time on one thread; the dictionary is 65536 bytes, trained on each 8th block and counted once above. A record is read by its path through two blocks, each read from its file (in the system's cache), its check computed, and searched: the median of 20000.
| blocks | stored bytes of each 100 | compressing MiB per s | decompressing MiB per s | a record read µs |
|---|---|---|---|---|
| as they are | 100 | 8.2 | ||
| compressed (zstd 3) | 56 | 172 | 538 | 16.2 |
| compressed with a dictionary (zstd 3) | 48 | 136 | 638 | 13.3 |
The service log by the number of accounts
| accounts | email and password | service log MiB | that log replayed at a start ms | start on the checkpoint ms | start bound ms | idle RSS MiB | compaction ms | peak RSS compacting MiB | RSS bound MiB |
|---|---|---|---|---|---|---|---|---|---|
| 10000 | no | 7.1 | 65 | 22 | 296 | 18.9 | 153 | 35.3 | 105.6 |
| 10000 | yes | 8.6 | 70 | 20 | 350 | 18.9 | 162 | 40.3 | 105.6 |
| 30000 | no | 21.4 | 149 | 22 | 303 | 19.1 | 433 | 69.5 | 183.7 |
| 30000 | yes | 25.8 | 182 | 24 | 299 | 19.1 | 517 | 84.5 | 183.8 |
| 100000 | no | 71.4 | 453 | 26 | 297 | 19.7 | 1501 | 175.0 | 457.2 |
| 100000 | yes | 85.9 | 491 | 25 | 305 | 19.9 | 1619 | 225.3 | 457.2 |
| 500000 | no | 357.6 | 4764 | 32 | 295 | 23.8 | 7833 | 750.3 | 2019.7 |
| 500000 | yes | 430.9 | 2597 | 36 | 296 | 24.4 | 8753 | 1000.2 | 2019.7 |
- A start on the checkpoint is 20 to 36 ms from 10000 accounts to 500000, and leaves 19 to 24 MiB resident: neither follows the accounts.
- Replaying a log goes at 75 to 170 MiB a second with the accounts' entries noted, so the service log is compacted at 8 MiB, whatever its checkpoint's size.
- The compaction timed is of a log that holds every account, which this part writes and no server is left with: what it holds follows that log, about 2 KiB an account in it.