# The store benchmark What the way the server keeps its data costs, measured apart from the rest of the server: its logs and tables (`crates/pds-store`) beside SQLite and RocksDB under the same commits, in writes, replays, reads and bytes on disk; a record's read taken a stage at a time, from the table to the server's `getRecord`; and how the server's start and memory follow the number of accounts in its service log. The PDS benchmark ([`../README.md`](../README.md)) measures whole servers, each with its own durability, in its own language and some in a container; this holds everything but the layout the same. The last results are in [`RESULTS.md`](RESULTS.md). It is a workspace of its own: SQLite, RocksDB and zstd are dependencies of the benchmark, not of the server, and no gate builds it. RocksDB (the `rocksdb` crate, RocksDB 11.8) is built from its C++ the first time, which takes three and a half minutes on a laptop's eight cores, and its bindings are made with libclang. fjall (the `fjall` crate, 3.1, over `lsm-tree` 3.1, its files in their format 3) is Rust, and builds in twenty seconds. ## Running it ```sh just store-bench # builds the server, then bench/store/run.sh just store-bench 10000 200000 # other counts of accounts for the last part ``` `run.sh` prints Markdown: the machine, then a table or a few for each part. Its files go under `BENCH_DIR`, by default `bench/store/target/data`, and are removed after each part; what a sync costs is that directory's disk's and file system's, so `BENCH_DIR` names the disk measured. It needs cargo, a C and a C++ compiler, libclang, Node and `ps`, and nothing of the harness's; without a server at `target/release/pds` (or `PDS_BIN`) the last part and the ladder's stage over HTTP are left out, and say so. The parts that need no server take about sixteen minutes on a laptop, at most 5 GiB under `BENCH_DIR` and 1.5 GiB of memory, and a limit on open files of 20,000 or more: the benchmark raises its limit to the most it is allowed, as the server does when it starts, and under a lower one the row of a RocksDB for each of 2000 accounts is left out, and says so. On the runner, the `store-bench` workflow runs the same script when the owner asks, on `main` or a branch (`knotless run rerun knotless://chadtmiller.com/monolith --ref refs/heads/main --workflow store-bench`): its log is the tables. Asked for when nothing else runs there, as it loads the runner's disk and the gates' runs beside it are held to times. A pull that changes `bench/store/` or `crates/pds-store` runs `store-bench-small` instead: the same script with `BENCH_SMALL` set, every part at a size that shows only that it builds and runs, whose numbers say nothing. Run it alone on the machine: every part times syncs, reads or starts. ## What it measures **Writes** (`store-bench write `). A commit is what a `createRecord` leaves: the commit's block, the record's, two MST nodes, and its event's frame, 4.8 KB in all, of random bytes, with the two nodes the account's commit before made taken out. Writers on threads of their own each write the next commit to an account drawn from 40, and wait until it is on disk: 1500 commits from 16 writers, as the PDS benchmark's `createRecord` part, and from 4. Each layout is warmed with a tenth as many first. The layouts: | layout | is | |---|---| | a log for each account | the server's: each commit one frame of JSON with holes, written where the last ended in its account's log (`pds_store::Wal`), commits to one account that wait at once sharing a sync. The frame's payload is the commit's entry and its event's as JSON text, with the commit's block, the tree's nodes and the event's frame each an empty string, a hole, and those bytes as they are after the text, its check XXH3-64 (`pds_store::frame_with_holes`; `holes` in `src/frame.rs`, which writes the server's fields in its order). A `Wal` gives its file room as it is written to: a frame that grows the file grows it by itself and by as many zeros as there are bytes of frames written to the log since it was opened, that frame's among them, 256 KiB at most, and the frames after are written over the zeros | | one log for all | the same frames in one log, so that commits to any accounts share a sync | | SQLite for each account | a database for each account in WAL mode, `synchronous=FULL`: a commit is a transaction of its blocks into a table by CID, the nodes taken out deleted, its record's path and its event into tables of their own | | one SQLite for all | the same tables in one database, on one connection | | one RocksDB for all | one database of three column families: an account's blocks by CID and its records' CIDs by path, each key beginning with its account, and events by seq (`src/rocks.rs`). A commit is one `WriteBatch`, its blocks put, the nodes taken out deleted, its path and its event, written with `WriteOptions::set_sync(true)`: RocksDB appends it to its log (the WAL the three families share), syncs that, and puts it in its memtables. Commits that wait at once are written and synced together by one of their threads, RocksDB's own group commit | Every layout's commit is synced to the disk before it is answered. On macOS that is `F_FULLFSYNC` for all of them (`PRAGMA fullfsync=ON` for SQLite), and two more rows show SQLite as it runs there by default, its `fsync` leaving the drive's cache unflushed; elsewhere `fsync` is the sync, and those rows would repeat the two above. RocksDB's sync is `F_FULLFSYNC` there only because this workspace builds it so (`.cargo/config.toml` defines `HAVE_FULLFSYNC`, as RocksDB's own Makefile and CMake do): `librocksdb-sys` leaves it undefined, and a synced write is then an `fdatasync`, or with `set_use_fsync(true)` an `fsync`, which leaves the drive's cache unflushed as SQLite's default does. Per layout: commits per second, the p50 and p99 of a commit's time, and for the logs and RocksDB, how many syncs were made and how long each took, RocksDB's by its own count. **Realistic commits.** Random bytes do not compress and are not what a record is, so every part below writes commits that are, the same for the same seed (`src/data.rs`): a post of about 300 bytes of DAG-CBOR (a `$type`, a text of words drawn from a few hundred common ones, `langs`, a `createdAt`, and three times in ten a reply's two CIDs and URIs); two MST nodes of about 1 KB each (entries of a key less the prefix it shares with the one before, a CID, and sometimes a subtree's); a commit block of about 190 bytes (a DID, a rev, the tree's CID, a signature of 64 random bytes); and its event's frame, about 3 KB: a `#commit` with those blocks again in a CAR, and the DID, rev and path as strings. A CID is the four bytes a CID of a DAG-CBOR block begins with and 32 random ones, so about half of a tree node's bytes do not compress, as in a real one. **Writes of realistic commits, by the number of writers** (`store-bench spread `). The write part again, of realistic commits, from 4 and 16 writers over 40 accounts (1500 commits) and from 64 and 256 over 2000 (2048 and 4096): with 2000 accounts the writers are spread as a server's are, an account's log gets a commit or two, and a log for each account shares no sync. Each layout is warmed with a tenth as many commits from the same writers; its files are made before, by 64 threads, and making them is not timed. The logs are of these kinds, for each account and one for all: | log | is | |---|---| | JSON frames with holes | the server's, as above, in a `Wal`, with the room it makes. For each account, as `pds serve` keeps them with `PDS_SYNC` `together` (the default): every log of one `pds_store::Files` pool, of the share of file descriptors the server takes, written to its journal too (`pds_store::Journal`, in the directory's `journal/`), a commit waiting for the journal's sync, which the commits waiting at once share whichever accounts they are to, and the journal's own thread making each log durable in its own file 2 seconds after the last commit or once the journal holds 16 MiB, as the server's does. Beside it, "the same, no pool and no journal, each log synced by itself (PDS_SYNC each)": the same logs with no pool and no journal, each synced by itself, commits to one account that wait at once sharing that sync | | JSON frames, bytes as hex | the same content as one JSON array, every byte string in hex (`entries` in `src/frame.rs`), a frame of `pds_store`'s first kind, its check SHA-256, in a `Wal` with no pool, each synced by itself: what writing the commit's bytes in hex costs, for each account alone | | binary frames | the same content with every length a varint and bytes as bytes (`src/frame.rs`), made with `pds_store::frame_of` and written with `Wal::append_frame_at`, so with a `Wal`'s room too: half the bytes of JSON frames, bytes as hex | | binary frames, each written at its offset | a log of this benchmark's own (`Placed` in `src/write.rs`): the same frames written with `write_at` where the last ended, in a file that grows as they are, and synced with `sync_data`, commits that wait at once sharing a sync as a `Wal`'s do. It is there to compare the next with: it differs from a `Wal` in what it leaves out (room, generations, a pool of files, mending a torn append) | | the same, allocated ahead | that log in a file allocated ahead a chunk at a time, so that a sync has no new size of the file to make durable: 16 MiB for one log for all; 64 KiB and 16 KiB, a row each, for an account's, since a server with many accounts, most of which write little, pays for each chunk begun. Where its frames end is found at a replay by the first that fails its check, and after each row every commit is replayed from it to see that they are there | | the same, its waiting commits answered in turn | that log, growing, for one log for all alone, sharing a sync as a `Wal` did before it woke its waiters together: the lock on what is durable held through each sync, and a commit learning that a sync covered it when it gets that lock in its turn, behind any commit that began the next sync first. The syncs are the same; how many commits each answers is the "log syncs" column against the commits | | the same, a sync first letting the commits begun be written | that log, growing, for one log for all alone: a commit says when it has begun, and a sync waits, for at most a millisecond, until every commit begun has written its frame, and so covers them. With writers that each begin their next commit as the last is answered, all of them are in every sync, where without it a commit that begins during a sync waits for the one after and each sync has half of them. RocksDB and fjall have a row with the same wait, so that the three are set beside each other under one way of sharing a sync | | a log for each account written and not synced, each commit made durable by one journal for all | a log for each account and beside them one journal (`Journaled` in `src/write.rs`), with the server's frames, JSON with holes, and with binary ones, a row each. A commit, under its account's lock, writes its frame with `write_at` where the last ended in its account's log, a plain file that grows, and then appends a record to the journal: a frame whose payload is the account's number (4 bytes), the frame's offset in the account's log (8) and the frame itself, its check XXH3-64. Outside the lock it waits for the journal's sync, and is answered then; no commit syncs an account's log. The journal is the log of "one log for all, the same, allocated ahead by 16 MiB of zeros", the same code: written at its offset, its room zeros, its waiting commits woken together by one sync. Since a record is written after its frame and under the same lock, the journal never holds a record of a frame further on in a log without those of the frames before it | The room of the logs allocated ahead is got two ways: as zeros written once, which leaves nothing but the frame's bytes to write, and with `fallocate`, on Linux alone (one row for each layout), which leaves the room unwritten to the file system, so that a frame first landing there may still give the file system a record of its own to change. A log allocated ahead has its first chunk as it is made, before any commit; a chunk used up, the next is got under the log's lock, inside a commit that is timed. macOS has no `fallocate`, and its `F_FULLFSYNC` flushes the drive whatever the file's size did: Linux is where allocating ahead is expected to matter. The rows of JSON frames and of binary frames not of this benchmark's own logs are a `Wal`'s, and have the room a `Wal` makes, which is none until it is written to: its file is made empty, the first commit to it grows the file by its frame and as much again, and a later one that the room left does not hold by its frame and as much as was written to the log so far. All of that is inside commits, warming or timed. So over 2000 accounts, where a log gets a commit or two, a `Wal`'s rows time the giving of room together with what it is worth, a file that has not grown under the second commit's sync; and the rows allocated ahead, every commit of which lands in room made before, time what it is worth alone. The "zeros written a commit" column is the bytes of zeros this benchmark's own logs wrote for each commit, the room a log began with and each chunk after: of a `Wal`'s rows it is empty, and what a `Wal` writes of zeros is never more than twice the bytes of its frames. The journal's rows have a thread that makes checkpoints. Whenever the journal has grown by 16 MiB since the last (`JOURNAL_CHECKPOINT`; the bytes `BENCH_JOURNAL_CHECKPOINT` names instead, which `run.sh` sets to 16 KiB for a small run), it makes every account's log durable as it then is: on Linux by one `syncfs` of the directory's file system, as the server's `sync_found` does; elsewhere by a plain `fsync` of each log written to since the last checkpoint and then one full sync of the journal. The records the journal held when a checkpoint began are after it not needed to recover the logs. A checkpoint takes no lock, only a flag of each log that says it was written to, and commits go on through it. The last three columns are the checkpoints made during the timed commits, how long one took and the longest; they are empty of every other row. One more is made after the commits, and under the table is how long it took. The journal's syncs, the "log syncs" column, leave the checkpoints' out. The bound on these rows is "one log for all, the same, allocated ahead by 16 MiB of zeros": the journal alone, of binary frames, its records 12 bytes shorter and no account's log written. There is no such row of the server's frames; what the encoding costs one log is the two rows of a `Wal` for all. After each of the journal's rows the journal is read, each record's check computed, and seen to hold a record for every commit, each at the offset where the one before it for its account ended; and each account's log is seen to be, byte for byte, the frames of its records and nothing else. SQLite has three rows. Two are as in the first part: one connection a database, a transaction a commit, so commits to one database wait in line and none shares a sync. The third is the one database as a server built on it would share a sync: one thread writes, taking every commit that waits and putting them all in one transaction, and each is answered when that is on disk. There is no such row for a database for each account: a transaction is one database's, so only commits to one account could share one, and a server makes those one after another. RocksDB has these rows. One database for all as the first part has it, every option RocksDB's own. The same with `max_write_batch_group_size_bytes` raised from 1 MiB to 64 MiB: the commits one sync covers are one write group, and a group that a small batch leads takes no more than an eighth of that option beyond its own batch, 128 KiB as it comes, twenty of these commits however many wait, and 8 MiB so raised. Its batches written unsynced and one `FlushWAL(true)` shared by the commits that wait, each sync first letting the commits begun be written, as the log's row above. Both of the first two again with its log files written over (`recycle_log_file_num=4`): a log file no memtable needs is kept and the next log written over it from its start, so that a sync has no new block and no new size to make durable, as a log allocated ahead has none; `max_total_wal_size` is 4 MiB there, so that the files come round within a row, and 3000 commits are written before the row is warmed. Under the table is how much of what those rows wrote to the log went past what its files held before, by their inodes and sizes. RocksDB has no option that gives its log blocks written ahead: it allocates with `fallocate`, and its `async_wal_precreate` reserves the next file empty. And a database for each account, as the server keeps a log for each, where there are 40 accounts and, of the tables of 2000, in that of 64 writers alone, since 2000 databases take over a minute to make: they share a block cache and the threads that write their files, a memtable of each holds 1 MiB where the one for all has RocksDB's 64 MiB, and no two share a sync. Under the table is what they take: the time to make them, the memory the process holds with them open, their files. A commit is written under its account's lock, which is held until RocksDB answers, as SQLite's is: RocksDB orders batches as they reach it, and a commit's deletes of the nodes before must follow their puts, so two commits to one account never share a sync, where a log's may. RocksDB's sync is its own, `fdatasync`, as a log's is `File::sync_data`. fjall (the `fjall` crate, 3.1, an LSM tree in Rust; `src/fj.rs`) has the same keys and values in three keyspaces, as it calls its column families, under one journal, a commit one batch across them. It has no group commit: a batch made durable (`PersistMode::SyncAll`, which is `File::sync_all`, so `F_FULLFSYNC` on macOS and `fsync` elsewhere) is written to the journal, synced, and put in the memtables under the journal's lock, one commit after another, and that is its first row. In the next two the benchmark makes group commit over it: a batch is committed with no durability and the commits that wait share one `Database::persist(SyncAll)`, begun as soon as one waits, or first letting the commits begun be written, as the log's row above. fjall holds the journal's lock through that sync too, so nothing is written while one goes on. The last row is a database for each account, where there are 40: each has one thread to write its files where the one for all has four, and memtables of 1 MiB; 2000 take over three minutes to make. **Writes with no sync** (`store-bench unsynced `). The same layouts with no sync at all, the logs never calling one (in the row of the server's journal, a commit's frame is written to the journal too, and its own thread makes its checkpoints as ever, which no commit waits for), RocksDB writing its log with `set_sync(false)`, fjall its journal with `PersistMode::Buffer`, and SQLite with `synchronous=OFF`, from 16 writers over 40 accounts and 256 over 2000: **nothing here is durable**. It is what the processor allows each layout, which a drive whose syncs cost well under a millisecond would approach and none would pass. The commits are made before any is written, 40,000 of them and a tenth more to warm, so that a writer does nothing but write; a commit's time is its frame made and written to the system's cache, or its transaction. A database writes each commit once, holding it by its key; a log writes them five times over, being too fast to time in one round, and the table says how many each row wrote. **A log replayed** (`store-bench scan ...`). A log of the first part's frames of the size given is written, then read as a start or a cold account's load reads it (`pds_store::scan_file`): each frame's check computed, its holes filled and its JSON parsed. The best of three, so the file is in the system's cache: what the replay itself costs, a floor under what loading an account with that much log takes. A second table has one account's realistic commits, as many as make that size of the server's frames, in a log of those and in a log of binary frames, with the bytes a commit takes in each. The binary log is read by a reader of its own over the same frame header: each frame's check computed, and every field walked, each string checked to be UTF-8. It makes no values of them, where the JSON reader makes a tree of values with every byte string in hex, its holes filled: a replay that kept what it read would allocate for it in either. **The firehose replayed** (`store-bench firehose [every]`). The events of 200,000 realistic commits over 2000 accounts, given in order of seq to something that counts them, from: - one log for all, of binary frames, whose order is the firehose's: read from its start, each frame's check computed and its fields walked, the event's bytes given as they lie in the frame. A sparse index, written as the log is, has the seq and offset of each 64th frame (`every`), 16 bytes each, and is held in memory; - the same frames with each other check in SHA-256's place (below), in a log of its own, and with no check computed, to tell the check's cost from the reading's; - SQLite: a table of events alone by seq, as a sequencer's, `SELECT seq, frame FROM event WHERE seq > ? ORDER BY seq` on one connection, its page cache SQLite's own 2 MiB; - RocksDB: its family of events alone, a key the seq, big end first, so that the order of its keys is the firehose's, a value the frame after its account and time; with its blocks compressed (Snappy, as RocksDB compresses when it is not told) and as they are, a row each. Read with an iterator sought to the seq after the cursor, each block's check (XXH3) computed as it is read from its file; its block cache is 32 MiB, what RocksDB gives itself, and a whole replay reads the files 2 MiB at a time (`set_readahead_size`) and leaves what it reads out of that cache (`fill_cache(false)`), as a long scan is told to; - fjall: its keyspace of events alone, the same keys and values, with its blocks compressed as it compresses when it is not told (LZ4 from its third level down, nothing above) and as they are, a row each; read with an iterator over the seqs from the one after the cursor (`Keyspace::range`), each block's check (XXH3) computed as it is read from its file, its block cache 32 MiB, what fjall gives itself. Its bytes on disk leave out its journal, which is 64 MiB allocated ahead whatever it holds; - a log for each account, the server's frames, as the server keeps them now, where the order has to be made again: every log read with `pds_store::scan_file`, its events' frames taken out of their hex and held in memory, all sorted by seq. Per source: its bytes on disk, the time of a whole replay (the faster of two), events and MiB of events per second, and the time to the first event after a cursor, the p50 and p99 over 200 cursors spread through the middle half, with the file, connection or database open and the statement prepared. The log starts at the frame its index names and reads forward, at most 63 frames before the one wanted, each checked; the last two columns are the same with an index entry for each frame, 16 bytes a commit held in memory, so that the first frame read is the one wanted. For the logs for each account the first event waits for all of them to be read and sorted, the one figure in both columns. All of it is on one thread. The one log holds whole commits, their blocks too, about twice the bytes of the events, and checks every byte; SQLite's table and RocksDB's family hold the events alone, SQLite checks nothing it reads, and RocksDB checks each block with XXH3. All of it favours them here, and is what each would do in a server. The files are written just before and never synced, and read from the system's cache where it has kept them; RocksDB's are as its own compactions leave them, which for keys that come in order is as they were first written. **The check.** A frame and a table's block each carry 8 bytes of check, computed whenever one is read: in `pds-store` a JSON frame's is the first 8 of SHA-256, a frame with holes' (a commit's) XXH3-64, and a table's block's, like its index's, its bloom filter's and its footer's, is XXH3-64 (the first 8 of SHA-256 in a table of the version before, which is still read, and whose footer has none). The firehose and read parts measure each of four in that place (`src/check.rs`): those two; CRC32C, with the processor's own instruction for it, 4 bytes of the 8; and the first 8 bytes of BLAKE3. Each finds a frame torn or a block rotted, CRC32C missing one in four thousand million where the others miss none to speak of; CRC32C and XXH3 do nothing against bytes put there on purpose, which a check kept beside the bytes it checks does little against whichever it is. **Reads** (`store-bench reads `). What 2000 accounts hold after 500 realistic commits each, their records and the tree over them, kept these ways with the same content (`src/stores.rs`): - a table for each account as the server writes its checkpoint (`pds_store::table::TableWriter`): under `B` and a CID a block with its count and rev, under `P` and a path its record's CID and rev, in data blocks of 4 KiB; read as the server reads one, with `Table::get` and `Table::iter` through one `BlockCache` for all, which keeps the blocks read and holds at most a share of the tables' files open: the share the server takes of the file descriptors the process may have, a quarter of them and at most 4096 (`pds_store::files::share`; `spec/LIMITS.md`, "Open files and threads"), which the row's name says; - SQLite for each account, the first part's `block` and `record` tables, one connection each; - one SQLite for all, a connection for each thread that reads; - one RocksDB for all, its families of blocks and of paths (`src/rocks.rs`), a value what the table has under the same key; with its blocks compressed (Snappy) and as they are, a store each. It is given its rows in the order the one SQLite database gets them, through its log, and left until it has no compaction of its own to do: none is asked of it. One handle on it serves every thread; - one fjall for all, its keyspaces of blocks and of paths, loaded with the RocksDB and let settle likewise, with its blocks compressed (LZ4 below its second level) and as they are, a store each. Its blocks are 4 KiB before compression, each checked with XXH3 as it is read, its files have bloom filters of 10 bits a key but in its first level, and its block cache is 1 GiB as the others': all but the cache as it comes. A record is two `get`s, and a listing an iterator over the paths and a `get` for each block: it has no read of many keys at once. None holds an event. Each store is measured in three processes of its own (the records one after another, with the first reads and the memory; the 16 threads; the listings), each started after the store's files are read through once: **cold** is such a process, which has read nothing, its accounts open; the files are in the system's cache if it has the memory to keep them, and that cache is never flushed. **Warm** is the same reads again in that process. No cache is let fill: the block cache may hold 1 GiB, each connection to the one database 1 GiB of pages, RocksDB's block cache 1 GiB (an LRU cache, the kind it makes for itself; its HyperClockCache read no faster from 16 threads here), and a connection to an account's database SQLite's own 2 MiB, more than the database; so warm is everything read held by the process, and the resident memory after the reads is what each holds of what it read, not what a server with a budget would keep. Measured: - a record by its path, 20,000 drawn from all accounts, one after another: two keys in the table (the path's CID, then the block), one prepared statement joining the two tables in SQLite, two `Get`s in RocksDB; p50 and p99; - the same from 16 threads, 5000 each: records per second (how each store is read from them is below); - a listing of 50 records in order from a path, 2000 of them, each record's bytes read: in RocksDB an iterator over the paths, then their blocks asked for together (`MultiGet`); - the first read of an account not yet open: the table opened (its index and bloom filter read and checked) or the account's database opened, and one record read. For the one database, SQLite's or RocksDB's, it is a read on a connection or a database already open, but for the first; - resident memory (`ps`) before anything is open, with every account open and one record read from each, and after the 20,000 records; - bytes on disk. The table reads blocks of 4 KiB and computes each one's check, XXH3-64, as it reads it, from a file its block cache holds open: of the 2000 tables' files at most a share are open at once, as in the server, the least recently read closed to make room, so that where the share is fewer than the accounts a read of an account whose file is not among them opens it again, once for the blocks it then reads. A read of a block the cache holds asks the bloom filter nothing; one that must read the file asks it first. SQLite checks nothing and reads 4 KiB pages from files it holds open. RocksDB reads blocks of 4 KiB before compression from files it holds open, all of them from when it is opened, and computes each block's check, XXH3, as it reads it; each of its files has a bloom filter of 10 bits a key, and its indexes and filters are held outside the block cache, as they are unless it is told otherwise. The databases get their rows in the order commits would give them and are not vacuumed, so their pages are part empty, as a database written over time has them; a table is written whole and dense. The `block` table is `WITHOUT ROWID`, as in the first part, which costs SQLite disk and not reads: see the next part. On 16 threads, the table's threads share the one block cache, which is in 64 parts, each with a lock, and RocksDB's share its block cache, in 64 parts likewise. The account databases have one connection each, behind a lock of its own, which a thread holds for a read: 16 threads over 2000 accounts seldom want the same one, and a connection for each thread and account would be 32,000 of them. The one database has a read connection for each thread, in WAL mode, its statement prepared once and kept. SQLite is built for it (`.cargo/config.toml`): without `rusqlite`'s `SQLITE_ENABLE_MEMORY_MANAGEMENT`, which puts every connection's page cache behind one lock, and with `SQLITE_DEFAULT_MEMSTATUS=0`, which takes its count of allocations from behind another. A last table reads the tables cold other ways, to tell what of a cold read is which choice. By `Table` with every table's file held open (`BlockCache::with_open`), whatever the share: what a share fewer than the accounts costs, and the same as the row before where the share is no fewer. By this benchmark's reader (`src/tables.rs`) from the same files, which holds every file open, keeps the blocks it reads in a map of each table's own, where `Table` has the one cache for all, has no bloom filter, and computes one of four checks over each block (above): XXH3-64 is the one the files hold, and is compared with it; each other is computed over the same bytes and compared with nothing, since a table with another check is of another version of its format. And by that reader from the layout before, written by this benchmark from the same entries (`tables::reblocked`): a table of the version before, its check SHA-256, in blocks of 16 KiB, each block's last key whole in the index, the file opened for each block read. That row has the layout, the check and the opens of the table as it was, and neither its bloom filter nor the one lock its block cache had, so its figures from 16 threads are not that table's. Per way: a record by its path, a listing of 50 and the first read of an account, cold, and records per second from 16 threads, cold and warm. **A record read, a stage at a time** (`store-bench ladder `, then `node ladder.mjs `). The read a `getRecord` makes, taken apart, so that when it gets slower the stage that did can be named: 40 accounts of 100 realistic records each, the same records at every stage, each stage the one before and one step more. | stage | is | |--:|---| | 1 | the record's CID and rev, taken from its path's entry in the account's table (`Table::get` through one `BlockCache`, as the server reads its tables) | | 2 | that, and the record's block by its CID from the same table, its bytes copied out as the server's `Repo::get` copies them | | 3 | that, and the block decoded as the server decodes it (`pds_repo::data::decode`) | | 4 | that, and the body `getRecord` answers: `{uri, cid, value}` with `value` from `pds_repo::data::to_json`, serialized to bytes, as the server's `record_json` makes it and its router writes it | | 5a | `GET /xrpc/_health` from the server: a call that reads no record, what HTTP, the server's router and the client cost | | 5b | `com.atproto.repo.getRecord` of a record from the server: stage 4 and all the server does around it | Stages 1 to 4 are in Rust (`src/ladder.rs`), in one process, over tables written as the read part writes them, every record first held to decode and to encode again to its own bytes. Stage 5 is in Node (`ladder.mjs`): the server (`target/release/pds`, or `PDS_BIN`), with a directory of DIDs that the script serves, is given the 40 accounts by `createAccount` and the same records, as `store-bench ladder-records` prints them, by `applyWrites`, an account's records 50 to a call, 16 calls at once: a commit and a sync a call, so that the writes take seconds where a `createRecord` each would take a sync each. They are written with `validate: false`: a reply in them holds its CIDs as links where `app.bsky.feed.post` has strings, which the lexicon refuses. Its backfill window is half a second (`PDS_REPO_BACKFILL_LIMIT_MS=500`), so that once the writes have stopped, one more write to each account folds its log into its table, which the script waits for: every record is then read from a table, as in stages 1 to 4. Each stage is measured after one pass over every record, so that the block cache holds all a read touches, the server's too: 20,000 reads drawn from all the records, one after another on one thread (for stage 5, on one kept-alive connection from Node's `http` client), their p50 and p99; and reads per second from 16 threads of 5000 each (16 connections at once, 2000 requests each). The gaps say: - 2 less 1: a second key found in the table and its bytes copied; - 3 less 2: about 300 bytes of DAG-CBOR made into values; - 4 less 3: rendering: the values made JSON, the CIDs base32, the URI made, and all of it serialized; - 5a: the least a call through HTTP costs here, with no store in it, the client's own share among it; - 5b less 5a less 4: the server's own work around the store: the query read and the call routed to its method, the account found and its store locked, the takedowns looked at, the answer's headers. When `getRecord` gets slower and stages 1 to 4 do not, it is here. The client is in 5a and 5b alike, and drops out of the difference; at 16 at once it shares the machine's cores with the server, so those figures are of both. **Size and compression** (`store-bench size `). 200 accounts of 500 realistic commits each, a record a commit, and what they take on disk: - as a log for each account before any compaction, the server's frames and binary ones. A log holds every commit whole: its event, and the tree nodes later commits took out; - as the tables and the SQLite databases of the part above, which hold what the commits leave, the records and the tree, and no event; - as the tables with each data block compressed with zstd at level 3, and with a dictionary of 64 KiB trained on each eighth block, counted once. A block is 4 KiB before, as the writer ends it, the index, bloom filter and footer as they were; - SQLite after every record is rewritten three times, each rewrite a transaction as a commit's (the record's block and each node's from it to the root taken out, new ones put in, the path's row replaced), as that leaves the files and then after `VACUUM`; - those vacuumed databases with the blocks in a table with a rowid and an index on the CID. The first part's table of blocks is `WITHOUT ROWID`, which keeps about 1 KB of a row in its page at most, so a tree node over that takes a 4 KiB page more of its own: this row says how much of SQLite's size is that choice of schema; - one RocksDB for all, the part above's, three ways: its blocks compressed with Snappy, as they are, and compressed with zstd at RocksDB's level for it, 3, to set beside the tables as they are and compressed. Each as the load leaves it once its own compactions are done; after every record is rewritten three times, each rewrite one batch (the same blocks deleted and put as SQLite's transaction, the path's value replaced), all accounts' at once from as many threads as there are cores, again once its own compactions are done; and after a compaction of every key is asked of it (`CompactRange`); - one fjall for all, two ways, its blocks compressed with LZ4 below its second level and as they are (it has no other compression), at the same three points, the last after `major_compact` of each keyspace; its journal is left out. It counts no bytes written, so the table under has no row for it. A table under it has what RocksDB wrote for what it was given: the bytes of the keys and values handed to it, and the bytes it wrote to its log, to the files it writes from its memtables, and to the files its compactions write, by its own count (its statistics), through the load and the rewrites, and then with the compaction of every key. Each byte given goes to the log once; what a memtable still holds when it is written out goes to a file, compressed; and each compaction a key's file is part of writes it again. The databases here are a few hundred MiB, which fills no level but the last: a database of many GiB has its keys pass through more levels, each a rewrite, and this table is a floor. The last table has what zstd costs: the stored bytes of each 100, MiB per second compressing and decompressing the blocks one at a time on one thread, and the median time of a record read by its path through a reader with no cache (`Reader` in `src/tables.rs`), the same for all three rows: two blocks, each read from its file, its check computed over what is stored, decompressed if it is, and searched. The files are in the system's cache. A table is not rewritten in place, so it has no row for rewrites: between compactions an account takes its table and its log, and a log may grow to its table's size, or 4 MiB if that is less, before it is compacted. **The service log by the number of accounts** (`node service.mjs [accounts...]`). For each count, without and with an email and a password on every account: the server (`target/release/pds`, with the scale run's `PDS_CACHE_BYTES`, 16 MiB) is started on an empty data directory, once unmeasured and then three times, the slowest the empty start; the accounts' entries are written into its service log as the server writes them, none with a repository; it is started on that log; one invite code is made, a write to the service log that makes its compaction due, and its resident memory sampled until the checkpoint is in place; then it is started on the checkpoint. The table has: - the start on that log, all of it replayed. The server compacts its service log long before it is that long, so no server starts on one: this is what replaying one costs, and is held to no bound; - the start on the checkpoint, as a server that holds those accounts starts, until `describeServer` answers, beside the bound `spec/STORE.md` gives a start (twice the empty start and 250 ms), and its resident memory then; - how long the compaction took, and the most resident memory while it ran, beside the bound `spec/STORE.md` gives memory (the empty start's, 32 MiB, the cache and 4 KiB an account). A figure over its bound is in bold. ## Reading the results - A machine's numbers compare only with the same machine's: the disk and how it syncs decide the write parts. `RESULTS.md` says which machine each table is from. - The write parts sync every commit and are the disk's; every other part reads files the system's cache holds, and is the processor's. None measures a read from a disk. - The accounts of the last part hold no sessions, app passwords, preferences or OAuth state, which a real account adds to the same log: its figures are a floor. - `service.mjs` writes the service log's own format, and the write parts the server's commit frames: a change to either format is a change here. The binary frame, the log written at offsets and the compressed table are this benchmark's alone: the server writes none of them. - SQLite is run as a careful server would run it for this: one connection a database for writes, prepared statements, a transaction a commit or one for all that wait, a connection a thread to read the one database, and built without the two locks `rusqlite`'s own build puts across every connection (`.cargo/config.toml`). Its page size, checkpoints and page cache are as they come but where a part says otherwise; it is not given memory-mapped reads. - What the logs and tables do that SQLite does not is counted against them: a check of every frame and block read. RocksDB checks each block it reads too, with XXH3. - RocksDB is run as a server built on it would run it: one database for all accounts, a column family for each kind of key over one log, a commit one batch and one sync, bloom filters, one block cache, a `MultiGet` for a listing's blocks, an iterator for the firehose. Its compression, block size, check, memtables and compaction are as they come (Snappy, 4 KiB, XXH3, 64 MiB, leveled) but where a row says otherwise. What is and is not fair to it, part by part: - Writes. It is one database for all, so commits to any accounts share its syncs: the rows to set it beside are one log for all and one SQLite for all with one writer. The server's log for each account shares no sync across accounts, and RocksDB beating it says what one log would buy, not what RocksDB would. Against it: a commit holds its account's lock until its sync ends, so two commits to one account take two syncs, which 16 writers over 40 accounts feel and writers over 2000 do not; and as it comes, a write group stops at twenty of these commits, which the row with larger groups takes away. For it: its sync flushes the drive on macOS only as this workspace builds it. The row of a database for each account is RocksDB put to a use it is not made for, and is there for what that costs. - Writes with no sync. It writes each commit to its log and to its memtables, and writes memtables out and compacts on other threads while the writers run; a log does the first alone, and its account's compaction, which no row here times, comes later. What RocksDB does in a commit it does for good: that is fair to both. - The firehose. Its family holds events alone, where the one log holds commits whole, twice the bytes: that favours it as it favours SQLite. Its files are minutes old and written in order of key, none overlapping another; a sequencer's would be the same, since events come only in order and old ones are dropped from the front. - Reads. Every account is open from the start, in a few files, where a table or a database for each account is opened at its first read: its first read of an account is a read like any other, and that is what one database for all gives. Against the tables it has this in its favour: it was loaded minutes before it is read, with nothing written since, and has settled into few files, where a database written to for months has more levels and files for a read to pass (a bloom filter turns it from most) and compactions running under its reads, which no part here has. And this against it: a table is written whole for one account and holds nothing else, where RocksDB's files hold every account's keys together. Its indexes and bloom filters are held outside its block cache, so the cache's size does not bound its memory unless it is told to keep them there. - Size. Its sizes are of databases let settle; between compactions it holds what the rewritten rows show. What it wrote for each byte given is of a database that fits its last level, and is a floor for one that does not. - fjall is run the same way as RocksDB, with what it has: one database, a keyspace for each kind of key, a commit one batch, bloom filters, one block cache. What differs is said where it does: it shares no sync among commits unless the benchmark does it for it, it has no `MultiGet` and no compression but LZ4, and a database of it for each of 2000 accounts takes too long to make to be in a table.