chadtmiller.com/monolith
An atproto PDS
Comparing with the reference
harness/compare.mjs holds the server to the reference PDS
(@atproto/pds), and you can run each of its runs by hand.
PDS.md says what's compared, and
each area's page has its own "How it is checked".
You'll need a clone of the repository, with the harness's packages installed and the server built (Contributing).
The comparison
spec/PDS.md says what the server does and how compare.mjs checks it.
The server and the reference PDS get the same random requests, and their
answers have to agree, apart from what differs between any two servers.
The reference's own library has to verify the server's whole repository,
and its MST root must equal the reference's.
Some requests go out at once: puts of different keys, and batches racing on the same commit (exactly one of those may land).
When the server forwards a request to the AppView, a chat service or any
service an atproto-proxy header names, both sides send it to stub
services (harness/lib/services.mjs). The stubs record what each side
sent. That way the harness compares the forwarded requests, their headers
and their service-auth tokens, as well as the answers.
An account on a third PDS that shares the PLC directory publishes OAuth's
permission sets
(OAUTH-PERMISSION-SETS.md), and
both servers point at it. So each one resolves the harness's sets over the
network, with their proofs, the same way it would any app's. The harness
compares PAR's answers, the page's data, the token's expanded scope at
sign-in and refresh, and what calls a set's session may make with the
reference.
Every run ends by checking the server's operations
(OPERATIONS.md). The harness checks its
/xrpc/_health against the reference's, its metrics against its log and
what the harness sent, and the admin's _logs and _stats against what
it wrote and counted. It checks every span against the request it belongs
to, at a stub OTLP receiver that the run takes away for a while. It checks
a shutdown, and starts at another log level, with a PLC directory that
fails, after a kill, and with variables that don't parse. Last, the whole
log has to hold a line for every request and none of the passwords,
tokens, codes, emails or keys the run handled.
An agent working independently of the first built a second server from
spec/. It passed the same harness, with the same repository encoding and
the same behaviour where the specs leave it open.
Running it
Run it from harness/, with the server's command after --:
cd harness && npm ci
node compare.mjs --steps 400 --seed 1 -- ../target/release/pds serve
A seed makes the same requests every run. Each flag below starts a different kind of run.
Crashes and the firehose
node compare.mjs --steps 1000 --seed 21 --crash-every 15 --firehose -- ../target/release/pds serve
--crash-every N makes the harness kill the server with SIGKILL every N
steps, often during a write. It starts the server again on the same data
and checks that every answered write survived, and that the interrupted
write landed whole or not at all. pds-server's own tests check what
survives a power cut, by cutting its log short at every byte of its last
write.
--firehose also subscribes the harness to both servers'
subscribeRepos. It checks each of the server's events the way a consumer
would (signature, the chain of since and prevData, the ops undone on
the tree in the event giving the previous root, and the reference
library's verifyProofs). It compares the server's commits with the
reference's by their ops, and replays from random cursors. When crashes
are on too, the stream has to carry on after each restart with no gap or
repeat.
Tools and commands
node compare.mjs --steps 400 --seed 31 --commands -- ../target/release/pds serve
node conformance.mjs -- ../target/release/pds serve
--commands sends some of its requests through the MCP tool, the CLI
command or pds xrpc instead of XRPC, in any of the three MCP revisions.
They're held to the reference like the rest. pds xrpc gets each argument
in the forms the request allows: ==, =, :=, an @path file, and -
with a Content-Type: header. Before that, the run checks every tool's
title, schema and hints, and every command's flags against the lexicons.
It also checks each legacy revision's handshake, pds xrpc on the
server's own routes and on a forwarded method, and its usage errors
(COMMANDS.md).
conformance.mjs runs MCP's conformance suite against the endpoint, for
2026-07-28 and 2025-11-25.
Invites and limits
node compare.mjs --steps 400 --seed 1 --invites -- ../target/release/pds serve
node compare.mjs --steps 400 --seed 1 --limits -- ../target/release/pds serve
--invites runs both the reference and the server with invite codes
required.
--limits turns the rate limits on and compares each with the
reference's, under both password sessions and OAuth tokens. It also starts
both with an upload limit of their own, and compares them at its edge.
Then it holds only the server to the rest of
LIMITS.md: file modes, the session and
app-password caps, firehose messages and subscribers, calls to the PLC
directory, timeouts, the free-space floor, the blob quota and its time for
unreferenced uploads, and starting again with more accounts than open
files allowed.
Lifecycle
node compare.mjs --lifecycle -- ../target/release/pds serve
--lifecycle (LIFECYCLE.md) deactivates
accounts, has the admin take them down, and deletes them by emailed token.
It checks what each state does to every method and to the firehose. When
you add --crash-every N, the harness kills the server every N changes of
state.
Accounts moving between servers
node compare.mjs --moves -- ../target/release/pds serve
--moves (MIGRATION.md) moves accounts to the
server from a third PDS running the reference, and from the server back
to it. Their handles are under the server's domains, or external ones
served by DNS and HTTPS. The harness compares every step with the
reference, holds both firehoses to what each move puts on them, and checks
the methods of a move by account state. It moves accounts but not blobs.
--migrate, below, moves blobs.
The store, scale and upgrades
node compare.mjs --steps 600 --seed 51 --crash-every 40 --store -- ../target/release/pds serve
node compare.mjs --scale -- ../target/release/pds serve
node compare.mjs --upgrade -- ../target/release/pds serve
--store holds the server to STORE.md. It gives
both sides a five-second backfill window, then rewrites a few accounts'
records, blobs and space records many times over. It kills the server
while it reorganizes its data, and compares every read and the firehose's
OutdatedCursor replays with the reference. Then it holds the data
directory to three times the live state and a constant for each account,
and each start to twice the median of the empty starts timed around it,
plus 250 ms. A start over that is tried again up to twice before it fails.
--scale holds only the server to STORE.md's
scale rules, with a 16 MiB cache. The harness
samples its resident memory while it holds 2000 accounts, touches each,
writes a 64 MiB repository, streams it back and replays the firehose. It
gives one account 4000 blobs and a space repo of 40000 records, and reads
them back after a start, the whole space repo among them. Then it holds
100000 accounts, finds one in 50 of them by DID, handle, email and
session, and refreshes 10000 sessions several times over. Through all of
that, memory has to stay within its base, the cache and 4 KiB an account.
The server has to start as fast with all of that as on an empty data
directory. After a kill, a shutdown and a start, the firehose's first
message from a cursor has to come within a second. Making the accounts
takes most of the run's time, some minutes.
--upgrade starts only the server on the data directories the last
release wrote, in harness/upgrade/ (one shut down and one killed). Every
read has to match what that release answered there, and the server's next
writes have to carry on from that release's
(STORE.md).
Blobs in a bucket
--migrate moves the server's blobs from its data directory into a bucket
and back with pds migrate-blobs. The command is killed at random moments
and run again, one move is turned back halfway, and one is stopped by a
missing object. After each, the harness reads every blob against the
reference's (BLOBS.md). It also holds
the data directory to one process at a time.
--s3 keeps both sides' blobs in buckets of an S3 store that the harness
runs in process, and the store checks every request's signature
(BLOBS.md). During the random
steps and crashes, or --lifecycle, the store fails uploads, reads and
the server's deletes for both sides alike, and the harness compares the
answers. The server's bucket has to match what its accounts hold within a
minute, and its data directory must hold no blob's bytes.
The authorization pages
node compare.mjs --pages --seed 1 -- ../target/release/pds serve
--pages drives the OAuth authorization pages in Chromium on both sides
(OAUTH-PAGES.md). It takes each flow the same
way on both: from a PAR to the client's redirect URI, signing in,
remembered accounts, the email code, two tabs of one request, denials and
backs. The harness compares the API calls each page made and where it
landed, and exchanges its codes. Only the server's pages are held to the
rest of OAUTH-PAGES.md: their views and words, the consent list for each
scope, focus, busy buttons, headers and HTML, assets, console, the origins
asked, the width on a phone, and axe-core's WCAG rules.
Spaces
The spaces run (SPACES.md) checks the server against the reference's pinned spaces alpha. It has its own packages:
cd harness/spaces && npm ci
node compare.mjs --seed 1 -- ../../target/release/pds serve
Besides the alpha's PDS and a second one for the spaces' peers, it starts
a third as the run's lexicon authority. That one publishes the space type
declarations that OAuth space: scopes name, and permission sets that
grant space permissions. The server is started with
PDS_LEXICON_AUTHORITY_DID set to that account's DID, and the alpha
resolves from it too (OAUTH-SCOPES.md, "How
it is checked"; OAUTH-PERMISSION-SETS.md,
"How it is checked").
Performance
node perf.mjs -- ../target/release/pds serve
perf.mjs runs one workload against both servers, each in its own
process. For each part it takes the server's best p99 and throughput as
ratios over the reference's. The server's p99 ratio must be within 1.5
times the baseline's in harness/perf-baseline.json, and never over 1.5.
Its throughput ratio must be at least 0.8 times the baseline's, and never
under 0.8.
The baseline is the weakest of at least five runs on the runner, from the
nightly or the perf workflow. The owner can ask for perf on any branch
(knotless run rerun knotless://chadtmiller.com/monolith --ref refs/heads/<branch> --workflow perf). Each run prints every part's ratios, then all of them on one line
the way the baseline holds them. A pull that makes the server faster
raises the baseline from those, and one that lowers it says why.
Then subscribers to the firehose come and go, 20 cohorts of 20. After the last cohort, the server must hold no more open file descriptors or threads than after the first, and no more than 8 MiB more memory (OPERATIONS.md).
Other PDSes
The benchmark
harness/bench.mjs benchmarks the server beside other PDSes on this
machine, one at a time. Every one runs in Docker the same way, with the
server as its image builds it. It measures:
- perf's workload with
applyWrites,listRecordsand more - the routes a client calls most
- reads and writes by the number of callers, and a mix of them at once
- subscribers coming and going
- each server's start time and memory
- what its client can send at all, so a rate near that is marked
A run refuses a busy machine unless you pass --busy, and it records the
load. Each implementation is an adapter in it, pinned in
bench/impls.json. bench/README.md says how to
run and extend it, and bench/RESULTS.md holds
the last results, with the machine and every caveat.
cd harness && node bench.mjs --setup && node bench.mjs --rounds 3 --json ../bench/results.json
just bench runs the same.
The store benchmark
bench/store/ benchmarks how the server keeps its data, separately from
the rest of the server. It runs the server's logs and tables
(crates/pds-store) beside SQLite, RocksDB and fjall under the same
commits, each synced before it's answered. Its parts are:
- writes by 4 to 256 writers, and with no sync at all
- a log replayed, and a firehose replayed from each store
- reads of a record, a listing and an account not yet open, on one thread and sixteen, with what the process then holds
- sizes on disk, compressed and not
- the server's start and memory by the number of accounts in its service log, beside STORE.md's bounds
Rows next to the server's own each change one thing about how it writes or
reads (frames as bytes, a log allocated ahead, a table's file held open,
smaller blocks, another check), to show what each is worth.
bench/store/README.md says what each part
measures and what's fair to each store, and
bench/store/RESULTS.md holds the last
results, by machine. The store-bench workflow runs it on the runner when
the owner asks, on main or a branch. A pull that changes it or
crates/pds-store runs it small (store-bench-small, BENCH_SMALL),
only to show that it builds and runs.
The first build compiles RocksDB from source, which takes some minutes.
Its bindings need libclang (apt-get install libclang-dev, and Xcode's
tools have it on macOS). On macOS, bench/store/.cargo/config.toml builds
it to sync with F_FULLFSYNC, like the logs do. Without that flag, its
writes there aren't as durable as the others'.
just store-bench # arguments: counts of accounts for its last part
The soak
bench/soak/soak.mjs runs the server, or several PDSes side by side,
under a steady mixed load for hours. As it goes, it checks that nothing
answered is lost. It reads every account's records back whole now and
then, and holds every answered commit to coming through on the firehose.
It's for what only hours show and the benchmark's short rounds don't:
memory over time, how the data directory grows and what it settles to,
and what a crash in the middle of a load costs. It needs only Node.
bench/soak/README.md says how to run it
against a server on this machine or ones on a network, and covers its
settled check of the data directory.
bench/soak/RESULTS.md holds the runs,
beside the reference.
OUT=/tmp/soak TARGET_URL=http://127.0.0.1:2594 TARGET_DOMAIN=test node bench/soak/soak.mjs
bench/soak/analyze.mjs makes every table of RESULTS.md from a run's
files: the soak's own, a sampler's where one ran beside it, and the
server's log cut to its requests and starts. Each run in RESULTS.md is
its description, its command and that command's output, so running the
command again checks it. You add a run the same way, with its files in a
directory of bench/soak/data/.
node bench/soak/analyze.mjs bench/soak/data/at-rest
Compatibility
harness/compat.mjs checks how closely any PDS behaves like the
reference, area by area. It runs the core of compare.mjs's checks (OAuth
and its scopes, sessions, accounts, records, validation, the repository,
sync, the firehose, identity, blobs) against each implementation with an
adapter in harness/lib/adapters/, and keeps going past a difference.
compat/README.md says which checks it runs and
how to add an implementation.
compat/RESULTS.md holds the last run, against
Monolith, zds, vlpds, pds.js and tranquil.
just compat
That's cd harness && node bench.mjs --setup && node compat.mjs --steps 200 --json ../compat/results.json. Arguments pass through, such as just compat --impls zds --areas oauth.