~ knotless Search/
↑↓ move↵ open⇧↵ all results
0
0
README.md

The PDS benchmark

harness/bench.mjs runs the same workload against several PDS implementations, one at a time, on one machine. It runs every one of them in Docker the same way, and writes the tables in RESULTS.md (with the raw numbers in results.json). Everything it runs comes from this repository and the pins in impls.json, so nothing on the machine is set up by hand.

What runs, pinned

implementationwhere it comes fromwhat pins it
Monoliththis repository's Dockerfile, the image it deploys as (deploying), built by --setup as monolith-bench-monolith:localthe last commit of what it's built from (its inputs in impls.json), which the image is labelled with
reference@atproto/pds 0.5.55 through @atproto/dev-env 0.6.26, from npm. The image docker/reference.Dockerfile builds it from the harness's packages as monolith-bench-reference:local, and harness/bench-reference.mjs starts itharness/package-lock.json, and Node 24.22.1 by its image's tag
vlpdsits release image ghcr.io/jazware/vlpds, v1.3.0 (commit 91fa452)image digest in impls.json
RustFS (vlpds's S3 store)rustfs/rustfs 1.0.4image digest in impls.json
zdstangled.org/zat.dev/zds v0.7.8, built by --setup with its own Dockerfile for this machine's architecture and tagged monolith-bench-zds:cef2c63repository URL and commit cef2c63 in impls.json
tranquiltangled.org/tranquil.farm/tranquil-pds main at b95f999 (10 commits past v0.8.0), built by --setup with its own Dockerfile for this machine's architecture and tagged monolith-bench-tranquil:b95f999repository URL and commit b95f999 in impls.json
PostgreSQL (tranquil's database)postgres 18.7 on Alpine, run the way tranquil's compose files run itimage digest in impls.json
pds.jsknotless://chadtmiller.com/pds.js main at 54f4e3a (35 commits past v2.5.2), built by --setup with its own Dockerfile and tagged monolith-bench-pdsjs:54f4e3a. Only the compatibility check runs it (compat/README.md), since it hosts one account and the workload makes 40repository URL and commit 54f4e3a in impls.json

--setup also builds Monolith and the reference outside Docker (cargo build --release -p pds, npm ci in harness/), because the compatibility check runs them that way.

zds's published image (atcr.io/zat.dev/zds:latest) is amd64 only and predates v0.7.8, so the bench builds its own from the pinned commit. While it builds, the Dockerfile fetches Zig 0.17.0 (checked against its SHA-256) and zds's Zig dependencies (pinned by hash in its build.zig.zon).

tranquil's published image (atcr.io/tranquil.farm/tranquil-pds) needs a login to pull, so the bench builds its own from the pinned commit, the way its docs build from source. While it builds, the Dockerfile fetches its pnpm packages and crates, plus UPX (checked against its SHA-256), which compresses the binary by default. tranquil runs on PostgreSQL, its default repository backend (the other is tranquil-store, its embedded one).

vlpds keeps all its data in an S3 bucket and needs strongly consistent conditional writes (If-None-Match: *, If-Match) from it. The store is RustFS. We chose it by running vlpds's own vlpds-bucket-probe (in the vlpds image) against it on a Docker network. All six correctness checks passed (conditional_create, compare_and_swap, race_create, race_cas, list_read_delete, multipart), with the verdict SAFE for vlpds. RustFS was the first candidate tried, ahead of Garage and SeaweedFS, so those weren't probed. MinIO, the store vlpds's docs name for tests, doesn't publish images any more. To probe it again:

docker network create probe
docker run -d --name probe-s3 --network probe --network-alias s3 -e RUSTFS_ACCESS_KEY=probeaccess1 -e RUSTFS_SECRET_KEY=probesecret1234567890 <rustfs image from impls.json>
# make the bucket `vlpds` (any S3 client), then:
docker run --rm --network probe -e VLPDS_S3_ENDPOINT=http://s3:9000 -e VLPDS_S3_BUCKET=vlpds -e VLPDS_S3_REGION=us-east-1 \
  -e VLPDS_S3_ACCESS_KEY=probeaccess1 -e VLPDS_S3_SECRET_KEY=probesecret1234567890 \
  --entrypoint vlpds-bucket-probe <vlpds image from impls.json>

Prerequisites

whatversion used for RESULTS.mdwhy
macOS or Linux, 8+ cores, 16+ GiBmacOS 26.5.2, Apple M1 Pro, 32 GiB
Node24 (24.22.1)runs the bench client and the PLC directory
npm11 (11.19.1)installs the harness's packages from harness/package-lock.json
rustup1.28+ (1.29.1)builds Monolith outside Docker on stable for the compatibility check (setup installs the toolchain)
Docker28 (Docker Desktop 28.0.1, with its Linux VM at 8 CPUs, 7.7 GiB)runs every server and builds the images of Monolith, the reference, zds, tranquil and pds.js. host.docker.internal must reach the host
gitany recent (2.50.1)fetches zds, tranquil and pds.js at their pinned commits, and names the commit Monolith's image is built from
git-remote-knotlessanylets git fetch pds.js's knotless:// repository
sccacheoptional (0.19.0)export RUSTC_WRAPPER=sccache before setup makes rebuilds outside Docker faster

Setup needs the network (crates.io, npm, Docker Hub, ghcr.io, tangled.org, ziglang.org, github.com). The bench itself talks only to loopback and Docker's local network.

You run it by hand on a dedicated machine with Docker. It can't run on the runner, because a job there is a container with no Docker of its own (no socket, and no room to start a daemon), so none of these servers could start. The runner's performance step (harness/perf.mjs, the nightly and perf workflows) compares the server with the reference there, outside Docker.

Commands

From a clone of this repository:

cd harness
node bench.mjs --setup                                     # build and pull everything pinned
node bench.mjs --rounds 3 --json ../bench/results.json     # run; Markdown tables on stdout

Or run both at once from the repository's root with just bench. Then copy the tables into RESULTS.md. node bench.mjs --report ../bench/results.json prints them again from a saved run. Every table puts the reference first, since it's the baseline the others are read against.

--setup does this:

  1. It checks for Node 24+, rustup, npm, git and a running Docker daemon, and stops with a message naming whatever's missing.
  2. It builds Monolith outside Docker (npm ci and npm run build in web/, then rustup run stable cargo build --release -p pds) and installs the reference (npm ci in harness/) for the compatibility check.
  3. It builds the images of Monolith (the repository's Dockerfile) and the reference (bench/docker/reference.Dockerfile, with harness/ as its context) every time. Each is labelled with the last commit that changed what it's built from (its inputs in impls.json), plus -dirty if any of that has uncommitted changes. Docker's cache leaves nothing to do when nothing changed, and Monolith's first build takes some minutes.
  4. Unless Docker already has the image monolith-bench-zds:cef2c63, it fetches zds at its pinned commit into bench/vendor/zds (gitignored) and builds that image with zds's own Dockerfile (docker build, a few minutes the first time). It does the same for tranquil, into bench/vendor/tranquil as monolith-bench-tranquil:b95f999 (a Rust workspace, and the longest build of them all). It does the same for pds.js, into bench/vendor/pds.js as monolith-bench-pdsjs:54f4e3a, for the compatibility check.
  5. It pulls the vlpds, RustFS and PostgreSQL images by digest, unless Docker already has them.

It's idempotent. A run refuses to start if an image it needs is missing, or if Monolith's or the reference's image was built from inputs other than the ones checked out. --impls monolith,vlpds runs only some of them. --rounds N sets the rounds and --clients N the client's threads (4). --storage-rounds N sets the storage rounds (1) and --backfill-ms W their backfill window (10000, see "What it measures").

A run also refuses to start on a busy machine. When the load average over the last minute is more than half the cores, every number would reflect the other work as much as the server. --busy runs anyway. Either way, the bench records the load at the start and end of each round, and RESULTS.md reports it.

What it measures

Each round is a fresh server on fresh data, registering accounts with a fresh in-memory PLC directory (@did-plc/server) that the bench runs. Each server runs in its own container, made with a fresh volume before the start clock and removed after the round. For vlpds, a round also makes a fresh Docker network, RustFS container and volume, and puts vlpds's cache on its own volume. For tranquil, it makes a fresh Docker network and PostgreSQL container and volume, ready for connections before the start clock. The bench warms the server first, running each part and route at a tenth of its size on separate accounts. Then the parts run in order:

partrequests per roundat oncewhat it does
createAccount404no invite codes, and includes the PLC registration
createRecord100016~100-byte records over the 40 accounts
applyWrites200810 creates per call
getRecord200016reads records written by createRecord
listRecords100016limit=50
getRepo10008~75 records per repo
uploadBlob1000864 KiB, application/octet-stream. Afterward, untimed, each account writes a record referring to its first blob
createRecordSerial10001one after another, like one person posting, so there's no other write to share a sync with
firehose100 writes20 subscribersputRecords one after another. Latency is the firehose's lag, from write answered to commit received
firehoseLoaded400 writes20 subscribersputRecords 16 at once, each to a different account. Latency is measured as for the firehose
replay4 subscribersstarts from cursor 0, like relays catching up, and reads every event of the round, then a write made once all are connected

For each part, the bench reports throughput (requests answered per second), p50 and a tail of request times, as the best and the median of the rounds. The tail is the highest percentile with at least ten times above it. That's p99 for 1000 times or more, p95 for 200 or more, p90 for 100 or more, and otherwise the slowest (max), which covers createAccount's 40 and replay's 4.

The firehose's live parts have no throughput, since their writes' throughput would just be the write path's, which the write parts already measure. Their times are the firehose's lag, from a write's answer to a subscriber having its commit, and 0 when the commit came first. replay's throughput is the events sent to its subscribers per second, until each has the commit of the write made once all are connected. Its times are each subscriber's, from asking to having that commit. It has 4 subscribers because zds and vlpds each serve 4 backfills at a time.

Routes. Next come the routes a client calls most besides those, each run by itself:

routerequests per roundat oncewhat it does
describeServer100016
oauthMetadata100016GET /.well-known/oauth-authorization-server
createSession2008logs in with an account's handle and password
getSession100016
getPreferences100016app.bsky.actor.getPreferences
getServiceAuth100016aud=did:web:api.bsky.app, lxm=app.bsky.feed.getTimeline
resolveHandle100016resolves the accounts' handles
describeRepo100016
getLatestCommit100016com.atproto.sync.getLatestCommit
syncGetRecord100016com.atproto.sync.getRecord, which returns a record and its proof as a CAR
listRepos100016limit=100
getBlob100016com.atproto.sync.getBlob of the blobs the records refer to
refreshSession100016each answer's tokens become the account's next ones. Each caller takes its own accounts in turn, so no account's tokens are in two requests at once. It runs last, since the old tokens may be spent

If a server answers a route with no 2xx in the warm-up, that route isn't served, and its table gives the answer. Proxied appview routes aren't measured, because there's no appview in the bench and their speed would be the appview's.

By the number of callers. The bench runs getRecord and createRecord at 1, 4, 16, 64 and 256 callers at once, over the same 40 accounts. getRecord sends 2000 requests or 40 per caller, whichever is more, and createRecord 500 or 4 per caller. The table shows how many are answered with a 2xx per second, their p50 and tail, and how many got some other answer. Those are counted rather than stopped at, so a server that pushes back under load shows it in the table.

The mix. 10000 requests come from 32 callers at once. Out of every 100, each route gets a fixed share (getSession 15, getPreferences 10, getRecord 25, listRecords 15, describeRepo 5, getBlob 10, getLatestCommit 5, createRecord 10, applyWrites 3, uploadBlob 2), spread through the 100 as evenly as they go. Meanwhile 2 subscribers follow the firehose like relays. The table puts each route's p95 in the mix beside its p95 on its own (every route has 200 requests or more in each). Where a read's p95 grows, it waited on the writes beside it. A server that doesn't serve one of the routes doesn't run the mix.

Subscribers coming and going. 11 cohorts of 20 subscribers connect, each from its own address. A cohort connects, gets one write's commit, and goes. The bench records the server's threads and RSS 2 s after the first cohort left and 2 s after the last, and how many subscribers it refused (each tries for 10 s). If those grow, the server didn't let go of what a subscriber held. A server that keeps a subscriber's place after it left refuses the next cohort.

The client's ceiling. Before the servers, the client sends to a responder in Docker, reached the same way as the servers. The responder answers every request with the same 2 bytes and does nothing else (Node's net, in the reference's image), and the client runs it at every concurrency the parts use. A server's rate at half that or more is marked ‡ in the tables. At that point, the client, Docker's port forwarding and the machine's cores the client takes are part of what limits it.

A connection a request. If a server closes each connection after its answer, the client has to open one for every request. Each closed connection holds its local port for a while (TIME_WAIT), so at a few hundred requests a second the client runs out of ports. A request that finds none waits 50 ms and is sent again, timed from when it's sent. The tables mark that rate ◊, since it's held to how fast the ports come back. That's a limit of one client machine, since a server's many clients each have their own ports.

For each server, the bench reports start time (from docker start of its already-made container to the first answered describeServer). It also reports RSS 2 s after start (idle) and the most sampled through the round (peak), from docker top inside the container. That covers only its own process (pds, Node's MainThread, vlpds, zds or tranquil-pds). vlpds's store's RSS is sampled the same way, summed over its container's processes. tranquil's database is measured as the sum of its processes' PSS, because they share its buffers and a sum of their RSS would count those in every process that touched them. Each gets its own table.

For each server the bench also reports what it keeps on disk, from its own storage round, run after its speed rounds (--storage-rounds N, 1 by default, with the median if there are more). That's a fresh server, as in a speed round, but started with a short backfill window W (--backfill-ms, 10 s by default). It's warmed with only the parts, then runs the parts, and their speed isn't kept. It's separate because a short window changes what the speed rounds measure. Replay from cursor 0 gets only the events within W instead of every event of the round, so the speed rounds run each server on its default window.

The window is Monolith's PDS_REPO_BACKFILL_LIMIT_MS and the reference's repoBackfillLimitMs (the same variable). For vlpds it's VLPDS_LOG_RETENTION (its log segment retention, which is its firehose backfill window), with VLPDS_LOG_RETENTION_INTERVAL at W / 2 so that a retention pass falls within the quiet. zds has no such setting (none of the ZDS_* variables its binary reads is a window or an event retention). tranquil has none that reaches W. Its window, FIREHOSE_BACKFILL_HOURS, is in whole hours, and it prunes events at most once a minute, never more recent than that window. So their storage rounds run on what they keep by default, and the table says so.

What a server keeps is the sum of the sizes of the files on its container's volume. The bench reads them with find -type f and stat inside the container. tranquil's image has no shell, so for it they run in a container of PostgreSQL's image given its volumes. The volumes are:

  • Monolith's /data, its data directory
  • the reference's /data, its dataDirectory and blobstoreDiskLocation
  • zds's /data, its database and blobs
  • vlpds's /var/lib/vlpds, which holds only its SST cache
  • tranquil's /var/lib/tranquil-pds/blobs, which holds only its blobs

Each server's log goes to docker logs instead. vlpds's bucket (the files on its RustFS container's volume) and tranquil's database (the files on its PostgreSQL container's /var/lib/postgresql) are measured the same way at the same moments. Like their RSS, they're reported in their own table.

  • Under load: once the parts end, after replay.
  • At rest, as spec/STORE.md, "Bounds", settles it: after the parts, there are no requests for 2 W + 1 s, then one createRecord to each account the round made (the warm-up's too, since they're in the same data directory). Then the bench samples the size every 500 ms for 10 s and keeps the least.
  • Live state is what the accounts hold, as "Bounds" defines it. It's the sum over those accounts of their getRepo bytes, the blobs listBlobs lists for them (getBlob's bytes), and the uploads no record references. Those are uploadBlob's, which each server holds for the blob TTL (an hour) and listBlobs doesn't list. The bench reads it once the at-rest samples are taken. The table gives at rest ÷ live state.

results.json keeps each implementation's storage rounds under its storage (rounds, in bytes, with the store's under beside, and window or noWindow). The window, quiet and sampling go under the top-level storage.

Fairness rules

  • One server runs at a time. Nothing of one is running while another is measured.

  • Every server runs in a container in the same Docker. The client reaches each through the same port forwarding, each writes to the same kind of volume, and each reaches the PLC directory the same way. On macOS that means Docker Desktop's Linux VM for all of them, and a sync is the VM's.

  • Each runs on its own defaults for a single-node local setup, changing only what a local benchmark needs: rate limits off, no invite codes, the bench's PLC directory, a loopback URL, and nothing sent off the machine.

    • vlpds runs as its docs' single-node tiny profile.
    • zds runs as its deployment docs run it (database and blobs under /data). It has no general rate limits to turn off, so the bench raises the one limit the workload hits, 4 full getRepo exports at once, to the 8 the workload runs.
    • tranquil runs as its container docs run it, on PostgreSQL at its defaults, with the same limit raised the same way. Two more of its gates are off. One holds an account's logins and record writes until it verifies its email (no mail leaves the machine to verify it). The other keeps it from fetching private addresses, which is where the bench's PLC directory is.

    Each adapter's info() records its exact configuration and durability. results.json keeps it, and RESULTS.md lists them.

  • Every server gets the same workload, warm-up and number of rounds (three).

  • A part a server can't serve isn't dropped. Its table says why.

  • Only the same work is set side by side. A route's rows are that route's for every server, and nothing stands in for a route a server doesn't serve.

  • The bench measures the client's ceiling and marks any rate near it.

  • It doesn't run on a busy machine unless told to, and it records the load either way.

Adding an implementation

  1. Pin it in impls.json under implementations. Give it one of these:

    • an image by digest (name@sha256:…) with its release and commit
    • a repo and commit, with the dockerfile in it to build and the image tag to give it (setup clones it into bench/vendor/<name>)
    • a dockerfile and context in this repository, and the tag

    Also give needs, the images it uses beside it, each under images by digest.

  2. Write its adapter in ADAPTERS in harness/bench.mjs. It has these fields:

    • comm, the name of its process in its container
    • volume, the directory in its container that holds everything it keeps
    • reader, an image with a shell to read that volume from, if its own image has none
    • setup({ port, plcUrl, dataDir, log, backfillMs }), which makes its container (and whatever runs beside it) before the start clock and answers { name, teardown }. It adds beside: { name, rssKiB, diskBytes } when what runs beside it has its own memory and data to report.
    • window, how backfillMs (given only in a storage round) sets its backfill window, or noWindow, why nothing does
    • start({ port, aux }), answering { url, exited, stop }
    • info() (version, commit, source, storage, durability, configuration)
    • optionally accountBody(i, tag) (handles under its domain, invite codes)
    • unsupported, each part it can't serve, with why
  3. Run node bench.mjs --setup, then node bench.mjs --impls <name> --rounds 1 to try it, then the full run.

Results depend on the machine

Absolute numbers depend on the CPU, the disk and its sync behaviour, the OS, Docker's VM and whatever else is running. RESULTS.md records the machine, the load on it, every pinned version, commit and digest it ran, and the caveats. Only compare runs from the same machine.