~ knotless Search/
↑↓ move↵ open⇧↵ all results
0
0
373 lines · 22.4 KB Raw History
# The PDS benchmark

`harness/bench.mjs` runs the same workload against several PDS implementations,
one at a time, on one machine. It runs every one of them in Docker the same
way, and writes the tables in [`RESULTS.md`](RESULTS.md) (with the raw numbers
in `results.json`). Everything it runs comes from this repository and the pins
in [`impls.json`](impls.json), so nothing on the machine is set up by hand.

## What runs, pinned

| implementation | where it comes from | what pins it |
|---|---|---|
| Monolith | this repository's `Dockerfile`, the image it deploys as ([deploying](../docs/guide/deploying.md)), built by `--setup` as `monolith-bench-monolith:local` | the last commit of what it's built from (its `inputs` in `impls.json`), which the image is labelled with |
| reference | `@atproto/pds` 0.5.55 through `@atproto/dev-env` 0.6.26, from npm. The image [`docker/reference.Dockerfile`](docker/reference.Dockerfile) builds it from the harness's packages as `monolith-bench-reference:local`, and `harness/bench-reference.mjs` starts it | `harness/package-lock.json`, and Node 24.22.1 by its image's tag |
| vlpds | its release image `ghcr.io/jazware/vlpds`, v1.3.0 (commit `91fa452`) | image digest in `impls.json` |
| RustFS (vlpds's S3 store) | `rustfs/rustfs` 1.0.4 | image digest in `impls.json` |
| zds | `tangled.org/zat.dev/zds` v0.7.8, built by `--setup` with its own `Dockerfile` for this machine's architecture and tagged `monolith-bench-zds:cef2c63` | repository URL and commit `cef2c63` in `impls.json` |
| tranquil | `tangled.org/tranquil.farm/tranquil-pds` main at `b95f999` (10 commits past v0.8.0), built by `--setup` with its own `Dockerfile` for this machine's architecture and tagged `monolith-bench-tranquil:b95f999` | repository URL and commit `b95f999` in `impls.json` |
| PostgreSQL (tranquil's database) | `postgres` 18.7 on Alpine, run the way tranquil's compose files run it | image digest in `impls.json` |
| pds.js | `knotless://chadtmiller.com/pds.js` main at `54f4e3a` (35 commits past v2.5.2), built by `--setup` with its own `Dockerfile` and tagged `monolith-bench-pdsjs:54f4e3a`. Only the compatibility check runs it (compat/README.md), since it hosts one account and the workload makes 40 | repository URL and commit `54f4e3a` in `impls.json` |

`--setup` also builds Monolith and the reference outside Docker
(`cargo build --release -p pds`, `npm ci` in `harness/`), because the
compatibility check runs them that way.

zds's published image (`atcr.io/zat.dev/zds:latest`) is amd64 only and
predates v0.7.8, so the bench builds its own from the pinned commit. While it
builds, the Dockerfile fetches Zig 0.17.0 (checked against its SHA-256) and
zds's Zig dependencies (pinned by hash in its `build.zig.zon`).

tranquil's published image (`atcr.io/tranquil.farm/tranquil-pds`) needs a
login to pull, so the bench builds its own from the pinned commit, the way its
docs build from source. While it builds, the Dockerfile fetches its pnpm
packages and crates, plus UPX (checked against its SHA-256), which compresses
the binary by default. tranquil runs on PostgreSQL, its default repository
backend (the other is `tranquil-store`, its embedded one).

vlpds keeps all its data in an S3 bucket and needs strongly consistent
conditional writes (`If-None-Match: *`, `If-Match`) from it. The store is
RustFS. We chose it by running vlpds's own `vlpds-bucket-probe` (in the vlpds
image) against it on a Docker network. All six correctness checks passed
(`conditional_create`, `compare_and_swap`, `race_create`, `race_cas`,
`list_read_delete`, `multipart`), with the verdict `SAFE for vlpds`. RustFS was
the first candidate tried, ahead of Garage and SeaweedFS, so those weren't
probed. MinIO, the store vlpds's docs name for tests, doesn't publish images
any more. To probe it again:

```sh
docker network create probe
docker run -d --name probe-s3 --network probe --network-alias s3 -e RUSTFS_ACCESS_KEY=probeaccess1 -e RUSTFS_SECRET_KEY=probesecret1234567890 <rustfs image from impls.json>
# make the bucket `vlpds` (any S3 client), then:
docker run --rm --network probe -e VLPDS_S3_ENDPOINT=http://s3:9000 -e VLPDS_S3_BUCKET=vlpds -e VLPDS_S3_REGION=us-east-1 \
  -e VLPDS_S3_ACCESS_KEY=probeaccess1 -e VLPDS_S3_SECRET_KEY=probesecret1234567890 \
  --entrypoint vlpds-bucket-probe <vlpds image from impls.json>
```

## Prerequisites

| what | version used for `RESULTS.md` | why |
|---|---|---|
| macOS or Linux, 8+ cores, 16+ GiB | macOS 26.5.2, Apple M1 Pro, 32 GiB | |
| Node | 24 (24.22.1) | runs the bench client and the PLC directory |
| npm | 11 (11.19.1) | installs the harness's packages from `harness/package-lock.json` |
| rustup | 1.28+ (1.29.1) | builds Monolith outside Docker on `stable` for the compatibility check (setup installs the toolchain) |
| Docker | 28 (Docker Desktop 28.0.1, with its Linux VM at 8 CPUs, 7.7 GiB) | runs every server and builds the images of Monolith, the reference, zds, tranquil and pds.js. `host.docker.internal` must reach the host |
| git | any recent (2.50.1) | fetches zds, tranquil and pds.js at their pinned commits, and names the commit Monolith's image is built from |
| git-remote-knotless | any | lets git fetch pds.js's `knotless://` repository |
| sccache | optional (0.19.0) | `export RUSTC_WRAPPER=sccache` before setup makes rebuilds outside Docker faster |

Setup needs the network (crates.io, npm, Docker Hub, ghcr.io, tangled.org,
ziglang.org, github.com). The bench itself talks only to loopback and
Docker's local network.

You run it by hand on a dedicated machine with Docker. It can't run on the
runner, because a job there is a container with no Docker of its own (no
socket, and no room to start a daemon), so none of these servers could
start. The runner's performance step (`harness/perf.mjs`, the `nightly` and
`perf` workflows) compares the server with the reference there, outside
Docker.

## Commands

From a clone of this repository:

```sh
cd harness
node bench.mjs --setup                                     # build and pull everything pinned
node bench.mjs --rounds 3 --json ../bench/results.json     # run; Markdown tables on stdout
```

Or run both at once from the repository's root with `just bench`. Then copy
the tables into `RESULTS.md`. `node bench.mjs --report ../bench/results.json`
prints them again from a saved run. Every table puts the reference first,
since it's the baseline the others are read against.

`--setup` does this:

1. It checks for Node 24+, rustup, npm, git and a running Docker daemon, and
   stops with a message naming whatever's missing.
2. It builds Monolith outside Docker (`npm ci` and `npm run build` in
   `web/`, then `rustup run stable cargo build --release -p pds`) and
   installs the reference (`npm ci` in `harness/`) for the compatibility
   check.
3. It builds the images of Monolith (the repository's `Dockerfile`) and the
   reference (`bench/docker/reference.Dockerfile`, with `harness/` as its
   context) every time. Each is labelled with the last commit that changed
   what it's built from (its `inputs` in `impls.json`), plus `-dirty` if any
   of that has uncommitted changes. Docker's cache leaves nothing to do when
   nothing changed, and Monolith's first build takes some minutes.
4. Unless Docker already has the image `monolith-bench-zds:cef2c63`, it
   fetches zds at its pinned commit into `bench/vendor/zds` (gitignored) and
   builds that image with zds's own `Dockerfile` (`docker build`, a few
   minutes the first time). It does the same for tranquil, into
   `bench/vendor/tranquil` as `monolith-bench-tranquil:b95f999` (a Rust
   workspace, and the longest build of them all). It does the same for
   pds.js, into `bench/vendor/pds.js` as `monolith-bench-pdsjs:54f4e3a`, for
   the compatibility check.
5. It pulls the vlpds, RustFS and PostgreSQL images by digest, unless Docker
   already has them.

It's idempotent. A run refuses to start if an image it needs is missing, or
if Monolith's or the reference's image was built from inputs other than the
ones checked out. `--impls monolith,vlpds` runs only some of them. `--rounds
N` sets the rounds and `--clients N` the client's threads (4). `--storage-rounds
N` sets the storage rounds (1) and `--backfill-ms W` their backfill window
(10000, see "What it measures").

A run also refuses to start on a busy machine. When the load average over
the last minute is more than half the cores, every number would reflect the
other work as much as the server. `--busy` runs anyway. Either way, the
bench records the load at the start and end of each round, and `RESULTS.md`
reports it.

## What it measures

Each round is a fresh server on fresh data, registering accounts with a fresh
in-memory PLC directory (`@did-plc/server`) that the bench runs. Each server
runs in its own container, made with a fresh volume before the start clock
and removed after the round. For vlpds, a round also makes a fresh Docker
network, RustFS container and volume, and puts vlpds's cache on its own
volume. For tranquil, it makes a fresh Docker network and PostgreSQL
container and volume, ready for connections before the start clock.
The bench warms the server first, running each part and route at a tenth of
its size on separate accounts. Then the parts run in order:

| part | requests per round | at once | what it does |
|---|--:|--:|---|
| createAccount | 40 | 4 | no invite codes, and includes the PLC registration |
| createRecord | 1000 | 16 | ~100-byte records over the 40 accounts |
| applyWrites | 200 | 8 | 10 creates per call |
| getRecord | 2000 | 16 | reads records written by createRecord |
| listRecords | 1000 | 16 | `limit=50` |
| getRepo | 1000 | 8 | ~75 records per repo |
| uploadBlob | 1000 | 8 | 64 KiB, `application/octet-stream`. Afterward, untimed, each account writes a record referring to its first blob |
| createRecordSerial | 1000 | 1 | one after another, like one person posting, so there's no other write to share a sync with |
| firehose | 100 writes | 20 subscribers | `putRecord`s one after another. Latency is the firehose's lag, from write answered to commit received |
| firehoseLoaded | 400 writes | 20 subscribers | `putRecord`s 16 at once, each to a different account. Latency is measured as for the firehose |
| replay | | 4 subscribers | starts from cursor 0, like relays catching up, and reads every event of the round, then a write made once all are connected |

For each part, the bench reports throughput (requests answered per second),
p50 and a tail of request times, as the best and the median of the rounds.
The tail is the highest percentile with at least ten times above it. That's
p99 for 1000 times or more, p95 for 200 or more, p90 for 100 or more, and
otherwise the slowest (`max`), which covers createAccount's 40 and replay's 4.

The firehose's live parts have no throughput, since their writes' throughput
would just be the write path's, which the write parts already measure. Their
times are the firehose's lag, from a write's answer to a subscriber having
its commit, and 0 when the commit came first. replay's throughput is the
events sent to its subscribers per second, until each has the commit of the
write made once all are connected. Its times are each subscriber's, from
asking to having that commit. It has 4 subscribers because zds and vlpds
each serve 4 backfills at a time.

**Routes.** Next come the routes a client calls most besides those, each run
by itself:

| route | requests per round | at once | what it does |
|---|--:|--:|---|
| describeServer | 1000 | 16 | |
| oauthMetadata | 1000 | 16 | `GET /.well-known/oauth-authorization-server` |
| createSession | 200 | 8 | logs in with an account's handle and password |
| getSession | 1000 | 16 | |
| getPreferences | 1000 | 16 | `app.bsky.actor.getPreferences` |
| getServiceAuth | 1000 | 16 | `aud=did:web:api.bsky.app`, `lxm=app.bsky.feed.getTimeline` |
| resolveHandle | 1000 | 16 | resolves the accounts' handles |
| describeRepo | 1000 | 16 | |
| getLatestCommit | 1000 | 16 | `com.atproto.sync.getLatestCommit` |
| syncGetRecord | 1000 | 16 | `com.atproto.sync.getRecord`, which returns a record and its proof as a CAR |
| listRepos | 1000 | 16 | `limit=100` |
| getBlob | 1000 | 16 | `com.atproto.sync.getBlob` of the blobs the records refer to |
| refreshSession | 1000 | 16 | each answer's tokens become the account's next ones. Each caller takes its own accounts in turn, so no account's tokens are in two requests at once. It runs last, since the old tokens may be spent |

If a server answers a route with no 2xx in the warm-up, that route isn't
served, and its table gives the answer. Proxied appview routes aren't
measured, because there's no appview in the bench and their speed would be
the appview's.

**By the number of callers.** The bench runs getRecord and createRecord at
1, 4, 16, 64 and 256 callers at once, over the same 40 accounts. getRecord
sends 2000 requests or 40 per caller, whichever is more, and createRecord
500 or 4 per caller. The table shows how many are answered with a 2xx per
second, their p50 and tail, and how many got some other answer. Those are
counted rather than stopped at, so a server that pushes back under load shows
it in the table.

**The mix.** 10000 requests come from 32 callers at once. Out of every 100,
each route gets a fixed share (getSession 15, getPreferences 10, getRecord 25,
listRecords 15, describeRepo 5, getBlob 10, getLatestCommit 5, createRecord
10, applyWrites 3, uploadBlob 2), spread through the 100 as evenly as they
go. Meanwhile 2 subscribers follow the firehose like relays. The table puts
each route's p95 in the mix beside its p95 on its own (every route has 200
requests or more in each). Where a read's p95 grows, it waited on the writes
beside it. A server that doesn't serve one of the routes doesn't run the mix.

**Subscribers coming and going.** 11 cohorts of 20 subscribers connect, each
from its own address. A cohort connects, gets one write's commit, and goes.
The bench records the server's threads and RSS 2 s after the first cohort
left and 2 s after the last, and how many subscribers it refused (each
tries for 10 s). If those grow, the server didn't let go of what a subscriber
held. A server that keeps a subscriber's place after it left refuses the next
cohort.

**The client's ceiling.** Before the servers, the client sends to a
responder in Docker, reached the same way as the servers. The responder
answers every request with the same 2 bytes and does nothing else (Node's
`net`, in the reference's image), and the client runs it at every
concurrency the parts use. A server's rate at half that or more is marked ‡
in the tables. At that point, the client, Docker's port forwarding and the
machine's cores the client takes are part of what limits it.

**A connection a request.** If a server closes each connection after its
answer, the client has to open one for every request. Each closed connection
holds its local port for a while (TIME_WAIT), so at a few hundred requests
a second the client runs out of ports. A request that finds none waits 50 ms
and is sent again, timed from when it's sent. The tables mark that rate ◊,
since it's held to how fast the ports come back. That's a limit of one client
machine, since a server's many clients each have their own ports.

For each server, the bench reports start time (from `docker start` of its
already-made container to the first answered `describeServer`). It also
reports RSS 2 s after start (idle) and the most sampled through the round
(peak), from `docker top` inside the container. That covers only its own
process (`pds`, Node's `MainThread`, `vlpds`, `zds` or `tranquil-pds`).
vlpds's store's RSS is sampled the same way, summed over its container's
processes. tranquil's database is measured as the sum of its processes' PSS,
because they share its buffers and a sum of their RSS would count those in
every process that touched them. Each gets its own table.

For each server the bench also reports what it keeps on disk, from its own
**storage round**, run after its speed rounds (`--storage-rounds N`, 1 by
default, with the median if there are more). That's a fresh server, as in a
speed round, but started with a short backfill window W (`--backfill-ms`,
10 s by default). It's warmed with only the parts, then runs the parts, and
their speed isn't kept. It's separate because a short window changes what
the speed rounds measure. Replay from cursor 0 gets only the events within W
instead of every event of the round, so the speed rounds run each server on
its default window.

The window is Monolith's `PDS_REPO_BACKFILL_LIMIT_MS` and the reference's
`repoBackfillLimitMs` (the same variable). For vlpds it's
`VLPDS_LOG_RETENTION` (its log segment retention, which is its firehose
backfill window), with `VLPDS_LOG_RETENTION_INTERVAL` at W / 2 so that a
retention pass falls within the quiet. zds has no such setting (none of the
`ZDS_*` variables its binary reads is a window or an event retention).
tranquil has none that reaches W. Its window, `FIREHOSE_BACKFILL_HOURS`, is
in whole hours, and it prunes events at most once a minute, never more recent
than that window. So their storage rounds run on what they keep by default,
and the table says so.

What a server keeps is the sum of the sizes of the files on its container's
volume. The bench reads them with `find -type f` and `stat` inside the
container. tranquil's image has no shell, so for it they run in a container
of PostgreSQL's image given its volumes. The volumes are:

- Monolith's `/data`, its data directory
- the reference's `/data`, its `dataDirectory` and `blobstoreDiskLocation`
- zds's `/data`, its database and blobs
- vlpds's `/var/lib/vlpds`, which holds only its SST cache
- tranquil's `/var/lib/tranquil-pds/blobs`, which holds only its blobs

Each server's log goes to `docker logs` instead. vlpds's bucket (the files
on its RustFS container's volume) and tranquil's database (the files on its
PostgreSQL container's `/var/lib/postgresql`) are measured the same way at
the same moments. Like their RSS, they're reported in their own table.

- Under load: once the parts end, after replay.
- At rest, as `spec/STORE.md`, "Bounds", settles it: after the parts, there
  are no requests for 2 W + 1 s, then one `createRecord` to each account the
  round made (the warm-up's too, since they're in the same data directory).
  Then the bench samples the size every 500 ms for 10 s and keeps the least.
- Live state is what the accounts hold, as "Bounds" defines it. It's the sum
  over those accounts of their `getRepo` bytes, the blobs `listBlobs` lists
  for them (`getBlob`'s bytes), and the uploads no record references. Those
  are uploadBlob's, which each server holds for the blob TTL (an hour) and
  `listBlobs` doesn't list. The bench reads it once the at-rest samples are
  taken. The table gives at rest ÷ live state.

`results.json` keeps each implementation's storage rounds under its
`storage` (`rounds`, in bytes, with the store's under `beside`, and `window`
or `noWindow`). The window, quiet and sampling go under the top-level
`storage`.

## Fairness rules

- One server runs at a time. Nothing of one is running while another is
  measured.
- Every server runs in a container in the same Docker. The client reaches
  each through the same port forwarding, each writes to the same kind of
  volume, and each reaches the PLC directory the same way. On macOS that
  means Docker Desktop's Linux VM for all of them, and a sync is the VM's.
- Each runs on its own defaults for a single-node local setup, changing only
  what a local benchmark needs: rate limits off, no invite codes, the bench's
  PLC directory, a loopback URL, and nothing sent off the machine.
  - vlpds runs as its docs' single-node `tiny` profile.
  - zds runs as its deployment docs run it (database and blobs under
    `/data`). It has no general rate limits to turn off, so the bench raises
    the one limit the workload hits, 4 full `getRepo` exports at once, to
    the 8 the workload runs.
  - tranquil runs as its container docs run it, on PostgreSQL at its
    defaults, with the same limit raised the same way. Two more of its gates
    are off. One holds an account's logins and record writes until it
    verifies its email (no mail leaves the machine to verify it). The other
    keeps it from fetching private addresses, which is where the bench's PLC
    directory is.

  Each adapter's `info()` records its exact configuration and durability.
  `results.json` keeps it, and `RESULTS.md` lists them.
- Every server gets the same workload, warm-up and number of rounds (three).
- A part a server can't serve isn't dropped. Its table says why.
- Only the same work is set side by side. A route's rows are that route's for
  every server, and nothing stands in for a route a server doesn't serve.
- The bench measures the client's ceiling and marks any rate near it.
- It doesn't run on a busy machine unless told to, and it records the load
  either way.

## Adding an implementation

1. Pin it in `impls.json` under `implementations`. Give it one of these:
   - an `image` by digest (`name@sha256:…`) with its `release` and `commit`
   - a `repo` and `commit`, with the `dockerfile` in it to build and the
     image `tag` to give it (setup clones it into `bench/vendor/<name>`)
   - a `dockerfile` and `context` in this repository, and the `tag`

   Also give `needs`, the images it uses beside it, each under `images` by
   digest.
2. Write its adapter in `ADAPTERS` in `harness/bench.mjs`. It has these
   fields:
   - `comm`, the name of its process in its container
   - `volume`, the directory in its container that holds everything it keeps
   - `reader`, an image with a shell to read that volume from, if its own
     image has none
   - `setup({ port, plcUrl, dataDir, log, backfillMs })`, which makes its
     container (and whatever runs beside it) before the start clock and
     answers `{ name, teardown }`. It adds `beside: { name, rssKiB,
     diskBytes }` when what runs beside it has its own memory and data to
     report.
   - `window`, how `backfillMs` (given only in a storage round) sets its
     backfill window, or `noWindow`, why nothing does
   - `start({ port, aux })`, answering `{ url, exited, stop }`
   - `info()` (version, commit, source, storage, durability, configuration)
   - optionally `accountBody(i, tag)` (handles under its domain, invite
     codes)
   - `unsupported`, each part it can't serve, with why
3. Run `node bench.mjs --setup`, then `node bench.mjs --impls <name> --rounds 1`
   to try it, then the full run.

## Results depend on the machine

Absolute numbers depend on the CPU, the disk and its sync behaviour, the OS,
Docker's VM and whatever else is running. `RESULTS.md` records the machine,
the load on it, every pinned version, commit and digest it ran, and the
caveats. Only compare runs from the same machine.