~ knotless Search/
↑↓ move↵ open⇧↵ all results
0
0
566 lines · 35.2 KB Raw History
# Benchmark: the reference PDS, Monolith, vlpds, zds and tranquil

`harness/bench.mjs`, run with `just bench` (`node bench.mjs --setup && node
bench.mjs --rounds 3 --json ../bench/results.json`), from a clean state (see
[README.md](README.md)). There are three rounds per implementation, one
implementation at a time, and every one runs in Docker the same way. Each
round is a fresh container on a fresh volume, registering with a fresh
in-memory PLC directory (`@did-plc/server`) run by the bench. Every round
warms the server the same way first: each part and route at a tenth of its
size, on accounts of its own. It then runs the parts, the routes, the curves
by the number of callers, the mix and the subscribers coming and going. A
storage round of each follows. README.md, "What it measures", says what each
of those is. The raw numbers of every round, and every server's recorded
configuration, are in `bench/results.json`.

## Machine and versions

- Apple M1 Pro, 8 cores, 32 GiB, macOS 26.5.2 (arm64), APFS on the internal
  SSD. Docker Desktop 28.0.1, its Linux VM given 8 CPUs and 7.7 GiB. Every
  server, and the client's ceiling responder, runs in that VM.
- The client: Node v24.22.1, 4 worker threads (`harness/lib/loadpool.mjs`).
- The load on the machine was not light (see "Caveats"). The 1-minute load
  average was 3.7 when the run started; at each round's start and end it
  was:
  - Monolith 6 to 13;
  - the reference 9 to 10;
  - vlpds 6 to 29;
  - zds 13 to 16;
  - tranquil 8 to 22.
- Pins, as `bench/impls.json` has them and `results.json` records:
  - **Monolith**: this repository's `Dockerfile` (Rust 1.98 on Debian
    trixie, as deployed), built from `42abe3c`. That is the last commit to
    change what the image is built from, and the commit the run was at.
    Image `sha256:044e78d7…`, linux/arm64.
  - **reference**: `@atproto/pds` 0.5.55 from npm, started through
    `@atproto/dev-env` 0.6.26's `TestPds` as `perf.mjs` does, pinned by
    `harness/package-lock.json`. It runs in the image
    `bench/docker/reference.Dockerfile` builds (Node 24.22.1 on Debian
    trixie), `sha256:fd421a48…`, linux/arm64.
  - **vlpds**: release v1.3.0 of `jazware/vlpds`, commit
    `91fa452993865b2f91ff5ed5b87fc28356796d0c`, its release image
    `ghcr.io/jazware/vlpds@sha256:e1aca7934017bb757ef3717bbd964ae758cf704dc623703262c512fd462c0477`
    (linux/arm64 here).
  - **RustFS** 1.0.4, vlpds's S3 store:
    `rustfs/rustfs@sha256:ed9210d6411d77ffceb70d94b964a3cb7d225983368bc25bebe01e995293dcc2`.
    It passed `vlpds-bucket-probe` (from the vlpds image): all six
    correctness checks, `SAFE for vlpds`.
  - **zds**: release v0.7.8 of `tangled.org/zat.dev/zds`, commit
    `cef2c63d7d6d7dd87b90100566307e9aa8a4d170`, built by `--setup` with its
    own `Dockerfile` (Zig 0.17.0, `-Doptimize=ReleaseSafe`) into
    `monolith-bench-zds:cef2c63`, image `sha256:124944cb…`, linux/arm64. Its
    published image (`atcr.io/zat.dev/zds:latest`) is amd64 only and
    predates v0.7.8, so it was not used.
  - **tranquil**: release v0.8.0 of `tangled.org/tranquil.farm/tranquil-pds`,
    commit `ce17e8a4948aa216c6c1056122966a11b8a693c8`, built by `--setup`
    with its own `Dockerfile` (Rust 1.99.1 on Debian trixie, the binary
    UPX-compressed, on distroless) into `monolith-bench-tranquil:ce17e8a`,
    image `sha256:a6a27312…`, linux/arm64. Its published image
    (`atcr.io/tranquil.farm/tranquil-pds`) needs a login to pull, so it was
    not used.
  - **PostgreSQL** 18.7, tranquil's database:
    `postgres@sha256:1018733457011bf88d81db98cf43aac6d5150afc18787dbe204771fedf80ff95`
    (`postgres:18-alpine`, as tranquil's compose files run it).

### How Monolith was configured

As its image runs it (`docs/guide/deploying.md`): `PDS_PORT=2583`,
`PDS_BIND_ADDRESS=::`, `PDS_DATA_DIR=/data`, with `/data` a fresh volume each
round. The bench adds only what a local benchmark needs:
- `PDS_HOSTNAME=localhost` and `PDS_PUBLIC_URL=http://127.0.0.1:<port>`;
- handle domains `.test,.example`;
- the bench's PLC directory (`http://host.docker.internal:<port>`), with a
  fresh secp256k1 rotation key each round;
- rate limits off and invites not required;
- every private range trusted as a proxy, so that subscribers' addresses in
  `X-Forwarded-For` count.

It is also given an AppView, `PDS_BSKY_APP_VIEW_URL=http://127.0.0.1:9` with
`did:web:api.bsky.app`, as every server on the network has one. Without it,
`getPreferences` is not served (`spec/PROXY.md`). Nothing in the workload is
forwarded there. Its DID cache is as it comes (`PDS_DID_CACHE_SECS`, an hour).
In the storage round only, `PDS_REPO_BACKFILL_LIMIT_MS=10000`.

### How the reference was configured

As `perf.mjs` runs it: dev-env's `TestPds` with `rateLimitsEnabled: false`,
`inviteRequired: false` and `hostname: localhost`, its `dataDirectory` and
`blobstoreDiskLocation` under `/data`, a fresh volume each round
(`harness/bench-reference.mjs`). It listens inside its container on the port
it is reached at, and registers DIDs with the bench's PLC directory through
`host.docker.internal`. In the storage round only, `repoBackfillLimitMs`
10000.

### How vlpds was configured

As its docs run a single node: the release image's `tiny` profile
(`deploy/ansible/roles/vlpds/defaults/main.yml` at v1.3.0, the profile
`docs/operations/deploy.md` describes for one node): a 2560 MiB container
memory limit (`--memory 2560m --memory-swap 2560m`, which sizes its caches),
`--stop-timeout 90`, `VLPDS_NODE_ID=single`, `VLPDS_SHARDS=1`,
`VLPDS_LEASE_TTL_MS=60000`, `VLPDS_SLATEDB_MANIFEST_POLL=60s`, a 4096 MiB SST
disk cache on a Docker volume, and tiny's firehose, backfill and export sizes
(`--firehose-ring-mb 64`, live ring 32, merge queue 32, max lag 64, backfill
cache 32 / read-ahead 16, 4 backfills, 4 exports, MST node cache 64, 1
firehose thread). Threads are its defaults (I/O = the 8 cores the VM shows,
workers = 4). `VLPDS_PLC_MODE=directory` with a fresh secp256k1 rotation key,
and fresh random JWT secret, admin and internal tokens and KEK each round
(not `--dev-mode`). The bench's own changes, as for the others: rate limits
off (`VLPDS_NO_RATE_LIMITS`), invites not required (its default), the bench's
PLC directory (`http://host.docker.internal:<port>`), a loopback URL
(`VLPDS_PUBLIC_URL=http://127.0.0.1:<port>`), and nothing sent off the
machine (`--crawlers=` for no relays, `--asn-lookup off`). Each round runs on
a Docker network of its own with a fresh RustFS container and volume, and the
bucket made through the S3 API; vlpds's container and RustFS's are made before
the start clock and removed, with their volumes and network, after the round.
In the storage round only, `VLPDS_LOG_RETENTION=10s` and
`VLPDS_LOG_RETENTION_INTERVAL=5s`.

### How zds was configured

As its deployment runs it (`docs/deployment.md`, `fly.toml`), on its
Dockerfile's `ENV` (`ZDS_HOST=0.0.0.0`, `ZDS_PORT=2583`,
`ZDS_DB=/data/zds.sqlite3`) with `ZDS_BLOBSTORE_PATH=/data/blobs`, `/data` a
fresh Docker volume each round; a fresh secp256k1 `ZDS_PLC_ROTATION_KEY`,
`ZDS_JWT_SECRET` and `ZDS_ADMIN_TOKEN` each round; no memory limit (its
deployment's 1 GiB is the Fly VM's, not a zds setting). zds has no dev mode.
Its defaults kept: handle domains `.test`, invites not required, permissioned
data off, mail provider `comail` (createAccount sends no mail, and with no
Comail key nothing is sent). The bench's own changes: its PLC directory
(`ZDS_PLC_DIRECTORY=http://host.docker.internal:<port>`), a loopback URL
(`ZDS_PUBLIC_URL=http://127.0.0.1:<port>`, from which it derives
`did:web:127.0.0.1%3A<port>`), no relays asked to crawl (`ZDS_CRAWLERS=`, in
place of bsky.network and vsky.network), and
`ZDS_MAX_CONCURRENT_REPO_EXPORTS=8`. zds has no general rate limits to turn
off; its cap on full `getRepo` exports at once (4 by default, a 429 past it)
is the one limit the workload hits, and it is raised to the 8 the workload
runs at once. Its container is made before the start clock and removed, with
its volume, after the round; zds is its container's PID 1 with no `SIGTERM`
handler, so the bench kills it rather than wait out `docker stop`.

### How tranquil was configured

As its container docs run it (`docs/2_INSTALL_CONTAINERS.md`,
`docker-compose.prod.yaml`): on PostgreSQL, its default repository backend,
with blobs as files under `/var/lib/tranquil-pds/blobs` (a fresh Docker
volume each round) and its in-process cache (ripple, one node); the image's
`SERVER_HOST=[::]`, `SERVER_PORT=3000`; `PDS_HOSTNAME=tranquil.test`, its
handles under that; fresh random `JWT_SECRET`, `DPOP_SECRET` and
`MASTER_KEY` each round. It signs PLC operations with each account's own
key, so it has no rotation key to give. The bench's own changes: rate limits
off (`DISABLE_RATE_LIMITING`), invites not required
(`INVITE_CODE_REQUIRED=false`), its PLC directory
(`PLC_DIRECTORY_URL=http://host.docker.internal:<port>`), private addresses
fetchable so that it reaches that directory (`ALLOW_PRIVATE_FETCH`), no
relays asked to crawl (`CRAWLERS=,`, a list of none), the gate that holds an
account's logins and record writes until it verifies an email off
(`DISABLE_ACCOUNT_VERIFICATION_GATE`), and `MAX_CONCURRENT_REPO_EXPORTS=8`,
raised from 4 as zds's is. With no sender address its email is off, so
createAccount sends no mail. Its passwords need an upper-case letter, so the
bench's accounts' have one. PostgreSQL runs at its defaults
(`max_connections` 100, `shared_buffers` 128 MB, `fsync` and
`synchronous_commit` on), as user `tranquil_pds` and database `pds` with a
fresh password. Each round runs on a Docker network of its own with a fresh
PostgreSQL container and volume, ready for connections before the start
clock; both containers are removed, with their volumes and network, after
the round. tranquil has no setting that makes a backfill window of 10 s, so
its storage round runs on its defaults.

## Results

### Machine and client

- Apple M1 Pro, 8 cores, 32 GiB, macOS 26.5.2 (arm64); Node v24.22.1; Docker 28.0.1, its Linux VM 8 CPUs, 8218316800 bytes.
- Load average (1 min): 3.7 at the start, 13.3 at the end; 6.2 to 28.5 at the rounds' starts and ends.
- The client: 4 worker threads, each with its own connections, kept alive. Its ceiling, a responder in Docker answering every request with 2 bytes and doing nothing else, requests per second (the best of three):

| at once | 1 | 4 | 8 | 16 | 32 | 64 | 256 |
|---|--:|--:|--:|--:|--:|--:|--:|
| per s | 1876 | 5351 | 8093 | 13634 | 18122 | 23098 | 28688 |

### Summary

Each part's requests per second (replay: events sent per second), the mix's, start time and memory, each the best round; for the firehose's live parts, the p50 of their lag, the best round. Detail follows.

| | reference | monolith | vlpds | zds | tranquil |
|---|--:|--:|--:|--:|--:|
| createAccount per s | 33 | 63 | 39 | 6 | 9 |
| createRecord per s | 129 | 3507 | 582 | 445 | 316 |
| applyWrites per s | 71 | 2205 | 303 | 323 | 136 |
| getRecord per s | 1155 | 8258 ‡ | 4343 | 2662 | 2454 |
| listRecords per s | 548 | 4230 | 2531 | 1953 | 1608 |
| getRepo per s | 365 | 2954 | 1191 | 690 | 108 |
| uploadBlob per s | 154 | 1175 | 131 | 405 | 315 |
| createRecordSerial per s | 95 | 576 | 68 | 223 | 55 |
| firehose lag p50 ms | 1.0 | 0.5 | 2.4 | 14.0 | 3.7 |
| firehoseLoaded lag p50 ms | 11.5 | 4.3 | 4.6 | not served | 21.0 |
| replay per s | 11825 | 41549 | 20274 | 53781 | 19302 |
| mix per s | 375 | 4090 | 1331 | 614 ◊ | 736 |
| start ms | 3146 | 180 | 1365 | 399 | 1105 |
| idle RSS MiB (median) | 376.1 | 19.3 | 177.1 | 16.5 | 37.6 |
| peak RSS MiB | 828.4 | 152.4 | 501.2 | 54.5 | 138.2 |

‡: at least 50% of the client's ceiling at the same concurrency (above): the client may be part of what limits it. ◊: the server closed each connection after its answer, the client opened one a request, and it ran out of local ports and waited for them (each closed connection holds its port for a while): the rate is held to how fast they came back, and the times are from when each request was sent.

#### createAccount

40 requests, 4 at once.

| impl | per s (best) | per s (median) | p50 ms (best) | p50 ms (median) | max ms (best) | max ms (median) |
|---|--:|--:|--:|--:|--:|--:|
| reference | 33 | 32 | 107.4 | 118.7 | 162.7 | 171.3 |
| monolith | 63 | 62 | 57.6 | 59.7 | 81.5 | 82.1 |
| vlpds | 39 | 34 | 101.8 | 114.4 | 147.8 | 175.3 |
| zds | 6 | 6 | 560.7 | 708.7 | 1444.9 | 1618.2 |
| tranquil | 9 | 8 | 431.6 | 459.5 | 487.3 | 628.8 |

#### createRecord

1000 requests, 16 at once.

| impl | per s (best) | per s (median) | p50 ms (best) | p50 ms (median) | p99 ms (best) | p99 ms (median) |
|---|--:|--:|--:|--:|--:|--:|
| reference | 129 | 127 | 121.9 | 123.6 | 160.5 | 174.2 |
| monolith | 3507 | 3365 | 3.8 | 3.9 | 20.5 | 20.9 |
| vlpds | 582 | 580 | 21.5 | 24.6 | 50.0 | 55.3 |
| zds | 445 | 440 | 29.6 | 29.9 | 99.0 | 102.0 |
| tranquil | 316 | 303 | 46.2 | 48.2 | 113.7 | 130.6 |

#### applyWrites

200 requests, 8 at once.

| impl | per s (best) | per s (median) | p50 ms (best) | p50 ms (median) | p95 ms (best) | p95 ms (median) |
|---|--:|--:|--:|--:|--:|--:|
| reference | 71 | 70 | 109.3 | 111.2 | 133.4 | 133.4 |
| monolith | 2205 | 1987 | 3.4 | 3.7 | 5.0 | 6.2 |
| vlpds | 303 | 283 | 21.8 | 25.3 | 42.4 | 44.5 |
| zds | 323 | 310 | 21.9 | 22.0 | 47.6 | 50.8 |
| tranquil | 136 | 127 | 55.3 | 57.3 | 81.6 | 90.6 |

#### getRecord

2000 requests, 16 at once.

| impl | per s (best) | per s (median) | p50 ms (best) | p50 ms (median) | p99 ms (best) | p99 ms (median) |
|---|--:|--:|--:|--:|--:|--:|
| reference | 1155 | 1140 | 12.7 | 12.7 | 31.9 | 32.0 |
| monolith | 8258 ‡ | 7860 | 1.6 | 1.7 | 5.9 | 6.8 |
| vlpds | 4343 | 4126 | 2.7 | 2.9 | 19.0 | 20.1 |
| zds | 2662 | 2252 | 4.6 | 5.1 | 21.9 | 27.5 |
| tranquil | 2454 | 2431 | 5.3 | 5.5 | 27.9 | 28.7 |

#### listRecords

1000 requests, 16 at once.

| impl | per s (best) | per s (median) | p50 ms (best) | p50 ms (median) | p99 ms (best) | p99 ms (median) |
|---|--:|--:|--:|--:|--:|--:|
| reference | 548 | 537 | 26.1 | 26.2 | 54.1 | 70.5 |
| monolith | 4230 | 4110 | 3.0 | 3.1 | 16.1 | 17.5 |
| vlpds | 2531 | 2391 | 4.8 | 4.9 | 23.1 | 30.1 |
| zds | 1953 | 1855 | 6.9 | 7.1 | 24.1 | 25.5 |
| tranquil | 1608 | 1406 | 8.7 | 9.8 | 33.2 | 36.3 |

#### getRepo

1000 requests, 8 at once.

| impl | per s (best) | per s (median) | p50 ms (best) | p50 ms (median) | p99 ms (best) | p99 ms (median) |
|---|--:|--:|--:|--:|--:|--:|
| reference | 365 | 361 | 19.4 | 19.9 | 42.5 | 48.3 |
| monolith | 2954 | 2644 | 2.4 | 2.4 | 5.9 | 12.5 |
| vlpds | 1191 | 954 | 4.9 | 6.4 | 28.6 | 28.6 |
| zds | 690 | 637 | 10.0 | 10.3 | 31.6 | 32.5 |
| tranquil | 108 | 104 | 69.4 | 70.0 | 128.8 | 154.4 |

#### uploadBlob

1000 requests, 8 at once, of 64 KiB each; then, not timed, a record of each account referring to its first blob, for getBlob.

| impl | per s (best) | per s (median) | p50 ms (best) | p50 ms (median) | p99 ms (best) | p99 ms (median) |
|---|--:|--:|--:|--:|--:|--:|
| reference | 154 | 152 | 48.1 | 48.9 | 90.4 | 93.7 |
| monolith | 1175 | 1136 | 5.3 | 5.4 | 21.0 | 21.5 |
| vlpds | 131 | 131 | 58.2 | 59.7 | 105.4 | 108.2 |
| zds | 405 | 378 | 16.0 | 17.4 | 53.2 | 57.8 |
| tranquil | 315 | 299 | 22.6 | 22.7 | 54.6 | 58.3 |

#### createRecordSerial

1000 requests, one after another, as one person posting: no other write to share a sync with.

| impl | per s (best) | per s (median) | p50 ms (best) | p50 ms (median) | p99 ms (best) | p99 ms (median) |
|---|--:|--:|--:|--:|--:|--:|
| reference | 95 | 93 | 9.2 | 9.4 | 28.6 | 29.6 |
| monolith | 576 | 572 | 1.4 | 1.5 | 6.0 | 6.4 |
| vlpds | 68 | 67 | 11.5 | 12.2 | 41.7 | 44.2 |
| zds | 223 | 214 | 3.6 | 3.6 | 20.4 | 22.1 |
| tranquil | 55 | 55 | 15.8 | 15.9 | 46.8 | 50.5 |

#### firehose

100 putRecords one after another, 20 subscribers; each time is the firehose's lag, from a write's answer to a subscriber having its commit, 0 when the commit came first. No rate: the writes' would be the write path's, which the write parts measure.

| impl | p50 ms (best) | p50 ms (median) | p99 ms (best) | p99 ms (median) |
|---|--:|--:|--:|--:|
| reference | 1.0 | 1.1 | 6.3 | 15.0 |
| monolith | 0.5 | 0.5 | 2.9 | 3.0 |
| vlpds | 2.4 | 2.6 | 17.9 | 21.2 |
| zds | 14.0 | 15.3 | 62.3 | 66.5 |
| tranquil | 3.7 | 3.8 | 23.6 | 24.9 |

#### firehoseLoaded

400 putRecords, 16 at once, each to a different account, 20 subscribers; each time is the firehose's lag, from a write's answer to a subscriber having its commit, 0 when the commit came first. No rate: the writes' would be the write path's, which the write parts measure.

| impl | p50 ms (best) | p50 ms (median) | p99 ms (best) | p99 ms (median) |
|---|--:|--:|--:|--:|
| reference | 11.5 | 13.0 | 47.1 | 53.1 |
| monolith | 4.3 | 10.2 | 22.9 | 40.0 |
| vlpds | 4.6 | 4.8 | 23.1 | 25.5 |
| zds | not served: its live firehose skips commits when writes come at once: of a warm-up of 40 putRecords, 4 at once to 4 accounts, each of 20 subscribers was sent 34 to 37 (its log: `subscribeRepos end start_cursor=466 end_cursor=506 frames=35`), so no lag can be timed for the rest ||||
| tranquil | 21.0 | 22.6 | 62.3 | 75.0 |

#### replay

4 subscribers from cursor 0, as relays catching up: each is sent every event the server made in the round, warm-up and all, then a write made once all are connected. Throughput is the events sent to them all up to that write's commit, per second; each time is a subscriber's, from asking to having that commit.

| impl | per s (best) | per s (median) | p50 ms (best) | p50 ms (median) | max ms (best) | max ms (median) |
|---|--:|--:|--:|--:|--:|--:|
| reference | 11825 | 11639 | 1079.7 | 1096.9 | 1079.7 | 1097.0 |
| monolith | 41549 | 40703 | 276.7 | 308.1 | 307.3 | 313.7 |
| vlpds | 20274 | 17750 | 606.0 | 693.2 | 611.8 | 701.7 |
| zds | 53781 | 45960 | 197.9 | 237.5 | 204.7 | 239.5 |
| tranquil | 19302 | 15850 | 663.0 | 811.1 | 670.6 | 816.7 |

### Routes

The routes a client calls most, besides the parts above, each measured alone: requests per second, then the p50 and the tail of their times, each the best round. A route a server answered with no 2xx in the warm-up is not served, with its answer.

| route | n, at once | reference per s | p50 ms | tail ms | monolith per s | p50 ms | tail ms | vlpds per s | p50 ms | tail ms | zds per s | p50 ms | tail ms | tranquil per s | p50 ms | tail ms |
|---|---|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|
| describeServer | 1000, 16 | 4285 | 3.1 | 20.0 (p99) | 7939 ‡ | 1.6 | 5.6 (p99) | 4944 | 2.4 | 17.8 (p99) | 3991 | 3.4 | 18.1 (p99) | 4718 | 2.6 | 19.5 (p99) |
| oauthMetadata | 1000, 16 | 3474 | 4.2 | 9.1 (p99) | 9120 ‡ | 1.5 | 4.2 (p99) | 2835 | 4.2 | 21.4 (p99) | 3630 | 3.5 | 11.3 (p99) | 5621 | 2.3 | 6.4 (p99) |
| createSession | 200, 8 | 22 | 355.5 | 362.7 (p95) | 22 | 354.7 | 364.0 (p95) | 72 | 109.3 | 146.8 (p95) | 29 | 234.4 | 426.5 (p95) | 15 | 525.9 | 697.9 (p95) |
| getSession | 1000, 16 | 2214 | 6.7 | 23.5 (p99) | 8260 ‡ | 1.7 | 5.2 (p99) | 4243 | 3.0 | 19.8 (p99) | 1848 | 6.4 | 30.4 (p99) | 2128 | 6.6 | 25.2 (p99) |
| getPreferences | 1000, 16 | 1394 | 10.6 | 27.8 (p99) | 8349 ‡ | 1.6 | 6.3 (p99) | 5067 | 2.6 | 18.4 (p99) | 2741 | 4.5 | 20.2 (p99) | 1992 | 6.8 | 28.1 (p99) |
| getServiceAuth | 1000, 16 | 700 | 20.7 | 44.2 (p99) | 9081 ‡ | 1.6 | 3.5 (p99) | 3754 | 3.0 | 21.6 (p99) | 2530 | 5.4 | 22.4 (p99) | 2232 | 6.0 | 25.4 (p99) |
| resolveHandle | 1000, 16 | 3171 | 4.8 | 18.8 (p99) | 8309 ‡ | 1.5 | 13.8 (p99) | 2944 | 4.1 | 32.4 (p99) | 3323 | 4.0 | 12.3 (p99) | 5548 | 2.1 | 19.6 (p99) |
| describeRepo | 1000, 16 | 1317 | 11.1 | 29.6 (p99) | 8176 ‡ | 1.6 | 6.1 (p99) | 2516 | 5.0 | 28.1 (p99) | 2956 | 4.3 | 20.6 (p99) | 2587 | 5.1 | 36.1 (p99) |
| getLatestCommit | 1000, 16 | 1367 | 10.5 | 26.4 (p99) | 9889 ‡ | 1.5 | 3.0 (p99) | 3940 | 2.9 | 21.4 (p99) | 3663 | 3.5 | 21.3 (p99) | 3257 | 4.1 | 27.2 (p99) |
| syncGetRecord | 1000, 16 | 513 | 27.9 | 63.0 (p99) | 8563 ‡ | 1.7 | 3.7 (p99) | 3005 | 4.1 | 20.4 (p99) | 947 | 14.3 | 43.8 (p99) | 1456 | 9.3 | 32.0 (p99) |
| listRepos | 1000, 16 | 1118 | 12.6 | 29.1 (p99) | 5798 | 2.2 | 10.2 (p99) | 640 | 21.7 | 69.8 (p99) | 1416 | 9.3 | 34.7 (p99) | 328 | 44.3 | 86.1 (p99) |
| getBlob | 1000, 16 | 578 | 24.8 | 52.9 (p99) | 1733 | 8.2 | 25.7 (p99) | 795 | 16.6 | 55.4 (p99) | 1544 | 9.0 | 25.8 (p99) | 1136 | 12.4 | 34.9 (p99) |
| refreshSession | 1000, 16 | 1087 | 11.0 | 44.6 (p99) | 4639 | 2.8 | 16.8 (p99) | 620 | 20.8 | 64.1 (p99) | 368 ◊ | 34.6 | 134.8 (p99) | 671 | 21.3 | 48.7 (p99) |

‡: at least 50% of the client's ceiling at the same concurrency (above): the client may be part of what limits it. ◊: the server closed each connection after its answer, the client opened one a request, and it ran out of local ports and waited for them (each closed connection holds its port for a while): the rate is held to how fast they came back, and the times are from when each request was sent.

### By the number of callers

Each at 1, 4, 16, 64, 256 callers at once: requests answered with a 2xx per second, and the p50 and tail of their times, the best round; and how many were answered otherwise, in the median round, counted rather than stopped at.

#### getRecord

| callers | reference | monolith | vlpds | zds | tranquil |
|--:|--:|--:|--:|--:|--:|
| 1 | 473/s, p50 1.9, p99 6.1 ms | 1309 ‡/s, p50 0.7, p99 1.9 ms | 557/s, p50 1.2, p99 17.1 ms | 518/s, p50 1.6, p99 7.9 ms | 365/s, p50 2.2, p99 17.0 ms |
| 4 | 1120/s, p50 3.1, p99 16.7 ms | 3855 ‡/s, p50 0.9, p99 2.6 ms | 1493/s, p50 1.9, p99 19.2 ms | 1452 ◊/s, p50 2.3, p99 11.4 ms | 1113/s, p50 2.9, p99 20.3 ms |
| 16 | 1250/s, p50 11.6, p99 29.1 ms | 8742 ‡/s, p50 1.5, p99 4.6 ms | 3993/s, p50 3.0, p99 20.6 ms | 238 ◊/s, p50 4.2, p99 18.5 ms | 2592/s, p50 5.2, p99 26.8 ms |
| 64 | 1289/s, p50 42.6, p99 202.0 ms | 15886 ‡/s, p50 3.7, p99 11.5 ms | 7865/s, p50 6.2, p99 28.5 ms | 317 ◊/s, p50 11.7, p99 42.1 ms | 3785/s, p50 14.0, p99 62.1 ms |
| 256 | 1370/s, p50 76.6, p99 4127.2 ms | 18286 ‡/s, p50 8.8, p99 34.0 ms | 12314/s, p50 19.4, p99 40.6 ms | 923 ◊/s, p50 40.6, p99 220.5 ms | 4260/s, p50 51.4, p99 122.2 ms |

#### createRecord

| callers | reference | monolith | vlpds | zds | tranquil |
|--:|--:|--:|--:|--:|--:|
| 1 | 94/s, p50 9.6, p95 17.2 ms | 604/s, p50 1.4, p95 2.6 ms | 71/s, p50 11.3, p95 30.3 ms | 214/s, p50 3.6, p95 9.7 ms | 48/s, p50 18.4, p95 38.9 ms |
| 4 | 112/s, p50 33.3, p95 51.7 ms | 1543/s, p50 2.1, p95 3.8 ms | 164/s, p50 20.5, p95 45.0 ms | 354/s, p50 8.1, p95 28.6 ms | 130/s, p50 26.5, p95 51.4 ms |
| 16 | 115/s, p50 135.5, p95 161.1 ms | 3331/s, p50 4.0, p95 10.5 ms | 596/s, p50 22.4, p95 46.1 ms | 341/s, p50 34.5, p95 97.5 ms | 246/s, p50 60.4, p95 87.7 ms |
| 64 | 119/s, p50 533.1, p95 579.6 ms | 4025/s, p50 11.6, p95 39.5 ms | 1655/s, p50 31.7, p95 56.1 ms | 334/s, p50 111.7, p95 476.9 ms | 292/s, p50 185.6, p95 341.4 ms |
| 256 | 111/s, p50 2185.5, p99 2600.0 ms | 4133/s, p50 48.0, p99 103.7 ms | 3281/s, p50 65.3, p99 135.9 ms | 336/s, p50 523.1, p99 2001.6 ms | 291/s, p50 778.0, p99 1315.1 ms |

### The mix

10000 requests, 32 at once, of each 100 so many of each route, with 2 subscribers to the firehose: requests per second, then each route's p95 in the mix beside its p95 measured alone (the parts' and the routes' tables), the best round, in ms.

| route | of 100 | reference p95 in mix | alone | monolith p95 in mix | alone | vlpds p95 in mix | alone | zds p95 in mix | alone | tranquil p95 in mix | alone |
|---|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|
| all, per s | | 375 | | 4090 | | 1331 | | 614 ◊ | | 736 | |
| getSession | 15 | 54.1 | 12.4 | 12.8 | 3.4 | 16.9 | 7.1 | 97.2 | 18.6 | 22.8 | 13.9 |
| getPreferences | 10 | 67.7 | 17.0 | 13.2 | 3.5 | 13.5 | 5.8 | 81.3 | 11.2 | 26.7 | 13.8 |
| getRecord | 25 | 70.0 | 20.0 | 13.6 | 3.3 | 15.9 | 8.1 | 81.0 | 11.8 | 20.8 | 10.4 |
| listRecords | 15 | 108.3 | 47.3 | 14.2 | 7.2 | 22.3 | 12.8 | 83.3 | 15.4 | 23.9 | 17.1 |
| describeRepo | 5 | 68.7 | 16.9 | 13.5 | 3.5 | 19.4 | 12.5 | 65.5 | 11.4 | 12.1 | 10.6 |
| getBlob | 10 | 249.8 | 45.1 | 19.8 | 17.3 | 22.0 | 42.0 | 77.7 | 19.2 | 17.3 | 28.8 |
| getLatestCommit | 5 | 63.0 | 18.4 | 12.2 | 2.4 | 10.9 | 9.2 | 75.1 | 9.7 | 12.3 | 7.8 |
| createRecord | 10 | 149.0 | 149.5 | 23.5 | 8.3 | 104.2 | 42.9 | 121.1 | 75.6 | 365.5 | 83.8 |
| applyWrites | 3 | 194.4 | 133.4 | 24.1 | 5.0 | 102.1 | 42.4 | 118.8 | 47.6 | 331.4 | 81.6 |
| uploadBlob | 2 | 242.1 | 71.4 | 18.1 | 14.3 | 146.1 | 89.2 | 92.4 | 41.1 | 54.7 | 45.3 |

### Subscribers coming and going

11 cohorts of 20 subscribers, each from an address of its own: a cohort connects, has one write's commit and goes. The server's threads and RSS 2 s after the first cohort went and 2 s after the last, and how many of the subscribers it refused, the median round.

| impl | threads after the first | after the last | RSS MiB after the first | after the last | refused, of all |
|---|--:|--:|--:|--:|--:|
| reference | 11 | 11 | 843.1 | 843.2 | 0 of 220 |
| monolith | 234 | 234 | 121.0 | 118.1 | 0 of 220 |
| vlpds | 30 | 30 | 505.0 | 504.5 | 0 of 220 |
| zds | 44 | 44 | 48.6 | 48.8 | 0 of 220 |
| tranquil | 13 | 13 | 93.4 | 94.1 | 0 of 220 |

### Start time and memory

Start: from `docker start` of the server's container, made already, to its first answered `describeServer`, on an empty volume. Idle RSS: 2 s after start, before any request. Peak RSS: the most sampled (every ~100 ms) through the warm-up and the whole round. Each the server's own process in its container, from `docker top`.

| impl | start ms (best) | start ms (median) | idle RSS MiB (median) | peak RSS MiB (best) | peak RSS MiB (median) |
|---|--:|--:|--:|--:|--:|
| reference | 3146 | 3193 | 376.1 | 828.4 | 843.6 |
| monolith | 180 | 257 | 19.3 | 152.4 | 188.8 |
| vlpds | 1365 | 1394 | 177.1 | 501.2 | 515.2 |
| zds | 399 | 434 | 16.5 | 54.5 | 54.6 |
| tranquil | 1105 | 1164 | 37.6 | 138.2 | 147.0 |

Not in the table above: what a server needs running beside it, its RSS sampled apart (the sum over its container's processes, from `docker top`, as fast as that answers; for PostgreSQL, whose processes share its buffers, the sum of their PSS, which counts each shared page once).

| impl | beside it | idle MiB (median) | peak MiB (best) | peak MiB (median) |
|---|---|--:|--:|--:|
| vlpds | RustFS | 172.0 | 435.4 | 487.4 |
| tranquil | PostgreSQL | 57.1 | 522.7 | 534.5 |

### Storage

From each implementation's storage round, apart from the rounds above: a fresh server started with a backfill window W of 10 s where it has a setting for one, the same warm-up of the parts and the parts, whose speed is not kept (a short window changes what replay sends). What it keeps on disk: the sum of the sizes of the files on its container's volume (its log goes to `docker logs`, not there). Under load: once the parts end, after replay. At rest: after the parts, no request for 2 W + 1 s (21 s), then one createRecord to each account the round made, the warm-up's too, then the size sampled every 500 ms for 10 s, the least kept (spec/STORE.md, "Bounds"). Live state: of each of those accounts, its getRepo, the blobs listBlobs lists and the uploads no record references.

| impl | backfill window | data under load MiB (median) | data at rest MiB (median) | live state MiB (median) | at rest ÷ live (median) |
|---|---|--:|--:|--:|--:|
| reference | W = 10 s | 94.8 | 95.4 | 69.8 | 1.37 |
| monolith | W = 10 s | 102.6 | 70.8 | 69.8 | 1.01 |
| vlpds | W = 10 s | 1.6 | 1.7 | 69.8 | 0.02 |
| zds | its default: no setting | 94.6 | 95.3 | 69.8 | 1.37 |
| tranquil | its default: no setting | 62.5 | 62.5 | 69.9 | 0.89 |

zds: zds has no setting for it: none of the ZDS_* variables its binary reads is a backfill window or an event retention, so at rest is measured with what it keeps by default.

tranquil: its backfill window is FIREHOSE_BACKFILL_HOURS, in whole hours (72 by default), and it prunes the events past EVENT_RETENTION_MAX_AGE_SECS (7 days by default, and never less than that window) once a minute at most: nothing sets a window of W with a pass in the quiet, so at rest is measured with what it keeps by default.

Not in the table above: what a server keeps in what runs beside it (vlpds: its bucket, the files on RustFS's volume; tranquil: its database, the files on PostgreSQL's volume), measured the same way at the same moments; the last column counts the server's own and this together.

| impl | beside it | under load MiB (median) | at rest MiB (median) | both at rest ÷ live (median) |
|---|---|--:|--:|--:|
| vlpds | RustFS | 76.6 | 78.7 | 1.15 |
| tranquil | PostgreSQL | 120.6 | 121.0 | 2.63 |

## Not run

pds.js is pinned, but only the compatibility check runs it: it hosts one
account, and the workload makes 40. Cocoon, millipds and rsky-pds are not
pinned, and tranquil ran only on PostgreSQL, not on `tranquil-store`, its
embedded store. `bench.mjs` takes any of them as an adapter (README.md,
"Adding an implementation").

Every server served every part, route and section, with one exception: zds
did not serve the firehose under load, as its table says.

## Caveats

- **The machine was not idle for the whole run.** The run waited until the
  1-minute load average was at most half the cores (3.7 at its start). Other
  work on the machine then rose: rounds started and ended at load averages
  of 6 to 29, highest while vlpds and tranquil ran. Every server's numbers
  are lower and noisier than an idle machine's would be. Best-of-three is
  reported for that reason, and the medians beside each best show the
  spread.
- **Every server ran in Docker Desktop's Linux VM, behind its port
  forwarding.** All five pay the same forwarding cost and see the same 8
  CPUs, the same volume driver and the same path to the PLC directory. The
  forwarding has a cost of its own, which the client's ceiling measures:
  about 1,900 requests a second one at a time, 13,600 at 16 at once, and
  28,700 at 256 (the first table). A rate marked ‡ is at least half of that,
  and the client and the forwarding may be part of what holds it:
  Monolith's getRecord at 16 at once, its fastest routes, and its getRecord
  curve at every number of callers.
- **Durability: every server's syncs land on the VM's virtual disk.**
  Whether Docker Desktop passes them through to the Mac's drive as a full
  flush was not established. On Linux, Rust's `sync_all`/`sync_data`
  (Monolith), SQLite's `fsync` (the reference, zds), PostgreSQL's WAL flush
  (tranquil) and RustFS's `fsync` (vlpds) are the same kind of call, so the
  five pay the same kind of sync. The write numbers are therefore far higher
  than a native run with macOS's `F_FULLFSYNC`, for Monolith most of all
  (createRecord 3,507 a second here, against 369 natively with
  `F_FULLFSYNC`). They show each server's write path under the same sync,
  not the cost of a sync through a drive's cache. Blobs: the reference
  writes them with `fs.writeFile` and zds with `fwrite`, and neither syncs
  them; Monolith syncs a blob's file when a commit first references it, not
  when it is uploaded (`spec/DURABILITY.md`, "Blobs"); tranquil syncs each
  to a temp file before renaming it into place; vlpds's are PUT to RustFS,
  which syncs them.
- **zds closes every connection after its answer** (`connection: close`),
  so the client opens a connection a request. At a few hundred a second the
  client ran out of local ports, as each closed connection holds its port
  for a while. A request then waited for one and was sent again (README.md,
  "A connection a request"). That is marked ◊: zds's getRecord curve at 4,
  16, 64 and 256 callers, its refreshSession and its mix. Its rate there is
  held to how fast the client's ports came back (238 a second at 16
  callers, against 2,662 in the getRecord part at 16 at once), not to zds.
  A server's many clients each have ports of their own.
- **zds's mix is its single connection.** zds keeps everything in one
  SQLite database on one connection. In the mix, every read's p95 rose to
  65–97 ms beside the writes, from 10–19 ms alone, where Monolith's,
  vlpds's and tranquil's reads stayed within 11–27 ms.
- **The reference's writes do not climb with callers.** Its createRecord
  stays at about 115 a second from 4 callers to 256, with p50 growing from
  33 ms to 2.2 s. Its getRecord stays at about 1,300 a second from 16
  callers on, with p99 4.1 s at 256.
- **vlpds's writes are batched across requests.** Every write on the node
  goes into one log, group-committed into segments that are PUT to the
  bucket. Its createRecord climbs with callers (71 a second at 1, 3,281 at
  256) as more writes share a segment, while one write at a time
  (createRecordSerial, 68 a second) pays a whole PUT.
- **tranquil's writes level off at about 300 a second.** Its createRecord
  is 48 a second at 1 caller and 291 at 256, with p50 growing to 778 ms; in
  the mix its createRecord and applyWrites p95 rose to 331–366 ms, from
  82–84 ms alone, while its reads stayed within 12–27 ms.
- **zds's firehose skips commits under concurrent writes.** Each of the 20
  subscribers of the warm-up of firehoseLoaded (40 putRecords, 4 at once,
  each to a different account) was sent 34 to 37 of the 40 commits, by its
  own log (`subscribeRepos end ... frames=`), though the sequence numbers ran
  on by 40. Written one at a time, every commit reached every subscriber. So
  the part is not served for it: no lag can be timed for a commit not sent.
- **zds's and tranquil's limits raised.** Their default cap of 4 full
  `getRepo` exports at once answers the workload's 8 at once with a
  refusal. It was raised to 8 (`ZDS_MAX_CONCURRENT_REPO_EXPORTS`,
  `MAX_CONCURRENT_REPO_EXPORTS`), as rate limits are off for the others.
  zds's other caps are not reached: 32 subscribeRepos connections (no part
  holds more than 20) and 4 cursor backfills at once (replay's 4).
- **tranquil's gates off.** Its default holds an account's logins and
  record writes until it verifies an email or other channel; with no mail
  leaving the machine, none could, and the other servers have no such gate.
  It is off for the run, as invite codes are for all of them.
- **replay's events are each server's own.** Every server makes its own
  events for the same round, since an account's creation is several events
  and not the same several everywhere. So replay compares events sent per
  second, not the time to send one fixed stream.
- **Storage.** vlpds keeps almost nothing on its own volume (its SST cache,
  1.7 MiB): its data is in the bucket, 79 MiB at rest, counted apart.
  tranquil keeps only its blobs on its own volume (62.5 MiB); its database,
  121 MiB at rest, is counted apart. zds and tranquil have no backfill
  window to set to 10 s, so their at-rest sizes include every event of the
  round. The live state is about the same for all five, 69.8 MiB, nearly all
  of it the uploads no record references, which each server holds for an
  hour.
- **Memory.** Every figure is the server's own process in its container,
  from `docker top`: `pds`, Node's `MainThread`, `vlpds`, `zds`,
  `tranquil-pds`. vlpds runs under a 2,560 MiB container limit, which sizes
  its caches; the others have none. RustFS's memory is on top of vlpds's and
  PostgreSQL's on top of tranquil's, each in a table of its own;
  PostgreSQL's is the sum of its processes' PSS, as they share its buffers.
  The PLC directory runs in the bench's process and is counted for none.
  The reference's figures include the Node runtime and dev-env's module
  graph. tranquil's binary is UPX-compressed, as its Dockerfile builds it,
  and unpacks into its memory at start.
- **createAccount** includes the PLC operation to the bench's directory,
  through `host.docker.internal`, for all five. createSession is about 22 a
  second for Monolith and the reference alike: it is their password hash.
  tranquil's is 15 a second.