~ knotless Search/
↑↓ move↵ open⇧↵ all results
0
0
358 lines · 24.4 KB Raw History
# Compatibility with the reference PDS

`harness/compat.mjs` checks how closely a PDS behaves like the reference
(`@atproto/pds`, the version `harness/package-lock.json` pins), area by
area. It runs the core of `harness/compare.mjs`'s checks against any
implementation that has an adapter in `harness/lib/adapters/`. It sends the
same random requests to the server and to a reference running beside it,
and compares their answers the way `compare.mjs` does. Unlike `compare.mjs`,
it keeps going past a difference. Each area records which of its checks
agree and which differ, plus the first difference in full.
[`RESULTS.md`](RESULTS.md) is a run against Monolith, zds, vlpds, pds.js and
tranquil.

```sh
just compat                                 # from the repository's root: everything below, in one
```

Or run it by hand:

```sh
cd harness
node bench.mjs --setup                      # once: Monolith's binary, and the other implementations' images (bench/README.md)
node compat.mjs --steps 200 --json ../compat/results.json > ../compat/tables.md
node compat.mjs --report ../compat/results.json   # the tables again
```

`--impls monolith,zds` and `--areas oauth,records` run only some
implementations or areas. `--seed` draws other requests, and `--jobs` sets
how many runs go at once (3). `just compat` passes these flags on
(`just compat --impls zds --areas oauth`), and a flag you give there wins
over the recipe's. The tables go to standard output, and `RESULTS.md` holds
them along with what they show.

Monolith's own gates are separate. `just gates` runs `compare.mjs` with
every check, and it stops at the first difference.

## What is checked: the map

`compare.mjs`'s checks fall into three kinds. **Core** checks compare the
server with the reference, or hold both to the same rule, and they run
against every implementation. **Not yet** checks compare with the reference
too, but in areas the first cut leaves out (email, invites, rate limits,
the admin lifecycle, proxying). **Monolith's own** checks hold Monolith to
its own pages where the reference can't say, or to the reference's
implementation rather than the protocol. They run only against Monolith, in
its gates.

| check (compare.mjs step) | module | kind | area |
|---|---|---|---|
| `checkOAuthMetadata` | oauth | core (but see below) | oauth |
| `oauthLogin`, `oauthParRefused`, `oauthDpopRefused`, `oauthTokenRefused`, `oauthRevokeRefused`, `oauthRefresh`, `oauthRevoke`, `oauthGetSession`, `oauthClientLibrary` | oauth | core | oauth |
| `oauthScopedCall`, `oauthScopedRefused`, `oauthScopedEdges` | oauth-scopes | core | oauth scopes |
| `oauthSetPar`, `oauthSetLogin`, `oauthSetCall` (the sets are published on a third PDS, which is the lexicon authority each server is given) | permission-sets | core, where an implementation can be given a lexicon authority | oauth permission sets |
| `oauthClientDocument`, `oauthClientCache`, `oauthClientLogin`, `oauthAssertionRefused`, `oauthConfidentialSession` | oauth-clients | core | oauth clients |
| `oauthAuthorizeRefused` (the page, its API and its redirect endpoint, each with one thing wrong) | oauth | Monolith's own (the reference's page API) | |
| `oauthDeviceCookies` (device cookies, renewed at sign-in, with 30 s of grace) | oauth | Monolith's own (the reference's page API) | |
| `oauthSilentSignIn` (a remembered device is sent back at once) | oauth-clients | Monolith's own (the reference's page API) | |
| `oauthDepartures`, `oauthClientDepartures` (spec/OAUTH.md's and spec/OAUTH-CLIENTS.md's departures, held to the spec) | oauth, oauth-clients | Monolith's own | |
| `oauthAccountAuth`, `oauthAccountSessions`, `oauthAccountDevices`, and with `--commands`, `oauthAccountCommands` (spec/OAUTH-ACCOUNT.md's routes, which the reference doesn't serve) | oauth-account | Monolith's own | |
| `passkeyAuth`, `passkeyAdding`, `passkeySigningIn`, `passkeySignInRefused`, `passkeyRemoving`, `passkeyStates`, and with `--commands`, `passkeyCommands` (spec/PASSKEYS.md's passkeys, which the reference doesn't have) | passkeys | Monolith's own | |
| `createSession`, `getSession`, `describeServer`, `refreshSession`, `deleteSession`, `wrongTokens`, `appPasswords`, `sessionRace` | accounts, session, sessions | core | sessions |
| `createAccount`, `createAccountRefused`, `accountRace` | accounts | core | accounts |
| `createRecord`, `putRecord`, `deleteRecord`, `applyWrites`, `getRecord`, `listRecords`, `describeRepo`, `createWithTid`, `concurrent`, `oddRequests`, `crossWrites` | records, odd, accounts | core | records |
| `validatedWrite`, `duplicates`, `validatedBatch`, `validatedElsewhere` | validation | core | validation |
| `checkRepos` (getRepo verified by the reference's library, the MST root, `since`, getLatestCommit) | repository | core | repository |
| `listRepos`, `getRepoStatus`, `getBlocks`, `getGoneBlocks`, `syncGetRecord`, `noCrawlsYet`, `checkCrawls` | federation | core | sync |
| `--firehose`: `startFirehose`, `checkFirehose`, `checkReplay` | firehose | core | firehose |
| `updateHandle`, `updateHandleRefused`, `updateHandleNoToken`, `resolveHandle`, `wellKnownHandle`, `recommendedCredentials`, `handleRace`, `describeRepo` | identity, records | core | identity |
| `uploadBlob`, `blobRecord`, `getBlob`, `listBlobs` | blobs | core | blobs |
| `requestEmailConfirmation`, `confirmEmail`, `requestEmailUpdate`, `updateEmail`, `emailFactor`, `passwordReset`, `appPasswordEmail` | email | not yet (email) | |
| `inviteCodes`, `--invites` | limits | not yet (invites) | |
| `bigBatch`, `bodyCaps`, `appPasswordScopes`, `--limits` | limits, oauth-limits | not yet (limits, though their exact numbers are Monolith's own) | |
| `loginTiming`, `oauthSignInTiming` | limits | Monolith's own (timing floors) | |
| `--lifecycle` | lifecycle | not yet (the admin lifecycle) | |
| `forwarded`, `refused`, `serviceAuth`, `chatCase`, `preferences`, `readAfterWrite`, `foreignRecord`, `feedAndPush` | proxy | not yet (proxying) | |
| `--commands`: `checkCommands`, `withoutAccount`, calls through MCP tools and CLI commands | commands, via | Monolith's own (spec/COMMANDS.md) | |
| `outage`, `checkOperations` (logs, traces, metrics, health, lifecycle restarts) | operations, logging | Monolith's own (spec/OPERATIONS.md) | |
| `--store`, `--scale` | store, scale | Monolith's own (spec/STORE.md) | |
| `--crash-every`: `crash` (a server killed and started again on its data) | crashes | core, but not in this first cut. It needs the server killed and restarted, which the adapters can do (`server.kill`, `server.start`), and it's left for later | |

Some parts of core checks are Monolith's own, and they hold only Monolith
(`cand.adapter.own` in the code):

| part | where | why |
|---|---|---|
| `describeServer`'s `spacesAlpha: true` | checks/session.mjs | spec/SPACES.md, not the reference |
| the `atproto_kem` key required in the genesis operation and in getRecommendedDidCredentials | checks/accounts.mjs, checks/identity.mjs | spec/SPACES.md (another implementation may have one, see below) |
| a blob at exactly the 5 MiB limit and one byte past it, plus `describeServer`'s `blobUploadLimit` | checks/blobs.mjs | the exact number of a limit |
| PAR bodies of exactly 100 KiB and one byte past it | checks/oauth.mjs | the exact number of a limit |
| OAuth access tokens' `expires_in` of 899 to 900 | checks/oauth.mjs | spec/OAUTH.md's departure (below) |
| the authorization server metadata's `request_parameter_supported: false` | checks/oauth.mjs | spec/OAUTH.md, held to the spec |
| a `GET` of the page API's `sign-in`, and a `POST` to an endpoint it doesn't have | checks/oauth.mjs | the reference's page API |

Each area runs its own steps on a fresh pair of servers, with two or three
accounts made first. It draws the steps by weight, the way `compare.mjs`
draws its own. The repository, sync and firehose areas need writes to have
something to check. So `createRecord`, `putRecord`, `deleteRecord` and
`applyWrites` are drawn among their steps as supporting steps (the firehose
also draws `createAccount` and `updateHandle`), and making the first
accounts is a supporting step in every area. A supporting step's difference
is reported separately and isn't counted in the area. The repository's
`checkRepos` and the firehose's checks run every `--check-every` steps (10)
and at the end. Sync's crawl checks run with the first accounts, every 10
steps and at the end.

A server that's built to host one account uses that account in every area
([`spec/COMPAT.md`](../spec/COMPAT.md), "One account"). Its adapter's
`claim` makes the account, and the reference makes its own beside it with
createAccount, both with the password the server was started with. The
checks that make an account (the accounts area, the firehose's
`createAccount`) or need a second one (`crossWrites`, `handleRace`, and
updateHandle's refusal of a handle another account has) don't run. Every
other check runs on the one account.

A check is one step. It keeps going past a difference (in a compatibility
run, `differ` in `lib/report.mjs` keeps the difference instead of ending
the run), and it counts as failed if anything in it differed. A check that
throws (on an answer of a shape it can't read) or takes more than 3 minutes
counts as one that couldn't go on. For each check, an area records how many
ran and failed, each thing that differed and how often, and its first
difference. That's the request and both answers, printed the way
`compare.mjs` prints them.

## The adapters

`harness/lib/adapters/` has one adapter per implementation, each a module
named in its `index.mjs`. `common.mjs` holds what they share (the server as
a process or a container, a rotation key, a browser's way through a form).
`compare.mjs --impl <name>` picks one (Monolith by default). An adapter says
how to make the server for the run's setup, and what that server is:

- `start(o)` makes the server for the setup `o` (port, data directory, PLC
  directory, the reference's recovery key, crawlers, AppView, the web of
  OAuth client documents, rate limits and invites on or off). It answers
  with the process the harness starts, kills and starts again, the URL the
  harness calls, the URL the server's DID documents name, and its OAuth
  issuer if that's a different one. It also gives the rotation keys it lists
  in each DID and its own key as a signer, its admin's `Authorization`, and
  `takeMail()`, which gives the emails it sent. A server that hosts one
  account also answers with its `password` and `claim`, which makes that
  account.
- `unsupported` lists each setting it can't be given, with the reason. The
  checks that depend on one are skipped, and the report says why.
- `pageApi`, or else `signIn` (below).
- `own` says whether Monolith's own rules hold it. Only Monolith's does.

The checks don't read any of this. When a check needs something from the
server beyond what the protocol says, it asks `harness/lib/server.mjs`.
That offers `own()` (whether Monolith's own rules hold it), `can(setting)`
(whether it can be given a setting `unsupported` names), `pageApi()` and
`issuer(side)`. For Monolith, each one answers so the check does exactly
what it does without asking. Anything only a compatibility run needs, such
as making a one-account server's account, lives in `harness/lib/areas.mjs`
(the compatibility run itself) instead of `checks/`.

**Monolith**'s adapter runs it the way the harness always has. It runs the
command after `--` with the `PDS_*` environment that `compare.mjs`'s header
lists, reads its mail from `PDS_DEV_MAIL_DIR`, and kills and restarts it as
a process.

**zds**, **vlpds**, **pds.js** and **tranquil** run in Docker, from the images
`bench.mjs --setup` builds and pulls (pinned in `bench/impls.json`). They're
configured the way the benchmark configures them (`harness/lib/impls.mjs`,
shared with `bench.mjs`), plus whatever a compatibility run needs. pds.js,
which the benchmark doesn't run, is configured the way its container runs
by default. The harness reaches each one on a loopback port, and each one
reaches the harness's PLC directory, AppView and crawlers through
`host.docker.internal`. The server is the container. The harness starts it,
kills it with a signal, and follows its log with `docker logs`.

**pds.js** hosts one account. Its adapter starts it with a fresh account
password (`PDS_PASSWORD`, which it requires) and claims the account the way
its onboarding page does. That's a same-origin
`POST /account/onboarding/create` with that password, the handle and the
email, then createSession by the new DID. pds.js names its endpoint
`https://<PDS_HOSTNAME>` in the DID documents it makes, and its OAuth issuer
is the URL it's called at. So the adapter gives `https://localhost` as its
public URL and the loopback URL as its issuer, and each is taken out of
answers the way a server's URL is.

**tranquil** runs on a fresh PostgreSQL of its own, on a Docker network
with it, as the benchmark runs it. It names itself `https://<PDS_HOSTNAME>`
everywhere: its OAuth issuer, the endpoint in its DID documents, and the
`htu` its DPoP proofs must name. So the adapter gives it the hostname
`localhost` and `https://localhost` as its public URL, which is its issuer
too, while the harness reaches it on a loopback port: the checks sign each
proof's `htu` with the issuer and send the request to the port. Its one
operator key, `PLC_ROTATION_KEY`, is a public `did:key`, listed first in
each DID it makes, so it takes the reference's recovery key; its other
rotation key is each account's own signing key, which it holds and signs
with, so the harness has none of its own.

| setting | Monolith | zds | vlpds | pds.js | tranquil |
|---|---|---|---|---|---|
| PLC directory | `PDS_PLC_URL` | `ZDS_PLC_DIRECTORY` | `VLPDS_PLC_URL`, `VLPDS_PLC_MODE=directory` | `PLC_URL` | `PLC_DIRECTORY_URL` |
| rotation key, the harness's | `PDS_PLC_ROTATION_KEY_HEX` | `ZDS_PLC_ROTATION_KEY` | `VLPDS_PLC_ROTATION_KEY_K256_PRIVATE_KEY_HEX` | **no**: only the account's signing key, which the server holds | **no**: each account's own signing key, which the server holds |
| the reference's recovery key | `PDS_RECOVERY_DID_KEY` | `ZDS_RECOVERY_DID_KEY` | `VLPDS_PLC_RECOVERY_DID_KEY` | **no**: the same | `PLC_ROTATION_KEY`, its one operator key |
| server DID `did:web:localhost` | `PDS_HOSTNAME` | `ZDS_SERVER_DID` | `VLPDS_SERVICE_DID` | `PDS_HOSTNAME` until it has its account, then that account's DID | `PDS_HOSTNAME` |
| hostname `localhost` (requestCrawl's) | `PDS_HOSTNAME` | **no**: it comes from `ZDS_PUBLIC_URL`, which must be the URL the harness calls (DPoP's `htu`) | **no**: the same, and it doesn't ask a relay on another host to crawl when its own host is local | **no**: it tells one relay (`RELAY_URL`) at most every 20 minutes | **no**: with the hostname `localhost` it asks no crawler to crawl |
| handle domains `.test`, `.example` | `PDS_HANDLE_DOMAINS` | `ZDS_HANDLE_DOMAINS` | **`.test` only**: `VLPDS_HANDLE_DOMAIN=test` (another domain needs two labels) | **no**: its hostname is its only one | `PDS_USER_HANDLE_DOMAINS=test,example`, which it names with no leading dot |
| accounts, more than one | yes | yes | yes | **no**: one, made by `claim` | yes |
| admin password `admin-pass` | `PDS_ADMIN_PASSWORD` | `ZDS_ADMIN_TOKEN` | **no**: its admin token must be 32 bytes or more | **no**: it has no admin methods | **no**: an admin is an account marked so in its database |
| invites off (or on) | `PDS_INVITE_REQUIRED` | `ZDS_INVITE_REQUIRED` | `VLPDS_INVITE_REQUIRED` | off, since it has none | `INVITE_CODE_REQUIRED` |
| rate limits off (or on) | `PDS_RATE_LIMITS_ENABLED` | **no** general limits to turn on, and full getRepo exports are raised to 8 at once | `VLPDS_NO_RATE_LIMITS` | **no** limits to turn on | `DISABLE_RATE_LIMITING` |
| the harness's AppView | `PDS_BSKY_APP_VIEW_*` | `ZDS_PROXY_SERVICE_URL`, `_DID` | `VLPDS_APPVIEW` | `APPVIEW_URL`, `APPVIEW_DID` | **no** setting: it proxies by the `atproto-proxy` header, resolving the DID that names |
| the harness's crawlers | `PDS_CRAWLERS` | `ZDS_CRAWLERS` | not given (see hostname) | not given (see hostname) | not given (see hostname) |
| OAuth client documents from the harness's web | `PDS_OAUTH_CLIENT_FETCH` | **no**: it fetches the client_id's own https URL | **no**: the same (http only with `--dev-mode`) | **no**: it fetches the client_id's own URL | **no**: it fetches the client_id's own URL |
| no SSRF guard, the way the reference runs | `PDS_DISABLE_SSRF_PROTECTION` | `ZDS_ALLOW_PRIVATE_FETCH` | its default for operator-set URLs | its default | `ALLOW_PRIVATE_FETCH` |
| mail | `PDS_DEV_MAIL_DIR` | `ZDS_MAIL_PROVIDER=outbox`, read with `dev.zat.admin.listMail` | **no**: it's readable only with `--dev-mode` | **no**: its container has no transport | **no**: it sends only to an SMTP relay and keeps none to read back |
| reached at the address it names | yes | yes | yes | yes | **no**: it names itself `https://localhost`, reached on a loopback port |
| consent to a narrower scope | the page API | **no**: there's no consent step | **no**: its form only takes scopes away | **no**: its form approves or denies the scope asked for | **no**: its consent takes the scopes approved, but the hook carries none (below) |

Each "no" skips these checks:

- the hostname: sync's crawl checks
- the handle domains: `describeServer`
- client documents: the whole oauth clients area
- a narrower scope: the narrowing in `oauthLogin` (the scope asked for is
  granted)
- more than one account: those above
- the harness's rotation key: its own PLC operation before an updateHandle
- the recovery key: `recommendedCredentials`
- the address it names: `oauthClientLibrary`, as the atproto client library
  refuses an https address on a loopback host

No core check needs mail, the admin password or rate limits.

## The OAuth sign-in hook

The harness signs in on the reference's authorization page through its page
API (`/@atproto/oauth-provider/~api/sign-in`, `consent`, `reject`, with its
CSRF cookie and device cookies). Monolith serves that API too, so with
Monolith every step of a sign-in is compared. No other server serves it, and
the protocol leaves the sign-in step to each server. So an adapter without
`pageApi` has a hook:

```js
signIn(account, authorizeUrl, consent)
// account: {identifier, password}; authorizeUrl: the page a PAR's request_uri names;
// consent: {approve}. Answers the answer that sends the browser back to the client,
// with its code: a redirect, or a form_post page.
```

**zds**'s hook posts its password form (`POST /oauth/authorize`:
`request_uri`, `username`, `password`, or `deny`), and zds grants at once
with a 303 to the client. **vlpds**'s hook loads the page (its device cookie
and `csrf` field), posts `/oauth/authorize/sign-in` (`identifier`,
`password`), then posts `/oauth/authorize/consent` (`did`, `action=allow` or
`deny`). That redirects to the client or answers with a `form_post` page.
**pds.js**'s hook posts its consent form (`POST /oauth/authorize`:
`request_uri`, `client_id`, `password`, `action=approve` or `deny`). The form
doesn't ask for an identifier, since the server has one account, and it
redirects to the client with a 302. **tranquil**'s page is a script calling
its JSON endpoints, and its hook calls them as the script does: it posts
`request_uri`, `username` and `password` to `/oauth/authorize`, which
answers the next page as `redirect_uri`. That's the consent page, whose
scopes `/oauth/authorize/consent` answers and then takes back approved, or
at once tranquil's `/oauth/authorize/redirect`, which redirects to the
client with a 303. To refuse, it posts `request_uri` to
`/oauth/authorize/deny`, which answers the client's URL with
`error=access_denied` as JSON, and the script navigates there; the hook
answers that as a 303 there. Its consent can approve fewer scopes than were
asked, but the hook is given no scope to approve, so it approves the scope
asked. None of them needs a browser.

When an adapter has a hook, the reference still signs in through its page
API, but its page's answers are its own and aren't compared. The server
signs in through its hook. What's compared is the protocol around the
sign-in: PAR, the redirect to the client (its target, `code`, `state` and
`iss`, or `error`), the token exchange with PKCE, DPoP and its nonces,
refresh, revocation, getSession, every scoped XRPC call with the session's
tokens, and the client documents' rules. `oauthClientLibrary` signs in
through the hook too, and compares only the pages' redirect status.

## What other implementations are let have

[`spec/COMPAT.md`](../spec/COMPAT.md) lists what they're allowed, section by
section, with the reason for each:

- the same things are taken out of every answer as for Monolith
- the KEM key is optional (held to Monolith's form if it's published)
- an OAuth access token may last 15 or 60 minutes
- only three of `describeServer`'s fields are compared
- the exact numbers of limits aren't checked
- sign-in happens on the server's own page, and it's compared from the
  redirect on
- Monolith's own departures aren't held against them
- a server that hosts one account uses that one
- a server with password rules of its own gets each area's first accounts
  with a password its adapter makes to fit them
- a server named by an https address it isn't reached at has the requests
  for that address sent where it is reached
- the settings an adapter can't give have their checks skipped and reported

The harness allows nothing else, and Monolith is held to all of
`spec/PDS.md`.

`spec/PDS.md` and the area pages list "what the reference does that the
specs leave open" (the order a write's checks refuse in, the 500s, and the
rest). Those aren't allowed to differ. Another implementation that does
otherwise differs from the reference, and `RESULTS.md` says where a
difference is one of those.

## The vlpds documentation the report cites as deliberate

- `listRepos`' order (`DESIGN.md:5905`): "**listRepos order** is
  `(slot, DID)`, because listings are served per shard, not by creation
  order."
- `getBlob` of an upload that no record names (`DESIGN.md:5906-5907`):
  "**Fresh uploads are readable.** Blobs are written under their final key
  at upload, so getBlob serves an upload before any record references it,
  until the blob GC collects it." Also `docs/blobs.md:65-66`: "Unlike the
  reference PDS, vlpds serves a blob through `getBlob` as soon as it's
  uploaded, before any record names it."

## Adding an implementation

`harness/checks/` is what Monolith's gates run against the reference.
Adding an implementation mustn't change it for Monolith, not by one request
and not by one draw from the seeded source. So an implementation brings
these, in order of preference:

1. An adapter, plus the settings it can't be given and why (their checks
   are skipped).
2. A capability in `spec/COMPAT.md`, where it differs from what the harness
   assumes in a way the protocol allows (one account, its own TLS). The
   capability is answered in `lib/server.mjs` or handled in
   `lib/areas.mjs`. It works the same for every implementation that needs
   it, and leaves Monolith unchanged.
3. Nothing else. A quirk of its own that no protocol rule allows is a
   difference or a skipped check. A check never gets bent around it.

No check names an implementation. A change of the second kind has to show
that it leaves Monolith's runs as they were. `compare.mjs --tally <file>`
writes how many requests of each kind a run made, and that's the same for
two runs of one seed. So you compare a few seeds' tallies from before and
after the change, and they come out the same.

Here are the steps:

1. Pin it in `bench/impls.json` and give it a benchmark adapter
   (`bench/README.md`), unless it can't serve the benchmark's workload
   (like pds.js, which hosts one account). Put its Docker configuration in
   `harness/lib/impls.mjs`.
2. Give it an adapter, a module of its own in `harness/lib/adapters/` named
   in its `index.mjs`. The adapter has `start`, `port`, `info`,
   `unsupported` with reasons, `signIn` unless it serves the reference's
   page API, and `readyMs` if it starts slowly. For a server that hosts one
   account, it also lists `accounts` in `unsupported` and gets `password`
   and `claim` from `start`. For a server that refuses passwords by a rule
   of its own, it has `passwordFor(p)` (spec/COMPAT.md, "Password rules").
   For a server that names itself by an https origin it isn't reached at,
   `start` answers that origin as `publicUrl` and `fronted: true`
   (spec/COMPAT.md, "Its own address").
3. Add its name to `IMPLS` in `harness/compat.mjs`, and add the differences
   its documentation says are on purpose to `DELIBERATE` there.
4. Run `node compat.mjs --impls <name> --steps 60` to try it, then do a full
   run.