# Compatibility with the reference PDS `harness/compat.mjs` checks how closely a PDS behaves like the reference (`@atproto/pds`, the version `harness/package-lock.json` pins), area by area. It runs the core of `harness/compare.mjs`'s checks against any implementation that has an adapter in `harness/lib/adapters/`. It sends the same random requests to the server and to a reference running beside it, and compares their answers the way `compare.mjs` does. Unlike `compare.mjs`, it keeps going past a difference. Each area records which of its checks agree and which differ, plus the first difference in full. [`RESULTS.md`](RESULTS.md) is a run against Monolith, zds, vlpds, pds.js and tranquil. ```sh just compat # from the repository's root: everything below, in one ``` Or run it by hand: ```sh cd harness node bench.mjs --setup # once: Monolith's binary, and the other implementations' images (bench/README.md) node compat.mjs --steps 200 --json ../compat/results.json > ../compat/tables.md node compat.mjs --report ../compat/results.json # the tables again ``` `--impls monolith,zds` and `--areas oauth,records` run only some implementations or areas. `--seed` draws other requests, and `--jobs` sets how many runs go at once (3). `just compat` passes these flags on (`just compat --impls zds --areas oauth`), and a flag you give there wins over the recipe's. The tables go to standard output, and `RESULTS.md` holds them along with what they show. Monolith's own gates are separate. `just gates` runs `compare.mjs` with every check, and it stops at the first difference. ## What is checked: the map `compare.mjs`'s checks fall into three kinds. **Core** checks compare the server with the reference, or hold both to the same rule, and they run against every implementation. **Not yet** checks compare with the reference too, but in areas the first cut leaves out (email, invites, rate limits, the admin lifecycle, proxying). **Monolith's own** checks hold Monolith to its own pages where the reference can't say, or to the reference's implementation rather than the protocol. They run only against Monolith, in its gates. | check (compare.mjs step) | module | kind | area | |---|---|---|---| | `checkOAuthMetadata` | oauth | core (but see below) | oauth | | `oauthLogin`, `oauthParRefused`, `oauthDpopRefused`, `oauthTokenRefused`, `oauthRevokeRefused`, `oauthRefresh`, `oauthRevoke`, `oauthGetSession`, `oauthClientLibrary` | oauth | core | oauth | | `oauthScopedCall`, `oauthScopedRefused`, `oauthScopedEdges` | oauth-scopes | core | oauth scopes | | `oauthSetPar`, `oauthSetLogin`, `oauthSetCall` (the sets are published on a third PDS, which is the lexicon authority each server is given) | permission-sets | core, where an implementation can be given a lexicon authority | oauth permission sets | | `oauthClientDocument`, `oauthClientCache`, `oauthClientLogin`, `oauthAssertionRefused`, `oauthConfidentialSession` | oauth-clients | core | oauth clients | | `oauthAuthorizeRefused` (the page, its API and its redirect endpoint, each with one thing wrong) | oauth | Monolith's own (the reference's page API) | | | `oauthDeviceCookies` (device cookies, renewed at sign-in, with 30 s of grace) | oauth | Monolith's own (the reference's page API) | | | `oauthSilentSignIn` (a remembered device is sent back at once) | oauth-clients | Monolith's own (the reference's page API) | | | `oauthDepartures`, `oauthClientDepartures` (spec/OAUTH.md's and spec/OAUTH-CLIENTS.md's departures, held to the spec) | oauth, oauth-clients | Monolith's own | | | `oauthAccountAuth`, `oauthAccountSessions`, `oauthAccountDevices`, and with `--commands`, `oauthAccountCommands` (spec/OAUTH-ACCOUNT.md's routes, which the reference doesn't serve) | oauth-account | Monolith's own | | | `passkeyAuth`, `passkeyAdding`, `passkeySigningIn`, `passkeySignInRefused`, `passkeyRemoving`, `passkeyStates`, and with `--commands`, `passkeyCommands` (spec/PASSKEYS.md's passkeys, which the reference doesn't have) | passkeys | Monolith's own | | | `createSession`, `getSession`, `describeServer`, `refreshSession`, `deleteSession`, `wrongTokens`, `appPasswords`, `sessionRace` | accounts, session, sessions | core | sessions | | `createAccount`, `createAccountRefused`, `accountRace` | accounts | core | accounts | | `createRecord`, `putRecord`, `deleteRecord`, `applyWrites`, `getRecord`, `listRecords`, `describeRepo`, `createWithTid`, `concurrent`, `oddRequests`, `crossWrites` | records, odd, accounts | core | records | | `validatedWrite`, `duplicates`, `validatedBatch`, `validatedElsewhere` | validation | core | validation | | `checkRepos` (getRepo verified by the reference's library, the MST root, `since`, getLatestCommit) | repository | core | repository | | `listRepos`, `getRepoStatus`, `getBlocks`, `getGoneBlocks`, `syncGetRecord`, `noCrawlsYet`, `checkCrawls` | federation | core | sync | | `--firehose`: `startFirehose`, `checkFirehose`, `checkReplay` | firehose | core | firehose | | `updateHandle`, `updateHandleRefused`, `updateHandleNoToken`, `resolveHandle`, `wellKnownHandle`, `recommendedCredentials`, `handleRace`, `describeRepo` | identity, records | core | identity | | `uploadBlob`, `blobRecord`, `getBlob`, `listBlobs` | blobs | core | blobs | | `requestEmailConfirmation`, `confirmEmail`, `requestEmailUpdate`, `updateEmail`, `emailFactor`, `passwordReset`, `appPasswordEmail` | email | not yet (email) | | | `inviteCodes`, `--invites` | limits | not yet (invites) | | | `bigBatch`, `bodyCaps`, `appPasswordScopes`, `--limits` | limits, oauth-limits | not yet (limits, though their exact numbers are Monolith's own) | | | `loginTiming`, `oauthSignInTiming` | limits | Monolith's own (timing floors) | | | `--lifecycle` | lifecycle | not yet (the admin lifecycle) | | | `forwarded`, `refused`, `serviceAuth`, `chatCase`, `preferences`, `readAfterWrite`, `foreignRecord`, `feedAndPush` | proxy | not yet (proxying) | | | `--commands`: `checkCommands`, `withoutAccount`, calls through MCP tools and CLI commands | commands, via | Monolith's own (spec/COMMANDS.md) | | | `outage`, `checkOperations` (logs, traces, metrics, health, lifecycle restarts) | operations, logging | Monolith's own (spec/OPERATIONS.md) | | | `--store`, `--scale` | store, scale | Monolith's own (spec/STORE.md) | | | `--crash-every`: `crash` (a server killed and started again on its data) | crashes | core, but not in this first cut. It needs the server killed and restarted, which the adapters can do (`server.kill`, `server.start`), and it's left for later | | Some parts of core checks are Monolith's own, and they hold only Monolith (`cand.adapter.own` in the code): | part | where | why | |---|---|---| | `describeServer`'s `spacesAlpha: true` | checks/session.mjs | spec/SPACES.md, not the reference | | the `atproto_kem` key required in the genesis operation and in getRecommendedDidCredentials | checks/accounts.mjs, checks/identity.mjs | spec/SPACES.md (another implementation may have one, see below) | | a blob at exactly the 5 MiB limit and one byte past it, plus `describeServer`'s `blobUploadLimit` | checks/blobs.mjs | the exact number of a limit | | PAR bodies of exactly 100 KiB and one byte past it | checks/oauth.mjs | the exact number of a limit | | OAuth access tokens' `expires_in` of 899 to 900 | checks/oauth.mjs | spec/OAUTH.md's departure (below) | | the authorization server metadata's `request_parameter_supported: false` | checks/oauth.mjs | spec/OAUTH.md, held to the spec | | a `GET` of the page API's `sign-in`, and a `POST` to an endpoint it doesn't have | checks/oauth.mjs | the reference's page API | Each area runs its own steps on a fresh pair of servers, with two or three accounts made first. It draws the steps by weight, the way `compare.mjs` draws its own. The repository, sync and firehose areas need writes to have something to check. So `createRecord`, `putRecord`, `deleteRecord` and `applyWrites` are drawn among their steps as supporting steps (the firehose also draws `createAccount` and `updateHandle`), and making the first accounts is a supporting step in every area. A supporting step's difference is reported separately and isn't counted in the area. The repository's `checkRepos` and the firehose's checks run every `--check-every` steps (10) and at the end. Sync's crawl checks run with the first accounts, every 10 steps and at the end. A server that's built to host one account uses that account in every area ([`spec/COMPAT.md`](../spec/COMPAT.md), "One account"). Its adapter's `claim` makes the account, and the reference makes its own beside it with createAccount, both with the password the server was started with. The checks that make an account (the accounts area, the firehose's `createAccount`) or need a second one (`crossWrites`, `handleRace`, and updateHandle's refusal of a handle another account has) don't run. Every other check runs on the one account. A check is one step. It keeps going past a difference (in a compatibility run, `differ` in `lib/report.mjs` keeps the difference instead of ending the run), and it counts as failed if anything in it differed. A check that throws (on an answer of a shape it can't read) or takes more than 3 minutes counts as one that couldn't go on. For each check, an area records how many ran and failed, each thing that differed and how often, and its first difference. That's the request and both answers, printed the way `compare.mjs` prints them. ## The adapters `harness/lib/adapters/` has one adapter per implementation, each a module named in its `index.mjs`. `common.mjs` holds what they share (the server as a process or a container, a rotation key, a browser's way through a form). `compare.mjs --impl ` picks one (Monolith by default). An adapter says how to make the server for the run's setup, and what that server is: - `start(o)` makes the server for the setup `o` (port, data directory, PLC directory, the reference's recovery key, crawlers, AppView, the web of OAuth client documents, rate limits and invites on or off). It answers with the process the harness starts, kills and starts again, the URL the harness calls, the URL the server's DID documents name, and its OAuth issuer if that's a different one. It also gives the rotation keys it lists in each DID and its own key as a signer, its admin's `Authorization`, and `takeMail()`, which gives the emails it sent. A server that hosts one account also answers with its `password` and `claim`, which makes that account. - `unsupported` lists each setting it can't be given, with the reason. The checks that depend on one are skipped, and the report says why. - `pageApi`, or else `signIn` (below). - `own` says whether Monolith's own rules hold it. Only Monolith's does. The checks don't read any of this. When a check needs something from the server beyond what the protocol says, it asks `harness/lib/server.mjs`. That offers `own()` (whether Monolith's own rules hold it), `can(setting)` (whether it can be given a setting `unsupported` names), `pageApi()` and `issuer(side)`. For Monolith, each one answers so the check does exactly what it does without asking. Anything only a compatibility run needs, such as making a one-account server's account, lives in `harness/lib/areas.mjs` (the compatibility run itself) instead of `checks/`. **Monolith**'s adapter runs it the way the harness always has. It runs the command after `--` with the `PDS_*` environment that `compare.mjs`'s header lists, reads its mail from `PDS_DEV_MAIL_DIR`, and kills and restarts it as a process. **zds**, **vlpds**, **pds.js** and **tranquil** run in Docker, from the images `bench.mjs --setup` builds and pulls (pinned in `bench/impls.json`). They're configured the way the benchmark configures them (`harness/lib/impls.mjs`, shared with `bench.mjs`), plus whatever a compatibility run needs. pds.js, which the benchmark doesn't run, is configured the way its container runs by default. The harness reaches each one on a loopback port, and each one reaches the harness's PLC directory, AppView and crawlers through `host.docker.internal`. The server is the container. The harness starts it, kills it with a signal, and follows its log with `docker logs`. **pds.js** hosts one account. Its adapter starts it with a fresh account password (`PDS_PASSWORD`, which it requires) and claims the account the way its onboarding page does. That's a same-origin `POST /account/onboarding/create` with that password, the handle and the email, then createSession by the new DID. pds.js names its endpoint `https://` in the DID documents it makes, and its OAuth issuer is the URL it's called at. So the adapter gives `https://localhost` as its public URL and the loopback URL as its issuer, and each is taken out of answers the way a server's URL is. **tranquil** runs on a fresh PostgreSQL of its own, on a Docker network with it, as the benchmark runs it. It names itself `https://` everywhere: its OAuth issuer, the endpoint in its DID documents, and the `htu` its DPoP proofs must name. So the adapter gives it the hostname `localhost` and `https://localhost` as its public URL, which is its issuer too, while the harness reaches it on a loopback port: the checks sign each proof's `htu` with the issuer and send the request to the port. Its one operator key, `PLC_ROTATION_KEY`, is a public `did:key`, listed first in each DID it makes, so it takes the reference's recovery key; its other rotation key is each account's own signing key, which it holds and signs with, so the harness has none of its own. | setting | Monolith | zds | vlpds | pds.js | tranquil | |---|---|---|---|---|---| | PLC directory | `PDS_PLC_URL` | `ZDS_PLC_DIRECTORY` | `VLPDS_PLC_URL`, `VLPDS_PLC_MODE=directory` | `PLC_URL` | `PLC_DIRECTORY_URL` | | rotation key, the harness's | `PDS_PLC_ROTATION_KEY_HEX` | `ZDS_PLC_ROTATION_KEY` | `VLPDS_PLC_ROTATION_KEY_K256_PRIVATE_KEY_HEX` | **no**: only the account's signing key, which the server holds | **no**: each account's own signing key, which the server holds | | the reference's recovery key | `PDS_RECOVERY_DID_KEY` | `ZDS_RECOVERY_DID_KEY` | `VLPDS_PLC_RECOVERY_DID_KEY` | **no**: the same | `PLC_ROTATION_KEY`, its one operator key | | server DID `did:web:localhost` | `PDS_HOSTNAME` | `ZDS_SERVER_DID` | `VLPDS_SERVICE_DID` | `PDS_HOSTNAME` until it has its account, then that account's DID | `PDS_HOSTNAME` | | hostname `localhost` (requestCrawl's) | `PDS_HOSTNAME` | **no**: it comes from `ZDS_PUBLIC_URL`, which must be the URL the harness calls (DPoP's `htu`) | **no**: the same, and it doesn't ask a relay on another host to crawl when its own host is local | **no**: it tells one relay (`RELAY_URL`) at most every 20 minutes | **no**: with the hostname `localhost` it asks no crawler to crawl | | handle domains `.test`, `.example` | `PDS_HANDLE_DOMAINS` | `ZDS_HANDLE_DOMAINS` | **`.test` only**: `VLPDS_HANDLE_DOMAIN=test` (another domain needs two labels) | **no**: its hostname is its only one | `PDS_USER_HANDLE_DOMAINS=test,example`, which it names with no leading dot | | accounts, more than one | yes | yes | yes | **no**: one, made by `claim` | yes | | admin password `admin-pass` | `PDS_ADMIN_PASSWORD` | `ZDS_ADMIN_TOKEN` | **no**: its admin token must be 32 bytes or more | **no**: it has no admin methods | **no**: an admin is an account marked so in its database | | invites off (or on) | `PDS_INVITE_REQUIRED` | `ZDS_INVITE_REQUIRED` | `VLPDS_INVITE_REQUIRED` | off, since it has none | `INVITE_CODE_REQUIRED` | | rate limits off (or on) | `PDS_RATE_LIMITS_ENABLED` | **no** general limits to turn on, and full getRepo exports are raised to 8 at once | `VLPDS_NO_RATE_LIMITS` | **no** limits to turn on | `DISABLE_RATE_LIMITING` | | the harness's AppView | `PDS_BSKY_APP_VIEW_*` | `ZDS_PROXY_SERVICE_URL`, `_DID` | `VLPDS_APPVIEW` | `APPVIEW_URL`, `APPVIEW_DID` | **no** setting: it proxies by the `atproto-proxy` header, resolving the DID that names | | the harness's crawlers | `PDS_CRAWLERS` | `ZDS_CRAWLERS` | not given (see hostname) | not given (see hostname) | not given (see hostname) | | OAuth client documents from the harness's web | `PDS_OAUTH_CLIENT_FETCH` | **no**: it fetches the client_id's own https URL | **no**: the same (http only with `--dev-mode`) | **no**: it fetches the client_id's own URL | **no**: it fetches the client_id's own URL | | no SSRF guard, the way the reference runs | `PDS_DISABLE_SSRF_PROTECTION` | `ZDS_ALLOW_PRIVATE_FETCH` | its default for operator-set URLs | its default | `ALLOW_PRIVATE_FETCH` | | mail | `PDS_DEV_MAIL_DIR` | `ZDS_MAIL_PROVIDER=outbox`, read with `dev.zat.admin.listMail` | **no**: it's readable only with `--dev-mode` | **no**: its container has no transport | **no**: it sends only to an SMTP relay and keeps none to read back | | reached at the address it names | yes | yes | yes | yes | **no**: it names itself `https://localhost`, reached on a loopback port | | consent to a narrower scope | the page API | **no**: there's no consent step | **no**: its form only takes scopes away | **no**: its form approves or denies the scope asked for | **no**: its consent takes the scopes approved, but the hook carries none (below) | Each "no" skips these checks: - the hostname: sync's crawl checks - the handle domains: `describeServer` - client documents: the whole oauth clients area - a narrower scope: the narrowing in `oauthLogin` (the scope asked for is granted) - more than one account: those above - the harness's rotation key: its own PLC operation before an updateHandle - the recovery key: `recommendedCredentials` - the address it names: `oauthClientLibrary`, as the atproto client library refuses an https address on a loopback host No core check needs mail, the admin password or rate limits. ## The OAuth sign-in hook The harness signs in on the reference's authorization page through its page API (`/@atproto/oauth-provider/~api/sign-in`, `consent`, `reject`, with its CSRF cookie and device cookies). Monolith serves that API too, so with Monolith every step of a sign-in is compared. No other server serves it, and the protocol leaves the sign-in step to each server. So an adapter without `pageApi` has a hook: ```js signIn(account, authorizeUrl, consent) // account: {identifier, password}; authorizeUrl: the page a PAR's request_uri names; // consent: {approve}. Answers the answer that sends the browser back to the client, // with its code: a redirect, or a form_post page. ``` **zds**'s hook posts its password form (`POST /oauth/authorize`: `request_uri`, `username`, `password`, or `deny`), and zds grants at once with a 303 to the client. **vlpds**'s hook loads the page (its device cookie and `csrf` field), posts `/oauth/authorize/sign-in` (`identifier`, `password`), then posts `/oauth/authorize/consent` (`did`, `action=allow` or `deny`). That redirects to the client or answers with a `form_post` page. **pds.js**'s hook posts its consent form (`POST /oauth/authorize`: `request_uri`, `client_id`, `password`, `action=approve` or `deny`). The form doesn't ask for an identifier, since the server has one account, and it redirects to the client with a 302. **tranquil**'s page is a script calling its JSON endpoints, and its hook calls them as the script does: it posts `request_uri`, `username` and `password` to `/oauth/authorize`, which answers the next page as `redirect_uri`. That's the consent page, whose scopes `/oauth/authorize/consent` answers and then takes back approved, or at once tranquil's `/oauth/authorize/redirect`, which redirects to the client with a 303. To refuse, it posts `request_uri` to `/oauth/authorize/deny`, which answers the client's URL with `error=access_denied` as JSON, and the script navigates there; the hook answers that as a 303 there. Its consent can approve fewer scopes than were asked, but the hook is given no scope to approve, so it approves the scope asked. None of them needs a browser. When an adapter has a hook, the reference still signs in through its page API, but its page's answers are its own and aren't compared. The server signs in through its hook. What's compared is the protocol around the sign-in: PAR, the redirect to the client (its target, `code`, `state` and `iss`, or `error`), the token exchange with PKCE, DPoP and its nonces, refresh, revocation, getSession, every scoped XRPC call with the session's tokens, and the client documents' rules. `oauthClientLibrary` signs in through the hook too, and compares only the pages' redirect status. ## What other implementations are let have [`spec/COMPAT.md`](../spec/COMPAT.md) lists what they're allowed, section by section, with the reason for each: - the same things are taken out of every answer as for Monolith - the KEM key is optional (held to Monolith's form if it's published) - an OAuth access token may last 15 or 60 minutes - only three of `describeServer`'s fields are compared - the exact numbers of limits aren't checked - sign-in happens on the server's own page, and it's compared from the redirect on - Monolith's own departures aren't held against them - a server that hosts one account uses that one - a server with password rules of its own gets each area's first accounts with a password its adapter makes to fit them - a server named by an https address it isn't reached at has the requests for that address sent where it is reached - the settings an adapter can't give have their checks skipped and reported The harness allows nothing else, and Monolith is held to all of `spec/PDS.md`. `spec/PDS.md` and the area pages list "what the reference does that the specs leave open" (the order a write's checks refuse in, the 500s, and the rest). Those aren't allowed to differ. Another implementation that does otherwise differs from the reference, and `RESULTS.md` says where a difference is one of those. ## The vlpds documentation the report cites as deliberate - `listRepos`' order (`DESIGN.md:5905`): "**listRepos order** is `(slot, DID)`, because listings are served per shard, not by creation order." - `getBlob` of an upload that no record names (`DESIGN.md:5906-5907`): "**Fresh uploads are readable.** Blobs are written under their final key at upload, so getBlob serves an upload before any record references it, until the blob GC collects it." Also `docs/blobs.md:65-66`: "Unlike the reference PDS, vlpds serves a blob through `getBlob` as soon as it's uploaded, before any record names it." ## Adding an implementation `harness/checks/` is what Monolith's gates run against the reference. Adding an implementation mustn't change it for Monolith, not by one request and not by one draw from the seeded source. So an implementation brings these, in order of preference: 1. An adapter, plus the settings it can't be given and why (their checks are skipped). 2. A capability in `spec/COMPAT.md`, where it differs from what the harness assumes in a way the protocol allows (one account, its own TLS). The capability is answered in `lib/server.mjs` or handled in `lib/areas.mjs`. It works the same for every implementation that needs it, and leaves Monolith unchanged. 3. Nothing else. A quirk of its own that no protocol rule allows is a difference or a skipped check. A check never gets bent around it. No check names an implementation. A change of the second kind has to show that it leaves Monolith's runs as they were. `compare.mjs --tally ` writes how many requests of each kind a run made, and that's the same for two runs of one seed. So you compare a few seeds' tallies from before and after the change, and they come out the same. Here are the steps: 1. Pin it in `bench/impls.json` and give it a benchmark adapter (`bench/README.md`), unless it can't serve the benchmark's workload (like pds.js, which hosts one account). Put its Docker configuration in `harness/lib/impls.mjs`. 2. Give it an adapter, a module of its own in `harness/lib/adapters/` named in its `index.mjs`. The adapter has `start`, `port`, `info`, `unsupported` with reasons, `signIn` unless it serves the reference's page API, and `readyMs` if it starts slowly. For a server that hosts one account, it also lists `accounts` in `unsupported` and gets `password` and `claim` from `start`. For a server that refuses passwords by a rule of its own, it has `passwordFor(p)` (spec/COMPAT.md, "Password rules"). For a server that names itself by an https origin it isn't reached at, `start` answers that origin as `publicUrl` and `fronted: true` (spec/COMPAT.md, "Its own address"). 3. Add its name to `IMPLS` in `harness/compat.mjs`, and add the differences its documentation says are on purpose to `DELIBERATE` there. 4. Run `node compat.mjs --impls --steps 60` to try it, then do a full run.