chadtmiller.com/monolith
An atproto PDS
Compatibility with the reference PDS
harness/compat.mjs checks how closely a PDS behaves like the reference
(@atproto/pds, the version harness/package-lock.json pins), area by
area. It runs the core of harness/compare.mjs's checks against any
implementation that has an adapter in harness/lib/adapters/. It sends the
same random requests to the server and to a reference running beside it,
and compares their answers the way compare.mjs does. Unlike compare.mjs,
it keeps going past a difference. Each area records which of its checks
agree and which differ, plus the first difference in full.
RESULTS.md is a run against Monolith, zds, vlpds, pds.js and
tranquil.
just compat # from the repository's root: everything below, in one
Or run it by hand:
cd harness
node bench.mjs --setup # once: Monolith's binary, and the other implementations' images (bench/README.md)
node compat.mjs --steps 200 --json ../compat/results.json > ../compat/tables.md
node compat.mjs --report ../compat/results.json # the tables again
--impls monolith,zds and --areas oauth,records run only some
implementations or areas. --seed draws other requests, and --jobs sets
how many runs go at once (3). just compat passes these flags on
(just compat --impls zds --areas oauth), and a flag you give there wins
over the recipe's. The tables go to standard output, and RESULTS.md holds
them along with what they show.
Monolith's own gates are separate. just gates runs compare.mjs with
every check, and it stops at the first difference.
What is checked: the map
compare.mjs's checks fall into three kinds. Core checks compare the
server with the reference, or hold both to the same rule, and they run
against every implementation. Not yet checks compare with the reference
too, but in areas the first cut leaves out (email, invites, rate limits,
the admin lifecycle, proxying). Monolith's own checks hold Monolith to
its own pages where the reference can't say, or to the reference's
implementation rather than the protocol. They run only against Monolith, in
its gates.
| check (compare.mjs step) | module | kind | area |
|---|---|---|---|
checkOAuthMetadata | oauth | core (but see below) | oauth |
oauthLogin, oauthParRefused, oauthDpopRefused, oauthTokenRefused, oauthRevokeRefused, oauthRefresh, oauthRevoke, oauthGetSession, oauthClientLibrary | oauth | core | oauth |
oauthScopedCall, oauthScopedRefused, oauthScopedEdges | oauth-scopes | core | oauth scopes |
oauthSetPar, oauthSetLogin, oauthSetCall (the sets are published on a third PDS, which is the lexicon authority each server is given) | permission-sets | core, where an implementation can be given a lexicon authority | oauth permission sets |
oauthClientDocument, oauthClientCache, oauthClientLogin, oauthAssertionRefused, oauthConfidentialSession | oauth-clients | core | oauth clients |
oauthAuthorizeRefused (the page, its API and its redirect endpoint, each with one thing wrong) | oauth | Monolith's own (the reference's page API) | |
oauthDeviceCookies (device cookies, renewed at sign-in, with 30 s of grace) | oauth | Monolith's own (the reference's page API) | |
oauthSilentSignIn (a remembered device is sent back at once) | oauth-clients | Monolith's own (the reference's page API) | |
oauthDepartures, oauthClientDepartures (spec/OAUTH.md's and spec/OAUTH-CLIENTS.md's departures, held to the spec) | oauth, oauth-clients | Monolith's own | |
oauthAccountAuth, oauthAccountSessions, oauthAccountDevices, and with --commands, oauthAccountCommands (spec/OAUTH-ACCOUNT.md's routes, which the reference doesn't serve) | oauth-account | Monolith's own | |
passkeyAuth, passkeyAdding, passkeySigningIn, passkeySignInRefused, passkeyRemoving, passkeyStates, and with --commands, passkeyCommands (spec/PASSKEYS.md's passkeys, which the reference doesn't have) | passkeys | Monolith's own | |
createSession, getSession, describeServer, refreshSession, deleteSession, wrongTokens, appPasswords, sessionRace | accounts, session, sessions | core | sessions |
createAccount, createAccountRefused, accountRace | accounts | core | accounts |
createRecord, putRecord, deleteRecord, applyWrites, getRecord, listRecords, describeRepo, createWithTid, concurrent, oddRequests, crossWrites | records, odd, accounts | core | records |
validatedWrite, duplicates, validatedBatch, validatedElsewhere | validation | core | validation |
checkRepos (getRepo verified by the reference's library, the MST root, since, getLatestCommit) | repository | core | repository |
listRepos, getRepoStatus, getBlocks, getGoneBlocks, syncGetRecord, noCrawlsYet, checkCrawls | federation | core | sync |
--firehose: startFirehose, checkFirehose, checkReplay | firehose | core | firehose |
updateHandle, updateHandleRefused, updateHandleNoToken, resolveHandle, wellKnownHandle, recommendedCredentials, handleRace, describeRepo | identity, records | core | identity |
uploadBlob, blobRecord, getBlob, listBlobs | blobs | core | blobs |
requestEmailConfirmation, confirmEmail, requestEmailUpdate, updateEmail, emailFactor, passwordReset, appPasswordEmail | not yet (email) | ||
inviteCodes, --invites | limits | not yet (invites) | |
bigBatch, bodyCaps, appPasswordScopes, --limits | limits, oauth-limits | not yet (limits, though their exact numbers are Monolith's own) | |
loginTiming, oauthSignInTiming | limits | Monolith's own (timing floors) | |
--lifecycle | lifecycle | not yet (the admin lifecycle) | |
forwarded, refused, serviceAuth, chatCase, preferences, readAfterWrite, foreignRecord, feedAndPush | proxy | not yet (proxying) | |
--commands: checkCommands, withoutAccount, calls through MCP tools and CLI commands | commands, via | Monolith's own (spec/COMMANDS.md) | |
outage, checkOperations (logs, traces, metrics, health, lifecycle restarts) | operations, logging | Monolith's own (spec/OPERATIONS.md) | |
--store, --scale | store, scale | Monolith's own (spec/STORE.md) | |
--crash-every: crash (a server killed and started again on its data) | crashes | core, but not in this first cut. It needs the server killed and restarted, which the adapters can do (server.kill, server.start), and it's left for later |
Some parts of core checks are Monolith's own, and they hold only Monolith
(cand.adapter.own in the code):
| part | where | why |
|---|---|---|
describeServer's spacesAlpha: true | checks/session.mjs | spec/SPACES.md, not the reference |
the atproto_kem key required in the genesis operation and in getRecommendedDidCredentials | checks/accounts.mjs, checks/identity.mjs | spec/SPACES.md (another implementation may have one, see below) |
a blob at exactly the 5 MiB limit and one byte past it, plus describeServer's blobUploadLimit | checks/blobs.mjs | the exact number of a limit |
| PAR bodies of exactly 100 KiB and one byte past it | checks/oauth.mjs | the exact number of a limit |
OAuth access tokens' expires_in of 899 to 900 | checks/oauth.mjs | spec/OAUTH.md's departure (below) |
the authorization server metadata's request_parameter_supported: false | checks/oauth.mjs | spec/OAUTH.md, held to the spec |
a GET of the page API's sign-in, and a POST to an endpoint it doesn't have | checks/oauth.mjs | the reference's page API |
Each area runs its own steps on a fresh pair of servers, with two or three
accounts made first. It draws the steps by weight, the way compare.mjs
draws its own. The repository, sync and firehose areas need writes to have
something to check. So createRecord, putRecord, deleteRecord and
applyWrites are drawn among their steps as supporting steps (the firehose
also draws createAccount and updateHandle), and making the first
accounts is a supporting step in every area. A supporting step's difference
is reported separately and isn't counted in the area. The repository's
checkRepos and the firehose's checks run every --check-every steps (10)
and at the end. Sync's crawl checks run with the first accounts, every 10
steps and at the end.
A server that's built to host one account uses that account in every area
(spec/COMPAT.md, "One account"). Its adapter's
claim makes the account, and the reference makes its own beside it with
createAccount, both with the password the server was started with. The
checks that make an account (the accounts area, the firehose's
createAccount) or need a second one (crossWrites, handleRace, and
updateHandle's refusal of a handle another account has) don't run. Every
other check runs on the one account.
A check is one step. It keeps going past a difference (in a compatibility
run, differ in lib/report.mjs keeps the difference instead of ending
the run), and it counts as failed if anything in it differed. A check that
throws (on an answer of a shape it can't read) or takes more than 3 minutes
counts as one that couldn't go on. For each check, an area records how many
ran and failed, each thing that differed and how often, and its first
difference. That's the request and both answers, printed the way
compare.mjs prints them.
The adapters
harness/lib/adapters/ has one adapter per implementation, each a module
named in its index.mjs. common.mjs holds what they share (the server as
a process or a container, a rotation key, a browser's way through a form).
compare.mjs --impl <name> picks one (Monolith by default). An adapter says
how to make the server for the run's setup, and what that server is:
start(o)makes the server for the setupo(port, data directory, PLC directory, the reference's recovery key, crawlers, AppView, the web of OAuth client documents, rate limits and invites on or off). It answers with the process the harness starts, kills and starts again, the URL the harness calls, the URL the server's DID documents name, and its OAuth issuer if that's a different one. It also gives the rotation keys it lists in each DID and its own key as a signer, its admin'sAuthorization, andtakeMail(), which gives the emails it sent. A server that hosts one account also answers with itspasswordandclaim, which makes that account.unsupportedlists each setting it can't be given, with the reason. The checks that depend on one are skipped, and the report says why.pageApi, or elsesignIn(below).ownsays whether Monolith's own rules hold it. Only Monolith's does.
The checks don't read any of this. When a check needs something from the
server beyond what the protocol says, it asks harness/lib/server.mjs.
That offers own() (whether Monolith's own rules hold it), can(setting)
(whether it can be given a setting unsupported names), pageApi() and
issuer(side). For Monolith, each one answers so the check does exactly
what it does without asking. Anything only a compatibility run needs, such
as making a one-account server's account, lives in harness/lib/areas.mjs
(the compatibility run itself) instead of checks/.
Monolith's adapter runs it the way the harness always has. It runs the
command after -- with the PDS_* environment that compare.mjs's header
lists, reads its mail from PDS_DEV_MAIL_DIR, and kills and restarts it as
a process.
zds, vlpds, pds.js and tranquil run in Docker, from the images
bench.mjs --setup builds and pulls (pinned in bench/impls.json). They're
configured the way the benchmark configures them (harness/lib/impls.mjs,
shared with bench.mjs), plus whatever a compatibility run needs. pds.js,
which the benchmark doesn't run, is configured the way its container runs
by default. The harness reaches each one on a loopback port, and each one
reaches the harness's PLC directory, AppView and crawlers through
host.docker.internal. The server is the container. The harness starts it,
kills it with a signal, and follows its log with docker logs.
pds.js hosts one account. Its adapter starts it with a fresh account
password (PDS_PASSWORD, which it requires) and claims the account the way
its onboarding page does. That's a same-origin
POST /account/onboarding/create with that password, the handle and the
email, then createSession by the new DID. pds.js names its endpoint
https://<PDS_HOSTNAME> in the DID documents it makes, and its OAuth issuer
is the URL it's called at. So the adapter gives https://localhost as its
public URL and the loopback URL as its issuer, and each is taken out of
answers the way a server's URL is.
tranquil runs on a fresh PostgreSQL of its own, on a Docker network
with it, as the benchmark runs it. It names itself https://<PDS_HOSTNAME>
everywhere: its OAuth issuer, the endpoint in its DID documents, and the
htu its DPoP proofs must name. So the adapter gives it the hostname
localhost and https://localhost as its public URL, which is its issuer
too, while the harness reaches it on a loopback port: the checks sign each
proof's htu with the issuer and send the request to the port. Its one
operator key, PLC_ROTATION_KEY, is a public did:key, listed first in
each DID it makes, so it takes the reference's recovery key; its other
rotation key is each account's own signing key, which it holds and signs
with, so the harness has none of its own.
| setting | Monolith | zds | vlpds | pds.js | tranquil |
|---|---|---|---|---|---|
| PLC directory | PDS_PLC_URL | ZDS_PLC_DIRECTORY | VLPDS_PLC_URL, VLPDS_PLC_MODE=directory | PLC_URL | PLC_DIRECTORY_URL |
| rotation key, the harness's | PDS_PLC_ROTATION_KEY_HEX | ZDS_PLC_ROTATION_KEY | VLPDS_PLC_ROTATION_KEY_K256_PRIVATE_KEY_HEX | no: only the account's signing key, which the server holds | no: each account's own signing key, which the server holds |
| the reference's recovery key | PDS_RECOVERY_DID_KEY | ZDS_RECOVERY_DID_KEY | VLPDS_PLC_RECOVERY_DID_KEY | no: the same | PLC_ROTATION_KEY, its one operator key |
server DID did:web:localhost | PDS_HOSTNAME | ZDS_SERVER_DID | VLPDS_SERVICE_DID | PDS_HOSTNAME until it has its account, then that account's DID | PDS_HOSTNAME |
hostname localhost (requestCrawl's) | PDS_HOSTNAME | no: it comes from ZDS_PUBLIC_URL, which must be the URL the harness calls (DPoP's htu) | no: the same, and it doesn't ask a relay on another host to crawl when its own host is local | no: it tells one relay (RELAY_URL) at most every 20 minutes | no: with the hostname localhost it asks no crawler to crawl |
handle domains .test, .example | PDS_HANDLE_DOMAINS | ZDS_HANDLE_DOMAINS | .test only: VLPDS_HANDLE_DOMAIN=test (another domain needs two labels) | no: its hostname is its only one | PDS_USER_HANDLE_DOMAINS=test,example, which it names with no leading dot |
| accounts, more than one | yes | yes | yes | no: one, made by claim | yes |
admin password admin-pass | PDS_ADMIN_PASSWORD | ZDS_ADMIN_TOKEN | no: its admin token must be 32 bytes or more | no: it has no admin methods | no: an admin is an account marked so in its database |
| invites off (or on) | PDS_INVITE_REQUIRED | ZDS_INVITE_REQUIRED | VLPDS_INVITE_REQUIRED | off, since it has none | INVITE_CODE_REQUIRED |
| rate limits off (or on) | PDS_RATE_LIMITS_ENABLED | no general limits to turn on, and full getRepo exports are raised to 8 at once | VLPDS_NO_RATE_LIMITS | no limits to turn on | DISABLE_RATE_LIMITING |
| the harness's AppView | PDS_BSKY_APP_VIEW_* | ZDS_PROXY_SERVICE_URL, _DID | VLPDS_APPVIEW | APPVIEW_URL, APPVIEW_DID | no setting: it proxies by the atproto-proxy header, resolving the DID that names |
| the harness's crawlers | PDS_CRAWLERS | ZDS_CRAWLERS | not given (see hostname) | not given (see hostname) | not given (see hostname) |
| OAuth client documents from the harness's web | PDS_OAUTH_CLIENT_FETCH | no: it fetches the client_id's own https URL | no: the same (http only with --dev-mode) | no: it fetches the client_id's own URL | no: it fetches the client_id's own URL |
| no SSRF guard, the way the reference runs | PDS_DISABLE_SSRF_PROTECTION | ZDS_ALLOW_PRIVATE_FETCH | its default for operator-set URLs | its default | ALLOW_PRIVATE_FETCH |
PDS_DEV_MAIL_DIR | ZDS_MAIL_PROVIDER=outbox, read with dev.zat.admin.listMail | no: it's readable only with --dev-mode | no: its container has no transport | no: it sends only to an SMTP relay and keeps none to read back | |
| reached at the address it names | yes | yes | yes | yes | no: it names itself https://localhost, reached on a loopback port |
| consent to a narrower scope | the page API | no: there's no consent step | no: its form only takes scopes away | no: its form approves or denies the scope asked for | no: its consent takes the scopes approved, but the hook carries none (below) |
Each "no" skips these checks:
- the hostname: sync's crawl checks
- the handle domains:
describeServer - client documents: the whole oauth clients area
- a narrower scope: the narrowing in
oauthLogin(the scope asked for is granted) - more than one account: those above
- the harness's rotation key: its own PLC operation before an updateHandle
- the recovery key:
recommendedCredentials - the address it names:
oauthClientLibrary, as the atproto client library refuses an https address on a loopback host
No core check needs mail, the admin password or rate limits.
The OAuth sign-in hook
The harness signs in on the reference's authorization page through its page
API (/@atproto/oauth-provider/~api/sign-in, consent, reject, with its
CSRF cookie and device cookies). Monolith serves that API too, so with
Monolith every step of a sign-in is compared. No other server serves it, and
the protocol leaves the sign-in step to each server. So an adapter without
pageApi has a hook:
signIn(account, authorizeUrl, consent)
// account: {identifier, password}; authorizeUrl: the page a PAR's request_uri names;
// consent: {approve}. Answers the answer that sends the browser back to the client,
// with its code: a redirect, or a form_post page.
zds's hook posts its password form (POST /oauth/authorize:
request_uri, username, password, or deny), and zds grants at once
with a 303 to the client. vlpds's hook loads the page (its device cookie
and csrf field), posts /oauth/authorize/sign-in (identifier,
password), then posts /oauth/authorize/consent (did, action=allow or
deny). That redirects to the client or answers with a form_post page.
pds.js's hook posts its consent form (POST /oauth/authorize:
request_uri, client_id, password, action=approve or deny). The form
doesn't ask for an identifier, since the server has one account, and it
redirects to the client with a 302. tranquil's page is a script calling
its JSON endpoints, and its hook calls them as the script does: it posts
request_uri, username and password to /oauth/authorize, which
answers the next page as redirect_uri. That's the consent page, whose
scopes /oauth/authorize/consent answers and then takes back approved, or
at once tranquil's /oauth/authorize/redirect, which redirects to the
client with a 303. To refuse, it posts request_uri to
/oauth/authorize/deny, which answers the client's URL with
error=access_denied as JSON, and the script navigates there; the hook
answers that as a 303 there. Its consent can approve fewer scopes than were
asked, but the hook is given no scope to approve, so it approves the scope
asked. None of them needs a browser.
When an adapter has a hook, the reference still signs in through its page
API, but its page's answers are its own and aren't compared. The server
signs in through its hook. What's compared is the protocol around the
sign-in: PAR, the redirect to the client (its target, code, state and
iss, or error), the token exchange with PKCE, DPoP and its nonces,
refresh, revocation, getSession, every scoped XRPC call with the session's
tokens, and the client documents' rules. oauthClientLibrary signs in
through the hook too, and compares only the pages' redirect status.
What other implementations are let have
spec/COMPAT.md lists what they're allowed, section by
section, with the reason for each:
- the same things are taken out of every answer as for Monolith
- the KEM key is optional (held to Monolith's form if it's published)
- an OAuth access token may last 15 or 60 minutes
- only three of
describeServer's fields are compared - the exact numbers of limits aren't checked
- sign-in happens on the server's own page, and it's compared from the redirect on
- Monolith's own departures aren't held against them
- a server that hosts one account uses that one
- a server with password rules of its own gets each area's first accounts with a password its adapter makes to fit them
- a server named by an https address it isn't reached at has the requests for that address sent where it is reached
- the settings an adapter can't give have their checks skipped and reported
The harness allows nothing else, and Monolith is held to all of
spec/PDS.md.
spec/PDS.md and the area pages list "what the reference does that the
specs leave open" (the order a write's checks refuse in, the 500s, and the
rest). Those aren't allowed to differ. Another implementation that does
otherwise differs from the reference, and RESULTS.md says where a
difference is one of those.
The vlpds documentation the report cites as deliberate
listRepos' order (DESIGN.md:5905): "listRepos order is(slot, DID), because listings are served per shard, not by creation order."getBlobof an upload that no record names (DESIGN.md:5906-5907): "Fresh uploads are readable. Blobs are written under their final key at upload, so getBlob serves an upload before any record references it, until the blob GC collects it." Alsodocs/blobs.md:65-66: "Unlike the reference PDS, vlpds serves a blob throughgetBlobas soon as it's uploaded, before any record names it."
Adding an implementation
harness/checks/ is what Monolith's gates run against the reference.
Adding an implementation mustn't change it for Monolith, not by one request
and not by one draw from the seeded source. So an implementation brings
these, in order of preference:
- An adapter, plus the settings it can't be given and why (their checks are skipped).
- A capability in
spec/COMPAT.md, where it differs from what the harness assumes in a way the protocol allows (one account, its own TLS). The capability is answered inlib/server.mjsor handled inlib/areas.mjs. It works the same for every implementation that needs it, and leaves Monolith unchanged. - Nothing else. A quirk of its own that no protocol rule allows is a difference or a skipped check. A check never gets bent around it.
No check names an implementation. A change of the second kind has to show
that it leaves Monolith's runs as they were. compare.mjs --tally <file>
writes how many requests of each kind a run made, and that's the same for
two runs of one seed. So you compare a few seeds' tallies from before and
after the change, and they come out the same.
Here are the steps:
- Pin it in
bench/impls.jsonand give it a benchmark adapter (bench/README.md), unless it can't serve the benchmark's workload (like pds.js, which hosts one account). Put its Docker configuration inharness/lib/impls.mjs. - Give it an adapter, a module of its own in
harness/lib/adapters/named in itsindex.mjs. The adapter hasstart,port,info,unsupportedwith reasons,signInunless it serves the reference's page API, andreadyMsif it starts slowly. For a server that hosts one account, it also listsaccountsinunsupportedand getspasswordandclaimfromstart. For a server that refuses passwords by a rule of its own, it haspasswordFor(p)(spec/COMPAT.md, "Password rules"). For a server that names itself by an https origin it isn't reached at,startanswers that origin aspublicUrlandfronted: true(spec/COMPAT.md, "Its own address"). - Add its name to
IMPLSinharness/compat.mjs, and add the differences its documentation says are on purpose toDELIBERATEthere. - Run
node compat.mjs --impls <name> --steps 60to try it, then do a full run.