chadtmiller.com/monolith
An atproto PDS
Federation: what a relay reads
The com.atproto.sync methods a relay uses besides getRepo,
getLatestCommit and the firehose (SYNC.md): listRepos,
getRepoStatus, getBlocks and getRecord; and the requestCrawl the
server sends to the relays it is configured with. What every method shares,
and how the server is checked, is in PDS.md; the repository they
serve is as REPO.md says.
Each names an account by did, which its lexicon requires to be a DID: a
handle there is 400 InvalidRequest, as is any other argument its lexicon
refuses, before anything else is checked. Then a DID no account here has is
400 RepoNotFound. An account that is not active is answered as
LIFECYCLE.md says; what follows is of an active one.
Environment
Besides those PDS.md and ACCOUNTS.md list:
| Variable | Is |
|---|---|
PDS_CRAWLERS | optional: the base URLs of the crawlers (relays) to notify, comma-separated, such as http://127.0.0.1:2470 |
listRepos
listRepos answers {repos, cursor}: every account whose repository has its
first commit, oldest first, in the order createAccount made them. Each is
{did, head, rev, active: true}, with no status: head the CID of the
repository's latest commit and rev its rev, as getLatestCommit answers
them.
limitis 1 to 1000, by default 500; outside that, or not an integer, it is400 InvalidRequest.- A page with repos has a
cursor, whether or not it is full; the next page, from that cursor, starts with the account after the page's last. A page with none has nocursor, so the one after the last account is{repos: []}. A cursor stays good: the same cursor asked again later, after a restart too, lists the accounts made since. An emptycursoris the same as none. - The server may choose what a cursor holds. The reference's is
<ms>::<did>, the account's creation time and DID, and one it cannot read is400 InvalidRequest. The harness sends each side only the cursors it gave out, or none.
One thing the reference does is a fault, and the server does not copy it.
The reference keeps each account's head and rev for listRepos apart
from its repository, and updates them after each commit; killed between the
two, it lists the commit before, until the account's next commit. The
server's listRepos names the latest commit, as getLatestCommit does,
after a restart too.
getRepoStatus
getRepoStatus answers {did, active: true, rev}, rev the repository's
latest commit's.
getBlocks
getBlocks answers the blocks of the account's repository whose CIDs cids
lists, as a CAR file (application/vnd.ipld.car, atproto/repository.md)
whose header has no roots, {version: 1, roots: []}. Each block is in it
once, however often cids names it, and in any order.
cidsis required, and each must be a CID: none, or one that does not parse, is400 InvalidRequest, before thedidis looked up.- The repository holds its latest commit and every MST node and record of its
current tree. If
cidsnames any CID it does not hold, the answer is400 InvalidRequest, with no blocks, not theBlockNotFoundthe lexicon names. So is a block only another account's repository holds, though the server has it. - Its cost is that of the CIDs it names, not of the repository: a CID is
looked up in an index, and one the repository does not hold is found
missing without reading the tree (
LIMITS.md). - A block the repository held before and no longer does (an earlier commit,
or a node or record a later commit took out of the tree) is answered as
one it does not hold,
400 InvalidRequest, as the reference answers it: the reference deletes such blocks, and the server need not keep them (STORE.md), and answers them so even while it still has them.
getRecord
getRecord (com.atproto.sync.getRecord, not the com.atproto.repo method
of the same name) answers a CAR file whose root is the account's latest
commit, with the blocks that prove the record at collection/rkey in it, or
prove that there is none:
- the commit;
- every MST node a search for the key
collection/rkeypasses through, from the root: in each node, the search ends at the entry whose key it is, or else goes on into the subtree just before the first entry whose key sorts after it (the node'slif that is its first entry, the last entry'stif no entry sorts after it), and ends where there is no such subtree; - the record, if the tree holds the key.
Nothing else, in any order. A record the tree does not hold is not an
error: the answer is 200 with the proof of its absence, though the lexicon
names RecordNotFound. A collection that is not an NSID, or an rkey that
is not a record key (atproto/record-key.md), is 400 InvalidRequest.
requestCrawl
With PDS_CRAWLERS set, the server tells each crawler that it has something
new: POST <url>/xrpc/com.atproto.sync.requestCrawl, with content-type: application/json and the body {"hostname": "<PDS_HOSTNAME>"}.
- It does so at the first firehose event after it starts, and then at the first event at least 20 minutes after it last did; at no other time. So not when it starts, and not while there are no events: a write that is refused, or that makes no commit, makes no event. Each event is a notification due, not a request on its own, so a burst of events is one request to each crawler.
- The requests go in the background: no answer of the server waits for one, and an answer that is an error, or a crawler that cannot be reached, is logged at most, not retried. Each crawler is told independently of the others.
- When it last told them need not survive a restart: started again, the server may tell them again at its first event.
How it is checked
The harness starts both sides with the same three crawlers, each its own:
one that answers 200, one that answers 500, and a URL where nothing
listens. Both crawlers that listen record every request.
- listRepos. Each side is paged through with its own cursors, at a
random
limit, and every page is compared, with each account's DID named asPDS.mdsays andheadandrevtaken out; on each side, each repo'sheadandrevmust be what that side'sgetLatestCommitanswers. The last page's cursor is kept, and at the next such step, asked again on each side: it must list the accounts made since, and only those. An empty cursor, and alimitout of range, are compared too. - getRepoStatus of an account, of a DID no one has, of a handle, and of
something that is no DID, compared with
revtaken out;revmust be the side's latest. - getBlocks of a random set of blocks of an account's repository, read
from the reference's
getRepo: MST nodes and records, the same on both sides since their trees are, and sometimes each side's own latest commit, sometimes with a CID named twice. Sometimes it adds a CID the repository never held (another account's latest commit, or a block of no repository, asdag-cbororraw), and sometimes it is a DID no one has, a CID that does not parse, or nocids. Errors are compared by status and name; a CAR must have no roots, and hold exactly the blocks asked, each the bytes its CID names. - getBlocks of blocks the repository no longer holds: MST nodes and
records the reference's tree had at an earlier such step and has not now,
and each side's own earlier commit, sometimes with a block it holds now.
Both sides must answer
400 InvalidRequest. - getRecord of a key of an account's repository, there or not, and of a
bad collection or record key or a DID no one has. Errors are compared; a
CAR's root must be the side's latest commit, the reference library's
verifyProofsmust verify it against the record's CID, or its absence, and the account's signing key, and its blocks besides the commit must be exactly the reference's. - requestCrawl. Before the first account is made, neither side has sent
one. Once the first accounts are made, and at the end of the run, each
crawler that listens must have had from each side the request above,
compared by method, path,
content-typeand body, whosehostnameislocalhoston both sides: the first within 5 seconds, and no more than one, plus one for every whole 20 minutes the run has lasted, and from the server one more for each restart.
With --crash-every, after each restart, once the repositories are checked,
listRepos and each account's getRepoStatus must list every account in
the order made, with the head and rev the repository check found.