chadtmiller.com/monolith
An atproto PDS
Sync and the firehose
The com.atproto.sync methods: getRepo, getLatestCommit and
subscribeRepos, the firehose. What every method shares, and how the server
is checked, is in PDS.md; the repository they serve is as
REPO.md says.
What the reference does that the specs leave open
getRepoandgetLatestCommitof a repository the server does not hold are400 RepoNotFound.getRepowithsince, a rev, answers a CAR file whose root is the latest commit and whose blocks are those the repository holds now that commits after that rev added: the latest commit, and every record and MST node of the current tree that the tree at that rev did not have. Nothing outside the current tree is among them, so a record written aftersinceand then replaced or deleted is not. A block the tree had at that rev is not among them either, unless a later commit removed it and another put it back, or wrote it again, as a record whose content another key holds: such a block may be among them or not. There are no blocks at all whensinceis the latest rev. Asincethat is not a TID is400 InvalidRequest.subscribeReposwith acursorthat is not an integer sends an error frame,InvalidRequest, and closes.
The firehose
The server also serves com.atproto.sync.subscribeRepos, as its lexicon,
atproto/event-stream.md and atproto/sync.md define it: a WebSocket at
/xrpc/com.atproto.sync.subscribeRepos, each message a DAG-CBOR header and
body.
- Every commit is a
#commitevent, in the order the commits were made, withseqincreasing. Itsblocksare a CAR file whose root is the signed commit, holding the record blocks it wrote and MST nodes enough for two things, which are what is checked: every op proved against the commit (each created or updated key present with its CID, each deleted key absent), and the ops undone on the tree in the event, in reverse, giving the treeprevDatanames (atproto/sync.md, the inversion it describes). The server may choose which other blocks it holds, and their order. Ops nameprev, the CID a record had, for updates and deletes.sinceis the previous commit'srev, andprevDataitsdata.rebaseandtooBigare false: a commit has at most 200 ops and 2000000 bytes of blocks, and one that would have more is refused (LIMITS.md). - A new account's stream starts as the reference's does:
#identity,#account, a#commitof the empty repository with no ops,sinceorprevData, then#sync. - An account's changes of state are
#accountevents, with#identityand#syncwhen it is activated, and its deletion drops its earlier events from what the stream replays (LIFECYCLE.md). - An account moving here (
MIGRATION.md):createAccountwith adid, andimportRepowhile the account is deactivated, make no events.submitPlcOperation, in any state, is#identitywith the DID and nohandle, and an active account'simportRepois#syncof its new head. - An
applyWritescommit has one op per write, in the batch's order, not one per key changed: a create then an update of the same key are two ops. An op'sprevis the CID the record had before the commit, whatever the action and whatever earlier writes in the batch did: an update of a record the batch created has none, and a create after the batch deleted the same record has the deleted record's. - With
cursor, the stream first replays every event after thatseq, not from it asatproto/event-stream.mdsays, then goes on live. Without one, it is live only. Acursorlater than the last event is refused as the reference refuses it: an error frame,FutureCursor. Acursorwhose next event is out of the backfill window,PDS_REPO_BACKFILL_LIMIT_MS, gets an#infomessage,OutdatedCursor, first, then the events of the window (STORE.md). - The client sends nothing: a message from it is ignored, up to 4096
bytes, and a longer one closes the connection. One client address may
hold 16 subscriptions, and the server 1024 (
LIMITS.md). - Events are kept with everything else: after a restart,
seqcarries on from the last event, and a cursor from before it in the backfill window replays without a gap (DURABILITY.md,STORE.md).
node compare.mjs --firehose ... subscribes to both servers from the start.
At each check it waits for the event of each side's latest commit, checks
every new event the way a consumer would (seq, signature, the chain of
since and prevData, undoing the ops, and the reference library's
verifyProofs), and compares the commits since the last check with the
reference's by their ops, ignoring their order, since writes sent at once
may be committed in either. Sometimes it also subscribes again from a cursor,
and from one in the future. With --crash-every, it subscribes again after
each restart from the last event it had, and the stream must carry on with
no gap or repeat.