# Sync and the firehose The `com.atproto.sync` methods: `getRepo`, `getLatestCommit` and `subscribeRepos`, the firehose. What every method shares, and how the server is checked, is in [`PDS.md`](PDS.md); the repository they serve is as [`REPO.md`](REPO.md) says. ## What the reference does that the specs leave open - `getRepo` and `getLatestCommit` of a repository the server does not hold are `400 RepoNotFound`. - `getRepo` with `since`, a rev, answers a CAR file whose root is the latest commit and whose blocks are those the repository holds now that commits after that rev added: the latest commit, and every record and MST node of the current tree that the tree at that rev did not have. Nothing outside the current tree is among them, so a record written after `since` and then replaced or deleted is not. A block the tree had at that rev is not among them either, unless a later commit removed it and another put it back, or wrote it again, as a record whose content another key holds: such a block may be among them or not. There are no blocks at all when `since` is the latest rev. A `since` that is not a TID is `400 InvalidRequest`. - `subscribeRepos` with a `cursor` that is not an integer sends an error frame, `InvalidRequest`, and closes. ## The firehose The server also serves `com.atproto.sync.subscribeRepos`, as its lexicon, `atproto/event-stream.md` and `atproto/sync.md` define it: a WebSocket at `/xrpc/com.atproto.sync.subscribeRepos`, each message a DAG-CBOR header and body. - Every commit is a `#commit` event, in the order the commits were made, with `seq` increasing. Its `blocks` are a CAR file whose root is the signed commit, holding the record blocks it wrote and MST nodes enough for two things, which are what is checked: every op proved against the commit (each created or updated key present with its CID, each deleted key absent), and the ops undone on the tree in the event, in reverse, giving the tree `prevData` names (`atproto/sync.md`, the inversion it describes). The server may choose which other blocks it holds, and their order. Ops name `prev`, the CID a record had, for updates and deletes. `since` is the previous commit's `rev`, and `prevData` its `data`. `rebase` and `tooBig` are false: a commit has at most 200 ops and 2000000 bytes of blocks, and one that would have more is refused (`LIMITS.md`). - A new account's stream starts as the reference's does: `#identity`, `#account`, a `#commit` of the empty repository with no ops, `since` or `prevData`, then `#sync`. - An account's changes of state are `#account` events, with `#identity` and `#sync` when it is activated, and its deletion drops its earlier events from what the stream replays (`LIFECYCLE.md`). - An account moving here (`MIGRATION.md`): `createAccount` with a `did`, and `importRepo` while the account is deactivated, make no events. `submitPlcOperation`, in any state, is `#identity` with the DID and no `handle`, and an active account's `importRepo` is `#sync` of its new head. - An `applyWrites` commit has one op per write, in the batch's order, not one per key changed: a create then an update of the same key are two ops. An op's `prev` is the CID the record had before the commit, whatever the action and whatever earlier writes in the batch did: an update of a record the batch created has none, and a create after the batch deleted the same record has the deleted record's. - With `cursor`, the stream first replays every event after that `seq`, not from it as `atproto/event-stream.md` says, then goes on live. Without one, it is live only. A `cursor` later than the last event is refused as the reference refuses it: an error frame, `FutureCursor`. A `cursor` whose next event is out of the backfill window, `PDS_REPO_BACKFILL_LIMIT_MS`, gets an `#info` message, `OutdatedCursor`, first, then the events of the window (`STORE.md`). - The client sends nothing: a message from it is ignored, up to 4096 bytes, and a longer one closes the connection. One client address may hold 16 subscriptions, and the server 1024 (`LIMITS.md`). - Events are kept with everything else: after a restart, `seq` carries on from the last event, and a cursor from before it in the backfill window replays without a gap (`DURABILITY.md`, `STORE.md`). `node compare.mjs --firehose ...` subscribes to both servers from the start. At each check it waits for the event of each side's latest commit, checks every new event the way a consumer would (seq, signature, the chain of `since` and `prevData`, undoing the ops, and the reference library's `verifyProofs`), and compares the commits since the last check with the reference's by their ops, ignoring their order, since writes sent at once may be committed in either. Sometimes it also subscribes again from a cursor, and from one in the future. With `--crash-every`, it subscribes again after each restart from the last event it had, and the stream must carry on with no gap or repeat.