~ knotless Search/
↑↓ move↵ open⇧↵ all results
0
0
528 lines · 29.4 KB Raw History

Spaces: write notifications, the writer set and syncers

How the hosts of a space keep each other, and its syncers, up to date: the repo host telling the space host of each commit (notifyRepoUpdate, sent), the space host taking those notifications (notifyRepoUpdate, served), keeping the writer set and serving it (listWriters), registering syncers (authorizeSpaceSync's notify, confirmed with confirmRegistration), and telling them of each change of the writer set, of their registration's expiry, and of the space's deletion (notifySpaceUpdate, sent). Spaces, their roles, keys and building blocks are in SPACES.md; the space repo and its commits in SPACES-REPO.md; space credentials, authorizeSpaceSync and service auth in SPACES-CREDENTIALS.md; policies, the space lifecycle and the writer set's other changes in SPACES-POLICIES.md; what every method shares in PDS.md. The protocol is spaces/proposal.md, "Write notifications", "The sync boundary (writer set)" and "Space deletion"; where this page and the proposal say the same thing, this page is the one to build from.

The lexicons of com.atproto.space.notifyRepoUpdate and listWriters, and of com.atproto.space.defs (#repoWrite, #repoRemoved), are in lexicons/com/atproto/space. com.atproto.space.confirmRegistration and notifySpaceUpdate are methods the server calls on syncers and does not serve: their lexicons are in spaces/lexicons/com/atproto/space.

Environment

VariableIs
PDS_DISABLE_SSRF_PROTECTIONas PROXY.md defines it, for every request the server sends to a URL others name: true or 1 turn protection off; false, 0, empty or unset leave it on; any other value makes pds serve exit 1

Every request the spaces pages describe the server making goes through PROXY.md's guard ("SSRF protection"): the repo update notifications, and the confirmations, updates and notices sent to syncers, here; the did:web documents of a space's authority, of the issuers of service auth, delegation tokens and space credentials (SPACES-CREDENTIALS.md, "Issuers' keys"), and of managing apps; a client attestation's metadata and its jwks_uri (SPACES-CREDENTIALS.md); and a managing app's checkAccess (SPACES-POLICIES.md). Only the directory at PDS_PLC_URL, which the operator names, is asked without it. Unless protection is off, such a request goes only to an https URL whose host is a public name, not an IP address and not localhost, resolving only to public unicast addresses; any port. One the server will not send is as one that got no answer: a repo update notification or a notice failed for now (below), an update given up, a registration not confirmed, a document that cannot be read. With protection on or off, none follows a redirect (a 3xx is as no answer), each is given up after 10 seconds, a DID document is read up to 65536 bytes and another JSON document up to 524288 (a longer one is as one that cannot be fetched), and each did:web document is fetched in the pool LIMITS.md keeps for them, apart from the directory's slots. With true, http URLs and every address are allowed, and a did:web on localhost is fetched over http (did:web:localhost%3A2583 from http://localhost:2583/.well-known/did.json); the spaces run sets it, as every host there is http on localhost, and the reference runs without SSRF protection.

An endpoint, wherever these pages send to one, is a string starting with http:// or https:// that parses as a URL: a service's serviceEndpoint in a DID document, or the endpoint a syncer registers (below); any other is no endpoint, and sending to it is as getting no answer. A request to it goes to <origin>/xrpc/<method>, <origin> the endpoint's scheme, host and port: its path, query and fragment are not used, so an endpoint https://syncer.example/base?x= is sent to at https://syncer.example/xrpc/com.atproto.space.notifySpaceUpdate. Nothing is sent to an endpoint with a user name or password: as one that got no answer.

The writer set

For each space it hosts, the space host keeps its writer set: one entry per account whose writes it tracks, {did, repoRev, data, spaceRev, removed}. did is the account, repoRev and data its space repo's commit rev and data as last notified, spaceRev a TID the space host gave the entry when it last changed, and removed whether the account was taken out of the set since (below, "Removing a writer"). The space's revision is the greatest spaceRev of its entries, removed ones included; a space whose set is empty has none. Each new spaceRev is the TID of the time it is given, or, if that is not later than the space's revision, the TID one microsecond after it, so a space's revisions only grow, and each change of the set has one of its own.

An entry is made or changed only by a notification the space host takes (below, "Taking a notification"), whether it came over HTTP or from a commit on this server, and marked removed only as "Removing a writer" says. Entries are never collected: a removed one stays until its account's next notification taken, or until the set is emptied. The policies part empties the set when the space is deleted, replaces it (importSpaceMeta) and writes out the entries not removed (exportSpaceMeta) (SPACES-POLICIES.md).

Telling the space host: notifyRepoUpdate, sent

The repo host's part. After every commit of an account's space repo on this server (on_commit, SPACES-REPO.md), clearing included, the space's host is told of it, at once:

  • The authority is an account on this server, the writer's own included: the notification is taken here directly, as below ("Taking a notification") from step 5 on, with no HTTP and no service auth. A notification refused there (an unknown or deleted space, a writer the policy does not authorize) changes nothing, and is not retried.
  • Otherwise it is sent over HTTP: POST <origin>/xrpc/com.atproto.space.notifyRepoUpdate, Content-Type: application/json, body exactly {space, repo, repoRev, data}: the space's URI, the writer's DID, the commit's rev, and its data as bytes ({"$bytes": …} in JSON). <origin> is that of the endpoint of the #atproto_pds service of the authority's DID document (a did:plc from the directory at PDS_PLC_URL, a did:web from its host), as "Environment" above says. It carries service auth (atproto/xrpc.md) from the writer: Authorization: Bearer <JWT>, the header {"typ": "JWT", "alg": "ES256K"}, the body exactly iss (the writer's DID), aud (<authority DID>#atproto_pds), lxm (com.atproto.space.notifyRepoUpdate), iat, exp (iat + 60) and jti (a fresh random string), signed with the writer's #atproto key, low-S.

Delivery and retries

A sent notification is delivered when the space host answers a 2xx. It failed for now when no answer came: the authority's DID document could not be read or has no #atproto_pds, the connection failed or was closed, or no answer came within 10 seconds; or when the answer is a redirect (3xx, not followed: as no answer, "Environment"); or when its status is 408, 425, 429, 500, 502, 503, 504, 522 or 524. Any other status is refused: not sent again. 400 FutureRev, a space host whose clock is behind the writer's, is refused so.

A notification that failed for now is kept in the retry queue, which holds at most one entry per writer and space: the latest commit's rev and data, with its schedule.

  • A later commit of the same repo replaces the entry, whatever its schedule: only the latest state is ever sent again, and its schedule starts afresh (its first retry, and 24 hours from that commit).
  • A notification delivered or refused takes out the entry if the entry's rev is not later than its own.
  • The nth retry of an entry is due min(60 · 2^(n−1), 3600) seconds, times a random factor from 0.5 to 1, after the attempt before it, and is sent within 10 seconds of being due: the first 30 to 60 seconds after the failure, the second 60 to 120 after the first retry, and so on to between 30 and 60 minutes. A retry is sent as above, with fresh service auth, and is delivered, failed for now (and rescheduled) or refused as above.
  • An entry not delivered 24 hours after the commit it holds is dropped.
  • The queue survives the server being killed (DURABILITY.md): an entry kept before is retried after, when due.

A space host that missed notifications is not told again otherwise: syncers catch up through listWriters, and from repo hosts.

Taking a notification: notifyRepoUpdate, served

com.atproto.space.notifyRepoUpdate takes {space, repo, repoRev, data} from a repo host, with service auth from the writer. It answers 200 with an empty body, or the first of these refusals:

  1. The service auth, as SPACES-CREDENTIALS.md checks a revocation's ("Revocation"), each 401 with its error, with lxm that must be com.atproto.space.notifyRepoUpdate (BadJwtLexiconMethod when absent or other): MissingJwt, BadJwt, BadJwtType, JwtExpired, BadJwtLexiconMethod, BadJwtIss, AuthenticationRequired, BadJwtSignature. A high-S signature is taken.
  2. The input against the lexicon: 400 InvalidRequest (a repoRev that is no TID, a space that is no space URI, …). Other properties, as spaceRev and prevSpaceRev, are ignored, and data may be any bytes.
  3. iss, as written, other than repo (a fragment too): 403 Forbidden.
  4. aud other than the space's authority's DID, bare or with #atproto_pds: 403 Forbidden.
  5. The authority not an account on this server: 400 SpaceNotFound.
  6. repoRev a TID more than 5 minutes later than now: 400 FutureRev.
  7. The space not live (never created, or deleted): 400 SpaceNotFound.
  8. The writer not authorized to write by the space's policy (authorize_write, SPACES-POLICIES.md): 403 Forbidden. A policy that cannot decide, a managedspace whose managing app does not answer, answers as that page says (502 UpstreamFailure), which a repo host retries, as a status that fails for now.

Then, if the writer's entry in the writer set, removed or not, has a repoRev not earlier than this one, nothing changes and nothing is sent, and the answer is 200 all the same. Otherwise the entry becomes {did: repo, repoRev, data, spaceRev}, not removed, with a new spaceRev: a removed writer is so added back. The change is sent to the space's syncers (below, "Sending updates"). Taking it does not depend on the authority's account status: a space whose authority is deactivated or taken down still takes notifications.

Removing a writer

remove_writer(space, did) is how the policies part takes an account out of the writer set: removeWriter, a member's write taken away by putMembers, and a member removed by removeMembers call it (SPACES-POLICIES.md). An entry there and not removed is marked removed, with a new spaceRev, keeping its repoRev and data, and the removal is sent to the space's syncers (below, "Sending updates"). An account with no entry, or one removed already, changes nothing and nothing is sent.

The writer set: listWriters

com.atproto.space.listWriters ?space=&limit=&cursor= takes a space credential (verify_space_credential, with the space's authority as audience; SPACES-CREDENTIALS.md), and answers {repos, cursor?}: the writer set's entries in ascending order of spaceRev, those whose spaceRev is after cursor (by bytes; any string is taken) when it is given, at most limit of them (1 to 1000, 100 by default). An entry not removed is

{"$type": "com.atproto.space.defs#repoWrite", "repo": <did>, "repoRev": <repoRev>, "data": {"$bytes": …}, "spaceRev": <spaceRev>}

and a removed one {"$type": "com.atproto.space.defs#repoRemoved", "repo": <did>, "spaceRev": <spaceRev>}; neither has prevSpaceRev. cursor is the last entry's spaceRev whenever there is one, and absent when there is none. It is the newest state of each writer, not a log: an entry changed while a syncer pages reappears later.

Its parameters are checked first (400 InvalidRequest), then the credential: none, or a bearer token, is 401 MissingJwt; an audience other than the authority 401 BadSpaceAudience; the other refusals the credentials part names. Then a credential of another space is 400 InvalidCredential; an authority that is not an account here 400 SpaceNotFound; an authority taken down or deactivated 400 RepoTakendown or 400 RepoDeactivated; and a space not live 400 SpaceNotFound. A credential of a deleted space still serves the space created again.

Syncers

A syncer is told of a space's updates at an endpoint of its own, with a token it chose, by registering them for the space when it authorizes (authorizeSpaceSync with notify, SPACES-CREDENTIALS.md). The space host then calls two methods there, each with the token in its body and no Authorization header: confirmRegistration once, before registering it, and notifySpaceUpdate for every update after.

Registering

A registration is of one space, and is named by its endpoint and token together, each the exact string the syncer gave: another token, or the same endpoint written otherwise (https://syncer.example/ for https://syncer.example), is another registration, told of every update apart. Each has an expiry, the exp of the credential its latest authorization was answered with: it is active until then, and lapsed from then on.

authorizeSpaceSync with notify: {endpoint, token} takes part in it at three of its steps (SPACES-CREDENTIALS.md, "authorizeSpaceSync"):

  • Refused at its step 8 or 9 (SpaceNotFound for the user perimeter, AppNotAuthorized), it drops the space's registration of the pair, active or lapsed, if there is one; an expiry notice owed it is not sent (below).
  • At step 10, unless the space has an active registration of the pair: if the space has 64 active registrations, 400 TooManyRegistrations, and nothing is sent to the endpoint. Then, unless the space has a lapsed registration of the pair, the endpoint is asked to confirm it: POST <origin>/xrpc/com.atproto.space.confirmRegistration, Content-Type: application/json, body exactly {space, token, challenge}, challenge 16 random bytes in base32 (RFC 4648's alphabet, lower case, no padding: 26 characters), fresh for each. It is confirmed when the endpoint answers a 2xx whose Content-Type is application/json, with a JSON object whose challenge is that string; other members of it are ignored. Anything else is 400 RegistrationNotConfirmed, and nothing is registered: another status, a redirect, a body that is no such object, no or another challenge, no answer within 10 seconds, a connection that fails or is closed, an endpoint that is no endpoint or has a user name or password ("Environment"), and one the guard will not send to.
  • At step 11, once the credential is minted, the registration is made, or renewed: its expiry becomes the credential's exp. A lapsed one so renewed is active again, and is owed an expiry notice again when it next lapses.

A registration ends only when it is dropped by a refusal (above), when its expiry notice is refused or given up (below, "Notices"), or with the space ("Deletion"): not when it lapses, not for a revocation (SPACES-CREDENTIALS.md, "Revocation"), and not while the authority is deactivated or taken down; then authorizeSpaceSync refuses, so no registration is renewed and each lapses at its expiry. Registrations survive the server being killed (DURABILITY.md).

Sending updates

Each change of the writer set, an entry made or changed by a notification taken or an entry removed, is sent, at once and in the background, to each active registration of the space: POST <origin>/xrpc/com.atproto.space.notifySpaceUpdate, Content-Type: application/json, body exactly {space, token, update}: the space's URI, the registration's token, and the update:

  • for an entry made or changed, {"$type": "com.atproto.space.defs#repoWrite", repo, repoRev, data, spaceRev, prevSpaceRev}, the entry as it now is;
  • for a removal, {"$type": "com.atproto.space.defs#repoRemoved", repo, spaceRev, prevSpaceRev}.

spaceRev is the entry's new one, and prevSpaceRev the space's revision before it, absent when it had none. Each is one attempt, given up after 10 seconds; whatever the syncer answers, it is not sent again: a syncer that missed one sees a gap in the prevSpaceRevs, and catches up with listWriters. At most 1024 updates wait to be sent, server-wide; one more is given up at once, as an attempt that failed.

The same change owes each registration of the space that has lapsed, and has not been sent an expiry notice since it last lapsed, an expiry notice in place of the update: {space, token, update: {"$type": "com.atproto.space.notifySpaceUpdate#registrationExpired", spaceRev}}, the change's spaceRev. So a lapsed registration is sent one expiry notice, however many changes follow, and nothing more but a deletion notice until it is renewed. It is delivered as "Notices" says.

Sending does not depend on the authority's account status.

Deletion

When a live space is deleted (deleteSpace, SPACES-POLICIES.md), each of its registrations, lapsed ones included, is owed a deletion notice, {space, token, update: {"$type": "com.atproto.space.notifySpaceUpdate#spaceDeleted"}}, delivered as "Notices" says; and the registrations are dropped and the writer set emptied, so a space created again starts with neither, and its first update has no prevSpaceRev. Deleting a space deleted already sends nothing. The server remembers the space as deleted (SPACES-POLICIES.md), so a syncer that missed the notice learns of it as SpaceDeleted from authorizeSpaceSync.

Notices

An expiry or deletion notice is sent at once, in the background, as an update is. It is delivered when the syncer answers a 2xx, whatever the body; it failed for now as a repo update notification does ("Delivery and retries": no answer, a redirect, or one of the nine statuses); any other status refuses it.

A notice that failed for now is kept in the notice queue, which holds at most one per space, endpoint and token, with its schedule:

  • A deletion notice replaces a notice kept for its registration, and its schedule starts afresh. An expiry notice never replaces one: one that fails for now while a notice is kept for its registration is dropped.
  • The nth retry is due a delay, times a random factor from 0.5 to 1, after the attempt before it, and is sent within 10 seconds of being due: 60 seconds for the first, 600 for the second, 3600 for the third, 21600 for the fourth, and 43200 for the fifth and every one after: the first 30 to 60 seconds after the failure.
  • Before a retry is sent: a notice whose space's authority is no longer an account here is dropped; one whose authority is deactivated or taken down is not sent, and is rescheduled as a retry that failed; an expiry notice whose registration has been renewed (it is active) or dropped since, and a deletion notice whose space has been created again since, are dropped, not sent.
  • A retry delivered is taken out; refused, the notice is given up; failed for now, it is rescheduled, unless 72 hours have passed since the notice was kept: then it is given up.
  • The queue survives the server being killed (DURABILITY.md): a notice kept before is retried after, when due.

An expiry notice refused, at once or on a retry, or given up drops its registration, unless it has been renewed since. A deletion notice given up changes nothing more: its registration went with the space.

Commands

The two served methods are MCP tools and CLI commands as COMMANDS.md makes them: space_notify_repo_update (pds space notify-repo-update) and space_list_writers; each answers JSON or nothing ({}), so each has a tool, with the hints of COMMANDS.md's "Hints": listWriters read-only, and notifyRepoUpdate only adding. notifyRepoUpdate's service-auth token is the bearer token a tool or command is given (PDS_TOKEN), as for notifyCredentialsRevoked. listWriters takes only a space credential and its HTTP signature, which a tool or command cannot present: given a bearer token, it answers as XRPC answers one, 401 MissingJwt.

For other slices

  • on_commit(space, repo, rev, data) (the space repo's) is where a commit enters this page: a notification sent or taken here, at once.
  • authorize_write(space, user_did) -> bool (the policies') is asked for each notification taken, after the checks before it (step 8).
  • remove_writer(space, did) (this page's, "Removing a writer") is called by the policies' removeWriter, putMembers and removeMembers. The writer set's emptying and replacement are the policies' (deleteSpace, importSpaceMeta); exportSpaceMeta writes out the entries not removed.
  • deleteSpace hands this page the space's registrations, each owed a deletion notice, and this page drops them.
  • authorizeSpaceSync (the credentials') hands this page its notify at its steps 9 to 11 ("Registering"), and answers the space's revision.

How it is checked

harness/spaces/sync.mjs, in the spaces run (SPACES.md), before the server's restart. It plays parties of its own: two syncers, HTTP servers that answer confirmRegistration for the tokens the harness gave them, each per space, and record every request; a writer whose notifications it signs, and a space host for an authority of its own, each a did:plc in the directory with #atproto_pds at an HTTP server the harness runs, answering as a check needs and recording what it is sent. Every token and key name is drawn from the run's seed. Requests go to both sides alike and the answers are compared as PDS.md says, each side held to this page.

  1. The server as space host, the authority on each side, writers on the second alpha PDS, on the same side, and the harness's writer:
    • authorizeSpaceSync of a space with an empty writer set, with no spaceRev; listWriters of it empty;
    • registrations: each new one confirmed once, POST of JSON to /xrpc/com.atproto.space.confirmRegistration with no Authorization, its body exactly {space, token, challenge} and each challenge 26 base32 characters and new; one at an endpoint with a path and a query, confirmed at the origin; the same pair again renewed with no confirmation; the same token at the endpoint with a / after it a registration of its own; a 512-byte token, and an answer with another member besides challenge, taken; RegistrationNotConfirmed for a wrong challenge, 500, text/plain, an empty 200, a closed connection, a token the syncer did not give, and endpoints ftp://, with a user and password, and a DID, the last three sent nothing; and InvalidRequest for an endpoint that is no URI, a 513-byte token and no token;
    • nothing confirmed for a request refused before step 10: a members-only space with no token, a space never made, one whose authority is elsewhere, a credentialKey that is no key;
    • writes by a peer, by an account on the same side, and by the authority, tracked in listWriters at the rev and data the writer's host answers; a reader's write neither tracked nor sent;
    • the harness writer's notifications, each refusal above in turn, and notifications taken: a bare aud, a high-S signature, a stale or equal repoRev (no change, nothing sent), one 4 minutes ahead (6 is FutureRev), 31 bytes of data, with spaceRev sent;
    • removals: removeWriter of a writer, of it again and of an account never in the set (nothing sent), the writer added back by its next write, a member's write taken away by putMembers and one removed by removeMembers, each shown in listWriters as #repoRemoved;
    • every registration sent each change once, POST of JSON to /xrpc/com.atproto.space.notifySpaceUpdate with no Authorization, the body exactly {space, token, update}, the updates chained by prevSpaceRev from none, each listWriters entry sent at its spaceRev, the refused registrations sent nothing; and authorizeSpaceSync's spaceRev the writer set's greatest;
    • listWriters' pages, cursors and refusals;
    • a refusal dropping a registration: a member's registration in a members-only space, then the same pair with a non-member's token (SpaceNotFound), after which the space's changes are not sent to it and it is confirmed anew; a token that is no JWT with the pair (InvalidDelegationToken) leaving it;
    • deletion: each registration told once, with nothing more for a second deleteSpace; authorizeSpaceSync (no confirmation), listWriters and notifyRepoUpdate of the deleted space; the space created again with no spaceRev, sending to no one, and its registrations confirmed anew.
  2. The server as repo host, writers on each side (the server's, the first alpha's): writes into a memberspace on the second alpha PDS, tracked there at the rev and data of each commit, clearing too; and writes into the harness space host's spaces, each notification exactly as above.
  3. Retries across a crash. The harness space host answers a writer's notifications in one space 503 three times (the latest kept), 429, closes the connection, or 400, 401 or 403; a syncer answers one space's deletion notice 503, and another's, created again at once, 503; and one registration answers updates 500. The server is then killed and started again, and the harness answers 200 from then on. Within 100 seconds of the failures, on each side: the 503 space's latest notification is delivered, once, and no earlier one; the 429 and closed ones are delivered; the 400, 401 and 403 ones, the update, and the notice of the space created again are not sent again; and the other notice is delivered. Each retry must come at least 25 seconds after its failure.

The limit on registrations is held to this page alone, the reference having none (below, "Where the reference departs from this page"): on the server, in a members-only space, a member's 64 registrations, each with a token drawn from the run's seed, are each confirmed by the harness's syncer, a 65th is TooManyRegistrations with nothing sent to its endpoint, while the first is renewed; once a non-member's refusal drops one, a new one is confirmed and taken.

harness/spaces/ssrf.mjs, near the end of the run, holds the server alone to the guard ("Environment") and to what the reference does otherwise (below). With protection off, as the run starts the server, an endpoint at http://localhost:<port> is confirmed and registered; one whose host answers 302 to another that would confirm it is RegistrationNotConfirmed, the redirect's target never asked; and one whose confirmation is 600000 bytes is RegistrationNotConfirmed, while one of 500000 bytes is registered. Then the server is started again without PDS_DISABLE_SSRF_PROTECTION, and plain TCP listeners on 127.0.0.1 count connections: authorizeSpaceSync with endpoints http://localhost:<port>, https://localhost:<port> and https://127.0.0.1:<port> is RegistrationNotConfirmed, and notifyRepoUpdate with service auth from did:web:localhost%3A<port> and did:web:127.0.0.1%3A<port> is AuthenticationRequired, and no listener sees a connection; and a change of the space registered before is not sent to that http endpoint. The server is then started again as before.

The policies run's writer-set stage (SPACES-POLICIES.md) compares the server's writer set too.

Not compared:

  • A registration's lapse, and so expiry notices, their schedule and what giving one up drops, and the renewal of a lapsed registration without a confirmation: a registration lapses with its credential, 600 seconds after it is made, which the run does not wait for.
  • Retries after the first, and the 24-hour and 72-hour horizons, which would take the run that long; the random factor, beyond each retry's window.
  • What a repo host does for a writer deactivated, taken down or deleted while its notification waits, and a notice whose authority is so.
  • The limit on registrations: the reference takes any number in a space.
  • SSRF protection, redirects and the size of documents and answers: the reference runs without the guard; the server is held to this page alone, as above.
  • The bound of 1024 updates waiting, and that a request is given up after 10 seconds: they are load and time, which the run does not make.
  • A public host name resolving to a private address, which needs a name the run does not control.
  • Whether a write's answer waits for its notification (the reference's does): it is not observable but as time.

Where the reference departs from this page

  • The reference follows redirects when it confirms a registration and when it sends updates and notices (its POST, redirected by a 302, arrives as a GET with no body, so a confirmation so redirected fails all the same); the server follows none.
  • The reference reads a syncer's answer whatever its size; the server reads a confirmation up to 524288 bytes, as "Environment" says.
  • The reference has no limit on a space's registrations; the server has 64 active ones.

The harness does not compare these.