# Spaces: a repo whole, its blobs, and migration What a repo host serves of a space repo beyond its records: the whole repo as a STAR archive, the blobs its records reference, and a repo imported from another host. Who reads them: the account itself, with a session, and a syncer, with a space credential, whose reads of every repo method are on this page. The space repo itself, its commits and its record methods are in [`SPACES-REPO.md`](SPACES-REPO.md); STAR, MACs and their headers in [`SPACES.md`](SPACES.md); credentials, and how a syncer gets one from the space host with `authorizeSpaceSync {space, credentialKey}`, in [`SPACES-CREDENTIALS.md`](SPACES-CREDENTIALS.md); what every method shares in [`PDS.md`](PDS.md). The methods, all `com.atproto.space`: `getRepo`, `getBlob`, `listBlobs` and `importRepo`. Their lexicons are in `lexicons/com/atproto/space`. ## Who may read Each repo read, `getRecord`, `listRecords`, `getLatestCommit` and `listRepoOps` (`SPACES-REPO.md`), and `getRepo`, `getBlob` and `listBlobs` here, takes either: - **a session**, the account's access token, from its password or an app password, which reads only the account's own repo: another `repo` is `400 RepoNotFound`, as `SPACES-REPO.md` says; or - **a space credential**, presented as `SPACES-CREDENTIALS.md` ("Presenting a credential") says, whose audience is the `repo` asked for. It reads any account's repo in the credential's space. With a credential, a read checks, in this order, and answers the first that fails: 1. its parameters against the lexicon, `400 InvalidRequest`; 2. the credential (`verify_space_credential`, `SPACES-CREDENTIALS.md`, "Verifying a credential"), with the `repo` as its audience: each `401` its table names, a wrong audience `401 BadSpaceAudience`; 3. the credential's space, which must be the `space` asked for: else `400 InvalidCredential`; 4. the `repo`: a DID no account here has is `400 RepoNotFound`; an account taken down is `400 RepoTakendown`, and one deactivated `400 RepoDeactivated` (below); 5. `Accept-Signature`, which a credential's read of a method that answers a commit or a record must send, asking for a MAC of the method's profile (`SPACES-REPO.md`, "MACs on request"): `atproto-space-record` for `getRecord`, `atproto-space-commit` for `getLatestCommit`, `listRepoOps` and `getRepo`. Without it, with one that asks for no MAC of that profile (an empty one, or one asking only for the other profile's), or with one `SPACES.md` refuses: `400 InvalidSpaceMac`. `listRecords`, `listBlobs` and `getBlob` take no MAC, and ignore the header, whatever it is; 6. then the method, as with a session, the MAC as `SPACES-REPO.md` says. So a repo never written answers a credential as it answers its account: `getRecord` `RecordNotFound`, `listRecords` `{records: []}`, `getLatestCommit`, `listRepoOps` and `getRepo` `RepoNotFound`, `listBlobs` `{cids: []}`, `getBlob` `BlobNotFound`. A credential is refused by the methods that take only a session, `listSpaces` and every write, as any scheme they do not take: `400 InvalidToken`. ## getRepo `getRepo ?space&repo` answers the repo whole: `200`, `Content-Type: application/vnd.dasl.star`, the lexicon's output encoding, whatever the request's `Accept`, and as the body the repo's STAR archive with the profile `atproto-space-repo`, as `SPACES.md` defines it: - its metadata the repo's commit, `{$type: "atproto-space-repo", ver: 1, repo, space, rev, data}`, as `getLatestCommit` answers it, encoded as DRISL, so its keys in DRISL's order; - then each record, its key the bytes of `{collection}/{rkey}` and its value the record's DRISL bytes, those its CID names, in strict ascending order of the keys; a repo with no records, as after `clearRecords`, has no pairs. So the archive of a commit and its records is one string of bytes, the one `spec.mjs`'s writer makes of them. `Accept-Signature` is taken as by `getLatestCommit` (`SPACES-REPO.md`, "MACs on request"), for the profile `atproto-space-commit`: with one that asks for a MAC, the answer has `Atproto-Space-Commit` of the archive's commit, `Signature-Input` and `Signature`, and `Access-Control-Expose-Headers` naming the three; one `SPACES.md` refuses is `400 InvalidSpaceMac`; every `200` has `Vary` naming `Accept-Signature`. A repo never written is `400 RepoNotFound`, after the header is checked. It is a query: `POST` is `400 InvalidRequest`. ## Blobs in space records A space record references blobs as a public record does: every blob ref in it is one, by the rules of `BLOBS.md` ("Blob refs in records"), and a blob an account holds is one store for both. So, in a space write: - each ref is checked last, after everything `SPACES-REPO.md` says a write checks, the space (`SpaceNotFound`) and the records' existence (`RecordAlreadyExists`, `RecordNotFound`) included, against the blobs the account held before the write: a blob it does not hold, another account's too, is `400 BlobNotFound`; one no record, public or in a space, referenced before must match the ref (`400 InvalidMimeType`, then `400 InvalidSize`); one some record referenced is not checked; - a blob is held while any record of the account references it, public or in any of its space repos, each record counted once; a write that deletes or replaces the last record that references it, `clearRecords` included, and puts none in its place, stops it being held, and a ref to it is `BlobNotFound` until it is uploaded again. The public methods count only public records: a blob that only space records reference is not served by `com.atproto.sync.getBlob` (`400`, as a blob no record references, `BLOBS.md`) and not listed by `com.atproto.sync.listBlobs`, so a blob behind a space's perimeter is read only through it. ## getBlob `getBlob ?space&repo&cid` answers a blob the repo's records in that space reference: - a `cid` that does not parse is `400 InvalidRequest`, with the other parameters; - a CID no record of the repo in that space references, whatever else the account holds (an upload, a blob of a public record or of another space), or no repo, is `400 BlobNotFound`; - a blob a record references whose bytes the account does not hold, as after `importRepo` (below), is `400 InvalidRequest`. Otherwise it answers `200` with the bytes, and the headers `BLOBS.md`'s `getBlob` answers: `Content-Type` the stored `mimeType` (compared up to its first `;`), `Content-Length`, `X-Content-Type-Options: nosniff`, `Content-Disposition: attachment; filename=""` and `Content-Security-Policy: default-src 'none'; sandbox`. ## listBlobs `listBlobs ?space&repo&limit?&cursor?` answers `{cids, cursor?}`: the CIDs of the blobs the repo's records in that space reference, each once, in ascending order of the CID string, at most `limit` (1 to 1000, 500 by default), whether or not the account holds their bytes. - With `cursor`, only CIDs greater than it, comparing strings; it need not be a CID, and none is refused. - A `since`, or any other parameter the lexicon does not name, is ignored, whatever its value: every blob is listed. - `cursor` in the answer is the page's last CID when the page is full (`limit` CIDs), and is left out otherwise. No repo answers `{cids: []}`. ## importRepo `importRepo` takes a space repo's STAR archive, as `getRepo` answers one, as the raw request body, and replaces the caller's repo in the space it names with it. It checks, in this order, and answers the first error: 1. The token: a session made with the account's password. None is `401 AuthMissing`; an app password's session, a refresh token, a credential or a token the server did not give out is `400 InvalidToken`. 2. The `Content-Type`: its media type, up to the first `;`, must be `application/vnd.dasl.star`, the lexicon's input encoding, whatever parameters follow; none, or another, `application/octet-stream` included, is `400 InvalidRequest`. 3. The size: a body of more than `PDS_BLOB_UPLOAD_LIMIT` bytes (`LIMITS.md`), 5242880 (5 MiB) unless it is set, is `413 PayloadTooLarge`, the name `uploadBlob`'s is; one of exactly the limit is taken. 4. The archive, read as `SPACES.md`'s reader reads one, with no MAC and no space or repo expected: any fault is `400 InvalidRepoFile` (no bytes, a bad magic, bytes after the end, a limit passed, metadata that is not an `atproto-space-repo` commit with `ver` `1`, its `repo` not a DID, its `space` not a `space-ref`, its `rev` not a TID, a key that is not a record path, keys out of order or repeated, a value that is not a DRISL map, the records' LtHash not `data`). Metadata fields beyond the commit's are ignored. 5. The commit's `repo`, which must be the caller's DID: else `400 RepoMismatch`. 6. The space, as a write's (`SPACES-REPO.md`): one of the caller's own must exist here, imported by `importSpaceMeta` or created (the policies part), else `400 SpaceNotFound`. Another authority's is not checked. 7. Each record's bytes, which must be its canonical DRISL encoding: else `400 InvalidRepoFile`. Then the caller's repo in that space becomes the archive's, all at once: exactly its records, as their bytes, the published collections too; an empty operation log; a new commit, its `rev` later than the repo's last (a TID of the time, as a write's), its `data` the archive's. A repo not there before comes to be, as with a first write (`listSpaces` lists its space). It answers `200` with no body. An imported record's blob refs are taken without being checked: each blob is referenced, and listed by `listBlobs`, whether the account holds it or not. `getBlob` of one it does not hold is `400 InvalidRequest` (above), until it is uploaded: `uploadBlob` of those bytes makes it held, as a record already references it, and `getBlob` serves it, with the `mimeType` of the upload. A blob the account held once and holds no longer, as after the last record that referenced it was deleted, is one it does not hold: the import's reference to it does not hold it again, and the `mimeType` it was uploaded with before is forgotten. An import is not a write the space host hears of: it calls no `on_commit` (`SPACES-REPO.md`), and the writer set keeps the `rev` it had. An account deactivated may import; one taken down may not (below). ## Takedown and deactivation For the account whose repo is read, deactivated (`com.atproto.server.deactivateAccount`) or taken down by the server's operator: | Read | Deactivated | Taken down | | --- | --- | --- | | with the account's session | answered as for an active account | its `allowTakendown` session (`createSession` with `allowTakendown: true`): `getRepo`, `listBlobs` and `listSpaces` answered as for an active account; `getBlob` not said (below); every other read, every write and `importRepo` `400 InvalidToken` | | with a credential | `400 RepoDeactivated` | `400 RepoTakendown` | An account's state is `LIFECYCLE.md`'s, taken down before deactivated. The reference's admin token (`Basic`) also reads these, and its sessions from before a takedown keep reading; neither is compared. Nor is `getBlob` with an `allowTakendown` session. *Note:* the reference answered it `400 BlobNotFound` in one run and served the blob in another. ## Durability As `DURABILITY.md` says of the rest: an import the server answered, and the blobs space records reference, survive it being killed; after a restart, `getRepo` answers the same bytes, `getBlob` the same blobs and `listBlobs` the same CIDs. ## Commands As `COMMANDS.md` makes them, group `space`: `listBlobs` answers JSON, so it is the tool `space_list_blobs` and the command `pds space list-blobs`. `getRepo` and `getBlob` answer bytes, so they have no tool, and their commands, `pds space get-repo` and `pds space get-blob`, write the bytes to standard output, as `pds sync get-repo` does. `importRepo` takes bytes, so it has no tool, and `pds space import-repo` sends standard input as the body, with `Content-Type: application/vnd.dasl.star`, the lexicon's input encoding; it takes no flags. None of them sends `Accept-Signature`. ## For other parts - **`on_commit`** is not called by `importRepo` (above). - **`importSpaceMeta`** (the policies part) must come before `importRepo` of the authority's own repo in a space it imports, as step 6 says. - **Takedown and deactivation** reach the repo reads through step 4 above and the table; the session side is the sessions' (`allowTakendown`). ## How it is checked In the spaces run (`SPACES.md`), beside the space repo's steps (`SPACES-REPO.md`), with the server and the alpha's first PDS each the repo host of their side's accounts, and the authorities' spaces on the alpha's second PDS, the **peer**, made with `publicRead`, so that the harness, as a syncer, gets credentials for them from the peer's `authorizeSpaceSync`, with no delegation token: - **Blobs.** Accounts upload blobs (`spec-pds blob ` bodies) and write space records referencing them, the uploads as answered and refs that may be wrong (another `mimeType` or `size`, a CID never uploaded, a blob no longer held); delete and replace them; and reference some from public records too. `getBlob` and `listBlobs` (paged, sometimes with a `since`, the side's own revs or no TID, which must be ignored) are compared, and the harness holds each side to the blobs its own model says the account holds and each repo references; the public `getBlob` of a blob only space records reference must be `400` and the public `listBlobs` must leave it out. - **getRepo** of every repo, by session and by credential, with `Accept-Signature` drawn as the space repo's reads draw it (`SPACES-REPO.md`, "How it is checked"). On each side the answer must be `application/vnd.dasl.star`, the archive must parse with `spec.mjs`'s reader, its MAC verify from the harness's key share and its `Atproto-Space-Commit` name its commit, with the response headers `SPACES-REPO.md` holds, its commit be `getLatestCommit`'s and its records the model's, and its bytes be exactly what `spec.mjs`'s writer makes of that commit and those records; a refused header, or a credential's read asking for no MAC, must be `400 InvalidSpaceMac`. Across the sides, the two archives must be the same but for the account's DID, the `rev`, and record keys each side made: the same records, byte for byte. - **Credential reads** of every repo method, with credentials from the peer: the account's own repo, another account's on the same host, one never written, a DID no account here has, a wrong audience, a credential of another space; with a good `Accept-Signature` mostly, and otherwise none, or one of `interop/spaces/mac.json`'s cases, where the method answers a MAC each side must answer `400 InvalidSpaceMac` unless a MAC is asked for, and `listRecords` must ignore the header and answer no MAC headers. - **importRepo**: each side's own `getRepo` archive imported back, archives the harness writes with `spec.mjs` (records with blob refs to blobs held, not held, held no longer and never uploaded, published collections, an empty archive, into a space with no repo yet), sent with `application/vnd.dasl.star`, with or without parameters, and archives with one thing wrong for each step above, each side held to the error that step names: a metadata field of another format, another account's `repo` (with a bad LtHash too, `InvalidRepoFile` first), a value not canonical DRISL into a space of the caller's own never created (`SpaceNotFound` first); app password and refresh tokens; a JSON or `application/octet-stream` `Content-Type`, or none; bodies of 5242880 and 5242881 bytes. After each, the repo is read whole, and `listBlobs`, `getBlob` and an upload of a blob it references are compared. - **A restart** of the server: every repo's `getRepo`, and the blobs, read again. - **Takedown and deactivation**: the reference alone, as above, until the server can be put in either state. Not compared: a `getBlob` answer's other headers, as `BLOBS.md` says; the admin token; sessions from before a takedown; the blobs an import replaced (the reference forgets their refs without dropping the blobs); the writer set after an import (the notifications part's run reads it); and `importRepo` with an OAuth token (`OAUTH-SCOPES.md`, "What each method needs"), which no run calls.