~ knotless Search/
↑↓ move↵ open⇧↵ all results
0
0
456 lines · 22.7 KB Raw History

Spaces

atproto spaces are permissioned data: records with an access perimeter, kept beside the public repository rather than in it, and read only with a credential the space's authority issues. This page is where spaces start: who takes part, the keys, what this server is in each part, and the building blocks every later part rests on, each specified exactly enough to build from and held to test vectors. What every method shares, and how the server is checked, is in PDS.md.

Spaces are an alpha. The protocol is spaces/proposal.md, the atproto proposal 0016 as vendored (spaces/README.md says which revision, and the other sources and their licences). Where this page and the proposal say the same thing, this page is the one to build from; where this page is silent, the proposal is. The reference for spaces is the pinned spaces alpha of the reference PDS, which the spaces run starts (below), not the latest reference the other pages compare with.

Addressing

A space is the triple of its authority, a DID; its type, an NSID; and its key (skey), a string with the syntax of a record key (atproto/record-key.md) and at most 512 bytes long. It is written as a URI, and a record in it as a longer one:

at://{authority}/space/{type}/{skey}
at://{authority}/space/{type}/{skey}/{author}/{collection}/{rkey}

space is a literal segment, which no collection can be, as an NSID always has dots. A string in the format space-ref is a space's URI and no more: exactly the first form, with each part valid as said.

Who takes part

RoleIsThis server
authoran account writing records into a space, each into its own space repo: one per account and spaceeach of its accounts
repo hostholds an author's space repos, serves their writes and reads, and tells the space host of each writethe repo host of its accounts' space repos
space authoritythe DID at the root of a space: its #atproto key signs the space's credentials, and it decides who may read the space and whose writes it tracksan account of its own, for the spaces that account creates
space hostthe authority's PDS (its #atproto_pds): authorizes syncers, issuing their credentials, keeps the writer set, sequences the space's updates and notifies the syncers registered for themthe space host of every space whose authority is one of its accounts
user's PDSmints the delegation token an app trades for a credentialthe PDS of its accounts
synceran app keeping a copy of a space: pulls repos from their hosts, is told of the space's updates by the space host at an endpoint of its ownnone: the harness plays it

The parts built on the building blocks below each have a page of their own:

  • the space repo: the com.atproto.space record writes and reads, its commit and operation log (SPACES-REPO.md);
  • credentials: delegation tokens, space credentials, revocation, in SPACES-CREDENTIALS.md, which verifies credentials for the methods that take them;
  • policies and the space lifecycle: creating, updating and deleting spaces, and member lists: SPACES-POLICIES.md;
  • full-state sync: a repo as a STAR archive (getRepo), blobs, and importing a repo: SPACES-EXPORT.md;
  • write notifications, the writer set, syncers' registrations and the space updates sent them: SPACES-SYNC.md.

Their lexicons join lexicons/ when they are served; spaces/lexicons/ has those the server calls on others and does not serve. com.atproto.space.defs (the commit, and the updates a writer set holds) is in lexicons/, with the space repo's methods.

Keys

Each account has two keys, both kept by the server for the account alone:

  • #atproto, the signing key ACCOUNTS.md gives it. For spaces, it is also the key that signs the delegation tokens the account's PDS mints, the credentials of the spaces the account is the authority of, and the service auth it sends as their authority (later parts).
  • #atproto_kem, a P-256 key of its own, below. It signs nothing: a syncer and the repo host agree a shared secret with it, under which the repo host authenticates what it serves with MACs.

The KEM key

createAccount makes the account a new P-256 keypair, its KEM key, with the signing key (ACCOUNTS.md), and keeps the private key as it keeps the signing key: it survives restarts (DURABILITY.md), and no other account's data holds it. Its public key, as a did:key (below), is:

  • verificationMethods.atproto_kem in the DID's genesis operation, beside verificationMethods.atproto;
  • verificationMethods.atproto_kem in what getRecommendedDidCredentials answers (IDENTITY.md);
  • in the DID document the directory serves, as the directory renders a verification method named atproto_kem. The proposal's form is a Multikey with id <did>#atproto_kem. The harness's directory renders it in the legacy form atproto/did.md describes: id #atproto_kem, type EcdsaSecp256r1VerificationKey2019, the uncompressed point, and https://w3id.org/security/suites/ecdsa-2019/v1 added to @context. The server serves what the directory serves, as it does for the signing key.

The KEM key does not change: an update operation the server posts, as for a handle change, keeps it, as it keeps every field but the one it changes.

describeServer

describeServer answers spacesAlpha: true besides what ACCOUNTS.md and BLOBS.md list: the reference's flag that it serves the spaces alpha.

Note: describeServer's lexicon does not name it; the reference answers it all the same, and calls it temporary.

The building blocks

Each is held to a file of test vectors in interop/spaces/ (below), and STAR to the published fixtures in interop/star/ as well. Hex in a vector is lower case; a vector's comment says what the case is about.

P-256 did:keys

A P-256 public key as a string is a did:key, as atproto/cryptography.md defines it: did:key:z, then base58btc of the bytes 0x80 0x24 (the multicodec p256-pub) followed by the 33-byte compressed point (0x02 or 0x03, then x). Every P-256 did:key starts did:key:zDn. A key share or credential key that is not a did:key, does not parse, is of another curve (such as secp256k1's 0xE7 0x01), is not 33 bytes of compressed point, or names an x with no point on P-256, is no P-256 key.

The uncompressed form of a point is 65 bytes: 0x04, x, y, each 32 bytes big-endian.

The shared secret

A syncer that wants a MAC sends a key share: a fresh P-256 public key, as a did:key. The repo host makes the shared secret from it and the author's KEM private key d, as HPKE's secret export (RFC 9180 §5.3, spaces/rfc9180.txt) in base mode, with the ciphersuite DHKEM(P-256, HKDF-SHA256), HKDF-SHA256 and the Export-only AEAD, the key share as enc, info the ASCII bytes atproto-mac, an empty exporter context and a length of 32. Spelled out, ‖ joining bytes, quoted parts ASCII, and u16(n) the two bytes of n big-endian:

  1. P is the key share's point (an error if it is no P-256 key, as above).
  2. dh is the 32-byte big-endian x of the point d·P (ECDH).
  3. enc is P uncompressed, and author is the author's KEM public key uncompressed, 65 bytes each.
  4. Extract(salt, ikm) is HMAC-SHA256 of ikm keyed with salt (an empty salt an empty key), and Expand(prk, info) is HMAC-SHA256 of info ‖ 0x01 keyed with prk: HKDF's extract, and its expand to 32 bytes.
  5. With kem the bytes "KEM" ‖ u16(0x0010) and suite the bytes "HPKE" ‖ u16(0x0010) ‖ u16(0x0001) ‖ u16(0xFFFF), LabeledExtract(id, salt, label, ikm) is Extract(salt, "HPKE-v1" ‖ id ‖ label ‖ ikm), and LabeledExpand(id, prk, label, info) is Expand(prk, u16(32) ‖ "HPKE-v1" ‖ id ‖ label ‖ info).
  6. kem_secret is LabeledExpand(kem, LabeledExtract(kem, "", "eae_prk", dh), "shared_secret", enc ‖ author): DHKEM's shared secret (RFC 9180 §4.1).
  7. context is 0x00 ‖ LabeledExtract(suite, "", "psk_id_hash", "") ‖ LabeledExtract(suite, "", "info_hash", "atproto-mac"), and exporter_secret is LabeledExpand(suite, LabeledExtract(suite, kem_secret, "secret", ""), "exp", context): the key schedule, with no pre-shared key.
  8. The shared secret is LabeledExpand(suite, exporter_secret, "sec", "").

The syncer makes the same secret from its private key and the author's public one. interop/spaces/kem.json: each valid case gives dh, kem_secret and shared_secret for its author key, key share and info; the first is RFC 9180's test vector A.3.1, whose kem_secret is the RFC's shared_secret, and the second is HPKE's own test vector for this ciphersuite, with its info. Each invalid key share must be refused.

The set hash, records and commits

A space repo's records are summed by LtHash:

  • The state is 2048 bytes, read as 1024 unsigned 16-bit lanes, little endian; it starts all zeros.
  • An element is a string. To add it, expand its UTF-8 bytes to 2048 bytes with BLAKE3 in its extendable-output mode (the plain hash, unkeyed, read to 2048 bytes), read those as 1024 little-endian lanes, and add each to the state's lane of the same index, modulo 65536. To remove it, subtract instead, modulo 65536.
  • The digest is SHA-256 of the 2048 bytes of the state.

So the state depends only on which elements are in it, not the order they came in, and the empty digest is e5a00aa9991ac8a5ee3109844d84a55583bd20572ad3ffcd42792f3c36b183ad. interop/spaces/lthash.json gives digests for elements added and removed, and the whole state after adding a.

A record in a space repo is the same DRISL (DAG-CBOR) value a public record is (REPO.md), with the same CID: version 1, codec 0x71, a SHA-256 multihash of its bytes, in base32. Its element is {collection}/{rkey}/{cid}, the CID as a string. A space repo's commit (com.atproto.space.defs#commit) is {ver: 1, repo, space, rev, data}: repo the author's DID, space the space's URI, rev a TID, and data the LtHash digest of the elements of every record the repo holds, 32 bytes. It has no signature.

HTTP Message Signatures

Both a request made with a space credential and a repo host's MAC are HTTP Message Signatures, as RFC 9421 (spaces/rfc9421.txt) defines them, with the choices below. What they share:

  • Signature-Input, Signature and Accept-Signature are RFC 9651 Dictionaries. A signature's label is its member's name, and carries no meaning: a signature is found by its tag parameter. The members tagged T are those whose value is an Inner List with a parameter tag that is the String T; any other member (another tag, no tag, a tag that is a Token, a value that is no Inner List) is passed over. A member, or a parameter, given more than once counts as its last, as RFC 9651 reads a Dictionary.
  • The signature base is one line per covered component, in the Inner List's order, then the line "@signature-params": and the Inner List with its parameters, serialized as RFC 9651 §4.1.1.1 serializes one, so ;sf=?1 is written ;sf; the lines are joined by a line feed, with none after the last, in UTF-8. A component's line is the component as serialized there, : , and its value.
  • alg, if a signature has it, must be the String the signature's kind names below; keyid, if it has it, must be a String, the key it is checked against.

A credential's signature. A request made with a space credential carries it in Authorization, and signs it with the key the credential is bound to (its cnf.kid, a P-256 did:key), as in:

Authorization: Atproto-Space-Credential <credential>
Atproto-Space-Audience: <a DID>
Signature-Input: sig=("authorization" "atproto-space-audience");tag="atproto-space-credential"
Signature: sig=:<64 bytes, base64>:

To verify one against the credential's key:

  1. Signature-Input must have exactly one member tagged atproto-space-credential, and Signature a member of the same label. In Signature-Input it must be an Inner List of exactly the Strings "authorization" and "atproto-space-audience", in that order, with no parameters on them. alg, if given, must be ecdsa-p256-sha256, and keyid, if given, the credential's key; other parameters are allowed (they are signed, in the base). In Signature the member must be a Byte Sequence with no parameters, of 64 bytes; base64 padding is optional.

  2. The base's two component lines are

    "authorization": <the Authorization value>
    "atproto-space-audience": <the Atproto-Space-Audience value>
    

    each value as received, less leading and trailing whitespace: Atproto-Space-Credential keeps the case it was sent in.

  3. The 64 bytes are r ‖ s, each 32 bytes big-endian. The signature must verify as ECDSA over P-256 with SHA-256 of the base, against the credential's key, which must be a P-256 did:key. Both low-S and high-S signatures verify; a DER signature is refused.

interop/spaces/http-signature.json: requests' headers, each valid or not against key_id, with the exact signature_base of each valid one. These hold the signature alone; the credential itself, and that the audience is the one the request needs, are the credentials' part (later).

MACs

A repo host authenticates a commit or a record to a syncer with a MAC: an HTTP Message Signature with the algorithm hmac-sha256 (RFC 9421 §3.3.3), keyed with their shared secret, over a response header that describes what it answers. There are two profiles, each a tag and the header it covers:

ProfileForCovers
atproto-space-commita commitAtproto-Space-Commit
atproto-space-recorda recordAtproto-Space-Record

Asking. A syncer asks for a MAC with the request header Accept-Signature (RFC 9421 §5.1), as in:

Accept-Signature: mac=("atproto-space-commit";sf);keyid="did:key:zDn...";tag="atproto-space-commit"

The repo host reads it for the one profile the method answers:

  1. No header, or one with no member tagged with the profile, asks for no MAC; members tagged otherwise are passed over.
  2. A header that is not a Dictionary is refused, and so is one with more than one member tagged with the profile.
  3. That member's Inner List must be exactly one item, the String naming the profile's header in lower case ("atproto-space-commit" or "atproto-space-record"), with exactly the one parameter sf, the Boolean true. Its parameters must have keyid, a String, the key share, and may have alg, which must be the String hmac-sha256; any other parameter is refused, created and nonce among them.
  4. The key share must be a P-256 key (above), or the request is refused.

Answering. The response has three headers more, and Vary naming Accept-Signature:

Atproto-Space-Commit: ver=1, repo="did:plc:...", space="at://...", rev="3mwcj3b6uak2f", data=:<the 32 bytes, base64 with padding>:
Signature-Input: mac=("atproto-space-commit";sf);keyid="did:key:zDn...";tag="atproto-space-commit"
Signature: mac=:<the 32 bytes of the MAC, base64 with padding>:
  • The covered header is a Dictionary, serialized as RFC 9651 serializes one: for a commit, its fields in the order ver (an Integer), repo, space and rev (Strings) and data (a Byte Sequence of its 32 bytes); for a record, uri, its full space URI, and cid, its CID as a string, both Strings, as in uri="at://...", cid="bafyrei...".

  • Signature-Input's one member has the request's label, and its value is the request's Inner List, serialized as above.

  • Signature's one member has the same label, and its value is the MAC as a Byte Sequence: HMAC-SHA256, keyed with the shared secret of the key share and the author's KEM key, of the signature base, whose one component line is the profile's, as in

    "atproto-space-commit";sf: ver=1, repo="did:plc:...", space="at://...", rev="3mwcj3b6uak2f", data=:...:
    "@signature-params": ("atproto-space-commit";sf);keyid="did:key:zDn...";tag="atproto-space-commit"
    

Checking, as a syncer does: Signature-Input has exactly one member tagged with the profile, an Inner List of exactly the one item above, with keyid, if given, the key share asked with and alg, if given, hmac-sha256 (other parameters are allowed: they are in the base); Signature has a member of its label, a Byte Sequence of 32 bytes with no parameters; the covered header, read as a Dictionary and serialized again (the sf flag), gives the base; the MAC matches it; and the header names what the answer holds: exactly the profile's fields, none with parameters, each of its type and equal to the answer's.

Which methods take Accept-Signature, and the error they answer when it is refused, are the space repo's (a later part).

interop/spaces/mac.json: for each commit and record, the shared secret, the request, the covered header, the signature base, the mac and the response headers; and accept_signature, request headers each with the profile it is read for and what it yields: the label, key share and serialized Inner List, null where no MAC is asked, or "refused".

STAR

A space repo travels as a STAR archive, as spaces/star.md defines STAR: the magic bytes 0x2A 0x6C 0x01 0x00; the metadata, a DRISL map with a String $type, after its length; then each pair as 0x01, the key's length, the key, the value's length and the value; then 0x00, and nothing after. Lengths are unsigned varints, minimally encoded, at most 9 bytes. Any fault refuses the whole archive. interop/star/ has STAR's published fixtures: each valid one must parse to its metadata and pairs, and each invalid one must be refused.

The profile atproto-space-repo is a space repo:

  • The metadata is the repo's commit, with $type: {$type: "atproto-space-repo", ver: 1, repo, space, rev, data}, data as DRISL bytes. A writer writes these six fields; a reader ignores any others STAR allows.
  • A pair is a record: its key the UTF-8 bytes of {collection}/{rkey}, its value the record's DRISL bytes, those its CID names. Pairs are in strict ascending order of the key bytes, so com.example.x.y/1 comes before com.example.x/10, as . is 0x2E and / is 0x2F; an empty repo has none. So the archive of a commit and its records is one string of bytes.
  • Limits: the metadata is at most 65536 bytes, a key 4096, a value 1048576.

A reader, given the space and repo it asked for, the key share it asked with, the shared secret and the response's headers, refuses the whole archive on any of these, and otherwise takes it as the repo:

  • a fault of STAR's, a limit passed, or metadata that is not canonical DRISL;
  • a $type other than atproto-space-repo, a ver other than 1, or a commit field missing or of another type;
  • a space or repo other than those asked for;
  • a commit MAC that does not check (above) against the response's headers, or an Atproto-Space-Commit header that names another commit than the metadata;
  • a key that is not a record path (an NSID, /, a record key), or one not after the key before it, duplicates included;
  • an LtHash digest of the records' elements, each CID computed from the value's bytes, other than data.

interop/spaces/repo-star.json: valid archives, each exactly the archive of its commit and records, to serialize and to read; one a reader must take though no writer here makes it; and archives to refuse, with their space, repo, key share, secret and response headers.

Test vectors

FileHolds
interop/spaces/kem.jsonthe shared secret
interop/spaces/lthash.jsonLtHash
interop/spaces/mac.jsoncommit and record MACs, and Accept-Signature read
interop/spaces/http-signature.jsona credential's HTTP Message Signature
interop/spaces/repo-star.jsonspace repos as STAR archives
interop/star/*.jsonSTAR's own fixtures, published with STAR

Each file says in its description how to read its cases. The server's own tests (cargo test) hold each building block to every case of its file. The spaces run holds every case to the reference's own library and to the harness's code written from this page (below), so each vector is one the reference and this page agree on; the one case the reference's library cannot take, kem.json's second, whose info is not atproto-mac, is held to the harness's code alone.

How it is checked

harness/spaces/compare.mjs is the spaces run. It starts the pinned spaces alpha (harness/spaces/package.json; pin-alpha.sh pins another): its PLC directory and its PDS, the reference here. It starts the server beside it on that directory, in the environment PDS.md lists, with the alpha's recovery key as PDS_RECOVERY_DID_KEY, and as PDS_LEXICON_AUTHORITY_DID the DID of the run's lexicon authority, a PDS of its own on that directory publishing space type declarations, which the alpha resolves from too (OAUTH-SCOPES.md, "How it is checked"):

cd harness/spaces && npm ci          # once
node compare.mjs --seed 1 [--accounts 3] -- <command that starts the server>

First it checks every vector, as above, and stops at the first that either side disagrees with. Then, on both sides, with each account's DID named alike as PDS.md says:

  • createAccount of a few accounts: the answers are compared, with keys, tokens and URLs taken out. Each genesis operation in the directory must be what ACCOUNTS.md says, with verificationMethods.atproto_kem a P-256 did:key of a point on the curve, other than the signing key; the answer's didDoc must be the directory's document; and that document must have the #atproto_kem verification method, its key the operation's.
  • getRecommendedDidCredentials of each: compared with keys and URLs taken out; on each side verificationMethods.atproto_kem must be the genesis operation's.
  • describeServer: did, availableUserDomains, inviteCodeRequired and spacesAlpha compared; spacesAlpha must be true on each side.
  • updateHandle of most of them: the operation it posts must keep every verificationMethods entry, and getRecommendedDidCredentials still name the same KEM key.
  • A restart: the server is killed and started again on its data, and every account's getRecommendedDidCredentials must name the KEM key it had, with no new operation in the directory.

The reference is held to each check too, which checks the check. The other runs, against the latest reference, hold what the KEM key changes to this page instead (PDS.md).

That the server makes the shared secret, MACs, signatures and archives the vectors hold, over HTTP, against a syncer, each later part checks through its own methods, in this run.