~ knotless Search/
↑↓ move↵ open⇧↵ all results
0
0
456 lines · 22.7 KB Raw History
# Spaces

atproto spaces are permissioned data: records with an access perimeter,
kept beside the public repository rather than in it, and read only with a
credential the space's authority issues. This page is where spaces start:
who takes part, the keys, what this server is in each part, and the
building blocks every later part rests on, each specified exactly enough
to build from and held to test vectors. What every method shares, and how
the server is checked, is in [`PDS.md`](PDS.md).

Spaces are an alpha. The protocol is `spaces/proposal.md`, the atproto
proposal 0016 as vendored (`spaces/README.md` says which revision, and the
other sources and their licences). Where this page and the proposal say
the same thing, this page is the one to build from; where this page is
silent, the proposal is. The reference for spaces is the pinned spaces
alpha of the reference PDS, which the spaces run starts (below), not the
`latest` reference the other pages compare with.

## Addressing

A space is the triple of its **authority**, a DID; its **type**, an NSID;
and its **key** (`skey`), a string with the syntax of a record key
(`atproto/record-key.md`) and at most 512 bytes long. It is written as a
URI, and a record in it as a longer one:

```
at://{authority}/space/{type}/{skey}
at://{authority}/space/{type}/{skey}/{author}/{collection}/{rkey}
```

`space` is a literal segment, which no collection can be, as an NSID always
has dots. A string in the format **`space-ref`** is a space's URI and no
more: exactly the first form, with each part valid as said.

## Who takes part

| Role | Is | This server |
| --- | --- | --- |
| author | an account writing records into a space, each into its own **space repo**: one per account and space | each of its accounts |
| repo host | holds an author's space repos, serves their writes and reads, and tells the space host of each write | the repo host of its accounts' space repos |
| space authority | the DID at the root of a space: its `#atproto` key signs the space's credentials, and it decides who may read the space and whose writes it tracks | an account of its own, for the spaces that account creates |
| space host | the authority's PDS (its `#atproto_pds`): authorizes syncers, issuing their credentials, keeps the **writer set**, sequences the space's updates and notifies the syncers registered for them | the space host of every space whose authority is one of its accounts |
| user's PDS | mints the delegation token an app trades for a credential | the PDS of its accounts |
| syncer | an app keeping a copy of a space: pulls repos from their hosts, is told of the space's updates by the space host at an endpoint of its own | none: the harness plays it |

The parts built on the building blocks below each have a page of their
own:

- the space repo: the `com.atproto.space` record writes and reads, its
  commit and operation log ([`SPACES-REPO.md`](SPACES-REPO.md));
- credentials: delegation tokens, space credentials, revocation, in
  [`SPACES-CREDENTIALS.md`](SPACES-CREDENTIALS.md), which verifies
  credentials for the methods that take them;
- policies and the space lifecycle: creating, updating and deleting
  spaces, and member lists: [`SPACES-POLICIES.md`](SPACES-POLICIES.md);
- full-state sync: a repo as a STAR archive (`getRepo`), blobs, and
  importing a repo: [`SPACES-EXPORT.md`](SPACES-EXPORT.md);
- write notifications, the writer set, syncers' registrations and the
  space updates sent them: [`SPACES-SYNC.md`](SPACES-SYNC.md).

Their lexicons join `lexicons/` when they are served; `spaces/lexicons/`
has those the server calls on others and does not serve.
`com.atproto.space.defs` (the commit, and the updates a writer set holds)
is in `lexicons/`, with the space repo's methods.

## Keys

Each account has two keys, both kept by the server for the account alone:

- **`#atproto`**, the signing key `ACCOUNTS.md` gives it. For spaces, it
  is also the key that signs the delegation tokens the account's PDS
  mints, the credentials of the spaces the account is the authority of,
  and the service auth it sends as their authority (later parts).
- **`#atproto_kem`**, a P-256 key of its own, below. It signs nothing: a
  syncer and the repo host agree a shared secret with it, under which the
  repo host authenticates what it serves with MACs.

### The KEM key

`createAccount` makes the account a new P-256 keypair, its **KEM key**,
with the signing key (`ACCOUNTS.md`), and keeps the private key as it keeps
the signing key: it survives restarts (`DURABILITY.md`), and no other
account's data holds it. Its public key, as a `did:key` (below), is:

- `verificationMethods.atproto_kem` in the DID's genesis operation, beside
  `verificationMethods.atproto`;
- `verificationMethods.atproto_kem` in what `getRecommendedDidCredentials`
  answers (`IDENTITY.md`);
- in the DID document the directory serves, as the directory renders a
  verification method named `atproto_kem`. The proposal's form is a
  `Multikey` with id `<did>#atproto_kem`. The harness's directory renders
  it in the legacy form `atproto/did.md` describes: id `#atproto_kem`,
  type `EcdsaSecp256r1VerificationKey2019`, the uncompressed point, and
  `https://w3id.org/security/suites/ecdsa-2019/v1` added to `@context`.
  The server serves what the directory serves, as it does for the signing
  key.

The KEM key does not change: an update operation the server posts, as for
a handle change, keeps it, as it keeps every field but the one it changes.

### describeServer

`describeServer` answers `spacesAlpha: true` besides what `ACCOUNTS.md`
and `BLOBS.md` list: the reference's flag that it serves the spaces alpha.

*Note:* `describeServer`'s lexicon does not name it; the reference answers
it all the same, and calls it temporary.

## The building blocks

Each is held to a file of test vectors in `interop/spaces/` (below), and
STAR to the published fixtures in `interop/star/` as well. Hex in a vector
is lower case; a vector's `comment` says what the case is about.

### P-256 did:keys

A P-256 public key as a string is a `did:key`, as
`atproto/cryptography.md` defines it: `did:key:z`, then base58btc of the
bytes `0x80 0x24` (the multicodec `p256-pub`) followed by the 33-byte
compressed point (`0x02` or `0x03`, then `x`). Every P-256 `did:key` starts
`did:key:zDn`. A key share or credential key that is not a `did:key`, does
not parse, is of another curve (such as secp256k1's `0xE7 0x01`), is not 33
bytes of compressed point, or names an `x` with no point on P-256, is no
P-256 key.

The uncompressed form of a point is 65 bytes: `0x04`, `x`, `y`, each 32
bytes big-endian.

### The shared secret

A syncer that wants a MAC sends a **key share**: a fresh P-256 public key,
as a `did:key`. The repo host makes the shared secret from it and the
author's KEM private key `d`, as HPKE's secret export (RFC 9180 §5.3,
`spaces/rfc9180.txt`) in base mode, with the ciphersuite
DHKEM(P-256, HKDF-SHA256), HKDF-SHA256 and the Export-only AEAD, the key
share as `enc`, `info` the ASCII bytes `atproto-mac`, an empty exporter
context and a length of 32. Spelled out, `‖` joining bytes, quoted parts
ASCII, and `u16(n)` the two bytes of `n` big-endian:

1. `P` is the key share's point (an error if it is no P-256 key, as above).
2. `dh` is the 32-byte big-endian `x` of the point `d·P` (ECDH).
3. `enc` is `P` uncompressed, and `author` is the author's KEM public key
   uncompressed, 65 bytes each.
4. `Extract(salt, ikm)` is HMAC-SHA256 of `ikm` keyed with `salt` (an
   empty `salt` an empty key), and `Expand(prk, info)` is HMAC-SHA256 of
   `info ‖ 0x01` keyed with `prk`: HKDF's extract, and its expand to 32
   bytes.
5. With `kem` the bytes `"KEM" ‖ u16(0x0010)` and `suite` the bytes
   `"HPKE" ‖ u16(0x0010) ‖ u16(0x0001) ‖ u16(0xFFFF)`,
   `LabeledExtract(id, salt, label, ikm)` is
   `Extract(salt, "HPKE-v1" ‖ id ‖ label ‖ ikm)`, and
   `LabeledExpand(id, prk, label, info)` is
   `Expand(prk, u16(32) ‖ "HPKE-v1" ‖ id ‖ label ‖ info)`.
6. `kem_secret` is `LabeledExpand(kem, LabeledExtract(kem, "", "eae_prk",
   dh), "shared_secret", enc ‖ author)`: DHKEM's shared secret (RFC 9180
   §4.1).
7. `context` is `0x00 ‖ LabeledExtract(suite, "", "psk_id_hash", "") ‖
   LabeledExtract(suite, "", "info_hash", "atproto-mac")`, and
   `exporter_secret` is `LabeledExpand(suite, LabeledExtract(suite,
   kem_secret, "secret", ""), "exp", context)`: the key schedule, with no
   pre-shared key.
8. The **shared secret** is `LabeledExpand(suite, exporter_secret, "sec",
   "")`.

The syncer makes the same secret from its private key and the author's
public one. `interop/spaces/kem.json`: each valid case gives `dh`,
`kem_secret` and `shared_secret` for its author key, key share and `info`;
the first is RFC 9180's test vector A.3.1, whose `kem_secret` is the
RFC's `shared_secret`, and the second is HPKE's own test vector for this
ciphersuite, with its `info`. Each invalid key share must be refused.

### The set hash, records and commits

A space repo's records are summed by **LtHash**:

- The state is 2048 bytes, read as 1024 unsigned 16-bit lanes, little
  endian; it starts all zeros.
- An element is a string. To add it, expand its UTF-8 bytes to 2048 bytes
  with BLAKE3 in its extendable-output mode (the plain hash, unkeyed, read
  to 2048 bytes), read those as 1024 little-endian lanes, and add each to
  the state's lane of the same index, modulo 65536. To remove it,
  subtract instead, modulo 65536.
- The digest is SHA-256 of the 2048 bytes of the state.

So the state depends only on which elements are in it, not the order they
came in, and the empty digest is
`e5a00aa9991ac8a5ee3109844d84a55583bd20572ad3ffcd42792f3c36b183ad`.
`interop/spaces/lthash.json` gives digests for elements added and removed,
and the whole state after adding `a`.

A record in a space repo is the same DRISL (DAG-CBOR) value a public
record is (`REPO.md`), with the same CID: version 1, codec `0x71`, a
SHA-256 multihash of its bytes, in base32. Its **element** is
`{collection}/{rkey}/{cid}`, the CID as a string. A space repo's
**commit** (`com.atproto.space.defs#commit`) is `{ver: 1, repo, space,
rev, data}`: `repo` the author's DID, `space` the space's URI, `rev` a TID,
and `data` the LtHash digest of the elements of every record the repo
holds, 32 bytes. It has no signature.

### HTTP Message Signatures

Both a request made with a space credential and a repo host's MAC are
HTTP Message Signatures, as RFC 9421 (`spaces/rfc9421.txt`) defines them,
with the choices below. What they share:

- `Signature-Input`, `Signature` and `Accept-Signature` are RFC 9651
  Dictionaries. A signature's **label** is its member's name, and carries
  no meaning: a signature is found by its `tag` parameter. The members
  **tagged** `T` are those whose value is an Inner List with a parameter
  `tag` that is the String `T`; any other member (another tag, no tag, a
  `tag` that is a Token, a value that is no Inner List) is passed over. A
  member, or a parameter, given more than once counts as its last, as RFC
  9651 reads a Dictionary.
- The **signature base** is one line per covered component, in the Inner
  List's order, then the line `"@signature-params": ` and the Inner List
  with its parameters, serialized as RFC 9651 §4.1.1.1 serializes one, so
  `;sf=?1` is written `;sf`; the lines are joined by a line feed, with none
  after the last, in UTF-8. A component's line is the component as
  serialized there, `: `, and its value.
- `alg`, if a signature has it, must be the String the signature's kind
  names below; `keyid`, if it has it, must be a String, the key it is
  checked against.

**A credential's signature.** A request made with a space credential
carries it in `Authorization`, and signs it with the key the credential is
bound to (its `cnf.kid`, a P-256 `did:key`), as in:

```
Authorization: Atproto-Space-Credential <credential>
Atproto-Space-Audience: <a DID>
Signature-Input: sig=("authorization" "atproto-space-audience");tag="atproto-space-credential"
Signature: sig=:<64 bytes, base64>:
```

To verify one against the credential's key:

1. `Signature-Input` must have exactly one member tagged
   `atproto-space-credential`, and `Signature` a member of the same label.
   In `Signature-Input` it must be an Inner List of exactly the Strings
   `"authorization"` and `"atproto-space-audience"`, in that order, with no
   parameters on them. `alg`, if given, must be `ecdsa-p256-sha256`, and
   `keyid`, if given, the credential's key; other parameters are allowed
   (they are signed, in the base). In `Signature` the member must be a Byte
   Sequence with no parameters, of 64 bytes; base64 padding is optional.
2. The base's two component lines are

   ```
   "authorization": <the Authorization value>
   "atproto-space-audience": <the Atproto-Space-Audience value>
   ```

   each value as received, less leading and trailing whitespace:
   `Atproto-Space-Credential` keeps the case it was sent in.
3. The 64 bytes are `r ‖ s`, each 32 bytes big-endian. The signature must
   verify as ECDSA over P-256 with SHA-256 of the base, against the
   credential's key, which must be a P-256 `did:key`. Both low-S and
   high-S signatures verify; a DER signature is refused.

`interop/spaces/http-signature.json`: requests' headers, each `valid` or
not against `key_id`, with the exact `signature_base` of each valid one.
These hold the signature alone; the credential itself, and that the
audience is the one the request needs, are the credentials' part (later).

### MACs

A repo host authenticates a commit or a record to a syncer with a MAC: an
HTTP Message Signature with the algorithm `hmac-sha256` (RFC 9421 §3.3.3),
keyed with their shared secret, over a response header that describes what
it answers. There are two **profiles**, each a tag and the header it
covers:

| Profile | For | Covers |
| --- | --- | --- |
| `atproto-space-commit` | a commit | `Atproto-Space-Commit` |
| `atproto-space-record` | a record | `Atproto-Space-Record` |

**Asking.** A syncer asks for a MAC with the request header
`Accept-Signature` (RFC 9421 §5.1), as in:

```
Accept-Signature: mac=("atproto-space-commit";sf);keyid="did:key:zDn...";tag="atproto-space-commit"
```

The repo host reads it for the one profile the method answers:

1. No header, or one with no member tagged with the profile, asks for no
   MAC; members tagged otherwise are passed over.
2. A header that is not a Dictionary is refused, and so is one with more
   than one member tagged with the profile.
3. That member's Inner List must be exactly one item, the String naming the
   profile's header in lower case (`"atproto-space-commit"` or
   `"atproto-space-record"`), with exactly the one parameter `sf`, the
   Boolean true. Its parameters must have `keyid`, a String, the **key
   share**, and may have `alg`, which must be the String `hmac-sha256`; any
   other parameter is refused, `created` and `nonce` among them.
4. The key share must be a P-256 key (above), or the request is refused.

**Answering.** The response has three headers more, and `Vary` naming
`Accept-Signature`:

```
Atproto-Space-Commit: ver=1, repo="did:plc:...", space="at://...", rev="3mwcj3b6uak2f", data=:<the 32 bytes, base64 with padding>:
Signature-Input: mac=("atproto-space-commit";sf);keyid="did:key:zDn...";tag="atproto-space-commit"
Signature: mac=:<the 32 bytes of the MAC, base64 with padding>:
```

- The covered header is a Dictionary, serialized as RFC 9651 serializes
  one: for a commit, its fields in the order `ver` (an Integer), `repo`,
  `space` and `rev` (Strings) and `data` (a Byte Sequence of its 32
  bytes); for a record, `uri`, its full space URI, and `cid`, its CID as a
  string, both Strings, as in `uri="at://...", cid="bafyrei..."`.
- `Signature-Input`'s one member has the request's label, and its value is
  the request's Inner List, serialized as above.
- `Signature`'s one member has the same label, and its value is the MAC as
  a Byte Sequence: HMAC-SHA256, keyed with the shared secret of the key
  share and the author's KEM key, of the signature base, whose one
  component line is the profile's, as in

  ```
  "atproto-space-commit";sf: ver=1, repo="did:plc:...", space="at://...", rev="3mwcj3b6uak2f", data=:...:
  "@signature-params": ("atproto-space-commit";sf);keyid="did:key:zDn...";tag="atproto-space-commit"
  ```

**Checking**, as a syncer does: `Signature-Input` has exactly one member
tagged with the profile, an Inner List of exactly the one item above, with
`keyid`, if given, the key share asked with and `alg`, if given,
`hmac-sha256` (other parameters are allowed: they are in the base);
`Signature` has a member of its label, a Byte Sequence of 32 bytes with no
parameters; the covered header, read as a Dictionary and serialized again
(the `sf` flag), gives the base; the MAC matches it; and the header names
what the answer holds: exactly the profile's fields, none with parameters,
each of its type and equal to the answer's.

Which methods take `Accept-Signature`, and the error they answer when it is
refused, are the space repo's (a later part).

`interop/spaces/mac.json`: for each commit and record, the shared secret,
the request, the covered header, the signature `base`, the `mac` and the
response headers; and `accept_signature`, request headers each with the
profile it is read for and what it yields: the label, key share and
serialized Inner List, `null` where no MAC is asked, or `"refused"`.

### STAR

A space repo travels as a STAR archive, as `spaces/star.md` defines STAR:
the magic bytes `0x2A 0x6C 0x01 0x00`; the metadata, a DRISL map with a
String `$type`, after its length; then each pair as `0x01`, the key's
length, the key, the value's length and the value; then `0x00`, and
nothing after. Lengths are unsigned varints, minimally encoded, at most 9
bytes. Any fault refuses the whole archive. `interop/star/` has STAR's
published fixtures: each valid one must parse to its metadata and pairs,
and each invalid one must be refused.

The profile `atproto-space-repo` is a space repo:

- **The metadata** is the repo's commit, with `$type`:
  `{$type: "atproto-space-repo", ver: 1, repo, space, rev, data}`, `data`
  as DRISL bytes. A writer writes these six fields; a reader ignores any
  others STAR allows.
- **A pair** is a record: its key the UTF-8 bytes of `{collection}/{rkey}`,
  its value the record's DRISL bytes, those its CID names. Pairs are in
  strict ascending order of the key bytes, so `com.example.x.y/1` comes
  before `com.example.x/10`, as `.` is `0x2E` and `/` is `0x2F`; an empty
  repo has none. So the archive of a commit and its records is one string
  of bytes.
- **Limits**: the metadata is at most 65536 bytes, a key 4096, a value
  1048576.

A reader, given the space and repo it asked for, the key share it asked
with, the shared secret and the response's headers, refuses the whole
archive on any of these, and otherwise takes it as the repo:

- a fault of STAR's, a limit passed, or metadata that is not canonical
  DRISL;
- a `$type` other than `atproto-space-repo`, a `ver` other than `1`, or a
  commit field missing or of another type;
- a `space` or `repo` other than those asked for;
- a commit MAC that does not check (above) against the response's
  headers, or an `Atproto-Space-Commit` header that names another commit
  than the metadata;
- a key that is not a record path (an NSID, `/`, a record key), or one not
  after the key before it, duplicates included;
- an LtHash digest of the records' elements, each CID computed from the
  value's bytes, other than `data`.

`interop/spaces/repo-star.json`: valid archives, each exactly the archive
of its commit and records, to serialize and to read; one a reader must
take though no writer here makes it; and archives to refuse, with their
space, repo, key share, secret and response headers.

## Test vectors

| File | Holds |
| --- | --- |
| `interop/spaces/kem.json` | the shared secret |
| `interop/spaces/lthash.json` | LtHash |
| `interop/spaces/mac.json` | commit and record MACs, and `Accept-Signature` read |
| `interop/spaces/http-signature.json` | a credential's HTTP Message Signature |
| `interop/spaces/repo-star.json` | space repos as STAR archives |
| `interop/star/*.json` | STAR's own fixtures, published with STAR |

Each file says in its `description` how to read its cases. The server's
own tests (`cargo test`) hold each building block to every case of its
file. The spaces run holds every case to the reference's own library and
to the harness's code written from this page (below), so each vector is one
the reference and this page agree on; the one case the reference's library
cannot take, `kem.json`'s second, whose `info` is not `atproto-mac`, is held
to the harness's code alone.

## How it is checked

`harness/spaces/compare.mjs` is the spaces run. It starts the pinned
spaces alpha (`harness/spaces/package.json`; `pin-alpha.sh` pins another):
its PLC directory and its PDS, the reference here. It starts the server
beside it on that directory, in the environment `PDS.md` lists, with the
alpha's recovery key as `PDS_RECOVERY_DID_KEY`, and as
`PDS_LEXICON_AUTHORITY_DID` the DID of the run's lexicon authority, a
PDS of its own on that directory publishing space type declarations,
which the alpha resolves from too (`OAUTH-SCOPES.md`, "How it is
checked"):

```sh
cd harness/spaces && npm ci          # once
node compare.mjs --seed 1 [--accounts 3] -- <command that starts the server>
```

First it checks every vector, as above, and stops at the first that either
side disagrees with. Then, on both sides, with each account's DID named
alike as `PDS.md` says:

- **createAccount** of a few accounts: the answers are compared, with keys,
  tokens and URLs taken out. Each genesis operation in the directory must
  be what `ACCOUNTS.md` says, with `verificationMethods.atproto_kem` a
  P-256 `did:key` of a point on the curve, other than the signing key; the
  answer's `didDoc` must be the directory's document; and that document
  must have the `#atproto_kem` verification method, its key the
  operation's.
- **getRecommendedDidCredentials** of each: compared with keys and URLs
  taken out; on each side `verificationMethods.atproto_kem` must be the
  genesis operation's.
- **describeServer**: `did`, `availableUserDomains`, `inviteCodeRequired`
  and `spacesAlpha` compared; `spacesAlpha` must be `true` on each side.
- **updateHandle** of most of them: the operation it posts must keep every
  `verificationMethods` entry, and `getRecommendedDidCredentials` still
  name the same KEM key.
- **A restart**: the server is killed and started again on its data, and
  every account's `getRecommendedDidCredentials` must name the KEM key it
  had, with no new operation in the directory.

The reference is held to each check too, which checks the check. The other
runs, against the `latest` reference, hold what the KEM key changes to
this page instead (`PDS.md`).

That the server makes the shared secret, MACs, signatures and archives the
vectors hold, over HTTP, against a syncer, each later part checks through
its own methods, in this run.