chadtmiller.com/pds.js

Switch repository

Loading…

# Regenerating `test/fixtures/space-vectors.json`

`test/fixtures/space-vectors.json` holds cross-implementation test vectors for the
AT Protocol permissioned data ("spaces") primitives: the LtHash set hash, a
commit and its MAC, a record MAC, the STAR archive of a repo, a KEM
decapsulation, the MAC headers and a credential signature. They are produced by
*running* the reference implementation, not by transcribing it. pds.js asserts
against them so that its primitives stay byte-for-byte compatible with upstream.

Reference pin: `bluesky-social/atproto` at
`704b9e2acd7df78c9a22a275c64ecf8befcf30b7`, published as
`@atproto/space@0.0.0-spaces-alpha-20261105160032`. The same SHA is recorded in
the fixture's `generatedFrom.sha`.

## Why a worktree

`~/code/atproto` normally carries uncommitted work on some other branch. Checking
out the pin in place would clobber it. Generation therefore happens in a
throwaway worktree, and the generator itself lives inside that worktree as an
untracked scratch file. **Nothing is ever committed to the atproto repo.**

## Procedure

### 1. Create the worktree and build the reference's dependencies

```bash
cd ~/code/atproto
git worktree add ~/code/atproto-space-ref 704b9e2ac
cd ~/code/atproto-space-ref
pnpm install --frozen-lockfile
pnpm --filter '@atproto/space^...' run build
pnpm --filter '@atproto-labs/rolldown-plugin-bundle-manifest' run build
pnpm --filter '@atproto/oauth-provider-api' run build
```

`packages/space` is a registered vitest project, so its own sources run straight
from TypeScript. Its *workspace dependencies* resolve through their `dist/`
entrypoints and must be built first. The same worktree serves
`scripts/space-reference/run.sh`.

Use `pnpm exec`, not `npx`: the repo pins pnpm via `devEngines` and `npx` aborts
with `EBADDEVENGINES`.

### 2. Write the generator

Create `packages/space/tests/gen-vectors.test.ts` in the worktree. It is a
scratch file: untracked, never committed, deleted with the worktree.

Rules the generator must keep:

- Import the primitives from `../src/index.js` (`LtHash`, `RepoCommit`,
  `verifyCommit`, `computeCommitMac`, `computeRecordMac`, `formatMacHeader`,
  `formatWantMacHeader`, `serializeRepo`, `verifyRepoStarFull`, `encapsulate`,
  `decapsulate`, `createSpaceSigHeaders`), `P256Keypair` from `@atproto/crypto`,
  `encode` from `@atproto/lex-cbor`, `cidForCbor` and `parseCid` from
  `@atproto/lex-data`.
- Never invent CIDs. The two in use are the CIDs of `{ text: 'hello' }` and
  `{ text: 'world' }`, lifted from the reference's own
  `packages/space/tests/repo-commit.test.ts`; the generator re-derives them and
  asserts they match before writing anything.
- LtHash inputs: the elements `a`,
  `app.example.post/3kabcdefghi/<hello cid>` and
  `com.atmoboards.thread/self/<world cid>`, singly and as the sets
  `[a, b]`, `[b, a]` and `[a, b, c]`.
- The commit is over those two records in
  `at://did:plc:aaaa/space/com.example.type/key1`, repo `did:plc:bbbb`, rev
  `3lkabcdefgh2x`, MACed under a fixed secret of 32 `0x07` bytes. Assert
  `verifyCommit` and `verifyRepoStarFull` over the serialized archive before
  writing, so a fixture upstream itself would reject never reaches disk.
- The record MAC is over the `app.example.post/3kabcdefghi` record's uri and CID
  under the same secret.
- The KEM vector is one `encapsulate` against a fresh exportable author key,
  recording the author's private key, the key share and the shared secret.
- Fail loudly if `VECTORS_OUT` or `REF_SHA` is unset.

The fixture's shape: `generatedFrom`, `lthash` (`empty`, `single`, `sets`),
`commit` (`space`, `repo`, `rev`, `records[]` with `valueHex`, `dataHex`,
`secretHex`, `macHex`, `macHeader`, `starHex`), `record` (`uri`, `cid`,
`secretHex`, `macHex`, `macHeader`), `kem` (`authorPrivateKeyHex`,
`authorDidKey`, `keyShare`, `sharedSecretHex`, `wantMacHeader`) and
`httpSignature` (`keyId`, `headers`).

### 3. Run it

```bash
cd ~/code/atproto-space-ref/packages/space
REF_SHA=$(git rev-parse HEAD) \
VECTORS_OUT=~/code/pds.js/test/fixtures/space-vectors.json \
  pnpm exec vitest run tests/gen-vectors.test.ts
```

### 4. Verify the fixture

```bash
cd ~/code/pds.js
npx vitest run packages/spaces/test
```

The LtHash, commit, MAC, STAR and KEM suites read the fixture. If the commit's
`dataHex` changes while the LtHash vectors do not, the commit encoding moved
upstream; if the LtHash vectors change, every downstream test is suspect, so
find out why before committing.

### 5. Clean up

```bash
git -C ~/code/atproto worktree remove --force ~/code/atproto-space-ref
```

## Resyncing with upstream

Regenerating this fixture **is** how pds.js resyncs with upstream spaces work.
When the reference moves, point the worktree at the new revision, rerun the
procedure, and let the diff in `space-vectors.json` tell you exactly which
primitive changed on the wire.

Whenever the fixture is regenerated at a new revision, update the pin in
`docs/superpowers/specs/2026-07-30-permissioned-data-design.md` §0 to match the
new `generatedFrom.sha`. The fixture and the spec pin must never disagree.

## Non-determinism

The KEM vector and the HTTP signature use fresh keys, so `kem.*` and
`httpSignature.*` change on every regeneration. They are for **verification**,
not reproduction: given the recorded private key and key share, an
implementation must derive the same shared secret, and the recorded signature
must verify against `keyId`. Everything else is deterministic and must not
change unless upstream changes.