~ knotless Search/
↑↓ move↵ open⇧↵ all results

chadtmiller.com/pds.js

An atproto PDS in JavaScript, for Cloudflare Workers and Node

0
0
main
624 lines · 28.5 KB Raw History

Permissioned data: spaces, permissioned repos, and the space auth chain

Date: 2026-07-30 Goal: implement AT Protocol proposal 0016 (permissioned data) in pds.js — spaces, per-user permissioned repos with LtHash commits, the delegation-token → space-credential auth chain, incremental sync, and com.atproto.simplespace management — behind a feature flag, in a new dependency-free @pdsjs/spaces package.

0. Upstream pins

Both upstream sources are drafts and will move. This spec is written against:

  • Proposal: bluesky-social/proposals, 0016-permissioned-data
  • Reference: bluesky-social/atproto PR #5187 ("Permissioned data", draft), head 24a81bf5ab8d39557da35a4c4c0978c9aa62c45a, branched from a9a13d2a6e60ce9035b5c1f4bc24a5a14623b4ae

The PR is fetched locally as pr-5187 in ~/code/atproto. Reference behavior is the design authority; adaptations to pds.js's single-actor, port-based architecture are documented below.

Resyncing with upstream is explicit periodic work, not something the feature flag makes free. The flag protects users from depending on a moving target; it does not stop upstream changes from breaking our tests.

1. Packaging and module boundaries

New workspace package @pdsjs/spaces, with no external dependencies and a workspace-only dependency on @pdsjs/core. This follows @pdsjs/core itself, which is dependency-free and built entirely on WebCrypto; of the existing packages only lexicon-resolver (@bigmoves/lexicon) and node (ws) carry external dependencies, and both are platform edges rather than protocol code.

packages/spaces/src/
  blake3.js        vendored BLAKE3 XOF — one export, blake3(bytes, outLen)
  lthash.js        LtHash: add/remove/state/digest/equals/isEmpty
  path.js          record path + set hash element formatting/parsing
  mac.js           encodeCommitCtx + deniable-signature MAC (HKDF/HMAC over commit ctx)
  commit.js        RepoCommit (add/remove/applyOp/sign) + verifyCommit
  memory-storage.js  in-memory SpaceStoragePort, for tests and handler unit tests
  uri.js           space AT-URI parse/format
  verifier.js      signature-verifier port; P-256 built in, k256 injectable
  scope.js         space: scope matching (grammar itself lives in core, §6)
  routes.js        space + simplespace route table
  handlers/
    space.js       repo-host + PDS endpoints
    simplespace.js space management
    notify.js      notification fan-out
  index.js         createSpaceRoutes(config) → route table

path.js and mac.js split out of what was originally sketched as parts of commit.js: path formatting has no dependency on commits or signatures, and the ctx-encoding/MAC logic is reused independently of RepoCommit. commit.js itself holds only the RepoCommit class and verifyCommit. scope.js, routes.js, handlers/, and the route-table shape of index.js are later phases (§9) and not yet implemented.

Dependency direction

@pdsjs/core must not depend on @pdsjs/spaces. Core exposes a hook; spaces plugs into it. Concretely:

  • pds.js:608 changes from routes[pathname] to this.routes[pathname].
  • The constructor gains config.spaceRoutes and sets this.routes = spaceRoutes ? { ...routes, ...spaceRoutes } : routes.
  • The platform packages (node, deno, cloudflare) do the wiring — they already construct the PDS and pass config, so they gain an optional import { createSpaceRoutes } from '@pdsjs/spaces'.

Core's package.json stays dependency-free, and a deployment that does not want spaces never installs the package.

This is also the answer to packages/core/src/pds.js being 4,304 lines: none of the 24 new endpoints land in it. It gains one line at the route lookup, one config field, and the changes in §6 and §7 that genuinely belong to core.

Handler convention

Existing route entries are PersonalDataServer.prototype.handleX, which cannot work from a separate package. Space routes instead carry plain functions taking an explicit context object assembled by core:

{ actorStorage, spaceStorage, keypair, did, hostname, verifier, resolveDid, fetch }

This differs from the existing convention. It is what keeps the packages decoupled, and it makes space handlers unit-testable without standing up a PDS.

Vendored crypto is a leaf

blake3.js imports nothing and exports one function. It is the only hand-rolled crypto in the tree. Nothing depends on its internals, and it is verified against the official BLAKE3 test vectors (§8).

2. Cryptographic primitives

Required surface

PrimitiveUsed forSource
SHA-256LtHash digest(), CIDscrypto.subtle (existing)
HMAC-SHA256commit maccrypto.subtle
HKDF-SHA256MAC key from ikmcrypto.subtle
ES256 sign/verifydelegation tokens, credentialsexisting crypto.js
BLAKE3 XOFLtHash element expansionvendored
ES256K verifyremote tokens and credentialsinjected port

WebCrypto has no BLAKE3 and no secp256k1. These are the only two gaps.

BLAKE3

The required surface is exactly one call — blake3(bytes, 2048), unkeyed, extended output, arbitrary-length input. No keyed mode, no derive-key mode.

Implement the full algorithm (compression function, chunk chaining, parent-node merging, root output with XOF counter), not just the single-chunk path. In practice the hashed elements are {collection}/{rkey}/{record_cid}, at most ~890 bytes, so a single chunk always suffices — but that is an undocumented consequence of unrelated length limits and must not be baked in silently.

digest() uses crypto.subtle.digest('SHA-256', …) and is therefore async, unlike the reference's synchronous digest(). This is a deliberate shape difference to avoid vendoring a second hash.

Signature verification port

verifier.verify(didKey, bytes, sig) → Promise<boolean>

@pdsjs/spaces ships a P-256 implementation over crypto.subtle. A secp256k1 verifier is injected by the deployment. The SignatureVerifierPort typedef is declared in packages/core/src/ports.js alongside SpaceStoragePort (§4), DidResolverPort (§6), and the existing LexiconResolverPort, so all ports are declared in one place. Rationale: hand-rolling ECDSA verify is a different risk class from hand-rolling a hash — a subtle bug means accepting a forged credential, not a failing test.

pds.js is P-256 only today (plc.js:47 encodes only P-256 did:keys; no zQ3s prefix or k256 code anywhere). did:plc accounts sign with secp256k1 by default, so without an injected k256 verifier, spaces interoperate only with other P-256 accounts. That is a coherent state for a flagged feature, and it is a deployment decision rather than a code change.

LtHash

Ported from packages/space/src/lthash.ts (80 lines): 1024 little-endian uint16 lanes over a 2048-byte buffer, Uint8Array and Uint16Array views over one ArrayBuffer so lane arithmetic wraps mod 2^16 natively. add and remove are lane-wise += / -=. Storing the state makes writes O(1) — a record is folded in or out without rehashing the repo.

Signed commits

Ported from packages/space/src/repo-commit.ts (171 lines).

ctx = "atproto-space-v1"
   || uint16be(len(space))  || space
   || uint16be(len(author)) || author
   || uint16be(len(rev))    || rev
   || uint16be(len(ikm))    || ikm

Length prefixes are big-endian (TLS variable-length vector convention), deliberately the opposite byte order from the set hash's lanes. Fields longer than 0xffff are an error.

mac = HMAC-SHA256(HKDF-SHA256(ikm, ctxBytes), hash)
sig = sign(ctxBytes)          -- never covers hash

The signature covers only ctx, never the digest, so a leaked commit proves nothing about what the author wrote; the digest is bound by a symmetric MAC keyed from the public ikm, which gives readers integrity and third parties nothing. A fresh 32-byte ikm per commit means each reader receives a distinct signature.

Whether ctxBytes is HKDF's salt or its info is not determinable from the reference source alone. This is settled empirically by the generated vectors (§8), not by guessing.

verifyCommit checks ver === 1, commit.rev === ctx.rev, recomputes and compares the MAC, then verifies the signature. All comparisons over byte arrays must be length-checked before the loop.

3. Feature flag

Config field config.spaces (boolean, default false), fed from PDS_ENABLE_SPACES, matching how readOnly and relayUrl already thread through.

The flag is a single gate at route registration. There are no if (this.spaces) checks inside handlers — that is the failure mode that makes flags rot. With the flag off, the space routes are simply absent from this.routes and fall through to the existing 404 path.

Schema DDL is CREATE TABLE IF NOT EXISTS and therefore idempotent, so it runs when the flag is enabled rather than unconditionally. There is no versioned migration system in pds.js (both backends use bare CREATE TABLE IF NOT EXISTS), so this carries none of the ordering hazards that would normally argue for running migrations up front. Toggling on creates tables; toggling off leaves them in place, inert.

The flag-off path is load-bearing

describeServer.json is unchanged in PR #5187 — there is no server-level capability advertisement. Space-host discovery is per-space via com.atproto.space.getSpace, and the proposal resolves the space host from #atproto_space_host falling back to #atproto_pds.

A space-unaware pds.js will therefore receive space requests from optimistic clients. With the flag off it must return a clean MethodNotImplemented, not throw. This is an explicit test, not an assumption.

The same fallback simplifies the flag-on case: pds.js is both the PDS and the space host at the same origin, so absent is the correct DID document state. No #atproto_space_host service entry, no PLC operation, nothing to unpublish when the flag is turned off. The flag stays purely local to the server.

4. Storage

Schema

Seven tables, ported from the reference's 002-space.ts, in the per-actor store (matching upstream's actor-store placement).

TablePrimary keyHolds
spaceurispaceDid, spaceType, isOwner, simplespace policy/appAccess config, createdAt, deletedAt
space_member(space, did)member list, host-internal, no rev/commit state
space_record(space, collection, rkey)cid, CBOR value, repoRev, indexedAt
space_repospacesetHash (2048-byte LtHash state), rev
space_record_oplog(space, rev, idx)action, collection, rkey, cid, prev — append-only
space_writer(space, did)writer set with rev + hash, maintained from notifyWrite
space_credential_recipient(space, serviceDid)registerNotify subscribers, serviceEndpoint, lastIssuedAt

Plus index space_record_rev_idx on (space, repoRev).

space_repo is keyed on space alone: one row per space, holding this actor's repo state in that space.

SpaceStoragePort

Storage follows the hexagonal convention used throughout pds.js. Spaces get a separate port, SpaceStoragePort, rather than ~20 new methods bolted onto ActorStoragePort. The typedef lives in packages/core/src/ports.js alongside the existing port definitions — that file is pure JSDoc ending in export {}, so core gains a type declaration and no runtime coupling to @pdsjs/spaces. storage-sqlite and the Cloudflare Durable Object each implement it, and from phase 3 the PDS config carries spaceStorage alongside actorStorage.

19 methods:

GroupMethods
SpacesgetSpace, putSpace, deleteSpace, listSpaces
MembersaddMember, removeMember, listMembers, isMember
Records (read)getSpaceRecord, getSpaceRecordCid, listSpaceRecords, listAllSpaceRecords
WritecommitSpaceWrite
Repo stategetSpaceRepo
OploglistSpaceOps
WritersputSpaceWriter, listSpaceWriters
RecipientsputCredentialRecipient, listCredentialRecipients

One contract, three implementations

Correct behaviour is defined once, in test/helpers/space-storage-conformance.js, as runSpaceStorageConformance(name, createStorage). Every implementation runs that same suite: the in-memory fake, storage-sqlite, and the Durable Object adapter (via a shim that presents better-sqlite3 through the DO sql.exec interface, so the adapter's SQL runs under plain vitest).

This is what makes "both backends in lockstep" affordable. Each adapter costs a test file of three lines rather than a re-derived suite, and a behavioural divergence between them fails at unit-test speed instead of surfacing in an e2e run — or not at all. Three findings during implementation came directly from it: missing mutation-immunity coverage for returned rows and for byte buffers, and a filtered-pagination bug where the SQL adapters silently truncated results while the in-memory one did not.

Two rules in the suite are load-bearing and easy to get wrong independently in each adapter, which is precisely why they are pinned centrally:

  • Cursors. A full page always yields a cursor, even when it is the last one; the next call then returns empty with a null cursor. In SQL this falls out of WHERE key > ? with an empty-string sentinel for the first page.
  • Mutation immunity. Nothing returned by the port may alias stored state, and nothing passed in may be retained by reference. The SQL adapters get this for free by serialising; the in-memory one must copy explicitly, including nested arrays and byte buffers.

The adapters never parse space URIs

SpaceRow carries spaceDid and spaceType alongside uri, and they are stored as indexed columns. They are redundant with uri, deliberately: parsing space URIs is domain logic that lives in @pdsjs/spaces, which no storage package may depend on. Rather than each adapter hand-rolling its own URI splitting to serve listSpaces's type and authority filters, the domain supplies the components and the adapters filter with a plain WHERE clause.

This keeps the boundary honest and is also simply better: the filter is indexed, and LIMIT applies to matches rather than to rows scanned. An earlier version filtered after fetching, which meant a sparse match set could under-fill a page and return a null cursor while matches remained — silently truncating pagination. The workaround was a scan-and-refill loop in both SQL adapters; moving the predicate into SQL deleted it.

The write method is commit-shaped

A space write must atomically update space_record, space_repo, and space_record_oplog. Exposing those as three port methods would force a generic transact(fn) escape hatch, which leaks transaction control out of the adapter and is precisely the kind of abstraction break the port exists to prevent. Instead:

commitSpaceWrite(space, { writes, setHash, rev }) → void

where writes is [{ action, collection, rkey, cid, value, prev }].

@pdsjs/spaces performs the LtHash arithmetic, builds the oplog entries, and derives the new rev; the adapter persists them in a single transaction. The interesting logic stays in the domain, the port speaks in domain terms, and four naive methods collapse into one.

The schema is declared once, in core

SPACE_SCHEMA_SQL in packages/core/src/space-schema.js holds the seven-table DDL, and both SQL adapters import it.

An earlier draft of this section accepted duplicating the DDL per backend, on the grounds that sharing it "would require core or the storage packages to import @pdsjs/spaces, which would invert the dependency direction." That is true of @pdsjs/spaces but not of @pdsjs/core, which already declares SpaceStoragePort a few paragraphs above — core stating the contract and adapters implementing it is the same move, not a new coupling. No storage package depends on @pdsjs/spaces, so spaces remains optional exactly as §1 requires, and the duplication is gone rather than merely policed.

packages/storage-sqlite gained a @pdsjs/core dependency as part of this; its dependencies had been empty even though its source already referenced @pdsjs/core/ports in JSDoc.

The backends cannot get atomicity the same way

commitSpaceWrite must be atomic across three tables, and the two SQL backends reach that differently:

  • storage-sqlite issues BEGIN / COMMIT / ROLLBACK through the driver's exec. It cannot use better-sqlite3's db.transaction(), because the driver typedef guarantees only exec and prepare — node:sqlite has no equivalent.
  • The Durable Object cannot use explicit transaction statements at all; DO SQL rejects them. It uses state.storage.transactionSync(callback) instead.

This is why the two factories differ: createSpaceStorage(db) for SQLite, createSpaceStorage(sql, storage) for the Durable Object, which needs state.storage and not just state.storage.sql.

Both validate every write before opening a transaction, so a malformed batch fails without a partial commit.

5. Endpoints

24 endpoints, all served from @pdsjs/spaces.

RoleEndpointsAuth
PDScreateRecord, putRecord, deleteRecord, applyWrites, listSpaces, getDelegationTokenOAuth + space: scope
Repo hostgetRecord, listRecords, getBlob, getRepo, listRepoOps, getLatestCommitOAuth (own repo) or space credential
Space hostgetSpace, getSpaceCredential, listRepos, registerNotify, notifyWrite, notifySpaceDeletedcredential / service auth
simplespacecreateSpace, updateSpace, deleteSpace, addMember, removeMember, listMembersOAuth + manage action

Three outbound calls are part of the work and are easy to miss when counting endpoints:

  • notifyWrite — as repo host → space authority, on every write. For shared spaces the repo host auto-registers the authority on first write.
  • notifySpaceDeleted — as authority → registered syncers.
  • checkUserAccess — as authority → managing app, for the managing-app policy. This is the one lexicon implemented as a client only; it is served by the application, not by us.

Space addressing

at://{spaceDid}/space/{spaceType}/{skey}/{authorDid}/{collection}/{rkey}

Permissioned URIs are distinguished from public ones by the literal space segment, which contains no dots, whereas a collection NSID always contains at least two.

Sync

listRepoOps takes a since rev and returns { rev, collection, rkey, cid, prev } per op — cid null for deletes, prev null for creates — inlining each created or updated record's value so a syncer advances without a getRecord round-trip per write. When the response includes the last available op it also carries the current signed commit.

Correctness rests on the set-hash comparison, not on receiving every op, so sync is self-healing: a missed operation is detected on a later sync.

getRepo returns a CAR with two roots in order — the signed commit block, then a DRISL (DAG-CBOR) map from "{collection}/{rkey}" to record CID — followed by record blocks in the index's order.

That order is canonical DAG-CBOR map key order: shortest key first, then bytewise — not plain lexicographic. com.e.p/a sorts before com.example.post/a because it is shorter, though lexicographically it would come after. A consumer walks the index as the CBOR encoder emitted its keys and expects the blocks to follow, so both use the same comparison, exported as byCanonicalKey from @pdsjs/spaces/car.

Note this is a different order from listAllSpaceRecords, whose SQL sorts on collection || '/' || rkey — plain lexicographic. The storage sort only has to be deterministic; the CAR order is a wire concern, so the serializer re-sorts rather than pushing the rule into the adapters. packages/core/src/repo.js's cborEncodeDagCbor already sorts map keys canonically, so the encoded index and the block sequence agree by construction.

buildCarFile(rootCid, blocks) in repo.js took a single root and wrapped it in an array. It now accepts string | string[], a backward-compatible widening mirroring upstream's own common/src/car.ts ("most callers have a single root; permissioned repos declare two"). Reading was already fine — car.js parses roots as an array.

6. Auth chain and scopes

app → PDS.getDelegationToken        OAuth, space:…?action=read
                                     → JWT, user's #atproto key,
                                       aud = authority#atproto_space_host,
                                       exp 60s, typ atproto-space-delegation+jwt
app → authority.getSpaceCredential
        verify delegation token      resolve user DID doc, verify sig
        verify client attestation    resolve client-metadata.json → JWKS  (optional)
        check policy + appAccess
                                     → JWT, authority's #atproto_space key,
                                       no aud, exp 2h, multi-use,
                                       typ atproto-space-credential+jwt
app → repo host.{getRepo,listRepoOps,…}   presents space credential

The delegation token asserts only user-to-app delegation and says nothing about which application is acting; app identity is established independently by the client attestation, which the authority verifies itself. Space credentials have no aud because they are presented to any repo host, and are signed by the authority so a repo host can verify them without contacting it.

Space authority resolution

Two DID document entries, with the proposal's fallbacks:

  • verification method #atproto_space → space signing key; absent falls back to the account's #atproto key.
  • service #atproto_space_host → space host endpoint; absent falls back to #atproto_pds.

Injected ports

resolveDid(did) → Promise<didDoc> is injected, not imported; its DidResolverPort typedef is declared in packages/core/src/ports.js with the others (§2). @pdsjs/lexicon-resolver already provides resolveDid and extractPdsEndpoint, but it carries an external dependency (@bigmoves/lexicon), and @pdsjs/spaces is dependency-free. The platform packages, which already depend on both, do the wiring — the same pattern as the verifier port.

space: OAuth scopes

space:<spaceType>[?authority=<did>][&skey=<skey>][&collection=<nsid>…]
                 [&action=<action>…][&manage=<op>…]
  • authority: DID, self (default), or *
  • skey: 1–512 chars or * (default)
  • action: read_self, read, create, update, delete (all by default)
  • manage: create, update, delete
  • collection: NSIDs or *; omitting it defaults to the collections declared by the space type's declaration, the same way a bare repo: grant permits the collections it names.

read grants whole-space access and access to getDelegationToken; read_self grants only the holder's own repo and yields no delegation token.

A request is authorized when its target space matches the grant's (authority, spaceType, skey), each component equal or covered by *.

The grammar lives in packages/core/src/scope.js, alongside the existing parseRepoScope / parseRpcScope / parseBlobScope, extending ScopePermissions and parseScopesForDisplay. This is pure string parsing with no dependency on @pdsjs/spaces — core gains a grammar, not a coupling — and it must be there regardless, because core owns the OAuth flow and the consent screen has to render what is being granted. Upstream's equivalent (space-permission.ts) is 331 lines, the largest single non-endpoint piece.

Consequence: with the flag off, core still parses and displays space: scopes. It must refuse to grant them — a check in the authorize path — so a PDS never issues a token for a capability it will not serve.

Space matching logic that needs space-type declarations (to resolve the default collection set) lives in @pdsjs/spaces/scope.js; only the grammar lives in core.

7. Space type declarations

Space type NSIDs resolve to Lexicon definitions with "type": "space":

{
  "lexicon": 1,
  "id": "com.atmoboards.forum",
  "defs": {
    "main": {
      "type": "space",
      "key": "any",
      "name": "AtmoBoards Forum",
      "collections": ["com.atmoboards.thread", "com.atmoboards.reply"]
    }
  }
}

name appears on OAuth consent screens; collections sets the default authorization target for a bare space: grant.

8. Testing

Vectors

A git worktree of pr-5187 in ~/code/atproto — the checkout has uncommitted work on another branch, so nothing is checked out in place — plus scripts/gen-space-vectors.js, which imports @atproto/space and emits test/fixtures/space-vectors.json. The fixture is committed and records the upstream SHA it was generated from. Regenerating it is the resync ritual.

Vectors cover:

  • LtHash add / remove / digest, including order-independence and add-then-remove round trips
  • encodeCommitCtx byte-for-byte
  • computeMac — which settles the HKDF salt-vs-info question empirically
  • full sign → verifyCommit round trips

BLAKE3 is verified against the official BLAKE3 test vectors instead, not the atproto tests: it needs multi-chunk and extended-output coverage that atproto's usage never exercises, since our inputs are always single-chunk.

Unit

@pdsjs/spaces against the vectors; handlers against an in-memory SpaceStoragePort backed by Maps, which needs no database and no dependencies.

e2e

Extends test/e2e.test.js, which already runs the PLATFORM=node|deno|cloudflare matrix, so both storage backends are covered by construction. Full path: create simplespace → add member → write records → listRepoOps incremental sync → getRepo CAR with both roots, verified against the commit hash.

Plus a flag-off test asserting clean MethodNotImplemented on space routes.

9. Phasing

Checkpoint after each phase.

#PhaseContents
1Primitivesvendored BLAKE3, LtHash, commit sign/verify, space AT-URI, vector harness
2StorageSpaceStoragePort typedef, DDL + implementation in both backends, in-memory fake
3Write pathcreateRecord/putRecord/deleteRecord/applyWrites, oplog, listSpaces
4Read/syncgetRecord, listRecords, listRepoOps, getRepo + buildCarFile widening, getLatestCommit, getBlob
5AuthgetDelegationToken, getSpaceCredential, getSpace, listRepos, space: grammar, consent display
6Managementsimplespace endpoints, notifyWrite/registerNotify/notifySpaceDeleted, checkUserAccess client

Phase 1 must not be rushed. Everything downstream inherits its correctness, and it is the only phase where a bug is silent rather than a failing test.

10. Risks

  • Upstream is a WIP draft. Pinned in §0. Resync is explicit periodic work. The flag protects users, not the test suite.
  • Vendored BLAKE3 is the only hand-rolled crypto in the tree. Mitigated by the official vectors; the file carries a comment saying so.
  • getSpaceCredential is the largest single handler upstream (173 lines) and does the most network I/O — DID resolution, JWKS fetch, possibly checkUserAccess. Most likely place for phase 5 to overrun.
  • The commit ctx encoding may change upstream. It carries a ver field, so a change is detectable rather than silent, but it invalidates every stored setHash. Because this ships behind a flag with no production data, the response is to drop the space tables and rebuild, not to migrate.
  • Both backends in lockstep doubles every storage step, and the hexagonal port means the doubling is real rather than mitigated: ~18 methods and seven tables land twice. This is an accepted cost of architectural consistency (§4). The e2e matrix catches drift, but it does not remove the work.

Non-goals

  • Syncer role. pds.js serves PDS, repo host, and space host. Consuming spaces (registerNotify as a caller, maintaining space copies) is an application's job.
  • End-to-end encryption. The proposal provides access control, not confidentiality, by design; services can read the data they handle. E2EE is a separate concern layered on top.
  • A secp256k1 implementation. Injected, not vendored (§2).
  • Space-type lexicon publishing. We resolve and consume space-type declarations; authoring tooling is out of scope.
  • Multi-tenant spaces. pds.js remains single-actor; spaces follow the existing per-actor store.
  • Versioned migrations. pds.js has none, and this feature does not introduce them (§3).