chadtmiller.com/knotless

Switch repository

Loading…

What happened

knotless-serve's backfill (crates/knotless-serve/src/lib.rs, accounts, pages, backfill, sweep) works like this:

• It asks the relay listReposByCollection once for each of the 14 or more indexed collections. If any one of those calls fails, it falls back to listRepos, which on a relay lists every account.
• For each account it pages com.atproto.repo.listRecords through every indexed collection, one account at a time, against that account's own PDS.
• Nothing it reads is checked against the commit the account signed.
• sweep runs the whole thing again every 10 minutes to recover what Jetstream dropped.
• A record a backfill reads can overwrite a newer write that Jetstream has already indexed: nothing compares revisions.

That costs about 15 requests per account per pass, spread over every PDS on the network, and grows with the number of accounts. For grain, which is moving onto a stack modelled on knotless, this would be the slow part.

opensocial already solves this in packages/group-sites/src/repoArchive.ts and networkIndex.ts:

1. Who to read. listReposByCollection from lightrail.microcosm.blue first, then the relay.
2. One archive per account, from a mirror. getRepo from hubble.microcosm.blue with Accept: application/x.microcosm.star-lite and zstd. STAR-lite is every key and record in key order with the commit in its header and no tree nodes. If the mirror is unreachable, a CAR from the account's PDS, one at a time, capped in size.
3. Verified while streaming. The MST root is rebuilt from the keys as they stream past, holding one node per layer, and must equal the commit's root. The commit must be signed by the key in the DID document. Collections not indexed are hashed and dropped.
4. A queue in SQLite. A table of repos with state, tries and retry_at, and a few workers. Retries back off at 1 minute per try, up to an hour. A 4xx gives up on that repo. The mirror's 429 pauses every worker for its retry-after. A watchdog drops a download that goes quiet for 30 s or runs past 10 minutes.
5. Newer rev wins. Each indexed record keeps the rev it came at, and a delete is remembered until its repo has been read, so a backfill never undoes a live write.

For knotless, step 5 could also replace the 10-minute full sweep: compare each repo's com.atproto.sync.getLatestCommit rev with the last one indexed, and queue only the repos that differ.

This would also settle two items in "knotless-serve: quirks in backfill, labels and caches": backfill aborting on the first failed account, and the listRepos fallback.

Reusing the STAR reader in space_sync.rs

crates/knotless/src/space_sync.rs already reads STAR for spaces, but a different version: magic *l\x01\x00, a header that is one frame holding the commit with its MAC, and an LtHash set hash checked against the commit. Hubble's STAR-lite is version 0: magic *l\x00, a 36-byte root CID, then a varint-length signed commit. The records after the header are framed the same way in both (varint key, varint value, in key order).

Reusable as is: Frames, the key-order check, splitting a path, DRISL decoding and cid(). New:

• the version 0 header
• rebuilding the MST root from the keys (opensocial's RootBuilder, about 80 lines)
• checking the commit's signature with the DID document's key
• zstd, and reading from a stream instead of a whole &[u8], so a large repo is never held in memory

Factoring that out of space_sync.rs into a STAR module both versions use would let grain take the same code.

Things to work out

• The mirror as a flag, empty for none, as the other feed flags are. The harness's reference PDS serves CARs, so the CAR path is what tests reach; STAR-lite needs a recorded fixture to hold the 100% gate.
• Whether lightrail's directory covers knotless's collections more completely than the relay's.

Done, in A backfill reads each account whole from its PDS, checked, and only those that changed and A backfill asks a directory which accounts to read, and reads them from a mirror as STAR-lite:

• Who to read: --directory (lightrail in prod) is asked listReposByCollection first, then the relay.
• One archive per account, from a mirror: --mirror (hubble in prod) serves STAR-lite or a CAR, and the account's PDS is the fallback.
• Verified: the MST root is rebuilt while the archive streams, and the commit's signature is checked against the DID document's key.
• Retries: each repository has its own retry_at and backs off.
• Newer rev wins: records keep the rev they came at.
• No full sweep: the 10-minute sweep compares getLatestCommit and reads only the repositories that changed.

The STAR reader was not shared with space_sync.rs. That can be its own change if grain needs it.