chadtmiller.com/knotless

Switch repository

Loading…

What changes

A backfill reads each account whole, as one CAR from its PDS. com.atproto.sync.getRepo, up to 256 MiB, replaces the 15 or more paged listRecords calls per account. Four accounts are read at a time instead of one.

Nothing read is believed until it checks out. The new archive.rs takes the CAR's root commit and checks three things: the commit is the account's, it is signed by the key the account's DID document names, and every block from the commit through the MST to each record hashes to its CID. Only the indexed collections' records are kept, each indexed as of the commit's revision, through the part this is stacked on.

Only accounts that changed are read. Each account is first asked for com.atproto.sync.getLatestCommit. One whose commit is the revision last read is passed over, so the 10-minute sweep reads only what changed since the last one. The index keeps each account's last revision in a new backfill table.

One account that fails no longer stops the rest. It is logged at warn as account not backfilled, and a later backfill tries it again, a minute later for each failure, up to an hour. The report counts accounts read, unchanged and failed.

Why

This is part 2 of Backfill is one listRecords call per collection per account, unverified. It is stacked on A backfill never puts back what the feed has since changed or deleted, whose commit it includes until that lands. The old backfill cost about 15 requests per account per pass, read nothing it could check, and stopped at the first account that failed.

The CAR comes from the account's own PDS in this part. A mirror's STAR-lite comes in part 3 and falls back to this. The CAR is the path the harness's reference PDS serves, so the tests reach it.

How it was checked

• Unit tests in archive.rs build signed repositories. A good one opens to its records, in key order, filtered by collection. These are each refused: another account's, one signed by another key, a commit changed after signing, a block changed in the CAR, a CAR cut short, and an empty one.
• The serve test's backfills now read through archive. A second backfill finds both accounts unchanged. A mock relay also names an account that cannot be resolved: it fails once, without stopping the other, and the next backfill puts it off.

CI runs these. Nothing was built here.

Shortcomings

• A repository is held in memory while it is read, at most 256 MiB, with four read at once. Streaming it is part 3's, with STAR-lite.
• A repository over 256 MiB is never read. It fails and is retried hourly.
• A record that is in the index but no longer in the account's repository, because its delete event was missed, is not taken out yet.
• The MST's shape is not checked: each node is checked by its hash from the signed root, and that is what makes the records the account's.

Changeset: none

🤖 Generated with Claude Code

This pull replaces the paged listRecords backfill with one getRepo CAR per account. It checks the commit's DID, its signature against the DID document's key, and each block's hash from the commit down through the MST to the records. It skips accounts whose getLatestCommit rev is the one last read. A failing account is now logged and retried with backoff instead of stopping the backfill.

I read archive.rs and the new backfill/read_account in lib.rs. I did not read the index.rs changes, the tests, or the CI result. I found nothing that must change.

• Every block is hashed before it is decoded.
• Cycles are impossible, because a block can't contain its own hash.
• The size cap is enforced while streaming.
• The retry_at and tries handling looks right. I could not see where tries resets after a successful read, so check that index.backfill_read clears it.
• Skipping an account on getLatestCommit trusts the PDS's word for the rev, which is fine because the stored rev always comes from the verified CAR.
• Your shortcomings list already covers the memory use, the 256 MiB limit, and stale records that are never removed.

Since my last review, the only change is one new commit, "The archive test reads a CAR whose first length never ends". It adds a test and doesn't touch the code I reviewed.

The open question from round 1 is settled. index.backfill_read writes the row with tries and retry_at set to 0, so a successful read clears the failure backoff.

I found nothing that must change.