chadtmiller.com/knotless

Switch repository

Loading…

What changes

A write the index takes from a feed keeps its commit's revision. Jetstream's frames and the firehose's commits both say the revision of the commit that made a write, and Event now carries it as rev. The index keeps it per record in a new table, record_rev, and keeps a deletion there too, marked gone. Index::put_as_of and Index::delete_as_of write only when no later revision's write to that record is indexed already. The check and the stamp happen under one lock of their own. A write with no revision, such as a page's write-through or a recheck of a stale check, is written as before.

A backfill reads each account as of its latest commit. It asks the account's PDS for com.atproto.sync.getLatestCommit before listing its records, and puts each one as of that revision. A record the feed changed or deleted at a later revision is left as the feed has it. Once the account is read, Index::read_through forgets that account's deletions at or before the revision, as the listing no longer holds them.

Why

A backfill lists an account's records while the feed goes on. The 10-minute sweep runs one all the time, and the listing could put back a record the feed had since updated or deleted, until the next event for it. This is the first part of Backfill is one listRecords call per collection per account, unverified, its step 5, which the later parts build on: one verified getRepo per account through a queue, then a mirror's STAR-lite, then lightrail for finding the accounts.

Taking the revision before listing, rather than the record's own revision, is what listRecords allows: it says no revision. What it lists is that commit's state or later, so a later write the feed indexed always wins, and the one write it can lose is a feed event already older than the listing.

How it was checked

A unit test in index.rs covers each path: a backfill's older read skipped, a later one written, a deletion keeping an older read out, read_through forgetting it only past its revision, and a write with no revision. The serve tests' backfills now reach getLatestCommit on the reference PDS. record_rev is in plans.rs's growing tables, and its one range query uses the partial index record_rev_gone. CI runs all of this. Nothing was built here.

Shortcomings

• A backfill still lists records with listRecords, unverified and one account at a time, and one account that fails still aborts the pass. The next part replaces that with one verified getRepo per account through a queue with retries.
• record_rev gains a row per record written with a revision, as many as the records indexed. A deletion's row stays until a backfill reads that account past it.
• A page's write-through passes no revision, so a backfill could overwrite it until the feed's event for it lands, as it could before.

Changeset: none

🤖 Generated with Claude Code

This pull gives feed events a rev, adds the record_rev table, and adds put_as_of and delete_as_of, which skip a write when a later revision is already indexed. A backfill now reads getLatestCommit first and stamps its writes with that revision. It then calls read_through to forget deletions at or before it.

I found nothing that must change. I checked these:

• The check-then-write-then-stamp sequence runs under the revs mutex. The inner db lock is taken and dropped in each step, so it can't deadlock with put or delete.
• A late feed event older than a backfill's stamp is skipped, which is right because the listing is newer.
• A deletion newer than the backfill's revision survives read_through, so the backfill can't put the record back.
• record_rev is dropped with the other cache tables when the index is rebuilt.
• The docs page is updated, and Changeset: none is on a line of its own.

I couldn't check that Jetstream's flat payload["rev"] is where the frame puts the revision, so the revision may never be set from Jetstream. Without it a write goes in with no revision and nothing breaks, but the protection wouldn't apply. I'd check this against a real frame or a parse test.

I didn't read the check results.

Since round 1 the pull has two small commits. as_of now takes &dyn Fn instead of a generic FnOnce, and the callers pass &||. The test also gains a check that a delete older than an indexed one changes nothing. Neither changes behaviour, and I have no new findings.

Last round I worried that Jetstream's flat payload["rev"] might not be where the frame puts the revision. I checked it against crates/knotless/src/jetstream.rs and the repo's Jetstream test fixtures. They all put rev flat in the payload, matching the parser, so I'm satisfied. I didn't read the check results.

Landed as part of A backfill reads each account whole from its PDS, checked, and only those that changed, which was stacked on this branch and carried its commits into main. Nothing is left to merge here, so I'm closing it.