chadtmiller.com/knotless

Switch repository

Loading…

What changes

A mirror sync reads refs from the index. While a feed (Jetstream or the firehose) is connected, a public repository's mirror syncs its record and live refs from the index, once a backfill has read the owner's account whole. Its packs are listed from the PDS only when the refs name commits the mirror lacks, as after a push. The index keeps each repository's live refs in a new live_ref table, from the feed and the backfill (hubble's STAR-lite or the PDS's CAR, both checked against the signed commit).

Ref and repository records come from the PDS, never from a Jetstream frame. Jetstream's frames are unsigned. For these two collections the inline record is dropped, and the index fetches the record from its PDS, as it already does for the firehose. A false event can still only cause a needless sync.

A mirror refreshes after the index has the event, rather than before. For thirty seconds after the pages write for an account, its mirrors still sync from its PDS, as the feed may not have brought the write yet.

An existing index reads every account whole again (its backfill revisions are cleared once, when the table is first made), so refs are read from the index only for accounts the index holds every ref of. Until the next sweep reads an account, its mirrors sync from the PDS as before.

Why

Every read of a repository waits on its mirror's sync, and a sync listed all of the repository's refs from its PDS, a hundred to a request, one after another. A deleted ref keeps its record, as its generation must never repeat (storage.md), so knotless's 594 ref records, 542 of them deleted branches, took 6–7 requests and about 1.5 s on prod for each sync. It syncs after every write to the account. On 2026-11-02 at 18:44 a sync took 6.9 s with the PDS slow, and 26 requests for the repository waited up to 18 s behind it. Deleting the old ref records was ruled out: it would let a generation repeat.

How it was checked

live_refs_once_an_account_is_read_whole (index): refs kept, a deleted ref and a deleted record dropped, none for an account no backfill read whole, and a reread asked for by an index that kept no refs before. a_mirror_syncs_from_the_refs_the_index_keeps (web): branches pushed and deleted show through a mirror synced from the index. Left to CI.

Shortcomings

A ref's mirror is as fresh as the index: a push's new tip shows once the feed brings it, as before. The repository settings page counts storage afresh after a sync from the index, which lists no packs. The pack collection is still read from the PDS when a push brought commits.

🤖 Generated with Claude Code

This pull makes a public repository's mirror sync its record and live refs from a new live_ref index table. It does so only while a feed is connected, the owner's account has been read whole, and the pages haven't written for that account in the last 30 seconds. Jetstream's inline ref and repo records are dropped so the index fetches them from the PDS. Existing indexes clear their backfill revisions once so every account is reread. I found no correctness problem: writes go through the as_of guard, deletions clear the row, and the fallback to the PDS is conservative. The tests and the serve.md update cover the change. I did not check CI. One small fix is below: a doc comment is attached to the wrong function.

The new commit only renames heard/heard_if to refresh_mirrors/refresh_if. That is fine. The one fix I asked for last round is still not done. In crates/knotless-serve/src/index.rs, live_ref was inserted between about's doc comment and fn about, so about has no docs and live_ref carries a wrong paragraph above its own. Move live_ref above that comment. Nothing else has changed since my approval. I did not check CI.