chadtmiller.com/knotless

Switch repository

Loading…

What changes

Reading a repository's ref records (Remote::refs, which every snapshot uses) first reads one page as before. A repository whose refs fit on that page costs the same one call. When the page is full, the records below it are read in up to eight ranges side by side, not as pages one after another. The ranges are cut at fixed first letters of the hash that ends each ref's key. Each range reads newest first from where it starts and keeps only records at or above its lower end. The records come back in the same order as before, newest first. scan_all stays as it was for the other collections read whole (packs and claims in compaction, checks, releases, reruns).

Why

The new mirror sync line shows that a sync of chadtmiller.com/knotless spends almost all of its 1.2–1.4s reading records (records_ms). That repository has 590 ref records and only 49 live branches. A deleted ref keeps its record so its generation never repeats (storage.md), so the count grows with every branch an agent pushes and deletes. They were read as 6 or 7 pages one after another at 200–330ms each. Measured from here against the same PDS, the same 590 records took 1.73s sequentially and 0.37s in 8 ranges side by side. Every read of the repository waits on its sync, and every push and clone reads the same records.

The other option was to have knotless-serve patch its last snapshot from the feed's events. But Jetstream sends a commit's records as separate events and never says how many there are, so a patch could miss a ref and never learn of it. Reading everything stays correct by construction.

How it was checked

A unit test, ranges_cover_what_is_below_the_first_page, checks where the ranges are cut. The library test many_refs_many_packs_many_repos takes a snapshot of 106 refs, which reads the first page and then the ranges below it, and checks it gets all 106. Every other test reads its refs in one page. Running them is left to CI. After deploy, the check is records_ms in the mirror sync lines for chadtmiller.com/knotless, against about 1,250ms.

Shortcomings

The cost still grows with the number of deleted branches, now divided among the ranges. Eight ranges allow about 800 records past the first page before any range takes a second page. Cutting finer would only add calls for most repositories.

🤖 Generated with Claude Code

This pull makes Remote::refs read the first page as before. If that page is full, it reads the rest in up to eight key ranges at once, cut at fixed first letters of the hash that ends each key.

I read it against scan and found no fault:

• The cursor is exclusive and keys sort descending, so a range starting at a cut key k.c returns keys below it. A range's records are >= low, so the ranges don't overlap and leave no gap.
• Joining the ranges highest first after the first page keeps the old newest-first order.
• scan_down stops paging once a page passes low, and scan already stops at keys that aren't this repository's.
• The pull includes the changeset and the storage.md update that AGENTS.md asks for.

I couldn't see the checks or the many_refs_many_packs_many_repos test, so I haven't confirmed that CI's 100% coverage gate passes. A range that fills a whole page without passing its low is rare and is covered only if some test reaches it. Check that check and test are green before queueing.

Nothing has changed in the code since my last review. The new commits are a merge of main, a changeset note, and a commit message that says scan_all stays for the other collections. The logic is the same as before: the first page is read as it was, and when it is full the rest is read in up to eight ranges at once. Each range keeps only keys at or above its low end, the ranges are joined highest first, and the old newest-first order holds. The changeset and the storage.md update are there, and the unit test covers where the ranges are cut. The check, test, e2e and docs runs were still queued when I looked, and the web, shots and lexicons runs were skipped. I haven't seen the CI coverage gate pass, so let the queue's checks finish before it lands.

The only change since my last review is in ranges in remote.rs. It now collects the range starts into a Vec before zipping them with the lows. The pairs it builds are the same as before, so the reading logic is unchanged. My earlier points still hold: the ranges don't overlap or leave gaps, order stays newest first, and the changeset and storage.md update are there. When I looked, check had passed and test, e2e and docs were still running. I haven't seen the coverage gate pass, so let those finish before it lands.