chadtmiller.com/knotless

Switch repository

Loading…

What changes

A branch's own caches. Of a workflow whose on.push.refs and on.pull.base both name a branch other than the default one, as both naming refs/heads/feature/* do, that branch keeps caches of its own. A passing push to it promotes into them, as a push to the default branch promotes into its own. Pulls into it, and the merge queue's batches on it, start from them. Under --pulls-only, a trusted pull into it promotes into them, ordered by the branch's own tips in marks.json.

Fallback. A branch with no base of its own yet reads the default branch's, so a new feature branch starts as warm as main. Its first passing push makes its first base from that one and what the push wrote. A layer kept over the default's base is taken only while the branch still has none and the default's has not moved on. A run of the branch reading the default's base keeps that base while main moves on, and it goes once the run ends.

Pulls landed on a feature branch are taken, not run again. A push to a branch at the same tree as a trusted pull's passing run into that branch takes the run's result and its kept layers, as a push to main does (Subject::by_tree covers every branch push, not just main's). grain merges into feature/rust by fast-forward, so a merged pull warms the branch without running again.

Garbage collection. Each pass over a repository drops the caches of branches it no longer has (Caches::drop_branches, beside drop_repo). A base still being read goes once its run ends. shed and the disk floor count branch bases like any others.

Layout. The default branch's caches keep their names, bases/knotless-cache-<repo>-<what>/<generation>. A branch's are knotless-cache-<repo>-<what>-<hash of branch ref>. So the bases on disk now are main's as they are, no migration is needed, and none goes cold on deploy. kept.json layers gain an optional read, set only when the layer was over the default's base. Files written without it read as before.

Nothing else changes. The default branch's behaviour is the same. Untrusted pulls' writes still reach no base. Batches still never write. A branch only pushed to, which no pull goes into, still reads main's caches and keeps nothing.

Why

Only a passing push to the default branch made a base, so a workflow that exists only on a feature branch never had one. grain's rust.yml, harness.yml and e2e.yml exist only on feature/rust, so every pull into it starts cold: Node downloaded, cargo from nothing, the spaces alpha rebuilt (about 10 of e2e's 12 minutes), Chromium installed. knotless's AGENTS.md sends big work to feature/<name> branches too.

Options for which branches get bases:

• feature/* hard-coded: bakes a naming convention into the runner.
• Any branch on.push names: under a workflow with no on, every topic branch pushed would grow a multi-GB base. It would also change what vol.yml in tests/run.rs relies on, where pushes to a and b keep nothing.
• on.push and on.pull.base both naming it (chosen): a base pays only where runs start from it. Pulls into a branch are where they do, and it is also what lets the branch's push take a pull's kept layers. It is opt-in per workflow, from the workflow file.

How it was checked

New tests in caches.rs:

• a_branch_starts_from_the_default_and_keeps_its_own: a branch falls back to main's base, makes its own on its first push, and stays put as main moves. A new branch starts from main as main is now. A kept pull layer over the fallback is taken into the branch's first base, and is not taken once the branch has its own base or main has moved. A read of main's base by a branch run holds that base. --pulls-only advances over the fallback. Reopening keeps each branch's own base.
• a_deleted_branchs_caches_go: drop_branches removes only the deleted branch's caches, a base a run reads once that run ends. The default branch's caches and another repository's stay.

In lib.rs, whose_branch_the_caches_are covers which runs read a branch's own caches, which promote, and how they are named. whose_layers_are_kept now also covers warms for pulls into a branch with its own caches and into one without.

The unit tests ran here once (cargo test -p knotless-run --lib -- caches:: whose_). The rest is left to CI.

Shortcomings

• A branch's bases cost disk beside main's: hard links where nothing changed, but a build's target diverges. They are bounded by --cache-max per base, by deletion of the branch, and by the disk floor's shedding, which sheds main's as well.
• A push to a branch at a pull's tree now looks up its tree on every branch push, not just main's: one rev-parse each.
• A branch renamed keeps nothing: its caches go with the old name, and the new one starts from main's.
• Changing the default branch does not carry its caches over.
• Not deployed. grain must add refs/heads/feature/* to on.push.refs of the workflows it wants warm before any of this applies there.

Review

The design: the rule for which branches get bases (own_branch in lib.rs), and Bases::start/ahead and take in caches.rs, where a layer over the fallback base is matched.

🤖 Generated with Claude Code

This gives a branch its own caches when its workflow names that branch in both on.push.refs and on.pull.base. The branch falls back to the default branch's base until its first passing push, and its caches are dropped when the branch is deleted.

I read own_branch, promotes, keeps and warms in lib.rs, and Bases::start, Bases::ahead, take, promoting, drop_branches and the Drop for Lease in caches.rs. I did not read the full 63k-character diff, the new tests or the Subject::by_tree change. I found nothing wrong in what I read:

• Readers are counted against the cache actually read (layer.read), so a fallback base is held until the run ends.
• take only accepts a kept layer while start still gives the same (read, generation).
• The default branch's cache names are unchanged.
• The read field on kept.json layers is optional, so existing files read as before.

CI is not done yet: the test, check and docs runs were still queued when I looked, and the other workflows were skipped. The caches.rs and lib.rs tests need to pass there before this merges.

Two small things to check, neither a blocker:

• drop_branches removes the caches of every branch missing from snap.live_refs(). That is safe as long as a snapshot's refs is always complete, which I did not check. If a snapshot can ever be partial, it would drop live branches' bases.
• repo_of and branch_of split names on -, so they rely on keys::ref_hash never producing a -.