chadtmiller.com/knotless

Switch repository

Loading…

What changes

When a run ends, knotless-run removes its copy-on-write layers under caches/runs/<run>. A removal that fails is now tried again, up to three times in all, instead of being dropped silently. Before, the layer stayed until the runner next started, and it counted against --cache-max until then.

Why

a_pull_runs_its_head failed in the test run of Coverage builds as many crates at once as a run's CPUs and memory allow and passed on retry, logged as FLAKY. It timed out at run.rs:2147, waiting for caches/runs to empty after an agent's pull ran.

In that test, the bot workflow writes a file into every run's layer, to show that nothing it writes is kept. When the pull's other workflow, pulls, finished while bot's loop was writing, the remove_dir_all in Lease::drop met a file made after it had emptied the directory. It failed with "directory not empty". let _ = ignored that, so the layer stayed for good.

On the runner hosts a run's container can't reach another run's layer, so this is mostly the test's race. Still, a removal that fails once shouldn't leave a layer behind. The retry is one any on one line, so the coverage gate doesn't depend on whether the race happens.

How it was checked

CI's test runs a_pull_runs_its_head on both nets. I don't know of a way to make the race happen at will.

Shortcomings

A run that keeps writing into another's layer could still outlast three tries. Only the shell executor allows that, which is the tests' executor.

Changeset: none

🤖 Generated with Claude Code

Lease::drop now tries remove(&self.dir) up to three times, stopping at the first success. I read remove in caches.rs. It returns Ok when the path is already gone, so a retry after a partial removal is safe. It also stops at the first success, so no extra attempts run in the normal case. The change is small and correct for the race you describe: remove_dir_all fails with "directory not empty" when a file appears after the directory was emptied, and the next pass clears it.

I have two minor points, neither blocking:

• The retries run back to back with no pause, so a writer that keeps going could outlast all three. You already list this under Shortcomings.
• I didn't look at the checks, and I can't confirm the flake is fixed. You say you can't reproduce the race at will, so the test run doesn't prove it either.