chadtmiller.com/knotless

Switch repository

Loading…

What changes

A floor of free disk. knotless-run keeps --disk-floor GiB free on the disk its --caches are on (20 by default, 0 for none). Before a run starts, it checks free space with statvfs, holding its job so no other run starts meanwhile. When the disk is below the floor, the run sheds first: it removes every kept run's layers and every cache's base no run reads, so those caches' next runs start afresh. If that is not enough, the run waits, checking every 30s, until the runs going end and free their layers. It stays queued and can be cancelled as usual. Its log says what was shed and how long it waited, and the runner logs disk short at warn.

Testable. Caches::open_with takes the source of free space, and the tests drive it with fake numbers under paused time.

Why

On 2026-10-31 prod's disk filled and knotless.dev answered 502 (No space left on device). The runner's caches (38 GB of bases) share that disk with the site's repositories and index, and one cargo build can write 10 GB at once. When it comes to space, the site must win over CI. 20 GiB is about two of the largest bases the host has held, so one run's build landing on the floor still leaves the site room. Shedding caches costs a cold build. Waiting costs time. Both are cheaper than the site going down.

How it was checked

New unit tests: short_of_disk_it_sheds_what_no_run_reads (caches.rs) checks that kept layers and unread bases go, while bases a run reads stay. a_run_waits_for_room_on_the_disk (lib.rs) covers the waits: room, no floor, an unreadable disk, shedding then waiting, and a shed that fails. Coverage and clippy are left to CI.

Shortcomings

The floor holds only at a run's start. A run already going can still fill the disk. A run waiting for room holds its job, so other runs wait behind it too. That is deliberate, but four jobs can sit idle while the disk stays short.

Changeset: none

🤖 Generated with Claude Code

This adds --disk-floor (20 GiB by default). Before a run starts, and while it holds its job, the runner checks free space with statvfs. If the disk is short, it sheds every kept run's layers and every base no run reads, then polls every 30s until there is room. The code matches the design, and the docs, CLI reference and tests are updated with it. I didn't run the tests or read the checks. I found one nit. Separately, a run can wait forever when nothing is running to free space, which you already list under Shortcomings. A short note in the docs saying so would help operators. Neither needs to hold up the merge.

This round only merges main and sets disk_floor: 0 in the integration tests' runner options. That keeps those runs from depending on the host's real free space, and it is correct. The floor logic is unchanged: shed once, then poll every 30s while holding the job. The docs, CLI reference and tests still match it. I didn't run the tests or read the checks.

My earlier nit is still open. The two-line "On paused time…" comment at lib.rs 3837–3838 sits above a_run_waits_for_room_on_the_disk and not above a_long_pass_is_watched, which it describes. Move the new test above that comment. It's cosmetic and doesn't need to hold up the merge.

This round reworks the disk-floor test so it counts looks at the disk instead of racing the clock. The first look finds the disk short, the shed happens, and then one look per 30s runs until there is room. The assertion waited 90s for room on the disk follows from that count. The floor logic is unchanged and still matches the docs. I didn't run the tests or read the checks.

My earlier nit is still open. The two-line "On paused time…" comment at lib.rs 3837–3838 is above a_run_waits_for_room_on_the_disk, but it describes a_long_pass_is_watched (the 5ms tick against the 30ms pass). Move the new test above that comment. It's cosmetic and doesn't need to hold up the merge.