chadtmiller.com/knotless

Switch repository

Loading…

What changes

deploy.sh ends by removing the images no tag names and no container runs (docker image prune -f). Those are the knotless:current images that earlier deploys replaced. If a container still runs one, as run may until its backgrounded restart finishes, that image goes with the next deploy.

Why

On 2026-10-31 prod's 150 GB disk filled, and knotless.dev answered 502 with git index-pack: fatal: write error: No space left on device. Each deploy leaves the 1.2 GB image it replaced untagged. The daily knotless-prune timer only removes images more than a week old, and dozens of deploys a day had left about 150 of them. Pruning those by hand freed 26 GB and brought the site back. Pruning the untagged images at each deploy keeps only what is running. The weekly prune of other images stays as it was.

How it was checked

Ran docker image prune -f by hand on the prod host: 26.37 GB freed and nothing running was touched. serve and pages stayed healthy, and the disk went from 100% to 74%. The deploy step itself first runs at the deploy after this lands.

Shortcomings

The runner's cache bases (/var/lib/knotless-run/caches/bases, 38 GB) and buildkit's cache (49 GB) are still the largest things on the disk. Both look bounded by their own cleanup, but nothing alerts before the disk fills.

🤖 Generated with Claude Code

This adds docker image prune -f at the end of deploy.sh, so each deploy removes the untagged images it replaced, and the README step is updated to match. Docker's prune skips images a container uses, so the running run image is safe. || true keeps a prune failure from failing a deploy that has already gone out. I found no rollback path in deploy/ that relies on keeping replaced images. The weekly knotless-prune timer is unchanged. I didn't run it. The author says the deploy step has only been run by hand on prod. Approving.