chadtmiller.com/knotless

Switch repository

Loading…

What changes

The prod host's runner runs two jobs of 3 CPUs and 6 GiB. deploy.yml gives it --queue --jobs 2 --cpus 3 --memory 6 rather than --queue alone, which left it the default four jobs with no limits. harness/cov.mjs already sizes cargo's jobs from the container's limits, so a test builds within them. A run that goes past its memory now fails its step, not the host.

setup.sh gives a host a 4 GB swap file, with swappiness 10, so it is used only under pressure. Running it again leaves it as it is.

Why

Prod stalled for several minutes today and had to be reset from Hetzner. The queue's batch and two queued pulls' pre-flight test runs went at once on the host that serves knotless.dev. One coverage test uncapped takes about 9.4 GiB and 7.5 of its 8 cores, and the host has 16 GB with no swap. With three running, it ran out of memory and kept dropping and rereading its programs from disk: 1.1 GB/s of reads while network traffic fell to nothing, and SSH timed out too.

deploy.yml's comment assumed only main's own runs happen here, but a queued pull's pre-flights run here too. Two capped jobs hold CI to 6 cores and 12 GB, leaving serve, pages and the bots 2 cores and about 4 GB. They use about 1 GB. Swap is the margin for the rest.

How it was checked

Both are already on prod by hand: the swap file is on, and /etc/knotless/.env has the new arguments, with run recreated. This pull makes them what the next deploy and setup.sh give a host. bash -n on setup.sh. The rest is left to CI.

Shortcomings

6 GiB at 3 CPUs for test is an estimate: the runner host's test needs more than 3 GiB, and passes at 7 GiB with 4 CPUs. If the next batch's test is killed for memory, raise it to 7. The runner host gets no swap here.

Changeset: none

🤖 Generated with Claude Code

This caps the prod runner at --queue --jobs 2 --cpus 3 --memory 6 in deploy.yml, and setup.sh now adds an idempotent 4 GB swap file with swappiness 10. The README and the script header are updated to match. I checked that --jobs, --cpus and --memory are real flags in knotless-run's lib.rs. The swap block is guarded so a second run changes nothing, and it adds the fstab entry only once. I found nothing that must change.

The 6 GiB / 3 CPU figure is an estimate, and the description says so. Its fallback is to raise memory to 7 if a batch's test is killed. I have not run the script.