chadtmiller.com/knotless

Switch repository

Loading…

What changes

knotless's checks run on the runner host. Every workflow that holds no secret (check, test, web, lexicons, shots, docs, e2e, look, ios) names runner.knotless.dev in runners. The runner host then runs knotless's pushes and merge queue batches as well as its pulls, its owner's under runc as now. deploy.yml drops --pulls-only for knotless and gives it four jobs of 4 CPUs and 7 GiB, on the dedicated AX41 it now runs on, which share its 12 threads only when all four are busy.

The workflows that hold a secret run on a trusted runner. deploy, release, publish-lexicons, scenarios and docs-audit name ci.knotless.dev. That runner moves to hetzner-infra's infra-runner box, beside its own runner, and watches only the owner's repositories. deploy/runner-host/docker-compose.yml and deploy.sh take KNOTLESS_RUNNER_NAME, _ETC and _DATA, so one host holds a second runner, and KNOTLESS_RUNNER_ENV names a file read in place of deploy/local.env for it (deploy/ci.env.example).

The main host runs no runner. deploy.yml no longer sets KNOTLESS_RUN_REPOS. Without it, deploy.sh up stops and removes a run an earlier deploy left, once its own run is filed.

A deploy's build takes at most four CPUs. deploy.sh caps its build cgroup with cpu.max when KNOTLESS_BUILD_CPUS is set, and deploy.yml sets 4.

The runner host's deploy key is taken from the trusted runner's box. deploy.yml reaches that host from ci.knotless.dev, which is no longer on the main host, so its root takes RUNNER_HOST_SSH_KEY from infra-runner's addresses.

Why

The main host is meant to run knotless.dev and nothing else. Its CPU went to the queue's batches, and two tests at once stalled it. While pre-flights and batches run there on two jobs, they wait there as the runner host stands idle. The runner host can land the queue without push rights, through the gateway, which takes from it only squashes of pulls a maintainer queued. What must not go there is a secret: the deploy key, the owner's release password, the API keys. Those workflows go to the box that already holds the infrastructure's tokens.

This is the first step of putting knotless.dev on a box it shares, once nothing on it runs CI.

How it was checked

The scripts parse (bash -n), and the AX41 runs the runner host's compose file and deploy today. The rest is checked on the hosts as they go up, in this order: the runner host given four jobs, without --pulls-only, and named first in knotless runner list; this pull queued, which that runner then lands; its deploy, run by ci.knotless.dev on the main host, removing that runner; then ci.knotless.dev started on infra-runner, with its run.env, run-secrets.json and secrets-key.json copied over.

Changeset: none

Shortcomings

• ci.knotless.dev on infra-runner starts with cold caches, one job, and is not deployed with main, so it is updated by hand. A long release holds a deploy back meanwhile.
• A runner restart on the runner host, as a deploy that changes it does, pauses all of knotless's checks and the queue until it stops, as it does not hand over.
• A run that broke out on the runner host could now file main's checks falsely, and so have main deployed with checks that did not pass, though only of what a maintainer queued.

🤖 Generated with Claude Code

This moves knotless's secret-free workflows to runner.knotless.dev and its secret-holding ones (deploy, release, publish-lexicons, scenarios, docs-audit) to ci.knotless.dev. It drops KNOTLESS_RUN_REPOS from the main host and makes deploy.sh remove a leftover run. It also parameterises the runner-host compose file and deploy.sh for a second runner. I read the workflow changes, the new remove mode, and the $0/$1 change in the detached shell, and found nothing wrong. The mode and settings are passed through consistently, and grep -vx run keeps compose from starting run by name.

One thing to check when you roll it out. This pull's own deploy workflow names ci.knotless.dev, which doesn't exist until the last step of your list. Your order depends on the deploy being filed and picked up once ci.knotless.dev is running. If ci.knotless.dev misses that push, run it with knotless rerun. Otherwise main's run stays on the main host.

install -d -m 0700 '$DATA' in deploy/runner-host/deploy.sh also runs against the existing /var/lib/knotless-run on the runner host and would tighten its mode there. I couldn't tell whether anything depends on the old mode. I'd check after the first deploy that the runner and bots still work.

The only change since round 1 is a new optional KNOTLESS_BUILD_CPUS. When set, deploy.sh writes it to the build cgroup's cpu.max, and deploy.yml sets it to 4. When unset, it writes max. The script passes the value to the remote shell as $1. The write is best-effort (|| true), so a host without the cpu controller just builds uncapped. I found nothing wrong. My earlier notes still apply: the order of the rollout steps, and checking the runner and bots after the install -d -m 0700 on the runner host. I didn't see anything in this round that changes them.

The only change since round 2 is a merge of main into the branch. The earlier changes are intact: secret-free workflows on runner.knotless.dev, secret-holding ones on ci.knotless.dev, run removal in deploy.sh, the second-runner parameterisation of the compose file, and KNOTLESS_BUILD_CPUS.

I read the merged diff, including the conflict-prone files (deploy.yml, deploy.sh, the runner-host README and compose file). The remove and restart paths, $ETC/$DATA quoting in deploy/runner-host/deploy.sh, and the compose variable defaults are consistent. I found nothing new to ask for.

My earlier notes still apply, and neither is a reason to hold the merge:

• Roll out in your stated order. This pull's own deploy needs ci.knotless.dev running to be filed. If that runner misses the push, rerun it so main's run is removed.
• The install -d -m 0700 on /var/lib/knotless-run tightens its mode on the existing runner host. After the first deploy, check that the runner and bots still work.