The runner host

A box of its own for running other people's workflows: other owners' repositories, agents' pulls, strangers' pulls. The main host keeps knotless-serve, its index, its data and the runner for knotless's own CI. A container escape there reaches all of that and the runner account's credentials. Here it reaches one runner account's app password and the runs going at the time.

What holds a run apart here:

gVisor. Every run's container of code nobody here vouches for starts with --runtime runsc, so its system calls go to gVisor's own kernel, not the host's. Most container escapes are kernel bugs, and those land in gVisor. It runs in systrap mode, which needs no nested virtualization, so a cloud VM will do.
Limits. Each run gets 2 CPUs, 3 GiB with no swap, 4,096 processes and an hour. Each owner gets 2 of the 4 jobs, and an allowance of 2,000 minutes of steps in 30 days (500 for private repositories), past which their runs go one at a time.
The network. Runs join knotless-runs, a bridge (kl-runs) whose containers cannot reach each other. Its egress rules drop the host itself, the private and link-local ranges (the cloud's metadata service at 169.254.169.254 among them) and mail ports (25, 465, 587). The rest of the internet is open: registries, package indexes, git hosts. Runs resolve names through public resolvers (/etc/knotless/resolv.conf, --resolv), since docker's own is out of gVisor's reach.
The operator's stop. /etc/knotless/paused.json pauses an owner, or everyone (*), at the next pass. See --paused in the CLI reference.
Nothing to steal but the runner's own. No serve data, no index, no other account's session. The runner's app password, its OAuth session if it has logged in (Private repositories), and the work under /var/lib/knotless-run are all there is.

Setting it up

Order the box. A Hetzner cloud CX43 (8 vCPU, 16 GB, 160 GB disk, €15.99 a month before VAT as of June 2026), Ubuntu 24.04, with your SSH key, in a location near the main host. In the console, or:
```
hcloud server create --name knotless-runner --type cx43 \
  --image ubuntu-24.04 --location fsn1 --ssh-key <your key>
```
Make the runner's account. A new atproto account, such as runner.knotless.dev, and an app password for it. Its checks and minutes records are written there, so it should not be anyone's own.
Set up the host. In deploy/local.env:
```
KNOTLESS_RUNNER_HOST=root@<its address>
```
then deploy/runner-host/setup.sh. It ends with a check run under gVisor on the runs' network, which should print:
```
internet: answers
metadata: blocked
private: blocked
mail: blocked
kernel: gVisor
```
If the internet does not answer, look at DNS first. Docker's resolver on a user-defined network is reached through NAT rules inside the container's network namespace, which gVisor's network stack does not apply, so runs are given /etc/knotless/resolv.conf instead (--resolv), naming public resolvers.
Deploy the runner. In deploy/local.env, once, its account:
```
KNOTLESS_RUNNER_IDENTIFIER=runner.knotless.dev
KNOTLESS_RUNNER_PASSWORD=xxxx-xxxx-xxxx-xxxx
```
then deploy/runner-host/deploy.sh. It builds the image on the host, as the main host's deploy does, and starts run. It watches every public repository whose record names its account (--discover), as knotless.dev lists them; KNOTLESS_RUNNER_REPOS names more, such as a private one. Its log: ssh root@<host> journalctl -f CONTAINER_TAG=knotless-runner-run-1.
Measure gVisor. Point a test repository's runners at the new account (knotless runners <url> add runner.knotless.dev), push a workflow that builds something real (a fork of knotless, say), and compare its steps' times with the same run on the main host. Expect cargo and npm 10–30% slower. Much worse than that, or a build that fails only under gVisor, is the case for Firecracker, which needs a dedicated server with KVM (an AX42). Check too that a workflow's cache: paths are kept between runs: they are overlay volumes the daemon mounts and gVisor sees as bind mounts.
Log it in, for private repositories. See below.
Invite an owner. They name the runner in their repository: knotless runners <url> add runner.knotless.dev. It is found within seconds, from the record Jetstream brings, and the next push runs. knotless runners <url> remove runner.knotless.dev lets it go. Their minutes show with knotless minutes <url>, and on the repository's checks tab.

Private repositories

An app password mints no delegation tokens, so it reads no one else's private repository. The runner logs in with OAuth instead, through a knotless-serve of its own on this host (bots in docker-compose.yml), which keeps the session and its client key here, never on the main host:

An A and an AAAA record for the name, such as runner.knotless.dev, to this host, not proxied, and ports 80 and 443 open in the cloud's firewall (setup.sh opens them in ufw).
In deploy/local.env:
```
KNOTLESS_RUNNER_DOMAIN=runner.knotless.dev
KNOTLESS_RUNNER_DID=did:plc:…
```
then deploy/runner-host/deploy.sh. It starts bots, and Caddy in front of it, which gets the name a certificate and answers only the bot login's three paths (/oauth/bot, /oauth/bot-callback, /oauth/bot-client-metadata.json).
Open https://runner.knotless.dev/oauth/bot?account=<the runner's handle>, signed in to the runner's account at its PDS, and approve. The session lasts two years if used every three months; run takes it from /var/lib/knotless-run/bots from its next start, and the app password then logs in only if it can no longer be refreshed.

A private repository made with knotless create --private, at an appview that names this runner, makes it a member of the space and invites it; knotless runners <url> add <the runner's handle> does the same for one made before. The runner reads its invites every sweep, as it lists public repositories. A login approved before the invite methods were in knotless's scope cannot read them (invites warnings in the log): open the login page again and approve. So too one from before dev.knotless.check and dev.knotless.minutes were: every run of it fails with Missing required scope "repo:dev.knotless.check?action=create". KNOTLESS_RUNNER_REPOS still names any more to watch; one it cannot read is logged and passed over.

Merge queues

With --queue in KNOTLESS_RUNNER_ARGS, this runner lands the merge queue of every repository that names it and no runner that may push to it, through knotless.dev's gateway once its owner has delegated writes there (Who lands a queue). The gateway takes from it only a batch of queued pulls, squashed in order: a run that broke out and took its account could land what someone queued without its checks passing, but push nothing else. knotless itself is --pulls-only here, so its queue stays on the main host.

knotless's own pulls

knotless's pulls run here too, so they no longer take the main host's CPU from serve. Its pushes, its merge queue's batches and its deploy stay on the main host, where the deploy's key is. In deploy/local.env:

KNOTLESS_RUNNER_ARGS="--queue --pulls-only chadtmiller.com/knotless --unmetered did:plc:bcgltzqazw5tb6k2g3ttenbj --trust did:plc:bcgltzqazw5tb6k2g3ttenbj --pull-jobs 2"
KNOTLESS_RUNNER_JOBS=2
KNOTLESS_RUNNER_MEMORY=6
KNOTLESS_RUNNER_CPUS=4

then deploy/runner-host/deploy.sh. This runner then runs the repository's pulls alone. Those of its owner and collaborators run under runc (--trust, and --trusted-runtime runc in the compose file), not gVisor: under gVisor its test took 18 minutes to the main host's 4, and the batch the pull goes into runs the same code there with no sandbox anyway. Anyone else's pull, run once the owner asks, is under gVisor as on any repository here. They count against no allowance and may take every job, past --owner-jobs. --pull-jobs 2 lets pulls take both jobs, not half: here nearly everything is a pull. Its test workflow builds every test binary with coverage and starts PDSes, which takes more than the default 3 GiB a run: past it the sandbox is killed (containerManager.WaitPID failed: EOF in the log, an OOM kill of gvisor_sentry in dmesg). So two jobs of 6 GiB and 4 CPUs each, on an 8-core, 16 GB host, rather than four that each fail. knotless names this runner's account among its runners (knotless runners knotless://chadtmiller.com/knotless add runner.knotless.dev), so its pulls' checks show. The main host's runner takes --queue without --pulls (KNOTLESS_RUN_ARGS in .knotless/workflows/deploy.yml). Nothing here can push to knotless or reach the main host. A run that broke out could file a false pull check, but the queue checks each batch again on the main host, and a deploy waits for main's own checks, run there too.

No push runs here, so its caches' bases are written by knotless's own pulls: an owner's or collaborator's passing run into main promotes its layers, as a push's would on the main host, but only over the base it started from and never from an older main than that base's (see Where pushes run elsewhere). The first pull over an empty cache builds in full; the ones after start from what it left, the test workflow's stored coverage runs among it.

Running it

Pause someone: edit /etc/knotless/paused.json on the host, as {"did:plc:…": "mining, by the looks of it"}, or {"*": "maintenance"} for everyone. No restart: the next pass reads it.
Grant a secret: /etc/knotless/run-secrets.json, as on the main host. Tenants' secrets are not offered yet.
Disk: images unused for a week are pruned daily; caches are bounded by --cache-max (16 GiB); checkouts go as their runs end.
Grow: when checks wait long for a job on most days, rescale the server to a CX53 (16 vCPU, 32 GB) in the Hetzner console: a reboot, the disk kept. Then raise KNOTLESS_RUNNER_JOBS to 8 and deploy again.

Not yet

Hostname filtering. Egress is filtered by address, not name, so known mining pools and proxies are not blocked. That wants an egress proxy.
Rootless docker. The daemon runs as root, and the runner holds its socket. gVisor is the wall between a run and the host. A user-namespaced daemon would be a second one, once the caches' overlays are shown to work under it.

chadtmiller.com/knotless Switch repository Loading…