The runner host

A box of its own for running workflows: other owners' repositories, agents' pulls, strangers' pulls, and every workflow of knotless's own that holds no secret, its pushes and merge queue among them. The main host keeps knotless-serve, its index and its data, and runs no runner. The workflows that hold a secret, deploying among them, run on the trusted runner, on another box again. A container escape here reaches one runner account's app password and the runs going at the time.

What holds a run apart here:

gVisor. Every run's container of code nobody here vouches for starts with --runtime runsc, so its system calls go to gVisor's own kernel, not the host's. Most container escapes are kernel bugs, and those land in gVisor. It runs in systrap mode, which needs no nested virtualization, so a cloud VM will do.
Limits. Each run gets 2 CPUs, 3 GiB with no swap, 4,096 processes and an hour. Each owner gets 2 of the 4 jobs, and an allowance of 2,000 minutes of steps in 30 days (500 for private repositories), past which their runs go one at a time.
The network. Runs join knotless-runs, a bridge (kl-runs) whose containers cannot reach each other. Its egress rules drop the host itself, the private and link-local ranges (the cloud's metadata service at 169.254.169.254 among them) and mail ports (25, 465, 587). The rest of the internet is open: registries, package indexes, git hosts. Runs resolve names through public resolvers (/etc/knotless/resolv.conf, --resolv), since docker's own is out of gVisor's reach.
The operator's stop. /etc/knotless/paused.json pauses an owner, or everyone (*), at the next pass. See --paused in the CLI reference.
Nothing to steal but the runner's own. No serve data, no index, no other account's session. The runner's app password, its OAuth session if it has logged in (Private repositories), and the work under /var/lib/knotless-run are all there is.

Setting it up

Order the box. A Hetzner cloud CX43 (8 vCPU, 16 GB, 160 GB disk, €15.99 a month before VAT as of June 2026), Ubuntu 24.04, with your SSH key, in a location near the main host. In the console, or:
```
hcloud server create --name knotless-runner --type cx43 \
  --image ubuntu-24.04 --location fsn1 --ssh-key <your key>
```
Make the runner's account. A new atproto account, such as runner.knotless.dev, and an app password for it. Its checks and minutes records are written there, so it should not be anyone's own.
Set up the host. In deploy/local.env:
```
KNOTLESS_RUNNER_HOST=root@<its address>
```
then deploy/runner-host/setup.sh. It ends with a check run under gVisor on the runs' network, which should print:
```
internet: answers
metadata: blocked
private: blocked
mail: blocked
kernel: gVisor
```
If the internet does not answer, look at DNS first. Docker's resolver on a user-defined network is reached through NAT rules inside the container's network namespace, which gVisor's network stack does not apply, so runs are given /etc/knotless/resolv.conf instead (--resolv), naming public resolvers.
Deploy the runner. In deploy/local.env, once, its account:
```
KNOTLESS_RUNNER_IDENTIFIER=runner.knotless.dev
KNOTLESS_RUNNER_PASSWORD=xxxx-xxxx-xxxx-xxxx
```
then deploy/runner-host/deploy.sh. It builds the image on the host, as the main host's deploy does, and starts run. It watches every public repository whose record names its account (--discover), as knotless.dev lists them; KNOTLESS_RUNNER_REPOS names more, such as a private one. Its log: ssh root@<host> journalctl -f CONTAINER_TAG=knotless-runner-run-1.
Measure gVisor. Point a test repository's runners at the new account (knotless runner add <url> runner.knotless.dev), push a workflow that builds something real (a fork of knotless, say), and compare its steps' times with the same run on the main host. Expect cargo and npm 10–30% slower. Much worse than that, or a build that fails only under gVisor, is the case for Firecracker, which needs a dedicated server with KVM (an AX42). Check too that a workflow's cache: paths are kept between runs: they are overlay volumes the daemon mounts and gVisor sees as bind mounts.
Log it in, for private repositories. See below.
Invite an owner. They name the runner in their repository: knotless runner add <url> runner.knotless.dev. It is found within seconds, from the record Jetstream brings, and the next push runs. knotless runner remove <url> runner.knotless.dev lets it go. Their minutes show with knotless runner minutes <url>, and on the repository's checks tab.

Private repositories

An app password mints no delegation tokens, so it reads no one else's private repository. The runner logs in with OAuth instead, through a knotless-serve of its own on this host (bots in docker-compose.yml), which keeps the session and its client key here, never on the main host:

An A and an AAAA record for the name, such as runner.knotless.dev, to this host, not proxied, and ports 80 and 443 open in the cloud's firewall (setup.sh opens them in ufw).
In deploy/local.env:
```
KNOTLESS_RUNNER_DOMAIN=runner.knotless.dev
KNOTLESS_RUNNER_DID=did:plc:…
```
then deploy/runner-host/deploy.sh. It starts bots, and Caddy in front of it, which gets the name a certificate and answers only the bot login's three paths (/oauth/bot, /oauth/bot-callback, /oauth/bot-client-metadata.json).
Open https://runner.knotless.dev/oauth/bot?account=<the runner's handle>, signed in to the runner's account at its PDS, and approve. The session lasts two years if used every three months; run takes it from /var/lib/knotless-run/bots from its next start, and the app password then logs in only if its server turns the refresh down (invalid_grant). A refresh that fails otherwise is logged (login refresh failed) and tried again, the OAuth session kept: a password login saved over it would read no private repository again, every one failing with Scope does not permit reading. Should that happen, approve the login again.

A private repository made with knotless repo create --private, at an appview that names this runner, makes it a member of the space and invites it; knotless runner add <url> <the runner's handle> does the same for one made before. The runner reads its invites every sweep, as it lists public repositories. A login approved before the invite methods were in knotless's scope cannot read them (invites warnings in the log): open the login page again and approve. So too one from before dev.knotless.check and dev.knotless.minutes were: every run of it fails with Missing required scope "repo:dev.knotless.check?action=create". A PDS grants a bare space: scope its type's collections only as the login is approved, so one approved at a PDS that did not do so yet, as pds.js before November 4, 2026, keeps a grant that writes nothing to a space, and every refresh keeps it too: each private repository's run fails with Scope does not permit update in at://…/space/…. Approve the login again once the PDS is updated. KNOTLESS_RUNNER_REPOS still names any more to watch; one it cannot read is logged and passed over.

Merge queues

With --queue in KNOTLESS_RUNNER_ARGS, this runner lands the merge queue of every repository that names it and no runner that may push to it, through knotless.dev's gateway once its owner has delegated writes there (Who lands a queue). The gateway takes from it only a batch of queued pulls, squashed in order: a run that broke out and took its account could land what someone queued without its checks passing, but push nothing else. knotless's own queue lands here too: this runner is the first its repository names, and none it names may push.

knotless's own CI

knotless's checks run here: its pushes, its pulls and its merge queue's batches, so none of them takes the main host's CPU from serve. Each of its workflows that holds no secret names this runner in runners; the rest name the trusted runner. In deploy/local.env:

KNOTLESS_RUNNER_ARGS="--queue --unmetered did:plc:bcgltzqazw5tb6k2g3ttenbj --trust did:plc:bcgltzqazw5tb6k2g3ttenbj --pull-jobs 4"
KNOTLESS_RUNNER_JOBS=4
KNOTLESS_RUNNER_MEMORY=12
KNOTLESS_RUNNER_CPUS=4

then deploy/runner-host/deploy.sh. Pushes to its owner's repositories, and the pulls and batches of its owner and collaborators, run under runc (--trust, and --trusted-runtime runc in the compose file), not gVisor: under gVisor its test took 18 minutes to runc's 4. Anyone else's pull, run once the owner asks, is under gVisor as on any repository here, and so is a batch holding one. They count against no allowance and may take every job, past --owner-jobs. --pull-jobs 4 lets pulls take every job, not half. Its test workflow builds every test binary with coverage and starts PDSes, which takes more than the default 3 GiB a run: past it the sandbox is killed (containerManager.WaitPID failed: EOF in the log, an OOM kill of gvisor_sentry in dmesg). So four jobs of 12 GiB and 4 CPUs each, on a dedicated 6-core, 12-thread, 64 GB host (an AX41), which share its threads only when all four are busy. Within them, cargo would still start a rustc for each of the host's cores, and knotless-serve's takes more than 3 GiB under coverage: harness/cov.mjs builds as many crates at once as the cgroup's CPUs, and one for each 4 GiB of its memory at most, so 3 here (rustc killed by oom-killer in dmesg is this, past those). knotless names this runner's account first among its runners (knotless runners knotless://chadtmiller.com/knotless), so it lands the queue. Its pushes to main run here, so they write its caches' bases, as on any host where a repository's pushes run.

Nothing here can push to knotless or reach the main host. A run that broke out could file a false check, or land through the gateway what a maintainer queued without its checks passing. A deploy waits for main's checks, filed here, so such a run could also have main deployed with checks that did not pass, though only of what was queued.

The trusted runner

The workflows that hold a secret run on a runner of their own, as ci.knotless.dev, on the box that already holds the infrastructure's tokens (hetzner-infra's infra-runner), beside that box's own runner: deploying knotless.dev (DEPLOY_SSH_KEY, RELEASE_PASSWORD), releasing the client, publishing the lexicons, the scenarios and the docs audit, and the owner's deploys of pds.js and grain. It runs no one's code but its owner's: it watches only the repositories it names (KNOTLESS_RUNNER_DISCOVER empty), and each workflow it runs names it in runners, the rest of those repositories' naming this runner. Every run there is trusted, so under runc.

It is a second project of this directory's compose file on that host, with a directory of its own for run.env, the secrets and the pause list, /etc/knotless-ci, and its own work, /var/lib/knotless-ci. From a deploy/ci.env (not committed; deploy/ci.env.example shows it):

KNOTLESS_RUNNER_ENV=deploy/ci.env deploy/runner-host/deploy.sh

It takes the image the main host built (KNOTLESS_RUNNER_IMAGE_FROM), as that box is too small to build knotless, and has no bot login: what it watches is public. Its secrets are those of /etc/knotless-ci/run-secrets.json, and those owners seal to its key (knotless secret <url> set <name> --runner ci.knotless.dev), kept in secrets-key.json under its work: moving it to another box means moving that file, or sealing them all again.

Deployed with main

Each deploy of main deploys this host too, last (.knotless/workflows/deploy.yml, its runner host step), so the runner that lands other repositories' merge queues reads their records as main does: an older one ejects a pull whose rule holds what it does not know (what this version does not know: gates; update knotless). It runs deploy.sh with KNOTLESS_RUNNER_IMAGE_FROM the main host, whose image it copies only when its runner's build (the fingerprint the main host's deploy compares) differs from this host's: a deploy that leaves the runner as it was restarts nothing here, as on the main host. Its settings are the KNOTLESS_RUNNER_* in that workflow, which deploy/local.env repeats for a deploy by hand.

It reaches the host with RUNNER_HOST_SSH_KEY, a key of its own, granted to pushes to main alone (knotless secret set … RUNNER_HOST_SSH_KEY --ref refs/heads/main). Root takes it only from the trusted runner's addresses, as setup.sh installs its public half:

KNOTLESS_RUNNER_DEPLOY_KEY="ssh-ed25519 AAAA… knotless deploy of main" \
KNOTLESS_RUNNER_DEPLOY_FROM="178.105.255.120,2a01:4f8:c016:89c6::1" \
  deploy/runner-host/setup.sh

Running it

Pause someone: edit /etc/knotless/paused.json on the host, as {"did:plc:…": "mining, by the looks of it"}, or {"*": "maintenance"} for everyone. No restart: the next pass reads it.
Grant a secret: /etc/knotless/run-secrets.json, as on the main host. Tenants' secrets are not offered yet.
Disk: images unused for a week are pruned daily; caches are bounded by --cache-max (16 GiB); checkouts go as their runs end.
Move: run setup.sh on the new host, with the deploy key. Point runner.knotless.dev at it, stop the old runner, copy /etc/knotless/run.env, run-secrets.json and paused.json, /var/lib/knotless-run's state.json, secrets-key.json, bots and config, and /var/lib/knotless-serve, then run deploy.sh. Caches build again on the first run.

Not yet

Hostname filtering. Egress is filtered by address, not name, so known mining pools and proxies are not blocked. That wants an egress proxy.
Rootless docker. The daemon runs as root, and the runner holds its socket. gVisor is the wall between a run and the host. A user-namespaced daemon would be a second one, once the caches' overlays are shown to work under it.

johnandersen777.bsky.social/knotless Switch repository Loading…