chadtmiller.com/knotless

Switch repository

Loading…

# The runner host

A box of its own for running other people's workflows: other owners'
repositories, agents' pulls, strangers' pulls. The main host keeps
knotless-serve, its index, its data and the runner for knotless's own CI.
A container escape there reaches all of that and the runner account's
credentials. Here it reaches one runner account's app password and the
runs going at the time.

What holds a run apart here:

- **gVisor.** Every run's container of code nobody here vouches for
  starts with `--runtime runsc`, so its
  system calls go to gVisor's own kernel, not the host's. Most container
  escapes are kernel bugs, and those land in gVisor. It runs in
  `systrap` mode, which needs no nested virtualization, so a cloud VM
  will do.
- **Limits.** Each run gets 2 CPUs, 3 GiB with no swap, 4,096 processes
  and an hour. Each owner gets 2 of the 4 jobs, and an allowance of
  2,000 minutes of steps in 30 days (500 for private repositories), past
  which their runs go one at a time.
- **The network.** Runs join `knotless-runs`, a bridge (`kl-runs`) whose
  containers cannot reach each other. Its egress rules drop the host
  itself, the private and link-local ranges (the cloud's metadata service
  at 169.254.169.254 among them) and mail ports (25, 465, 587). The rest
  of the internet is open: registries, package indexes, git hosts. Runs
  resolve names through public resolvers (`/etc/knotless/resolv.conf`,
  `--resolv`), since docker's own is out of gVisor's reach.
- **The operator's stop.** `/etc/knotless/paused.json` pauses an owner, or
  everyone (`*`), at the next pass. See `--paused` in
  [the CLI reference](../../docs/reference/cli.md).
- **Nothing to steal but the runner's own.** No serve data, no index, no
  other account's session. The runner's app password, its OAuth session
  if it has logged in (Private repositories), and the work under
  `/var/lib/knotless-run` are all there is.

## Setting it up

1. **Order the box.** A Hetzner cloud CX43 (8 vCPU, 16 GB, 160 GB disk,
   €15.99 a month before VAT as of June 2026), Ubuntu 24.04, with your
   SSH key, in a location near the main host. In the console, or:

   ```sh
   hcloud server create --name knotless-runner --type cx43 \
     --image ubuntu-24.04 --location fsn1 --ssh-key <your key>
   ```

2. **Make the runner's account.** A new atproto account, such as
   `runner.knotless.dev`, and an app password for it. Its checks and
   minutes records are written there, so it should not be anyone's own.

3. **Set up the host.** In `deploy/local.env`:

   ```sh
   KNOTLESS_RUNNER_HOST=root@<its address>
   ```

   then `deploy/runner-host/setup.sh`. It ends with a check run under
   gVisor on the runs' network, which should print:

   ```
   internet: answers
   metadata: blocked
   private: blocked
   mail: blocked
   kernel: gVisor
   ```

   If the internet does not answer, look at DNS first. Docker's resolver
   on a user-defined network is reached through NAT rules inside the
   container's network namespace, which gVisor's network stack does not
   apply, so runs are given `/etc/knotless/resolv.conf` instead
   (`--resolv`), naming public resolvers.

4. **Deploy the runner.** In `deploy/local.env`, once, its account:

   ```sh
   KNOTLESS_RUNNER_IDENTIFIER=runner.knotless.dev
   KNOTLESS_RUNNER_PASSWORD=xxxx-xxxx-xxxx-xxxx
   ```

   then `deploy/runner-host/deploy.sh`. It builds the image on the host,
   as the main host's deploy does, and starts `run`. It watches every
   public repository whose record names its account (`--discover`), as
   knotless.dev lists them; `KNOTLESS_RUNNER_REPOS` names more, such as a
   private one. Its log:
   `ssh root@<host> journalctl -f CONTAINER_TAG=knotless-runner-run-1`.

5. **Measure gVisor.** Point a test repository's `runners` at the new
   account (`knotless runners <url> add runner.knotless.dev`), push a
   workflow that builds something real (a fork of knotless, say), and
   compare its steps' times with the same run on the main host. Expect
   cargo and npm 10–30% slower. Much worse than that, or a build that fails
   only under gVisor, is the case for Firecracker, which needs a dedicated
   server with KVM (an AX42). Check too that a workflow's `cache:` paths
   are kept between runs: they are overlay volumes the daemon mounts and
   gVisor sees as bind mounts.

6. **Log it in, for private repositories.** See below.

7. **Invite an owner.** They name the runner in their repository:
   `knotless runners <url> add runner.knotless.dev`. It is found within
   seconds, from the record Jetstream brings, and the next push runs.
   `knotless runners <url> remove runner.knotless.dev` lets it go. Their
   minutes show with `knotless minutes <url>`, and on the repository's
   checks tab.

## Private repositories

An app password mints no delegation tokens, so it reads no one else's
private repository. The runner logs in with OAuth instead, through a
knotless-serve of its own on this host (`bots` in docker-compose.yml),
which keeps the session and its client key here, never on the main host:

1. An A and an AAAA record for the name, such as `runner.knotless.dev`, to
   this host, not proxied, and ports 80 and 443 open in the cloud's
   firewall (setup.sh opens them in ufw).
2. In `deploy/local.env`:

   ```sh
   KNOTLESS_RUNNER_DOMAIN=runner.knotless.dev
   KNOTLESS_RUNNER_DID=did:plc:…
   ```

   then `deploy/runner-host/deploy.sh`. It starts `bots`, and Caddy in
   front of it, which gets the name a certificate and answers only the
   bot login's three paths (`/oauth/bot`, `/oauth/bot-callback`,
   `/oauth/bot-client-metadata.json`).
3. Open `https://runner.knotless.dev/oauth/bot?account=<the runner's
   handle>`, signed in to the runner's account at its PDS, and approve.
   The session lasts two years if used every three months; `run` takes
   it from `/var/lib/knotless-run/bots` from its next start, and the app
   password then logs in only if it can no longer be refreshed.

A private repository made with `knotless create --private`, at an appview
that names this runner, makes it a member of the space and invites it;
`knotless runners <url> add <the runner's handle>` does the same for one
made before. The runner reads its invites every sweep, as it lists
public repositories. A login approved before the invite methods were in
knotless's scope cannot read them (`invites` warnings in the log): open
the login page again and approve. So too one from before `dev.knotless.check`
and `dev.knotless.minutes` were: every run of it fails with `Missing required
scope "repo:dev.knotless.check?action=create"`. `KNOTLESS_RUNNER_REPOS` still names
any more to watch; one it cannot read is logged and passed over.

## Merge queues

With `--queue` in `KNOTLESS_RUNNER_ARGS`, this runner lands the merge
queue of every repository that names it and no runner that may push to
it, through knotless.dev's gateway once its owner has delegated writes
there ([Who lands a
queue](../../docs/design/merge-queue.md#who-lands-a-queue)). The gateway
takes from it only a batch of queued pulls, squashed in order: a run
that broke out and took its account could land what someone queued
without its checks passing, but push nothing else. knotless itself is
`--pulls-only` here, so its queue stays on the main host.

## knotless's own pulls

knotless's pulls run here too, so they no longer take the main host's
CPU from serve. Its pushes, its merge queue's batches and its deploy stay
on the main host, where the deploy's key is. In `deploy/local.env`:

```sh
KNOTLESS_RUNNER_ARGS="--queue --pulls-only chadtmiller.com/knotless --unmetered did:plc:bcgltzqazw5tb6k2g3ttenbj --trust did:plc:bcgltzqazw5tb6k2g3ttenbj --pull-jobs 2"
KNOTLESS_RUNNER_JOBS=2
KNOTLESS_RUNNER_MEMORY=6
KNOTLESS_RUNNER_CPUS=4
```

then `deploy/runner-host/deploy.sh`. This runner then runs the
repository's pulls alone. Those of its owner and collaborators run under
`runc` (`--trust`, and `--trusted-runtime runc` in the compose file),
not gVisor: under gVisor its `test` took 18 minutes to the main host's
4, and the batch the pull goes into runs the same code there with no
sandbox anyway. Anyone else's pull, run once the owner asks, is under
gVisor as on any repository here. They count
against no allowance and may take every job, past `--owner-jobs`.
`--pull-jobs 2` lets pulls take both jobs, not half: here nearly
everything is a pull. Its `test` workflow builds every test binary with
coverage and starts PDSes, which takes more than the default 3 GiB a run:
past it the sandbox is killed (`containerManager.WaitPID failed: EOF` in
the log, an OOM kill of `gvisor_sentry` in `dmesg`). So two jobs of 6 GiB and 4 CPUs
each, on an 8-core, 16 GB host, rather than four that each fail. knotless names this runner's account among its
`runners` (`knotless runners knotless://chadtmiller.com/knotless add
runner.knotless.dev`), so its pulls' checks show. The main host's runner
takes `--queue` without `--pulls` (`KNOTLESS_RUN_ARGS` in
`.knotless/workflows/deploy.yml`). Nothing here can push to knotless or
reach the main host. A run that broke out could file a false pull check,
but the queue checks each batch again on the main host, and a deploy
waits for main's own checks, run there too.

No push runs here, so its caches' bases are written by knotless's own
pulls: an owner's or collaborator's passing run into `main` promotes its
layers, as a push's would on the main host, but only over the base it
started from and never from an older `main` than that base's (see
[Where pushes run
elsewhere](../../docs/design/runner.md#where-pushes-run-elsewhere)). The
first pull over an empty cache builds in full; the ones after start from
what it left, the `test` workflow's stored coverage runs among it.

## Running it

- **Pause someone:** edit `/etc/knotless/paused.json` on the host, as
  `{"did:plc:…": "mining, by the looks of it"}`, or `{"*": "maintenance"}`
  for everyone. No restart: the next pass reads it.
- **Grant a secret:** `/etc/knotless/run-secrets.json`, as on the main
  host. Tenants' secrets are not offered yet.
- **Disk:** images unused for a week are pruned daily; caches are bounded
  by `--cache-max` (16 GiB); checkouts go as their runs end.
- **Grow:** when checks wait long for a job on most days, rescale the
  server to a CX53 (16 vCPU, 32 GB) in the Hetzner console: a reboot, the
  disk kept. Then raise `KNOTLESS_RUNNER_JOBS` to 8 and deploy again.

## Not yet

- **Hostname filtering.** Egress is filtered by address, not name, so
  known mining pools and proxies are not blocked. That wants an egress
  proxy.
- **Rootless docker.** The daemon runs as root, and the runner holds its
  socket. gVisor is the wall between a run and the host. A user-namespaced
  daemon would be a second one, once the caches' overlays are shown to
  work under it.