johnandersen777.bsky.social/knotless
git that lives in your Atmosphere account
Forked from chadtmiller.com/knotless
0
0
# The runner host
A box of its own for running workflows: other owners' repositories,
agents' pulls, strangers' pulls, and every workflow of knotless's own
that holds no secret, its pushes and merge queue among them. The main
host keeps knotless-serve, its index and its data, and runs no runner.
The workflows that hold a secret, deploying among them, run on [the
trusted runner](#the-trusted-runner), on another box again. A container
escape here reaches one runner account's app password and the runs going
at the time.
What holds a run apart here:
- **gVisor.** Every run's container of code nobody here vouches for
starts with `--runtime runsc`, so its
system calls go to gVisor's own kernel, not the host's. Most container
escapes are kernel bugs, and those land in gVisor. It runs in
`systrap` mode, which needs no nested virtualization, so a cloud VM
will do.
- **Limits.** Each run gets 2 CPUs, 3 GiB with no swap, 4,096 processes
and an hour. Each owner gets 2 of the 4 jobs, and an allowance of
2,000 minutes of steps in 30 days (500 for private repositories), past
which their runs go one at a time.
- **The network.** Runs join `knotless-runs`, a bridge (`kl-runs`) whose
containers cannot reach each other. Its egress rules drop the host
itself, the private and link-local ranges (the cloud's metadata service
at 169.254.169.254 among them) and mail ports (25, 465, 587). The rest
of the internet is open: registries, package indexes, git hosts. Runs
resolve names through public resolvers (`/etc/knotless/resolv.conf`,
`--resolv`), since docker's own is out of gVisor's reach.
- **The operator's stop.** `/etc/knotless/paused.json` pauses an owner, or
everyone (`*`), at the next pass. See `--paused` in
[the CLI reference](../../docs/reference/cli.md).
- **Nothing to steal but the runner's own.** No serve data, no index, no
other account's session. The runner's app password, its OAuth session
if it has logged in (Private repositories), and the work under
`/var/lib/knotless-run` are all there is.
## Setting it up
1. **Order the box.** A Hetzner cloud CX43 (8 vCPU, 16 GB, 160 GB disk,
€15.99 a month before VAT as of June 2026), Ubuntu 24.04, with your
SSH key, in a location near the main host. In the console, or:
```sh
hcloud server create --name knotless-runner --type cx43 \
--image ubuntu-24.04 --location fsn1 --ssh-key <your key>
```
2. **Make the runner's account.** A new atproto account, such as
`runner.knotless.dev`, and an app password for it. Its checks and
minutes records are written there, so it should not be anyone's own.
3. **Set up the host.** In `deploy/local.env`:
```sh
KNOTLESS_RUNNER_HOST=root@<its address>
```
then `deploy/runner-host/setup.sh`. It ends with a check run under
gVisor on the runs' network, which should print:
```
internet: answers
metadata: blocked
private: blocked
mail: blocked
kernel: gVisor
```
If the internet does not answer, look at DNS first. Docker's resolver
on a user-defined network is reached through NAT rules inside the
container's network namespace, which gVisor's network stack does not
apply, so runs are given `/etc/knotless/resolv.conf` instead
(`--resolv`), naming public resolvers.
4. **Deploy the runner.** In `deploy/local.env`, once, its account:
```sh
KNOTLESS_RUNNER_IDENTIFIER=runner.knotless.dev
KNOTLESS_RUNNER_PASSWORD=xxxx-xxxx-xxxx-xxxx
```
then `deploy/runner-host/deploy.sh`. It builds the image on the host,
as the main host's deploy does, and starts `run`. It watches every
public repository whose record names its account (`--discover`), as
knotless.dev lists them; `KNOTLESS_RUNNER_REPOS` names more, such as a
private one. Its log:
`ssh root@<host> journalctl -f CONTAINER_TAG=knotless-runner-run-1`.
5. **Measure gVisor.** Point a test repository's `runners` at the new
account (`knotless runner add <url> runner.knotless.dev`), push a
workflow that builds something real (a fork of knotless, say), and
compare its steps' times with the same run on the main host. Expect
cargo and npm 10–30% slower. Much worse than that, or a build that fails
only under gVisor, is the case for Firecracker, which needs a dedicated
server with KVM (an AX42). Check too that a workflow's `cache:` paths
are kept between runs: they are overlay volumes the daemon mounts and
gVisor sees as bind mounts.
6. **Log it in, for private repositories.** See below.
7. **Invite an owner.** They name the runner in their repository:
`knotless runner add <url> runner.knotless.dev`. It is found within
seconds, from the record Jetstream brings, and the next push runs.
`knotless runner remove <url> runner.knotless.dev` lets it go. Their
minutes show with `knotless runner minutes <url>`, and on the repository's
checks tab.
## Private repositories
An app password mints no delegation tokens, so it reads no one else's
private repository. The runner logs in with OAuth instead, through a
knotless-serve of its own on this host (`bots` in docker-compose.yml),
which keeps the session and its client key here, never on the main host:
1. An A and an AAAA record for the name, such as `runner.knotless.dev`, to
this host, not proxied, and ports 80 and 443 open in the cloud's
firewall (setup.sh opens them in ufw).
2. In `deploy/local.env`:
```sh
KNOTLESS_RUNNER_DOMAIN=runner.knotless.dev
KNOTLESS_RUNNER_DID=did:plc:…
```
then `deploy/runner-host/deploy.sh`. It starts `bots`, and Caddy in
front of it, which gets the name a certificate and answers only the
bot login's three paths (`/oauth/bot`, `/oauth/bot-callback`,
`/oauth/bot-client-metadata.json`).
3. Open `https://runner.knotless.dev/oauth/bot?account=<the runner's
handle>`, signed in to the runner's account at its PDS, and approve.
The session lasts two years if used every three months; `run` takes
it from `/var/lib/knotless-run/bots` from its next start, and the app
password then logs in only if its server turns the refresh down
(`invalid_grant`). A refresh that fails otherwise is logged (`login
refresh failed`) and tried again, the OAuth session kept: a password
login saved over it would read no private repository again, every one
failing with `Scope does not permit reading`. Should that happen, approve
the login again.
A private repository made with `knotless repo create --private`, at an appview
that names this runner, makes it a member of the space and invites it;
`knotless runner add <url> <the runner's handle>` does the same for one
made before. The runner reads its invites every sweep, as it lists
public repositories. A login approved before the invite methods were in
knotless's scope cannot read them (`invites` warnings in the log): open
the login page again and approve. So too one from before `dev.knotless.check`
and `dev.knotless.minutes` were: every run of it fails with `Missing required
scope "repo:dev.knotless.check?action=create"`. A PDS grants a bare
`space:` scope its type's collections only as the login is approved, so
one approved at a PDS that did not do so yet, as pds.js before November
4, 2026, keeps a grant that writes nothing to a space, and every refresh
keeps it too: each private repository's run fails with `Scope does not
permit update in at://…/space/…`. Approve the login again once the PDS
is updated. `KNOTLESS_RUNNER_REPOS` still names
any more to watch; one it cannot read is logged and passed over.
## Merge queues
With `--queue` in `KNOTLESS_RUNNER_ARGS`, this runner lands the merge
queue of every repository that names it and no runner that may push to
it, through knotless.dev's gateway once its owner has delegated writes
there ([Who lands a
queue](../../docs/design/merge-queue.md#who-lands-a-queue)). The gateway
takes from it only a batch of queued pulls, squashed in order: a run
that broke out and took its account could land what someone queued
without its checks passing, but push nothing else. knotless's own queue
lands here too: this runner is the first its repository names, and none
it names may push.
## knotless's own CI
knotless's checks run here: its pushes, its pulls and its merge queue's
batches, so none of them takes the main host's CPU from serve. Each of
its workflows that holds no secret names this runner in `runners`; the
rest name [the trusted runner](#the-trusted-runner). In
`deploy/local.env`:
```sh
KNOTLESS_RUNNER_ARGS="--queue --unmetered did:plc:bcgltzqazw5tb6k2g3ttenbj --trust did:plc:bcgltzqazw5tb6k2g3ttenbj --pull-jobs 4"
KNOTLESS_RUNNER_JOBS=4
KNOTLESS_RUNNER_MEMORY=12
KNOTLESS_RUNNER_CPUS=4
```
then `deploy/runner-host/deploy.sh`. Pushes to its owner's repositories,
and the pulls and batches of its owner and collaborators, run under
`runc` (`--trust`, and `--trusted-runtime runc` in the compose file),
not gVisor: under gVisor its `test` took 18 minutes to runc's 4. Anyone
else's pull, run once the owner asks, is under gVisor as on any
repository here, and so is a batch holding one. They count against no
allowance and may take every job, past `--owner-jobs`. `--pull-jobs 4`
lets pulls take every job, not half. Its `test` workflow builds every
test binary with coverage and starts PDSes, which takes more than the
default 3 GiB a run: past it the sandbox is killed
(`containerManager.WaitPID failed: EOF` in the log, an OOM kill of
`gvisor_sentry` in `dmesg`). So four jobs of 12 GiB and 4 CPUs each, on a
dedicated 6-core, 12-thread, 64 GB host (an AX41), which share its threads
only when all four are busy. Within them, cargo would still start a rustc for
each of the host's cores, and knotless-serve's takes more than 3 GiB under
coverage: `harness/cov.mjs` builds as many crates at once as the
cgroup's CPUs, and one for each 4 GiB of its memory at most, so 3 here
(`rustc` killed by `oom-killer` in `dmesg` is this, past those).
knotless names this runner's account first among its `runners`
(`knotless runners knotless://chadtmiller.com/knotless`), so it lands
the queue. Its pushes to `main` run here, so they write its caches'
bases, as on any host where a repository's pushes run.
Nothing here can push to knotless or reach the main host. A run that
broke out could file a false check, or land through the gateway what a
maintainer queued without its checks passing. A deploy waits for main's
checks, filed here, so such a run could also have main deployed with
checks that did not pass, though only of what was queued.
## The trusted runner
The workflows that hold a secret run on a runner of their own, as
ci.knotless.dev, on the box that already holds the infrastructure's
tokens (hetzner-infra's `infra-runner`), beside that box's own runner:
deploying knotless.dev (`DEPLOY_SSH_KEY`, `RELEASE_PASSWORD`), releasing
the client, publishing the lexicons, the scenarios and the docs audit,
and the owner's deploys of pds.js and grain. It runs no one's code but
its owner's: it watches only the repositories it names
(`KNOTLESS_RUNNER_DISCOVER` empty), and each workflow it runs names it in
`runners`, the rest of those repositories' naming this runner. Every run
there is trusted, so under `runc`.
It is a second project of this directory's compose file on that host,
with a directory of its own for `run.env`, the secrets and the pause
list, `/etc/knotless-ci`, and its own work, `/var/lib/knotless-ci`. From
a `deploy/ci.env` (not committed; `deploy/ci.env.example` shows it):
```sh
KNOTLESS_RUNNER_ENV=deploy/ci.env deploy/runner-host/deploy.sh
```
It takes the image the main host built (`KNOTLESS_RUNNER_IMAGE_FROM`), as
that box is too small to build knotless, and has no bot login: what it
watches is public. Its secrets are those of
`/etc/knotless-ci/run-secrets.json`, and those owners seal to its key
(`knotless secret <url> set <name> --runner ci.knotless.dev`), kept in
`secrets-key.json` under its work: moving it to another box means moving
that file, or sealing them all again.
## Deployed with main
Each deploy of main deploys this host too, last
(`.knotless/workflows/deploy.yml`, its `runner host` step), so the
runner that lands other repositories' merge queues reads their records
as main does: an older one ejects a pull whose rule holds what it does
not know (`what this version does not know: gates; update knotless`).
It runs `deploy.sh` with `KNOTLESS_RUNNER_IMAGE_FROM` the main host,
whose image it copies only when its runner's build (the fingerprint
the main host's deploy compares) differs from this host's: a deploy
that leaves the runner as it was restarts nothing here, as on the main
host. Its settings are the `KNOTLESS_RUNNER_*` in that workflow, which
`deploy/local.env` repeats for a deploy by hand.
It reaches the host with `RUNNER_HOST_SSH_KEY`, a key of its own,
granted to pushes to main alone (`knotless secret set …
RUNNER_HOST_SSH_KEY --ref refs/heads/main`). Root takes it only from the
trusted runner's addresses, as `setup.sh` installs its public half:
```sh
KNOTLESS_RUNNER_DEPLOY_KEY="ssh-ed25519 AAAA… knotless deploy of main" \
KNOTLESS_RUNNER_DEPLOY_FROM="178.105.255.120,2a01:4f8:c016:89c6::1" \
deploy/runner-host/setup.sh
```
## Running it
- **Pause someone:** edit `/etc/knotless/paused.json` on the host, as
`{"did:plc:…": "mining, by the looks of it"}`, or `{"*": "maintenance"}`
for everyone. No restart: the next pass reads it.
- **Grant a secret:** `/etc/knotless/run-secrets.json`, as on the main
host. Tenants' secrets are not offered yet.
- **Disk:** images unused for a week are pruned daily; caches are bounded
by `--cache-max` (16 GiB); checkouts go as their runs end.
- **Move:** run `setup.sh` on the new host, with the deploy key. Point
`runner.knotless.dev` at it, stop the old runner, copy `/etc/knotless/run.env`,
`run-secrets.json` and `paused.json`, `/var/lib/knotless-run`'s `state.json`,
`secrets-key.json`, `bots` and `config`, and `/var/lib/knotless-serve`, then
run `deploy.sh`. Caches build again on the first run.
## Not yet
- **Hostname filtering.** Egress is filtered by address, not name, so
known mining pools and proxies are not blocked. That wants an egress
proxy.
- **Rootless docker.** The daemon runs as root, and the runner holds its
socket. gVisor is the wall between a run and the host. A user-namespaced
daemon would be a second one, once the caches' overlays are shown to
work under it.