# The runner

`knotless-run` is knotless's CI runner. It watches a set of repositories
and runs their workflows when a ref moves or a pull changes. It reports
each run as a `dev.knotless.check` record in its own atproto account.

A workflow is a YAML or JSON file under `.knotless/workflows/`, read at
the commit being checked. Its format is in
[the workflow reference](../reference/workflows.md), and the runner's
flags are in [the CLI reference](../reference/cli.md).

Some background. A knotless repository is records in its owner's atproto
account, kept on the account's server, its PDS. Each ref is one `dev.knotless.ref` record holding
the ref's target and a generation, a counter that goes up with every push
to that ref. The owner's PDS is the authority on where a ref points
([storage.md](storage.md)). Pulls, and the states that open and close
them, are records in each author's own account. knotless-serve can index
them as an appview, a service that lists records across accounts
([collaboration.md](collaboration.md), [serve.md](serve.md)). The
[atproto site](https://atproto.com) has the protocol itself.

## What it does

The runner logs in as one account, from `KNOTLESS_IDENTIFIER` and
`KNOTLESS_PASSWORD` (an app password), or with `knotless login`. An app
password reads no private repositories, so a runner that watches one
logs in with OAuth: on a server, through the gateway beside it
([Bots](gateway.md#bots)), whose session lasts two years. Its
arguments name the repositories to watch, each as `<handle or
did>/<name>`; a private one is read as that account, a member of it (see
[Private repositories](#private-repositories)). The string as
given is the repository's label, which names it in logs, in
`KNOTLESS_RUN_REPO` and in the secrets file. Internally the runner knows
a repository by `<owner did>/<repo key>`, where the key is a TID (a
time-ordered record key) fixed when the repository was made, so a rename
or a new repository under the old name does not fool it.

Each run's check record holds its status, each step's result, a log and
any images the run made. `knotless checks`, the web pages and
`knotless merge` read them from the runner's PDS: from its repo in the
space, for a private repository.

## What shapes the design

Six problems decide most of how the runner works.

1. Events cannot be trusted. The runner learns of pushes from Jetstream, a
   public relay of every account's record changes. Its frames are
   unsigned, so anyone can forge one, and it has dropped events.
2. Pulls carry code from strangers. Anyone can open a pull, and its
   workflows and build scripts are theirs to write.
3. Deploys must not be cut off. A deploy is a workflow too, and stopping
   it halfway can leave a site down.
4. Every merge moves the base of every open pull. A pull is checked as
   merging would leave it, so each merge retests every open pull, and that
   burst must not starve the main branch's own runs.
5. Caches cross trust levels. A cache written by a stranger's pull must
   not reach a main-branch build, and one written by a run that had
   secrets must not reach a pull.
6. The runner stops and starts. It is redeployed, it crashes, and its
   host restarts. No run may be lost, and no check may stay `running`
   forever.

## Finding out that something changed

Jetstream is only a doorbell. A frame tells the runner which repository to
look at again, never what to run. The runner then reads the repository
record, its refs and its rerun requests back from the owner's PDS, and runs
the targets the PDS gives. A forged frame can make the runner read, but
never run something the owner did not push.

To ring a repository's door is the runner's term for waking that
repository's reconcile task. Each watched repository has exactly one. A
reconcile pass reads the repository and starts whatever is due. Rings that
arrive while a pass is reading coalesce into one more pass.

A frame rings a watched repository's door when it changes:

- a `dev.knotless.ref` or `dev.knotless.rerun` record of the owner's for
  that repository;
- the same kinds of record in a repository that an open pull into the
  watched one comes from (known only with `--pulls`, from reading pulls);
- a `dev.knotless.pull` or `dev.knotless.state` record naming the
  repository, so a new, reopened or retargeted pull runs;
- a `dev.knotless.queue` record, for the merge queue;
- a `dev.knotless.repo` record of the owner's, which may no longer name
  the runner (see [Repositories that name it](#repositories-that-name-it)).

Because a frame only rings, a missed one loses no work, and the runner
rings doors in other ways too:

- It connects without a cursor, and every connection rings every door.
  What passed while it was disconnected runs on reconnect, and a replayed
  event finds nothing new. A connection silent for 60 seconds gets a
  ping, and silent another 60 is given up and made again, as
  [knotless-serve's](serve.md) is: a half-open connection never ends,
  and the runner would hear nothing from it.
- It rings every door every `--sweep` seconds (600). Jetstream once
  dropped a whole account's events for about 15 minutes on every
  instance. A pass runs only what has not run, so a sweep costs reads,
  not runs.
- A pass that fails, say because the PDS is down, rings its own door
  again after `min(--sweep, 60 s)`.
- With `--queue`, a change to the merge queue's records, re-read every
  `--notice` seconds (30), rings the door.
- A private repository's space written to rings its door, as the
  appview passes on its authority's word of the write, at once; or, asked
  itself every `--poll` seconds (10), when anything was written there
  since the last ask. Nothing written in a space reaches Jetstream (see
  [Private repositories](#private-repositories)).

```mermaid
flowchart TD
  frame[Jetstream frame naming a watched repo] --> ring
  connect[each Jetstream connection] --> ring
  sweep[every --sweep seconds] --> ring
  notice[queue records changed, every --notice seconds] --> ring
  poll[a private repo's space written to: the appview says, or every --poll seconds] --> ring
  retry[a failed pass, after at most 60 s] --> ring
  ring[ring the repo's door] --> pass[reconcile pass]
  pds[(owner's PDS: repo record, refs, reruns)] --> pass
  appview[(appview: open pulls, merge queue)] --> pass
  pass --> due{due, and trusted?}
  due -->|no| seen[recorded as seen]
  due -->|yes| file[file every check as running]
  file --> line[wait in line for a job]
  line --> steps[run the steps]
  steps --> ended[file the check as ended]
```

A pass is logged as it ends, with how long it took. One that hung for 13
minutes once said nothing of where, so a pass still going after 5 minutes
logs the stage it is at, and again every 5 minutes, and a slow git command
is logged as a slow HTTP call is ([What a pass
logs](../reference/runner-internals.md#what-a-pass-logs)).

### Repositories that name it

With `--discover`, the runner watches, besides those its arguments name,
every repository whose record names its account in `runners`. That
is how a hosted runner serves whoever names it, with no list kept by
hand. It finds them three ways, and reads each before watching it:

- At the start, and every `--sweep` seconds after, it asks the appview
  (`KNOTLESS_APPVIEW`) for `dev.knotless.listRepos?runner=<its DID>`, and
  takes those it found before from the state file too, so a start does
  not wait on the appview.
- A Jetstream frame writing a `dev.knotless.repo` record has it read that
  repository at once, so one named meanwhile is found within seconds.
- A private repository's invites, read every `--sweep` seconds from the
  appview, from where the last read ended (`invited` in the state file):
  see [Private repositories](#private-repositories).

Each found is read from its owner's PDS, or its space, as any ring is,
before it is watched: the appview, the frame and the invite only say
where to look. One whose record there does not name the runner, or that is
archived, is passed over, and one watched that becomes either is left.
Unarchiving writes the record again, which a Jetstream frame brings, so
it is watched again within seconds.

A repository found is labelled `<owner did>/<name>`, in logs, in
`KNOTLESS_RUN_REPO` and in the secrets file. Its first pass starts from
where its refs are, as any repository's does, but for its default
branch's tip, which runs once: a repository made by its first push shows
that push's checks, and one that names the runner later shows the runner
works. What else was pushed before it was found waits for its ref to
move. A pass that finds its record gone, or no
longer naming the runner, stops watching it: what it was running
finishes, and its door and the loops that ring it go. One renamed is found
again under its new name, by its key, and watched by that instead. A
repository named on the command line stays as it is, whatever its record
says.

A new repository names a runner of this kind without its owner doing
anything. An appview started with `--default-runner <did>` names it in
`dev.knotless.describeServer`, and `knotless create`, MCP's
`create_repo` and a first push write it into a new repository's
`runners`, unless the owner names others (`--runner`, or `--no-runner`
for none); a private one's are made members of it and invited, and
`knotless create` says so. knotless.dev names its runner host's account. It names one
only: each runner a repository names runs all its checks, so two would
run each twice. The record
still says it all: the owner sees the runner with `knotless runners
<url>` and lets it go with `knotless runners <url> remove`.

### A reconcile pass

A pass reads the repository record, then acts on new rerun requests, then
runs what is due for refs that moved, then for pulls (with `--pulls`),
then works the merge queue (with `--queue`), then, at most once an hour,
prunes the repository's checks (see [Pruning](#pruning)). If the record's key has
changed and no repository in the owner's account has the old key, the
repository was deleted: the pass clears the checks, secrets and caches it
left and stops watching it. A discovered repository whose record no
longer names the runner is left. Otherwise a changed key fails the pass:
the label no longer names the repository the runner was watching.

A ref is due when its generation is past the one the runner last ran, or
it is new to the runner, or its target matches a run dropped at a stop. A
deleted ref never runs. Each due ref syncs the runner's mirror, a local
bare git repository kept in step with the PDS, once per pass. After a
ref's runs are filed, the runner records its generation in the state file.

The first time the runner watches a repository, the refs, open pulls and
rerun requests already there are its starting point: it runs none of
them. A repository with many branches and pulls would otherwise run them
all at once, and so would every repository again whenever the state file
is lost, a deploy workflow's run on main included. `knotless rerun`
runs any of them at once.

Details:

- A watched repository that cannot be found stops the runner starting.
  Later, an error in one repository is logged and the others go on.
- The default branch is the repository record's `defaultBranch`, or
  `main`.

## What a push runs

For a ref that moved, the runner checks out the new target and loads the
workflows at that commit, so a change to a workflow is tested by the push
that makes it. A workflow runs on a push when its `on.push.refs` matches
the ref.

Every workflow's check is filed before any run starts, as `queued` or
already ended, so a reader never sees some of a commit's checks passed
while another is missing. A run waits `queued` for its turns, and is
filed again as `running`, its `startedAt` then, once it has its job and
is about to run its first step. A pull's pre-flight that another runner
the repository names has started or passed at the same head and base
meanwhile is not run then: its check is deleted
([merge-queue.md](merge-queue.md#pre-flights)). A check is filed already ended for:

- a workflow that does not parse, or two that share a name: `error`,
  whether or not it would have run;
- a workflow whose `paths` match nothing that changed (below);
- a run on the default branch that
  [reuses a pull's pass](#reusing-a-pulls-pass-at-the-push).

Each commit keeps its own checks. A push leaves the previous commit's
checks standing, and a rerun replaces only that commit's.

### Paths

A `push` or `pull` trigger may list `paths` globs, and the workflow runs
only if a changed file matches one ([syntax](../reference/workflows.md#paths)).
For a push, the change set runs from where the runner's mirror had the ref
before this pass synced it to the new target. When that is unknown, as
for a new ref, nothing is skipped.

A workflow left out is filed at once as `success`, every step `skipped`,
with a note naming the paths. It takes no job and no cache, and the merge
gate and deploys read it as passed. Runs asked for by name ignore `paths`.

## What a pull runs

With `--pulls`, the runner checks each open pull into a watched
repository at its head merged with the current tip of its base branch, as
merging would leave it. A pull that breaks the base only beside another
pull merged since then fails before it is merged.

The runner lists open pulls from the appview (`KNOTLESS_APPVIEW`),
counting only states written by the owner or a collaborator. Drafts are
among them: a draft is checked as any pull is, so its author sees whether
it passes before marking it ready, though it cannot merge or be queued
until then. It reads
each head from its owner's PDS and fetches it into the watched
repository's mirror, as `knotless merge` does. A pull whose head branch,
head repository or base branch is gone is treated as closed.

If the head already contains the base, the runner checks the head itself.
Otherwise it builds a merge commit with `git merge-tree` and
`git commit-tree`, with a fixed author, committer and time. The same head
and base always make the same commit, so the runner and `knotless merge`
agree on what was checked without talking to each other. The workflows
that run are those in that commit whose `on.pull.base` matches the base.

If the head conflicts with the base, there is no merge commit to load
workflows from. The runner loads them at the head, and files each that
runs on a pull into the base as `failure` without running it, naming the
files and asking for a merge or rebase.

A pull's check names the head branch as its `ref`, the head as its
`commit`, the base tip as `baseCommit`, and the pull. `KNOTLESS_RUN_SHA`
is the head too, not the merge commit checked out, and `KNOTLESS_RUN_BASE`
the base tip, as it is a batch's base in a batch's run.

A repository named with `--pulls-only` runs only its pulls here. Its
pushes and its merge queue run on another runner, one without `--pulls`
that it also names. The pulls' runs then load a machine of their own, and
the runs that land and deploy stay where the deploy's secrets are. That
split is the operator's; an owner makes one of their own by naming the
runners a workflow runs on, in its `runners`, and giving its secrets to
those alone (`knotless secret set --runner`). A runner not named files no
check of it. A push
to the default branch can take the pass of a batch from the runner that
lands the queue, but not the pass of a pull run elsewhere. Since no push
runs here, a trusted pull's passing run writes the caches' bases instead,
so the pulls here start warm (see
[Where pushes run elsewhere](#where-pushes-run-elsewhere)).
As another runner lands the queue, the pulls run here as on that one:
a queued pull does not run on its own, and the rest run again only as
their head moves, not their base.

The state file holds the head and base each open pull last ran at, so
on a runner that does not land the queue, when the base moves, every open
trusted pull into it runs again. The sections on
[scheduling](#who-gets-a-job-first) and the
[merge queue](#the-merge-queue) bound what that costs. A pull that runs as
it moves, trusted or an agent's, seen again after a gap, such as on reopening, does not run if the runner
already has checks of it at that head and base.

## Whose code runs

A pull runs as it moves only if it is trusted: its author and its head
repository's owner both write to the repository, each being the owner or
listed in the repository record's `collaborators`. A merge-queue batch is
trusted when every pull in it is.

An untrusted pull runs only when the owner asks, with `knotless rerun
--pull`. It runs once, where it is at that moment, and until then
`knotless merge` finds it not checked. A stranger cannot spend the
runner's time, or reach its host, without the owner looking first.

Between the two are the accounts the repository record names in
`agents`, such as a coding agent pushing to its own fork. A pull whose
author and head repository's owner are each an agent or may push runs as
it moves, so an agent hears from the checks without waiting for the
owner, but it runs untrusted: [caches](#across-trust-levels) apart for a
workflow asking for secrets, nothing it writes kept, and so no pass a push
can [reuse](#reusing-a-pulls-pass-at-the-push). What an agent writes can
be steered by whoever wrote what it works from, so a collaborator who set
it going does not vouch for it. An account named as an agent counts as
one even if it is a collaborator too, and a batch holding an agent's pull
is untrusted.

A pull that brings [shared steps](reusable-workflows.md) from an account
the repository does not vouch for is untrusted the same way: its lock
names an account the base's lock does not, that is neither the owner nor
a collaborator, or whose handle does not resolve. So is a batch holding
it. Its log says so, naming the accounts.

Trust decides when code runs, not what it is given. Every pull and batch
run gets no secrets, trusted or not, and only the docker executor runs
them: `--pulls` and `--queue` refuse `--executor shell`.

The shell executor runs each step as the runner's own user, who can read
the secrets file and the runner's session. Watch a repository under it
only if you trust everyone who can push to it with the host.

## Who gets a job first

All watched repositories share one pool of `--jobs` jobs (4). A
seconds-long workflow never waits behind a minutes-long one in a queue of
its own.

A run takes everything it needs at once, when all of it is free: a job,
and any [turn](#deploys-superseded-finish) on a ref it must hold. It never
holds one thing while waiting for another, so two runs never wait on each
other.

Waiting runs are ranked by class, then by the order they were started. A
run ranked earlier has first claim on what it waits for and on a free job.
A later one goes ahead only with what nothing before it waits for, and a
job left over. A running run is never stopped for one ranked before it, so
main's runs wait only for runs already going.

| Class | Runs in it |
| --- | --- |
| `Main` | pushes to the default branch, and the merge-queue batch that lands next |
| `Other` | pull runs, and pushes to other refs |
| `Later` | pulls not in the queue, where this runner lands the queue or runs the pulls alone, and a batch built ahead of one still checking |
| `Asked` | runs asked for by name, after every other waiting run, however old |

The classes other than `Main` together hold at most `--pull-jobs` jobs,
half of `--jobs` by default. A merge that reruns every open pull starts
only that many, and main's next run always has a job.

Within a class, the owners of the repositories take turns. Each run of an
owner's comes the turn after its last one still waiting, or, with none
waiting, at the turn of the last run to start. A burst of one owner's
runs, such as every open pull run again, goes one at a time between other
owners' runs rather than all ahead of them, and one owner's runs keep the
order they were started in. `--owner-jobs` caps the jobs one owner's runs
hold at once, all of them by default. Past it, that owner's next run waits
while other owners' runs take the jobs free.

`--allowance <minutes>` sets how many minutes of steps each owner's runs
may take in 30 days. Time is counted from the steps' own durations, not
from when a check was filed, since that includes the wait for a job. Past
its allowance, an owner's runs go one at a time, so they slow down but
still finish. Each such run's log begins by saying so, and how long it
waited for a job. The runner's own account is not held to it, nor is an
owner named with `--unmetered`, and without the flag no one is. `--private-allowance <minutes>` does the same for private
repositories' runs, counted apart from public ones.

`--paused <file>` is the operator's stop: a JSON object of owner DIDs,
or `*` for every owner but the runner's own account, each to why, such as
`{"did:plc:…": "mining, by the looks of it"}`. The runner reads it again
on every pass, so an edit takes effect at the next pass, with no restart.
A paused owner's runs are not run: every workflow at the commit, whether
or not it would have run there, is filed `error`, saying `paused by this
runner's operator: <why>`, never silently dropped. A file that
cannot be read pauses no one; one that is not such an object fails the
pass, and is logged.

The runner files each owner's public minutes as a `dev.knotless.minutes`
record in its own account, keyed by the owner's DID: the minutes in the
last 30 days, the allowance, and when minutes counted now stop counting.
It writes the record again as the owner's runs end, at most every five
minutes, or at once when the owner reaches 80% of the allowance or passes
it. Private repositories' minutes are never in it, since the record is
public and would show that private repositories exist and how busy they
are.

The same split reaches the CPU. Under docker, every run gets
`--cpu-shares 256`, a weight of about 10 against the default 100, except
pushes to the default branch and the batch that lands next. A burst of
pull runs leaves the site, deploys and main's runs the CPU they need.

## When a newer commit arrives

By default a workflow is `superseded: cancel`. A newer commit's run of it
on the same ref or pull cancels the older run, waiting or running, since
the older result no longer matters. Cancelling kills the running step and
its process group, skips the rest, and files the check as `cancelled`. A
run cancelled while waiting is filed `cancelled` without starting.

A full rerun of the same commit leaves the older run going, but a run
asked for by name cancels the previous run of that workflow at the same
commit, as both would file one check. Runs on different refs are left to
finish.

### Deploys: `superseded: finish`

A workflow that deploys can say `superseded: finish`. Such a run takes the
workflow's turn on the ref or pull as soon as it is filed, if no run holds
it. From then on it is kept: a newer commit's run does not cancel it. The
newer run waits, as `queued`, for the turn, and a third commit's run
cancels the one waiting, so at most one waits. Deploys go out in order,
and the newest commit is the next to go.

The turn is held until the check is filed as ended, so the next deploy
files after it.

A kept run is still cancelled when its result can no longer mean
anything or its wait would be unbounded: its pull moves to another head or
base, closes, or loses a branch; its merge-queue batch is dropped; or a
stop runs out of time. A later cancel never overwrites one of these
reasons.

## Stopping and starting again

### A stop

`SIGTERM` or `SIGINT` stops the runner, and is heard even during
startup. A stop:

1. waits for any pass that is filing or starting runs, and stops reading
   events;
2. cancels runs asked for by name, recording them to run again on start;
3. waits up to `--stop-wait` seconds (900) for every run to end,
   landing each merge-queue batch whose runs have then all passed
   ([Dropped runs](merge-queue.md#when-something-moves));
4. cancels every run still going, and waits for those to end.

A stop never waits forever on a hung step. One did once, when a deploy's
`docker compose up` failed and nothing came to kill the runner.

Since a stop costs the runs still waiting, and can cut off a merge-queue
batch still being checked, which is then checked again, knotless.dev's
deploys stop the runner only when what it runs changed: the sources it
is built from, or its settings. Each deploy rebuilds the image, so the
image alone cannot tell; when unsure, it restarts the runner
([When a deploy restarts the runner](../../deploy/README.md#when-a-deploy-restarts-the-runner)).

A run that has not started its steps when the stop comes is dropped: its
check is filed as `cancelled`, and the state file records it. The exception is a run
at the same commit and base as a running `superseded: finish` run, since a
deploy may wait for its own commit's other checks. It is kept until that
run ends, and dropped then if still not started.

On the next start, a dropped run runs again for only those workflows, and
only if its ref or pull is still where it was. If it moved on, the newer
commit runs all its workflows. A pull whose base alone moved runs all its
workflows on the base as it is then: a runner landing the queue runs a
pull again only as its head moves, so it would otherwise not run at all. A dropped batch run is not reported to the
queue as a failure.

### A start

It reads each repository its arguments name first. One it cannot read,
gone or private to it, is logged (`failed`, with the reason) and passed
over until the next start, so one bad name does not keep it from the
rest. Only when it can read none of them, and has none to find without
`--discover`, does it stop, with that reason.

Before any run, the runner ends as `error` every check of a watched
repository that its account left `queued` or `running`, saying the
runner stopped before the run started or finished. `knotless rerun` runs it again.

It also removes leftover containers: those labelled with its DID, and,
with `--work`, those mounting a path under the work directory. It does
this again every `--sweep`, sparing the containers of its own runs. One
was once left running for seven hours. It never touches other containers.

### The state file

The state file (`--state`) is JSON, written atomically. A file that
cannot be read, or is corrupt, counts as empty, and every repository is
then watched as if for the first time. Each field is keyed by repository:

| Field | Holds |
| --- | --- |
| `repos` | each ref's generation last run |
| `pulls` | each open pull's head and base last run at, or passed over at |
| `reruns` | the newest rerun request acted on |
| `asked` | pulls asked for and not yet run |
| `asked_only` | of those, the ones asked for only some workflows, by name |
| `passed` | trusted pull and batch runs that passed, for reuse, at most 256 |
| `dropped` | the workflows dropped at a stop, per ref, pull or queue, and commit and base |
| `wanted` | the workflows asked for by name and not yet run, per ref, with their variables |
| `used` | the milliseconds each owner's public repositories' runs' steps took, per day, for the last 30 days |
| `used_private` | the same for private repositories |

Two more are the runner's own, not by repository: `discovered`, the
repositories found naming the runner, watched again at a start before the
appview is asked, and `invited`, where the reading of invites ended.

## Where a step runs

Each run gets a temporary directory under `--work`, holding a clone of the
mirror at the commit, the log, and an `outputs/` directory, all removed
afterwards. The clone hard-links objects rather than using alternates, so
git works inside a container that sees only the checkout.

Under the `docker` executor, the command line's default (the library's
`Options::new` defaults to `shell`), each run gets one container from
the workflow's `image`, or `--image`. The checkout is mounted at `/work`.
Each step is a `docker exec` of `sh -ec <script>` in that container, so a
step finds what earlier steps left, installed tools included. Variable
values reach the container through the docker client's environment,
never its arguments. Secrets are not among them: each step's `docker
exec` is given the secrets that step may see, the same way, so the
container never holds one and a step cannot read another's from
`/proc/1/environ`. Starting the container, image pull included, writes
to the run's log and is limited to `--step-timeout`. A container that does
not start makes the check `error`.

Under `shell`, each step is `sh -ec <script>` in the checkout. When the
run ends, the container is removed, which kills anything a step
left behind, and then its cache overlays. The job is released once the
container is gone, before the log and check are written, so a slow PDS
does not hold a job.

### Steps for some refs

A step may name `on: {refs: [...]}`, matched as `on.push.refs` are,
against the ref a push moved or a run asked for by name is at
([reference](../reference/workflows.md#steps-for-some-refs)). A step it
does not match is not run: the log says why under its `==> <step>` line,
and its check lists it `skipped`, as a step after a failure is, the run
still passing. On a shared step, its `on` holds for each step it comes
to, beside their own.

A pull's run and a merge-queue batch's run match no ref, whatever
`KNOTLESS_RUN_REF` holds. A pull's holds its head branch, which a fork
may call `main`, and a batch's holds its base branch, which has not
moved yet: matching either would run a step meant for the default
branch before anything landed there. So a step for `refs/heads/main`
runs only at the push that lands the change. That is why a run that left
a step out is never kept for that push to reuse (below).

### Time limits

`--step-timeout` (1800 seconds) limits each step. A step's own `timeout`
can only shorten it, except in a run asked for by name, where it may
exceed it. `--run-timeout`, unset by default, limits a run's steps
together: each step's limit is the shorter of its own and what is left of
the run's, and a step killed because the run's time is up says so in the
log (`killed as the run's <n>s were up`), the steps after it skipped. A
run asked for by name is not held to it, as it may outrun
`--step-timeout`.

### What a run may take

Under docker, `--cpus`, `--memory` (whole GiB, no swap beyond it) and
`--pids` limit each run's container, and `--network` names the network
it joins. All are unset by default, leaving a run as much of the host as
it can get, as before they existed; a runner checking code it does not
trust sets them all, so no run takes the machine from the others or from
what else the host serves. They apply to every run alike, and come on
top of the lower CPU weight runs other than a push to the default branch
already get. The network is made, if missing, as a bridge with traffic
between its containers turned off
(`com.docker.network.bridge.enable_icc=false`), so a run reaches neither
another run nor the containers on the host's other networks, which
docker keeps apart from each other; one
there already is used as it is, and one another run's start makes
meanwhile is as good. A network that cannot be made fails the run's
start, an `error` check, its overlays removed. It does not stop a run
reaching the host itself, or anything else the host can reach.

`--runtime` names the container runtime runs start with, as docker's own
`--runtime` does: `runsc` runs them under gVisor, whose kernel takes a
run's system calls instead of the host's. The runner host for other
people's code runs every run so, with the limits above and egress rules on
its network that keep runs off the host and the private ranges: see
[the runner host](../../deploy/runner-host/README.md).

`--trusted-runtime` names another, for code an owner `--trust` names
vouches for: a push to their repository, or a pull or merge-queue batch
every writer of which they vouch for, as [whose code runs](#whose-code-runs)
reckons it. gVisor costs such a run two to five times
its time, the tests above all, and buys nothing for it: once the pull is
queued, its batch runs the same code on the owner's own runner with no
sandbox at all. Anyone else's code, a pull its owner asked to run among
it, keeps `--runtime`.

`--resolv` names a file on the docker host that runs see as their
`/etc/resolv.conf`, mounted read-only. On a network docker made, a
container's resolver is docker's own at 127.0.0.11, reached through NAT
rules in the container's network namespace. gVisor's network stack does
not apply them, so under `runsc` a run resolves nothing until it names
resolvers of its own, as the runner host's file does.

A timeout or a cancel sends `SIGKILL` to the step's process group through
a system call, not a `kill` command, since the runner's own image has
none. Without that, a cancelled or timed-out step ran on, and a hung one
held its job and a stop for good. Under docker the process killed is the
`docker exec` client, and what it started goes with the container, which
is removed right after.

A check is `cancelled` if a step was cancelled, else `failure` if a step
failed, else `success`. An error of the runner's own makes it `error`.

### Environment and outputs

A step sees the workflow's `env`, the secrets granted to the run, and the
runner's `KNOTLESS_RUN_*` variables. Under `shell` it also sees a few of
the runner's own variables, such as `PATH` and `HOME`. Under docker those
reach only the docker client, and the container keeps its image's
environment. The [workflow reference](../reference/workflows.md#environment)
lists them.

After the steps, passed or failed, the images a run left in
`KNOTLESS_RUN_OUTPUTS` become the check's
[`outputs`](../reference/workflows.md#outputs), as blobs (binary data
stored beside records). A PDS can take seconds over each blob, so the
runner uploads four at a time, and files with the same bytes once, as
one blob. `outputs` is an open union, so readers pass over kinds they do
not know.

## Caches

A workflow's `cache` lists paths in the container to keep between runs.
Caches exist only under docker. They have to be fast, and must not carry
anything across trust levels.

Each cache path has a base, a read-only directory per repository,
workflow and path. A run never writes it. The run sees it through an
overlay of its own, an overlayfs mount whose upper layer takes the run's
writes. Runs never wait on each other for a cache, and two builds never
share output. A new cache starts as an empty base, and the image's own
files at that path show through only where the base has nothing.

```mermaid
flowchart BT
  base[base, generation n, never written]
  a[run on the default branch: its upper layer] --> base
  p[trusted pull's run: its upper layer] --> base
  u[untrusted pull's run: its upper layer] --> apart[an apart base, always empty]
  a -->|the run passes| next[base, generation n+1]
  p -->|the run passes| kept[kept layers, by check key]
  kept -->|the push at the same tree reuses the pass| next
  p -->|passes, under --pulls-only| next
  u -->|removed when the run ends| gone[nothing kept]
```

Only a passing push to the default branch promotes, or, on a repository
whose pushes run on another runner, a trusted pull's passing run
([below](#where-pushes-run-elsewhere)). Each non-empty upper
layer is applied over a hard-link clone of its base, as the base's next
generation, and runs started after that start from it. The apply follows
overlayfs's markings: a whiteout deletes, a directory not marked opaque
merges, and anything else replaces. Files are moved, not copied.

A base is never changed while a run reads it. An old generation goes once
no run reads it and it is not current. A new base over `--cache-max`
(16 GiB) is not made current, and the next run starts afresh.

### Across trust levels

No untrusted pull, nor batch, promotes. If an untrusted pull, or a batch holding one,
runs a workflow that asks for secrets, it reads its caches apart: from
bases nothing writes, so always empty, because a push's run with secrets
may have left things in the shared base. An untrusted run's layers are
never kept, so its writes reach no base.

Every other run's upper layer is removed when the run ends, with one
exception. A trusted pull's or batch's passing run of a workflow that
could be [reused](#reusing-a-pulls-pass-at-the-push) keeps its upper
layers, so the push that lands it takes them as if it had made them.

Kept layers go after 3 days, beyond 16 runs, or beyond `--cache-max` in
total, oldest first. They are also forgotten once taken, once their pull
moves or closes, or, for a batch, once the next batch on the branch
passes. A batch running ahead does not displace the kept passes of the
batch it runs ahead of until that one lands. A push takes only the layers
written over a base that is still current, and the log says the next run
warms the rest.

On start, the runner removes its leftover overlays and half-made bases,
and sweeps no old base while an old overlay is still mounted.

### Room on the disk

The caches share their disk with whatever else the host runs. On
knotless.dev's own host that includes the site's repositories and index.
On 2026-10-31 a full disk took the site down. So the runner keeps
`--disk-floor` free on the disk `--caches` is on: 20 GiB by default. That
is about two of the largest bases the host has held, so one run's build
can land on the floor without the site running out.

Before a run starts, holding its job so no other run starts meanwhile, the
runner checks the free space. When it is below the floor, the runner sheds
first: every kept run's layers, and every cache's base no run reads. Those
caches' next runs start afresh, as after a dropped base. If that is not
enough, the run waits, checking again every 30 seconds, until the runs
going end and free their layers. It stays queued, and can be cancelled as
usual. Its log says what was shed and how long it waited. The runner logs
`disk short` at warn each time it looks and finds the disk short. A disk
it cannot read holds nothing back.

The floor holds only at a run's start. A run already going can still fill
the disk.

### Where pushes run elsewhere

A repository named with `--pulls-only` has its pushes run by another
runner, with caches of its own, so here no base would ever be written
and no kept layer taken: every pull would start cold. Instead a trusted
pull's passing run into the default branch, of a workflow that could be
[reused](#reusing-a-pulls-pass-at-the-push), promotes its layers itself,
as a push's run would, and keeps none. Two rules keep the base from
going backwards:

- A layer is promoted only over the base it was written over, if that is
  still current, checked as the new base is made current. Of two pulls'
  runs over one base, the first to pass makes the next one, and the
  other's layers go.
- A base a pull made is marked with the default branch's generation at
  the tip the pull merged with (`marks.json` under the caches). A run
  merged with an older tip than the mark does not promote, so the base
  never goes back to an older `main`. A base a push made has no mark.

The pull's log says which of its paths it made the base and which it
did not. Pulls into another branch, untrusted pulls (an agent's among
them) and workflows that ask for secrets or are `superseded: finish`
never promote.

This keeps the rule above that no untrusted run's writes reach a base:
only the owner's or a collaborator's pull promotes, code that may be
pushed to the default branch anyway. What changes is that it reaches the
base before it is merged. That is safe because on a `--pulls-only`
runner the base is read only by pulls, which get no secrets, and no
workflow that asks for secrets promotes. The worst a bad base can do is
make a later pull's check here wrong; the queue checks each batch again
where pushes run, from caches no pull wrote, and a deploy waits for
main's own checks there.

## Reusing a pull's pass at the push

When a trusted pull merges, the merge commit often has exactly the tree
the pull was checked at, and running the same workflows again would test
nothing new. So a push to the default branch whose tree equals the tree
of a trusted pull's passing run into that branch, or a trusted batch's,
takes that run's result. The check is filed `success` with the earlier
steps and outputs, and its log names the run it took, then copies that
run's log, so it still reads whole once the earlier check is pruned. The tree holds the
workflows, so they were the same workflows.

A pass is not reused when:

- the workflow asks for secrets, since the pull never had them;
- the workflow is `superseded: finish`, since a deploy runs for what it
  does, not what it checks;
- the earlier check is no longer `success`;
- the run was asked for by name;
- the run left a step out for its ref (`on`), which the push, at another
  ref, may run.

A batch can likewise take a trusted pull's pass at the same tree. Passes
stay in the state file after their cache layers are forgotten, so a check
can still be reused without its caches.

## Secrets

Secrets come from two places. The file named by `--secrets`, keyed by the
repository's label exactly as on the command line
([format](../reference/workflows.md#secrets-file)), is the operator's. The
owner's are those `knotless secret` sealed to the runner's key and left at
the appview ([The runner's key](#the-runners-key)): for a name the workflow
lists and the file does not grant, the runner asks the appview
(`dev.knotless.listSecrets`, with its own service token) for those sealed
to it for the repository, by owner and key, and opens each with its keys,
rebuilding what it was bound to from where it found it. One that does not
open, or whose `refs` do not reach the ref, is not given. The file wins a
name both grant, since an operator who grants one means it. An owner can
give the hosted runner a secret without handing it to whoever runs it.
An appview that does not answer makes the check `error`, as an unreadable
file does. A runner that logs in as the repository's owner is listed
every secret, as owners are, with the values of those sealed to it, so it
opens its own as any runner does.

A secret reaches a step
only if the workflow lists it, the file or the owner grants it to the repository and,
when the secret names `refs`, to the ref, and the run is not a pull's or a
batch's. A secret limited to `refs/heads/main` cannot be read from another
branch, which matters because a branch can change the workflow itself. A
secret not granted is absent from the step's environment, and the log
names it before the steps (`==> NAME is not granted to <repo> at <ref>`),
so a workflow that fails for want of one says why. The line names the
secret, never a value.

A step put in place by a [shared step](reusable-workflows.md) sees only
the secrets its `uses` step passes it, by name, of those the step file
asks for. Every other step sees every secret the run is granted. In a
workflow that uses shared steps, what a step given secrets leaves
running is ended once the step ends, so that no process holds them for
a later step to read.

The runner re-reads the file on every run that needs secrets, so a change
takes effect without a restart. A workflow that asks for none never reads
it. A file that cannot be read makes the check `error`.

Before the log is uploaded, each granted value is replaced by `***`, and
so is each trimmed line of a multi-line value such as a PEM key. The
longest values go first, so a secret inside another leaves nothing behind.
Values under 4 bytes are left alone, since masking them would mask
ordinary output. Masking catches a secret only as written. A step can
still leak one transformed, encoded, or a bit at a time.

### The runner's key

Owners will give a runner secrets without the operator's hands: each
sealed to the runner, so whatever holds it on the way holds only
ciphertext. For that the runner keeps a P-256 key, made on its first
start, in the file `--secrets-key` names (`secrets-key.json` beside the
state file by default, readable by its owner alone), and publishes its
public half as its `dev.knotless.runnerKey` record, as a `did:key`. It
writes the record again only when the key differs.

A value is sealed with HPKE (RFC 9180) in base mode: DHKEM(P-256,
HKDF-SHA256), HKDF-SHA256 and AES-128-GCM, as GitHub seals a repository's
Actions secrets to a key of its own. What it is for is bound to it as the
AEAD's associated data: the owner's DID, the repository's key (its record
key, so a rename keeps its secrets and a repository made later under the
old name gets none), the runner's DID, the secret's name and the refs it
is limited to. The runner rebuilds those from where it found the value,
so one copied to another repository, name, runner or refs does not open:
nobody can carry an owner's sealed token into a repository of their own
to print it. Base mode does not say who sealed a value, so whoever holds
the sealed values could swap one in, though never read one.

The file holds a list of keys, the newest first: the runner publishes the
newest and opens what was sealed to any of them, each value naming the
key it was sealed to, so a key can be replaced while values sealed to the
old one are set again.

## Check records

Checks are `dev.knotless.check` records in the runner's account, written
with `putRecord`, so a run replaces its own earlier record. The record key
says what was checked:

| Subject | Record key |
| --- | --- |
| a push | `<h(owner did)>.<repo key>.<h(full ref)>.<commit>.<workflow>` |
| a pull | `<h(owner did)>.<repo key>.<h(pull AT-URI)>.<head>.<base>.<workflow>` |
| a merge-queue batch | `<h(owner did)>.<repo key>.<h("queue refs/heads/<branch>")>.<tip>.<base>.<workflow>` |

`h` is the first 26 characters of the lowercase base32 SHA-256. Record
keys sort as strings, so a prefix can be listed with one `listRecords`
scan: all of one runner's checks of a repository share
`<h(owner did)>.<repo key>`, and a pull's, a queue's or a ref's pushes
share a narrower prefix, as do a ref's checks of one commit: `knotless
checks --ref` scans only those, as deploy.yml's wait does. The owner's hash keeps one owner from naming another's repository
with a key of their own. A pull run against a newer base is a check of its
own.

The record's fields are in the `dev.knotless.check` lexicon. Two points
it understates: `baseCommit` is set for a batch's run as well as a
pull's, and `cancelled` covers every reason a run is cancelled, not only
a newer commit.

The runner retries a write while the PDS fails with a 5xx, a 429 or no
status at all, three times, waiting 0.5, 1 and 2 seconds. A write given
up would leave a check that never ends, or, for the log uploaded as a run
ends, file a run that passed as an error. So a 429 that says when its
rate limit lets writes in again (`RateLimit-Reset`) is waited out until
then instead, for up to ten minutes in all for the one write, while the
run keeps its job.

### Output as it is printed

A check's record is written as the run starts and as it ends, and its log
is a blob uploaded at the end, so nothing between shows in the record.
With an appview (`KNOTLESS_APPVIEW`), the runner also sends it what the
run prints as it prints it (`dev.knotless.putRunOutput`), for the check's
page to show while the run goes ([serve](serve.md#live)). It reads the
log file as the steps write it, every second while the appview says a page
follows the run and every five seconds otherwise, and at once as a step
starts or ends, with the steps ended so far, the name of the one
going and when the run started, on every send, so a page names the step going even when the
appview never had its `==>` line (it started before the appview did, or
before the runner could reach it). Only whole lines go (a line ends at a
newline or a carriage return), each masked with the run's secrets as the
log is (see [Secrets](#secrets)), so a secret printed in pieces is never
sent split where masking would miss it; the masked text is then cut into
pieces of at most 16,000 bytes. A line still unended past a megabyte is let
go unsent. A send that fails stops that round, and what it held goes again
in the next, so a page sees what the run printed while the appview was
out of reach; past a megabyte held, as much as the appview keeps, the
oldest is let go. The last send, with the rest of the log and saying the
run ended, goes before the check is filed, given five seconds at most, and
is not sent again if it fails: the log is what is kept, and a run never
waits on the appview. A private repository's runs
are sent too: the appview shows them only to its members, as it does
their logs. A runner logged in with OAuth before these methods were in
knotless's scope gets no token for them until it logs in again: until
then its runs' output is not sent, and its private repositories' spaces
are asked every `--poll` seconds, as with no appview.

### Pruning

Nothing else deletes a check, and most are soon of no use: on 2026-10-25
our account held 4,041 checks of knotless, three quarters of them pull
runs of heads or bases since moved, and batches. So the runner deletes its
own, at the end of a pass, at most once an hour per repository, in
`applyWrites` of 200 deletes. A check still running, or finished less than
a day ago, stays. Of the rest, it deletes:

- a merge-queue batch's run;
- a pull's run, once the pull is closed or merged, or a newer run of the
  same pull and workflow was filed;
- a run on a ref other than the default branch, once the ref is gone or a
  newer run of the same ref and workflow was filed;
- a run on the default branch older than `--keep-checks` days (30),
  unless it is the newest of its workflow there.

A push that takes a pull's or a batch's pass copies that run's log (see
[Reusing a pull's pass](#reusing-a-pulls-pass-at-the-push)), so the
default branch's checks read whole once the run they took is gone. Blobs
go with their records: the PDS reaps a blob no record references.
`--keep-checks 0` prunes nothing, and neither does a runner with no
appview (`KNOTLESS_APPVIEW` set empty) for a public repository, whose open
pulls it cannot then know.

### Whose checks count

Anyone can write a check record about any repository, so the repository
record lists, in `runners`, the accounts whose checks readers trust, and
readers show only theirs. Readers read `runners` when they show checks, so
a runner added later shows what it reported before. The owner sets the list
with `knotless runners`.

### Private repositories

A check says which commits a repository has, what its workflows are
called and what their steps printed, so a private repository's checks
must be as private as its code. The runner reads a private repository in
its [space](access.md#private-repositories), as its own account, a member
there, and files each check with the space's `putRecord` into its own repo
in the space. Logs and images are blobs those records reference, which
only members can read. `knotless create --private`, for the runners it
names, and `knotless runners <url> add` make the runner a member, as
reading the space and writing to it both need; `remove` leaves it one,
since it may be a member for itself too.

The appview indexes no private repository, and Jetstream carries nothing
written in a space, so a runner would never find one by itself. Both
commands also leave the runner an invite at the appview
(`dev.knotless.inviteRunner`, with the owner's service token), and a
runner with `--discover` reads its own (`dev.knotless.listRunnerInvites`,
with its token) every `--sweep` seconds, keeping where the last page
ended. An invite only says where to look: anyone can leave one, and the
runner watches the repository only if it can read its record in the
space, as a member, and the record names it. An owner whose invite could
not be left, with no appview or a login from before the methods were in
knotless's scope, is warned (`<did> was not told of it: …`); `knotless
runners <url> add` invites it again once they log in again, and naming
the repository on the runner's command line works as ever.

Readers read a private repository's checks from each runner's repo in the
space, with a space credential, which even the owner needs to read
another member's repo there. Reruns need nothing new: the owner writes
them where all of their records about the repository are, in the space.

Nothing written in a space reaches Jetstream. The runner asks the space's
authority instead, every `--poll` seconds, with `com.atproto.space.listRepos`
from where it last looked: the members' repos written since, if any,
whatever was written. One that answers rings the door, and the pass reads
what moved as it reads a public repository's. Reading the space takes a
credential, so a runner of a private repository logs in with OAuth.

Asking every space every ten seconds is slow to hear of a push and costs a
request per space whether anything was written or not. So, with an appview,
the runner also asks each space's authority, as a member, to tell the
appview of every write there (`com.atproto.space.registerNotify`, naming
the appview's `#knotless_appview` service), every twelve hours, as the
authority keeps the ask for a day. The appview passes each write on to the
runners the space's owner invited (`dev.knotless.awaitSpaceWrites`): the
runner asks from where it last heard, and is answered as soon as one of
its spaces is written to, or with none after 25 seconds. A write passed on
rings that repository's door; an answer saying the runner may have missed
some, as the first does and one after the appview restarted, rings every
private repository's. The runner registers rather than being told
directly because a service a space's authority calls needs a DID document
and an address it can reach, which a runner on a laptop or behind NAT has
not; the appview has both. A word passed on is only a reason to look, as
an invite is.

While the appview answers, each space is asked itself only once a minute,
for a write the appview missed. With no appview, or one that does not
answer, it is asked every `--poll` seconds as before, and so is one whose
owner invited no runner (one named only on the runner's command line).

A private repository that is deleted, or no longer names the runner, is
left, and its space is asked no more: one runner asked a deleted one
every few seconds for hours.

A repository found deleted, its owner holding no repository record with
its key (one renamed still does), is no longer watched, and the runner
clears what it kept of it: the checks it filed of it, with their logs and
outputs, the grants the secrets file gives its label, so that a
repository made under that name later does not inherit them, and its
caches' bases. A cache is named for its repository, workflow and path,
each hashed, so a repository's are found by name; one named as before
that, by a single hash, is removed at start.

## Reruns and runs by name

The owner asks for a run with a `dev.knotless.rerun` record in their own
account, keyed by the repository's key and a TID, written by
`knotless rerun`, the MCP `rerun` tool, or the pages: Re-run on a pull's
checks and on a check's page, and Run workflow on the checks page, which
names a ref, one of the workflows `knotless workflows` says may be asked
for there, and its variables. On every pass the runner reads
the owner's rerun records from the owner's PDS. Only the owner's count,
and a forged frame cannot fake one. The appview does not index them.

A request does one of four things:

- Naming a ref: the runner forgets the ref's last generation and dropped
  runs, so the next pass runs all its workflows at its current target.
- Naming a ref and a workflow: the next pass runs that one workflow at the
  ref, even if the ref did not move.
- Naming a pull: the runner forgets where the pull last ran, and runs all
  its workflows where it is now, even if it is queued or untrusted. A pull
  run dropped at a stop runs again on the same terms.
- Naming a pull and a workflow: the next pass runs only that workflow
  where the pull is, as a run dropped at a stop runs only what was
  dropped, leaving its other checks standing. It is for one check that
  failed alone, or that reads the pull's body, such as a screenshot check
  reading which pages may move. A pull asked for in full, or moved, runs
  all of its workflows anyway. Variables go only to a run at a ref.

A request naming neither a ref nor a pull does nothing. Without `--pulls`, requests for pulls are recorded but not acted
on.

### By name

A run asked for by name lets a costly workflow, such as a long scenario
suite, run only when someone wants it. A workflow may run by name at a ref
if it runs on a push to that ref or its `on.asked.refs` matches. A
workflow with only `on.asked` runs in no other way. Asking for one that is
not there, or may not run at that ref, files an `error` check saying so.

The request may carry variables, which replace the workflow's `env`
entries of the same name. The runner drops any named as one of the
workflow's secrets, and writes the variables at the top of the public
log. These runs rank last, may exceed `--step-timeout` with a step's own
`timeout`, and are cancelled at once by a stop and run again on the next
start. Workflows wanted at a ref that no longer exists are forgotten.

One whose `on.asked` names `commit` globs writes back. Once it passes, the
runner reads `git status` in its checkout, commits the files changed
among those globs on top of the commit it ran at, fetches the commit into
its mirror and pushes it to the branch, as with a landing. It refuses the
default branch, a protected one, and a branch that moved while it ran,
failing the check with `not written back: <why>`. See
[Writing back](../reference/workflows.md#writing-back).

## The merge gate

In a repository that names runners, `knotless merge` and the MCP `merge`
tool merge a pull only once each workflow that runs on it has a `success`
check at its head and its base as it is now. The workflows are those of
the head merged with the base, plus any that fail to load, and the commit
is the head for a fast-forward, else the same merge commit the runner
checks.

For each workflow the newest check at that head and base decides. A status
other than `success`, `queued`, `running` or `cancelled` fails the gate,
and that wins. Otherwise a `queued` or `running` check means the checks
are still running.
Otherwise a missing or `cancelled` check means the pull is not checked
yet. Otherwise it passes, including a `success` skipped by `paths`.

With `--wait`, merge polls while checks are running or missing. When the
owner waits on a pull that is not checked, merge writes one rerun
request for it, since a runner started after the pull last moved has not
run it. `--force`, the owner's alone, merges whatever the checks say. A
repository naming no runners merges without a gate.

The gateway, through which collaborators push to the owner's account,
holds a collaborator's merge to the gate as well, reading the pull's checks
as `merge --check` does when the push arrives: a push whose commit ends
with a pull's AT-URI, as a merge's does, and is that pull's merge, must be
one that may merge ([gateway.md](gateway.md#protected-branches)). A branch whose rule asks for reviews holds a
merge to those too, read before the checks ([required
reviews](collaboration.md#required-reviews)); the runner landing the queue
ejects a queued pull that lacks them before it goes into a batch. `knotless merge --check`, MCP `merge` with
`check`, and a pull page's merge box read the same gate without merging:
`passed`, `failed`, `running`, or `pending` for a workflow with no check
there yet.

## The merge queue

The merge queue keeps a merge from retesting every open pull: queued
pulls are checked together, as a batch, and land together. A runner lands
a repository's queue only with `--queue`, and only if its own account
writes to the repository, or the repository names it and no runner that
does: then it lands through the gateway
([merge-queue.md](merge-queue.md#who-lands-a-queue)). [merge-queue.md](merge-queue.md) has the design.

The runner stacks up to `--batch` queued pulls on the branch tip and runs
the batch once at its tip, in the `Main` class, without secrets. If every
run passes, it lands the batch in one push, which refuses if the branch is
anywhere but where the batch was built. If not, it halves the batch to
find the pull that broke it. With `--depth 2`, the default, it runs the
next batch ahead, built on the one being checked, in the `Later` class.
A pull goes into a batch only once its pre-flights, the workflows that
say `queue: first`, have passed at its head; with none passed or
running, the runner runs them on the pull itself, as a pull's run.

While a runner lands a repository's queue, or runs its pulls alone with
`--pulls-only`, a queued pull does not run on its own: its batch is its
run. Pulls not in the queue rank `Later` and run again only when their
head moves, not their base. A trusted batch's passes
are kept like a trusted pull's, so the push that lands it reuses them and
their cache layers.