# knotless-run internals

What `knotless-run` keeps on disk, and the containers and volumes it makes:
for operating a runner and debugging one. How it decides what to run is in
[the design](../design/runner.md); its flags are in the [command
reference](cli.md#knotless-run).

## The state file

The state file is pretty-printed JSON, written through a temporary file.
Each field is keyed by repository, as `<owner did>/<repo key>`:

| Field | Holds |
| --- | --- |
| `repos` | each ref's generation last run |
| `pulls` | each open pull's URI, to the `"<head> <base>"` it last ran at |
| `reruns` | the record key of the newest rerun request acted on |
| `asked` | the URIs of untrusted pulls the owner asked to run, not run yet |
| `asked_only` | of those, each asked for only some workflows, to their names |
| `passed` | each passing trusted pull's or batch's run: tree, workflow, base ref, check key, pull, and the head and base the pull was at; oldest first, at most 256 a repository |
| `dropped` | runs dropped as the runner stopped, by ref, pull URI or `queue <branch ref>`, to `"<commit> <base>"`, to workflows |
| `wanted` | workflows asked for by name and not run yet, by ref, with the variables asked with |

## Caches on disk

Under `--caches`:

| Path | |
| --- | --- |
| `bases/<cache>/<generation>` | each generation of a cache's base; `<cache>` is `knotless-cache-<h1>-<h2>`, `<h1>` a hash of the repository and `<h2>` of the workflow, mount path and, for a cache read apart, `apart`; a deleted repository's go |
| `runs/<run>/<i>/{upper,work}` | a run's overlay layers; `<run>` is the runner's boot time in milliseconds and a counter |
| `kept/<hash of check key>/` | a passing trusted run's kept layers: `kept.json`, and `<i>/upper` per cache |
| `marks.json` | of each cache whose current base a trusted pull made under `--pulls-only`, that base's generation and the default branch's generation at the tip the pull merged with: `{"<cache>": [<base>, <tip>]}` |

## Containers and volumes

Each overlay is a docker `local` volume of type `overlay`
(`lowerdir=<base>,upperdir=<upper>,workdir=<work>,userxattr`), named
`knotless-overlay-<run>-<i>` and labelled `dev.knotless.overlay=<run>`.
Each run's container is labelled `dev.knotless.runner=<did>` and
`dev.knotless.run=<check key>`, runs with `--init`, mounts the checkout
at `/work` and the outputs at `/outputs`, and, outside pushes to the
default branch and the batch landing next, has `--cpu-shares 256`. With
the runner's limits set it also has `--cpus <n>`, `--memory <bytes>
--memory-swap <bytes>`, `--pids-limit <n>` and `--network <name>`; before
it starts, `docker network inspect <name>` asks for that network, and if
it is missing `docker network create --driver bridge --opt
com.docker.network.bridge.enable_icc=false <name>` makes it, asking again
when that fails, in case another run's start made it meanwhile.

## Cleanup at start

At start the runner removes, before any run, containers labelled with its
DID or (with `--work`) mounting a path under `--work`, volumes labelled
`dev.knotless.overlay`, and `knotless-cache-*` volumes from before
overlays. Runs whose overlays are still mounted keep their directories,
and while any are, no old base is removed. Half-made `*.new` bases, and
kept directories not wanted or broken, are removed, and the newest
generation of each cache is the current one. A base named `knotless-cache-<hash>`, as runners named them before a cache's name held its repository's, is removed. Marks of a base no longer
current are let go.

## Free space

The free space is `statvfs(2)` of `--caches`: the blocks available to an
unprivileged writer (`f_bavail`) times the fragment size (`f_frsize`).
Below `--disk-floor`, a run about to start removes every `kept/*`
directory and, for each cache whose current base no run reads, that base
and its older generations no run reads, forgetting the cache's mark. It
does this once for each run that finds the disk short. The run's log
starts `==> <n> GiB free on the disk, under the runner's floor of <n>
GiB: shed <n> caches' bases no run read and <n> kept runs' layers`, then
says `==> this one waited <n>s for room on the disk` if it waited 30s or
more. The runner logs `disk short: shedding caches` and `disk short:
waiting for room` at warn, with the free space in `reason`.

## What a pass logs

Each pass over a repository, as its door rings, logs `pass ended` with
`repo`, `status` (`ok` or `failed`) and `duration_ms`: at debug, or at
info when it took `KNOTLESS_SLOW_MS` (1000) or longer.

A pass still going after 5 minutes logs `pass running long` at warn, with
`repo`, `duration_ms` and `stage`, where the pass last said it was, and
again every 5 minutes while it goes on. The stages, in the order a pass
meets them:

| Stage | |
| --- | --- |
| `waiting for a stop` | waiting for a stop under way, which holds every pass back |
| `reading the repository` | reading the repository's record and refs |
| `reading reruns` | reading the reruns asked for |
| `running refs` | filing and starting the runs of refs that moved |
| `running pulls` | with `--pulls`, reading open pulls and starting their runs |
| `reading the queue` | with `--queue`, reading the merge queue's records |
| `syncing the mirror` | bringing the mirror the queue builds batches in up to date |
| `landing a batch` | landing a batch whose checks passed |
| `ejecting a pull` | writing a pull's ejection |
| `checking pre-flights` | reading and starting the pre-flights of pulls waiting |
| `stacking a batch` | merging a batch's pulls onto the branch, in the mirror |
| `launching a batch` | filing and starting a batch's runs |
| `pruning checks` | deleting checks of no more use |

Each git command logs `git` (target `knotless::git`), with `command`, its
subcommand (`pack-objects`), `path`, the repository's directory, and
`duration_ms`: at debug, or at warn when it took `KNOTLESS_SLOW_MS` or
longer, as a slow HTTP call logs `outgoing` at warn. A git command is never
cut short.

A failure of a repository's work logs `failed` at warn, with `repo` and
`error`, the first time that error is seen for that repository. The same
error again is logged only once a minute while it goes on, with `count`,
the times it was seen since last logged. Another error is logged at once,
and so is the same one after any of the repository's work succeeds, or
once it has not been seen for an hour.