knotless-run internals

What knotless-run keeps on disk, and the containers and volumes it makes: for operating a runner and debugging one. How it decides what to run is in the design; its flags are in the command reference.

The state file

The state file is pretty-printed JSON, written through a temporary file. Each field is keyed by repository, as <owner did>/<repo key>:

repos

each ref’s generation last run

pulls

each open pull’s URI, to the "<head> <base>" it last ran at

reruns

the record key of the newest rerun request acted on

asked

the URIs of untrusted pulls the owner asked to run, not run yet

asked_only

of those, each asked for only some workflows, to their names

passed

each passing trusted pull’s or batch’s run: tree, workflow, base ref, check key, pull, and the head and base the pull was at; oldest first, at most 256 a repository

dropped

runs dropped as the runner stopped, by ref, pull URI or queue <branch ref>, to "<commit> <base>", to workflows

wanted

workflows asked for by name and not run yet, by ref, with the variables asked with

Caches on disk

Under --caches:

bases/<cache>/<generation>

each generation of a cache’s base; <cache> is knotless-cache-<h1>-<h2>, <h1> a hash of the repository and <h2> of the workflow, mount path and, for a cache read apart, apart; a deleted repository’s go

runs/<run>/<i>/{upper,work}

a run’s overlay layers; <run> is the runner’s boot time in milliseconds and a counter

kept/<hash of check key>/

a passing trusted run’s kept layers: kept.json, and <i>/upper per cache

marks.json

of each cache whose current base a trusted pull made under --pulls-only, that base’s generation and the default branch’s generation at the tip the pull merged with: {"<cache>": [<base>, <tip>]}

Containers and volumes

Each overlay is a docker local volume of type overlay (lowerdir=<base>,upperdir=<upper>,workdir=<work>,userxattr), named knotless-overlay-<run>-<i> and labelled dev.knotless.overlay=<run>. Each run’s container is labelled dev.knotless.runner=<did> and dev.knotless.run=<check key>, runs with --init, mounts the checkout at /work and the outputs at /outputs, and, outside pushes to the default branch and the batch landing next, has --cpu-shares 256. With the runner’s limits set it also has --cpus <n>, --memory <bytes> --memory-swap <bytes>, --pids-limit <n> and --network <name>; before it starts, docker network inspect <name> asks for that network, and if it is missing docker network create --driver bridge --opt com.docker.network.bridge.enable_icc=false <name> makes it, asking again when that fails, in case another run’s start made it meanwhile.

Cleanup at start

At start the runner removes, before any run, containers labelled with its DID or (with --work) mounting a path under --work, volumes labelled dev.knotless.overlay, and knotless-cache-* volumes from before overlays. Runs whose overlays are still mounted keep their directories, and while any are, no old base is removed. Half-made *.new bases, and kept directories not wanted or broken, are removed, and the newest generation of each cache is the current one. A base named knotless-cache-<hash>, as runners named them before a cache’s name held its repository’s, is removed. Marks of a base no longer current are let go.

Free space

The free space is statvfs(2) of --caches: the blocks available to an unprivileged writer (f_bavail) times the fragment size (f_frsize). Below --disk-floor, a run about to start removes every kept/* directory and, for each cache whose current base no run reads, that base and its older generations no run reads, forgetting the cache’s mark. It does this once for each run that finds the disk short. The run’s log starts ==> <n> GiB free on the disk, under the runner's floor of <n> GiB: shed <n> caches' bases no run read and <n> kept runs' layers, then says ==> this one waited <n>s for room on the disk if it waited 30s or more. The runner logs disk short: shedding caches and disk short: waiting for room at warn, with the free space in reason.

What a pass logs

Each pass over a repository, as its door rings, logs pass ended with repo, status (ok or failed) and duration_ms: at debug, or at info when it took KNOTLESS_SLOW_MS (1000) or longer.

A pass still going after 5 minutes logs pass running long at warn, with repo, duration_ms and stage, where the pass last said it was, and again every 5 minutes while it goes on. The stages, in the order a pass meets them:

Stage
waiting for a stop	waiting for a stop under way, which holds every pass back
reading the repository	reading the repository’s record and refs
reading reruns	reading the reruns asked for
running refs	filing and starting the runs of refs that moved
running pulls	with --pulls, reading open pulls and starting their runs
reading the queue	with --queue, reading the merge queue’s records
syncing the mirror	bringing the mirror the queue builds batches in up to date
landing a batch	landing a batch whose checks passed
ejecting a pull	writing a pull’s ejection
checking pre-flights	reading and starting the pre-flights of pulls waiting
stacking a batch	merging a batch’s pulls onto the branch, in the mirror
launching a batch	filing and starting a batch’s runs
pruning checks	deleting checks of no more use

Each git command logs git (target knotless::git), with command, its subcommand (pack-objects), path, the repository’s directory, and duration_ms: at debug, or at warn when it took KNOTLESS_SLOW_MS or longer, as a slow HTTP call logs outgoing at warn. A git command is never cut short.

A failure of a repository’s work logs failed at warn, with repo and error, the first time that error is seen for that repository. The same error again is logged only once a minute while it goes on, with count, the times it was seen since last logged. Another error is logged at once, and so is the same one after any of the repository’s work succeeds, or once it has not been seen for an hour.