knotless-run internals

What knotless-run keeps on disk, and the containers and volumes it makes: for operating a runner and debugging one. How it decides what to run is in the design; its flags are in the command reference.

The state file

The state file is pretty-printed JSON, written through a temporary file. Each field is keyed by repository, as <owner did>/<repo key>:

Field	Holds
`repos`	each ref's generation last run
`pulls`	each open pull's URI, to the `"<head> <base>"` it last ran at
`reruns`	the record key of the newest rerun request acted on
`asked`	the URIs of untrusted pulls the owner asked to run, not run yet
`asked_only`	of those, each asked for only some workflows, to their names
`passed`	each passing trusted pull's or batch's run: tree, workflow, base ref, check key, pull, and the head and base the pull was at; oldest first, at most 256 a repository
`dropped`	runs dropped as the runner stopped, by ref, pull URI or `queue <branch ref>`, to `"<commit> <base>"`, to workflows
`wanted`	workflows asked for by name and not run yet, by ref, with the variables asked with

Caches on disk

Under --caches:

Path
`bases/<cache>/<generation>`	each generation of a cache's base; `<cache>` is `knotless-cache-<h1>-<h2>`, `<h1>` a hash of the repository and `<h2>` of the workflow, mount path and, for a cache read apart, `apart`; a deleted repository's go
`runs/<run>/<i>/{upper,work}`	a run's overlay layers; `<run>` is the runner's boot time in milliseconds and a counter
`kept/<hash of check key>/`	a passing trusted run's kept layers: `kept.json`, and `<i>/upper` per cache
`marks.json`	of each cache whose current base a trusted pull made under `--pulls-only`, that base's generation and the default branch's generation at the tip the pull merged with: `{"<cache>": [<base>, <tip>]}`

Containers and volumes

Each overlay is a docker local volume of type overlay (lowerdir=<base>,upperdir=<upper>,workdir=<work>,userxattr), named knotless-overlay-<run>-<i> and labelled dev.knotless.overlay=<run>. Each run's container is labelled dev.knotless.runner=<did> and dev.knotless.run=<check key>, runs with --init, mounts the checkout at /work and the outputs at /outputs, and, outside pushes to the default branch and the batch landing next, has --cpu-shares 256. With the runner's limits set it also has --cpus <n>, --memory <bytes> --memory-swap <bytes>, --pids-limit <n> and --network <name>; before it starts, docker network inspect <name> asks for that network, and if it is missing docker network create --driver bridge --opt com.docker.network.bridge.enable_icc=false <name> makes it, asking again when that fails, in case another run's start made it meanwhile.

Cleanup at start

At start the runner removes, before any run, containers labelled with its DID or (with --work) mounting a path under --work, volumes labelled dev.knotless.overlay, and knotless-cache-* volumes from before overlays. Runs whose overlays are still mounted keep their directories, and while any are, no old base is removed. Half-made *.new bases, and kept directories not wanted or broken, are removed, and the newest generation of each cache is the current one. A base named knotless-cache-<hash>, as runners named them before a cache's name held its repository's, is removed. Marks of a base no longer current are let go.

Free space

The free space is statvfs(2) of --caches: the blocks available to an unprivileged writer (f_bavail) times the fragment size (f_frsize). Below --disk-floor, a run about to start removes every kept/* directory and, for each cache whose current base no run reads, that base and its older generations no run reads, forgetting the cache's mark. It does this once for each run that finds the disk short. The run's log starts ==> <n> GiB free on the disk, under the runner's floor of <n> GiB: shed <n> caches' bases no run read and <n> kept runs' layers, then says ==> this one waited <n>s for room on the disk if it waited 30s or more. The runner logs disk short: shedding caches and disk short: waiting for room at warn, with the free space in reason.

What a pass logs

Each pass over a repository, as its door rings, logs pass ended with repo, status (ok or failed) and duration_ms: at debug, or at info when it took KNOTLESS_SLOW_MS (1000) or longer.

A pass still going after 5 minutes logs pass running long at warn, with repo, duration_ms and stage, where the pass last said it was, and again every 5 minutes while it goes on. The stages, in the order a pass meets them:

Stage
`waiting for a stop`	waiting for a stop under way, which holds every pass back
`reading the repository`	reading the repository's record and refs
`reading reruns`	reading the reruns asked for
`running refs`	filing and starting the runs of refs that moved
`running pulls`	with `--pulls`, reading open pulls and starting their runs
`reading the queue`	with `--queue`, reading the merge queue's records
`syncing the mirror`	bringing the mirror the queue builds batches in up to date
`landing a batch`	landing a batch whose checks passed
`ejecting a pull`	writing a pull's ejection
`checking pre-flights`	reading and starting the pre-flights of pulls waiting
`stacking a batch`	merging a batch's pulls onto the branch, in the mirror
`launching a batch`	filing and starting a batch's runs
`pruning checks`	deleting checks of no more use

Each git command logs git (target knotless::git), with command, its subcommand (pack-objects), path, the repository's directory, and duration_ms: at debug, or at warn when it took KNOTLESS_SLOW_MS or longer, as a slow HTTP call logs outgoing at warn. A git command is never cut short.

A failure of a repository's work logs failed at warn, with repo and error, the first time that error is seen for that repository. The same error again is logged only once a minute while it goes on, with count, the times it was seen since last logged. Another error is logged at once, and so is the same one after any of the repository's work succeeds, or once it has not been seen for an hour.

chadtmiller.com/knotless Switch repository Loading…