The runner

knotless-run is knotless’s CI runner. It watches a set of repositories and runs their workflows when a ref moves or a pull changes. It reports each run as a dev.knotless.check record in its own atproto account.

A workflow is a YAML or JSON file under .knotless/workflows/, read at the commit being checked. Its format is in the workflow reference, and the runner’s flags are in the CLI reference.

Some background. A knotless repository is records in its owner’s atproto account, kept on the account’s server, its PDS. Each ref is one dev.knotless.ref record holding the ref’s target and a generation, a counter that goes up with every push to that ref. The owner’s PDS is the authority on where a ref points (storage.md). Pulls, and the states that open and close them, are records in each author’s own account. knotless-serve can index them as an appview, a service that lists records across accounts (collaboration.md, serve.md). The atproto site has the protocol itself.

What it does

The runner logs in as one account, from KNOTLESS_IDENTIFIER and KNOTLESS_PASSWORD (an app password), or with knotless login. An app password reads no private repositories, so a runner that watches one logs in with OAuth: on a server, through the gateway beside it (Bots), whose session lasts two years. Its arguments name the repositories to watch, each as <handle or did>/<name>; a private one is read as that account, a member of it (see Private repositories). The string as given is the repository’s label, which names it in logs, in KNOTLESS_RUN_REPO and in the secrets file. Internally the runner knows a repository by <owner did>/<repo key>, where the key is a TID (a time-ordered record key) fixed when the repository was made, so a rename or a new repository under the old name does not fool it.

Each run’s check record holds its status, each step’s result, a log and any images the run made. knotless checks, the web pages and knotless merge read them from the runner’s PDS: from its repo in the space, for a private repository.

What shapes the design

Six problems decide most of how the runner works.

Events cannot be trusted. The runner learns of pushes from Jetstream, a public relay of every account’s record changes. Its frames are unsigned, so anyone can forge one, and it has dropped events.
Pulls carry code from strangers. Anyone can open a pull, and its workflows and build scripts are theirs to write.
Deploys must not be cut off. A deploy is a workflow too, and stopping it halfway can leave a site down.
Every merge moves the base of every open pull. A pull is checked as merging would leave it, so each merge retests every open pull, and that burst must not starve the main branch’s own runs.
Caches cross trust levels. A cache written by a stranger’s pull must not reach a main-branch build, and one written by a run that had secrets must not reach a pull.
The runner stops and starts. It is redeployed, it crashes, and its host restarts. No run may be lost, and no check may stay running forever.

Finding out that something changed

Jetstream is only a doorbell. A frame tells the runner which repository to look at again, never what to run. The runner then reads the repository record, its refs and its rerun requests back from the owner’s PDS, and runs the targets the PDS gives. A forged frame can make the runner read, but never run something the owner did not push.

To ring a repository’s door is the runner’s term for waking that repository’s reconcile task. Each watched repository has exactly one. A reconcile pass reads the repository and starts whatever is due. Rings that arrive while a pass is reading coalesce into one more pass.

A frame rings a watched repository’s door when it changes:

a dev.knotless.ref or dev.knotless.rerun record of the owner’s for that repository;
the same kinds of record in a repository that an open pull into the watched one comes from (known only with --pulls, from reading pulls);
a dev.knotless.pull or dev.knotless.state record naming the repository, so a new, reopened or retargeted pull runs;
a dev.knotless.queue record, for the merge queue;
a dev.knotless.repo record of the owner’s, which may no longer name the runner (see Repositories that name it).

Because a frame only rings, a missed one loses no work, and the runner rings doors in other ways too:

It connects without a cursor, and every connection rings every door. What passed while it was disconnected runs on reconnect, and a replayed event finds nothing new. A connection silent for 60 seconds gets a ping, and silent another 60 is given up and made again, as knotless-serve’s is: a half-open connection never ends, and the runner would hear nothing from it.
It rings every door every --sweep seconds (600). Jetstream once dropped a whole account’s events for about 15 minutes on every instance. A pass runs only what has not run, so a sweep costs reads, not runs.
A pass that fails, say because the PDS is down, rings its own door again after min(--sweep, 60 s).
With --queue, a change to the merge queue’s records, re-read every --notice seconds (30), rings the door.
A private repository’s space written to rings its door, as the appview passes on its authority’s word of the write, at once; or, asked itself every --poll seconds (10), when anything was written there since the last ask. Nothing written in a space reaches Jetstream (see Private repositories).

A pass is logged as it ends, with how long it took. One that hung for 13 minutes once said nothing of where, so a pass still going after 5 minutes logs the stage it is at, and again every 5 minutes, and a slow git command is logged as a slow HTTP call is (What a pass logs).

Repositories that name it

With --discover, the runner watches, besides those its arguments name, every repository whose record names its account in runners. That is how a hosted runner serves whoever names it, with no list kept by hand. It finds them three ways, and reads each before watching it:

At the start, and every --sweep seconds after, it asks the appview (KNOTLESS_APPVIEW) for dev.knotless.listRepos?runner=<its DID>, and takes those it found before from the state file too, so a start does not wait on the appview.
A Jetstream frame writing a dev.knotless.repo record has it read that repository at once, so one named meanwhile is found within seconds.
A private repository’s invites, read every --sweep seconds from the appview, from where the last read ended (invited in the state file): see Private repositories.

Each found is read from its owner’s PDS, or its space, as any ring is, before it is watched: the appview, the frame and the invite only say where to look. One whose record there does not name the runner, or that is archived, is passed over, and one watched that becomes either is left. Unarchiving writes the record again, which a Jetstream frame brings, so it is watched again within seconds.

A repository found is labelled <owner did>/<name>, in logs, in KNOTLESS_RUN_REPO and in the secrets file. Its first pass starts from where its refs are, as any repository’s does, but for its default branch’s tip, which runs once: a repository made by its first push shows that push’s checks, and one that names the runner later shows the runner works. What else was pushed before it was found waits for its ref to move. A pass that finds its record gone, or no longer naming the runner, stops watching it: what it was running finishes, and its door and the loops that ring it go. One renamed is found again under its new name, by its key, and watched by that instead. A repository named on the command line stays as it is, whatever its record says.

A new repository names a runner of this kind without its owner doing anything. An appview started with --default-runner <did> names it in dev.knotless.describeServer, and knotless create, MCP’s create_repo and a first push write it into a new repository’s runners, unless the owner names others (--runner, or --no-runner for none); a private one’s are made members of it and invited, and knotless create says so. knotless.dev names its runner host’s account. It names one only: each runner a repository names runs all its checks, so two would run each twice. The record still says it all: the owner sees the runner with knotless runners <url> and lets it go with knotless runners <url> remove.

A reconcile pass

A pass reads the repository record, then acts on new rerun requests, then runs what is due for refs that moved, then for pulls (with --pulls), then works the merge queue (with --queue), then, at most once an hour, prunes the repository’s checks (see Pruning). If the record’s key has changed and no repository in the owner’s account has the old key, the repository was deleted: the pass clears the checks, secrets and caches it left and stops watching it. A discovered repository whose record no longer names the runner is left. Otherwise a changed key fails the pass: the label no longer names the repository the runner was watching.

A ref is due when its generation is past the one the runner last ran, or it is new to the runner, or its target matches a run dropped at a stop. A deleted ref never runs. Each due ref syncs the runner’s mirror, a local bare git repository kept in step with the PDS, once per pass. After a ref’s runs are filed, the runner records its generation in the state file.

The first time the runner watches a repository, the refs, open pulls and rerun requests already there are its starting point: it runs none of them. A repository with many branches and pulls would otherwise run them all at once, and so would every repository again whenever the state file is lost, a deploy workflow’s run on main included. knotless rerun runs any of them at once.

Details:

A watched repository that cannot be found stops the runner starting. Later, an error in one repository is logged and the others go on.
The default branch is the repository record’s defaultBranch, or main.

What a push runs

For a ref that moved, the runner checks out the new target and loads the workflows at that commit, so a change to a workflow is tested by the push that makes it. A workflow runs on a push when its on.push.refs matches the ref.

Every workflow’s check is filed before any run starts, as queued or already ended, so a reader never sees some of a commit’s checks passed while another is missing. A run waits queued for its turns, and is filed again as running, its startedAt then, once it has its job and is about to run its first step. A pull’s pre-flight that another runner the repository names has started or passed at the same head and base meanwhile is not run then: its check is deleted (merge-queue.md). A check is filed already ended for:

a workflow that does not parse, or two that share a name: error, whether or not it would have run;
a workflow whose paths match nothing that changed (below);
a run on the default branch that reuses a pull’s pass.

Each commit keeps its own checks. A push leaves the previous commit’s checks standing, and a rerun replaces only that commit’s.

Paths

A push or pull trigger may list paths globs, and the workflow runs only if a changed file matches one (syntax). For a push, the change set runs from where the runner’s mirror had the ref before this pass synced it to the new target. When that is unknown, as for a new ref, nothing is skipped.

A workflow left out is filed at once as success, every step skipped, with a note naming the paths. It takes no job and no cache, and the merge gate and deploys read it as passed. Runs asked for by name ignore paths.

What a pull runs

With --pulls, the runner checks each open pull into a watched repository at its head merged with the current tip of its base branch, as merging would leave it. A pull that breaks the base only beside another pull merged since then fails before it is merged.

The runner lists open pulls from the appview (KNOTLESS_APPVIEW), counting only states written by the owner or a collaborator. Drafts are among them: a draft is checked as any pull is, so its author sees whether it passes before marking it ready, though it cannot merge or be queued until then. It reads each head from its owner’s PDS and fetches it into the watched repository’s mirror, as knotless merge does. A pull whose head branch, head repository or base branch is gone is treated as closed.

If the head already contains the base, the runner checks the head itself. Otherwise it builds a merge commit with git merge-tree and git commit-tree, with a fixed author, committer and time. The same head and base always make the same commit, so the runner and knotless merge agree on what was checked without talking to each other. The workflows that run are those in that commit whose on.pull.base matches the base.

If the head conflicts with the base, there is no merge commit to load workflows from. The runner loads them at the head, and files each that runs on a pull into the base as failure without running it, naming the files and asking for a merge or rebase.

A pull’s check names the head branch as its ref, the head as its commit, the base tip as baseCommit, and the pull. KNOTLESS_RUN_SHA is the head too, not the merge commit checked out, and KNOTLESS_RUN_BASE the base tip, as it is a batch’s base in a batch’s run.

A repository named with --pulls-only runs only its pulls here. Its pushes and its merge queue run on another runner, one without --pulls that it also names. The pulls’ runs then load a machine of their own, and the runs that land and deploy stay where the deploy’s secrets are. That split is the operator’s; an owner makes one of their own by naming the runners a workflow runs on, in its runners, and giving its secrets to those alone (knotless secret set --runner). A runner not named files no check of it. A push to the default branch can take the pass of a batch from the runner that lands the queue, but not the pass of a pull run elsewhere. Since no push runs here, a trusted pull’s passing run writes the caches’ bases instead, so the pulls here start warm (see Where pushes run elsewhere). As another runner lands the queue, the pulls run here as on that one: a queued pull does not run on its own, and the rest run again only as their head moves, not their base.

The state file holds the head and base each open pull last ran at, so on a runner that does not land the queue, when the base moves, every open trusted pull into it runs again. The sections on scheduling and the merge queue bound what that costs. A pull that runs as it moves, trusted or an agent’s, seen again after a gap, such as on reopening, does not run if the runner already has checks of it at that head and base.

Whose code runs

A pull runs as it moves only if it is trusted: its author and its head repository’s owner both write to the repository, each being the owner or listed in the repository record’s collaborators. A merge-queue batch is trusted when every pull in it is.

An untrusted pull runs only when the owner asks, with knotless rerun --pull. It runs once, where it is at that moment, and until then knotless merge finds it not checked. A stranger cannot spend the runner’s time, or reach its host, without the owner looking first.

Between the two are the accounts the repository record names in agents, such as a coding agent pushing to its own fork. A pull whose author and head repository’s owner are each an agent or may push runs as it moves, so an agent hears from the checks without waiting for the owner, but it runs untrusted: caches apart for a workflow asking for secrets, nothing it writes kept, and so no pass a push can reuse. What an agent writes can be steered by whoever wrote what it works from, so a collaborator who set it going does not vouch for it. An account named as an agent counts as one even if it is a collaborator too, and a batch holding an agent’s pull is untrusted.

A pull that brings shared steps from an account the repository does not vouch for is untrusted the same way: its lock names an account the base’s lock does not, that is neither the owner nor a collaborator, or whose handle does not resolve. So is a batch holding it. Its log says so, naming the accounts.

Trust decides when code runs, not what it is given. Every pull and batch run gets no secrets, trusted or not, and only the docker executor runs them: --pulls and --queue refuse --executor shell.

The shell executor runs each step as the runner’s own user, who can read the secrets file and the runner’s session. Watch a repository under it only if you trust everyone who can push to it with the host.

Who gets a job first

All watched repositories share one pool of --jobs jobs (4). A seconds-long workflow never waits behind a minutes-long one in a queue of its own.

A run takes everything it needs at once, when all of it is free: a job, and any turn on a ref it must hold. It never holds one thing while waiting for another, so two runs never wait on each other.

Waiting runs are ranked by class, then by the order they were started. A run ranked earlier has first claim on what it waits for and on a free job. A later one goes ahead only with what nothing before it waits for, and a job left over. A running run is never stopped for one ranked before it, so main’s runs wait only for runs already going.

Main: pushes to the default branch, and the merge-queue batch that lands next
Other: pull runs, and pushes to other refs
Later: pulls not in the queue, where this runner lands the queue or runs the pulls alone, and a batch built ahead of one still checking
Asked: runs asked for by name, after every other waiting run, however old

The classes other than Main together hold at most --pull-jobs jobs, half of --jobs by default. A merge that reruns every open pull starts only that many, and main’s next run always has a job.

Within a class, the owners of the repositories take turns. Each run of an owner’s comes the turn after its last one still waiting, or, with none waiting, at the turn of the last run to start. A burst of one owner’s runs, such as every open pull run again, goes one at a time between other owners’ runs rather than all ahead of them, and one owner’s runs keep the order they were started in. --owner-jobs caps the jobs one owner’s runs hold at once, all of them by default. Past it, that owner’s next run waits while other owners’ runs take the jobs free.

--allowance <minutes> sets how many minutes of steps each owner’s runs may take in 30 days. Time is counted from the steps’ own durations, not from when a check was filed, since that includes the wait for a job. Past its allowance, an owner’s runs go one at a time, so they slow down but still finish. Each such run’s log begins by saying so, and how long it waited for a job. The runner’s own account is not held to it, nor is an owner named with --unmetered, and without the flag no one is. --private-allowance <minutes> does the same for private repositories’ runs, counted apart from public ones.

--paused <file> is the operator’s stop: a JSON object of owner DIDs, or * for every owner but the runner’s own account, each to why, such as {"did:plc:…": "mining, by the looks of it"}. The runner reads it again on every pass, so an edit takes effect at the next pass, with no restart. A paused owner’s runs are not run: every workflow at the commit, whether or not it would have run there, is filed error, saying paused by this runner's operator: <why>, never silently dropped. A file that cannot be read pauses no one; one that is not such an object fails the pass, and is logged.

The runner files each owner’s public minutes as a dev.knotless.minutes record in its own account, keyed by the owner’s DID: the minutes in the last 30 days, the allowance, and when minutes counted now stop counting. It writes the record again as the owner’s runs end, at most every five minutes, or at once when the owner reaches 80% of the allowance or passes it. Private repositories’ minutes are never in it, since the record is public and would show that private repositories exist and how busy they are.

The same split reaches the CPU. Under docker, every run gets --cpu-shares 256, a weight of about 10 against the default 100, except pushes to the default branch and the batch that lands next. A burst of pull runs leaves the site, deploys and main’s runs the CPU they need.

When a newer commit arrives

By default a workflow is superseded: cancel. A newer commit’s run of it on the same ref or pull cancels the older run, waiting or running, since the older result no longer matters. Cancelling kills the running step and its process group, skips the rest, and files the check as cancelled. A run cancelled while waiting is filed cancelled without starting.

A full rerun of the same commit leaves the older run going, but a run asked for by name cancels the previous run of that workflow at the same commit, as both would file one check. Runs on different refs are left to finish.

Deploys: `superseded: finish`

A workflow that deploys can say superseded: finish. Such a run takes the workflow’s turn on the ref or pull as soon as it is filed, if no run holds it. From then on it is kept: a newer commit’s run does not cancel it. The newer run waits, as queued, for the turn, and a third commit’s run cancels the one waiting, so at most one waits. Deploys go out in order, and the newest commit is the next to go.

The turn is held until the check is filed as ended, so the next deploy files after it.

A kept run is still cancelled when its result can no longer mean anything or its wait would be unbounded: its pull moves to another head or base, closes, or loses a branch; its merge-queue batch is dropped; or a stop runs out of time. A later cancel never overwrites one of these reasons.

Stopping and starting again

A stop

SIGTERM or SIGINT stops the runner, and is heard even during startup. A stop:

waits for any pass that is filing or starting runs, and stops reading events;
cancels runs asked for by name, recording them to run again on start;
waits up to --stop-wait seconds (900) for every run to end, landing each merge-queue batch whose runs have then all passed (Dropped runs);
cancels every run still going, and waits for those to end.

A stop never waits forever on a hung step. One did once, when a deploy’s docker compose up failed and nothing came to kill the runner.

Since a stop costs the runs still waiting, and can cut off a merge-queue batch still being checked, which is then checked again, knotless.dev’s deploys stop the runner only when what it runs changed: the sources it is built from, or its settings. Each deploy rebuilds the image, so the image alone cannot tell; when unsure, it restarts the runner (When a deploy restarts the runner).

A run that has not started its steps when the stop comes is dropped: its check is filed as cancelled, and the state file records it. The exception is a run at the same commit and base as a running superseded: finish run, since a deploy may wait for its own commit’s other checks. It is kept until that run ends, and dropped then if still not started.

On the next start, a dropped run runs again for only those workflows, and only if its ref or pull is still where it was. If it moved on, the newer commit runs all its workflows. A pull whose base alone moved runs all its workflows on the base as it is then: a runner landing the queue runs a pull again only as its head moves, so it would otherwise not run at all. A dropped batch run is not reported to the queue as a failure.

A start

It reads each repository its arguments name first. One it cannot read, gone or private to it, is logged (failed, with the reason) and passed over until the next start, so one bad name does not keep it from the rest. Only when it can read none of them, and has none to find without --discover, does it stop, with that reason.

Before any run, the runner ends as error every check of a watched repository that its account left queued or running, saying the runner stopped before the run started or finished. knotless rerun runs it again.

It also removes leftover containers: those labelled with its DID, and, with --work, those mounting a path under the work directory. It does this again every --sweep, sparing the containers of its own runs. One was once left running for seven hours. It never touches other containers.

The state file

The state file (--state) is JSON, written atomically. A file that cannot be read, or is corrupt, counts as empty, and every repository is then watched as if for the first time. Each field is keyed by repository:

Field	Holds
`repos`	each ref’s generation last run
`pulls`	each open pull’s head and base last run at, or passed over at
`reruns`	the newest rerun request acted on
`asked`	pulls asked for and not yet run
`asked_only`	of those, the ones asked for only some workflows, by name
`passed`	trusted pull and batch runs that passed, for reuse, at most 256
`dropped`	the workflows dropped at a stop, per ref, pull or queue, and commit and base
`wanted`	the workflows asked for by name and not yet run, per ref, with their variables
`used`	the milliseconds each owner’s public repositories’ runs’ steps took, per day, for the last 30 days
`used_private`	the same for private repositories

Two more are the runner’s own, not by repository: discovered, the repositories found naming the runner, watched again at a start before the appview is asked, and invited, where the reading of invites ended.

Where a step runs

Each run gets a temporary directory under --work, holding a clone of the mirror at the commit, the log, and an outputs/ directory, all removed afterwards. The clone hard-links objects rather than using alternates, so git works inside a container that sees only the checkout.

Under the docker executor, the command line’s default (the library’s Options::new defaults to shell), each run gets one container from the workflow’s image, or --image. The checkout is mounted at /work. Each step is a docker exec of sh -ec <script> in that container, so a step finds what earlier steps left, installed tools included. Variable values reach the container through the docker client’s environment, never its arguments. Secrets are not among them: each step’s docker exec is given the secrets that step may see, the same way, so the container never holds one and a step cannot read another’s from /proc/1/environ. Starting the container, image pull included, writes to the run’s log and is limited to --step-timeout. A container that does not start makes the check error.

Under shell, each step is sh -ec <script> in the checkout. When the run ends, the container is removed, which kills anything a step left behind, and then its cache overlays. The job is released once the container is gone, before the log and check are written, so a slow PDS does not hold a job.

Steps for some refs

A step may name on: {refs: [...]}, matched as on.push.refs are, against the ref a push moved or a run asked for by name is at (reference). A step it does not match is not run: the log says why under its ==> <step> line, and its check lists it skipped, as a step after a failure is, the run still passing. On a shared step, its on holds for each step it comes to, beside their own.

A pull’s run and a merge-queue batch’s run match no ref, whatever KNOTLESS_RUN_REF holds. A pull’s holds its head branch, which a fork may call main, and a batch’s holds its base branch, which has not moved yet: matching either would run a step meant for the default branch before anything landed there. So a step for refs/heads/main runs only at the push that lands the change. That is why a run that left a step out is never kept for that push to reuse (below).

Time limits

--step-timeout (1800 seconds) limits each step. A step’s own timeout can only shorten it, except in a run asked for by name, where it may exceed it. --run-timeout, unset by default, limits a run’s steps together: each step’s limit is the shorter of its own and what is left of the run’s, and a step killed because the run’s time is up says so in the log (killed as the run's <n>s were up), the steps after it skipped. A run asked for by name is not held to it, as it may outrun --step-timeout.

What a run may take

Under docker, --cpus, --memory (whole GiB, no swap beyond it) and --pids limit each run’s container, and --network names the network it joins. All are unset by default, leaving a run as much of the host as it can get, as before they existed; a runner checking code it does not trust sets them all, so no run takes the machine from the others or from what else the host serves. They apply to every run alike, and come on top of the lower CPU weight runs other than a push to the default branch already get. The network is made, if missing, as a bridge with traffic between its containers turned off (com.docker.network.bridge.enable_icc=false), so a run reaches neither another run nor the containers on the host’s other networks, which docker keeps apart from each other; one there already is used as it is, and one another run’s start makes meanwhile is as good. A network that cannot be made fails the run’s start, an error check, its overlays removed. It does not stop a run reaching the host itself, or anything else the host can reach.

--runtime names the container runtime runs start with, as docker’s own --runtime does: runsc runs them under gVisor, whose kernel takes a run’s system calls instead of the host’s. The runner host for other people’s code runs every run so, with the limits above and egress rules on its network that keep runs off the host and the private ranges: see the runner host.

--trusted-runtime names another, for code an owner --trust names vouches for: a push to their repository, or a pull or merge-queue batch every writer of which they vouch for, as whose code runs reckons it. gVisor costs such a run two to five times its time, the tests above all, and buys nothing for it: once the pull is queued, its batch runs the same code on the owner’s own runner with no sandbox at all. Anyone else’s code, a pull its owner asked to run among it, keeps --runtime.

--resolv names a file on the docker host that runs see as their /etc/resolv.conf, mounted read-only. On a network docker made, a container’s resolver is docker’s own at 127.0.0.11, reached through NAT rules in the container’s network namespace. gVisor’s network stack does not apply them, so under runsc a run resolves nothing until it names resolvers of its own, as the runner host’s file does.

A timeout or a cancel sends SIGKILL to the step’s process group through a system call, not a kill command, since the runner’s own image has none. Without that, a cancelled or timed-out step ran on, and a hung one held its job and a stop for good. Under docker the process killed is the docker exec client, and what it started goes with the container, which is removed right after.

A check is cancelled if a step was cancelled, else failure if a step failed, else success. An error of the runner’s own makes it error.

Environment and outputs

A step sees the workflow’s env, the secrets granted to the run, and the runner’s KNOTLESS_RUN_* variables. Under shell it also sees a few of the runner’s own variables, such as PATH and HOME. Under docker those reach only the docker client, and the container keeps its image’s environment. The workflow reference lists them.

After the steps, passed or failed, the images a run left in KNOTLESS_RUN_OUTPUTS become the check’s outputs, as blobs (binary data stored beside records). A PDS can take seconds over each blob, so the runner uploads four at a time, and files with the same bytes once, as one blob. outputs is an open union, so readers pass over kinds they do not know.

Caches

A workflow’s cache lists paths in the container to keep between runs. Caches exist only under docker. They have to be fast, and must not carry anything across trust levels.

Each cache path has a base, a read-only directory per repository, workflow and path. A run never writes it. The run sees it through an overlay of its own, an overlayfs mount whose upper layer takes the run’s writes. Runs never wait on each other for a cache, and two builds never share output. A new cache starts as an empty base, and the image’s own files at that path show through only where the base has nothing.

Only a passing push to the default branch promotes, or, on a repository whose pushes run on another runner, a trusted pull’s passing run (below). Each non-empty upper layer is applied over a hard-link clone of its base, as the base’s next generation, and runs started after that start from it. The apply follows overlayfs’s markings: a whiteout deletes, a directory not marked opaque merges, and anything else replaces. Files are moved, not copied.

A base is never changed while a run reads it. An old generation goes once no run reads it and it is not current. A new base over --cache-max (16 GiB) is not made current, and the next run starts afresh.

Across trust levels

No untrusted pull, nor batch, promotes. If an untrusted pull, or a batch holding one, runs a workflow that asks for secrets, it reads its caches apart: from bases nothing writes, so always empty, because a push’s run with secrets may have left things in the shared base. An untrusted run’s layers are never kept, so its writes reach no base.

Every other run’s upper layer is removed when the run ends, with one exception. A trusted pull’s or batch’s passing run of a workflow that could be reused keeps its upper layers, so the push that lands it takes them as if it had made them.

Kept layers go after 3 days, beyond 16 runs, or beyond --cache-max in total, oldest first. They are also forgotten once taken, once their pull moves or closes, or, for a batch, once the next batch on the branch passes. A batch running ahead does not displace the kept passes of the batch it runs ahead of until that one lands. A push takes only the layers written over a base that is still current, and the log says the next run warms the rest.

On start, the runner removes its leftover overlays and half-made bases, and sweeps no old base while an old overlay is still mounted.

Room on the disk

The caches share their disk with whatever else the host runs. On knotless.dev’s own host that includes the site’s repositories and index. On 2026-10-31 a full disk took the site down. So the runner keeps --disk-floor free on the disk --caches is on: 20 GiB by default. That is about two of the largest bases the host has held, so one run’s build can land on the floor without the site running out.

Before a run starts, holding its job so no other run starts meanwhile, the runner checks the free space. When it is below the floor, the runner sheds first: every kept run’s layers, and every cache’s base no run reads. Those caches’ next runs start afresh, as after a dropped base. If that is not enough, the run waits, checking again every 30 seconds, until the runs going end and free their layers. It stays queued, and can be cancelled as usual. Its log says what was shed and how long it waited. The runner logs disk short at warn each time it looks and finds the disk short. A disk it cannot read holds nothing back.

The floor holds only at a run’s start. A run already going can still fill the disk.

Where pushes run elsewhere

A repository named with --pulls-only has its pushes run by another runner, with caches of its own, so here no base would ever be written and no kept layer taken: every pull would start cold. Instead a trusted pull’s passing run into the default branch, of a workflow that could be reused, promotes its layers itself, as a push’s run would, and keeps none. Two rules keep the base from going backwards:

A layer is promoted only over the base it was written over, if that is still current, checked as the new base is made current. Of two pulls’ runs over one base, the first to pass makes the next one, and the other’s layers go.
A base a pull made is marked with the default branch’s generation at the tip the pull merged with (marks.json under the caches). A run merged with an older tip than the mark does not promote, so the base never goes back to an older main. A base a push made has no mark.

The pull’s log says which of its paths it made the base and which it did not. Pulls into another branch, untrusted pulls (an agent’s among them) and workflows that ask for secrets or are superseded: finish never promote.

This keeps the rule above that no untrusted run’s writes reach a base: only the owner’s or a collaborator’s pull promotes, code that may be pushed to the default branch anyway. What changes is that it reaches the base before it is merged. That is safe because on a --pulls-only runner the base is read only by pulls, which get no secrets, and no workflow that asks for secrets promotes. The worst a bad base can do is make a later pull’s check here wrong; the queue checks each batch again where pushes run, from caches no pull wrote, and a deploy waits for main’s own checks there.

Reusing a pull’s pass at the push

When a trusted pull merges, the merge commit often has exactly the tree the pull was checked at, and running the same workflows again would test nothing new. So a push to the default branch whose tree equals the tree of a trusted pull’s passing run into that branch, or a trusted batch’s, takes that run’s result. The check is filed success with the earlier steps and outputs, and its log names the run it took, then copies that run’s log, so it still reads whole once the earlier check is pruned. The tree holds the workflows, so they were the same workflows.

A pass is not reused when:

the workflow asks for secrets, since the pull never had them;
the workflow is superseded: finish, since a deploy runs for what it does, not what it checks;
the earlier check is no longer success;
the run was asked for by name;
the run left a step out for its ref (on), which the push, at another ref, may run.

A batch can likewise take a trusted pull’s pass at the same tree. Passes stay in the state file after their cache layers are forgotten, so a check can still be reused without its caches.

Secrets

Secrets come from two places. The file named by --secrets, keyed by the repository’s label exactly as on the command line (format), is the operator’s. The owner’s are those knotless secret sealed to the runner’s key and left at the appview (The runner’s key): for a name the workflow lists and the file does not grant, the runner asks the appview (dev.knotless.listSecrets, with its own service token) for those sealed to it for the repository, by owner and key, and opens each with its keys, rebuilding what it was bound to from where it found it. One that does not open, or whose refs do not reach the ref, is not given. The file wins a name both grant, since an operator who grants one means it. An owner can give the hosted runner a secret without handing it to whoever runs it. An appview that does not answer makes the check error, as an unreadable file does. A runner that logs in as the repository’s owner is listed every secret, as owners are, with the values of those sealed to it, so it opens its own as any runner does.

A secret reaches a step only if the workflow lists it, the file or the owner grants it to the repository and, when the secret names refs, to the ref, and the run is not a pull’s or a batch’s. A secret limited to refs/heads/main cannot be read from another branch, which matters because a branch can change the workflow itself. A secret not granted is absent from the step’s environment, and the log names it before the steps (==> NAME is not granted to <repo> at <ref>), so a workflow that fails for want of one says why. The line names the secret, never a value.

A step put in place by a shared step sees only the secrets its uses step passes it, by name, of those the step file asks for. Every other step sees every secret the run is granted. In a workflow that uses shared steps, what a step given secrets leaves running is ended once the step ends, so that no process holds them for a later step to read.

The runner re-reads the file on every run that needs secrets, so a change takes effect without a restart. A workflow that asks for none never reads it. A file that cannot be read makes the check error.

Before the log is uploaded, each granted value is replaced by ***, and so is each trimmed line of a multi-line value such as a PEM key. The longest values go first, so a secret inside another leaves nothing behind. Values under 4 bytes are left alone, since masking them would mask ordinary output. Masking catches a secret only as written. A step can still leak one transformed, encoded, or a bit at a time.

The runner’s key

Owners will give a runner secrets without the operator’s hands: each sealed to the runner, so whatever holds it on the way holds only ciphertext. For that the runner keeps a P-256 key, made on its first start, in the file --secrets-key names (secrets-key.json beside the state file by default, readable by its owner alone), and publishes its public half as its dev.knotless.runnerKey record, as a did:key. It writes the record again only when the key differs.

A value is sealed with HPKE (RFC 9180) in base mode: DHKEM(P-256, HKDF-SHA256), HKDF-SHA256 and AES-128-GCM, as GitHub seals a repository’s Actions secrets to a key of its own. What it is for is bound to it as the AEAD’s associated data: the owner’s DID, the repository’s key (its record key, so a rename keeps its secrets and a repository made later under the old name gets none), the runner’s DID, the secret’s name and the refs it is limited to. The runner rebuilds those from where it found the value, so one copied to another repository, name, runner or refs does not open: nobody can carry an owner’s sealed token into a repository of their own to print it. Base mode does not say who sealed a value, so whoever holds the sealed values could swap one in, though never read one.

The file holds a list of keys, the newest first: the runner publishes the newest and opens what was sealed to any of them, each value naming the key it was sealed to, so a key can be replaced while values sealed to the old one are set again.

Check records

Checks are dev.knotless.check records in the runner’s account, written with putRecord, so a run replaces its own earlier record. The record key says what was checked:

Subject	Record key
a push	`<h(owner did)>.<repo key>.<h(full ref)>.<commit>.<workflow>`
a pull	`<h(owner did)>.<repo key>.<h(pull AT-URI)>.<head>.<base>.<workflow>`
a merge-queue batch	`<h(owner did)>.<repo key>.<h("queue refs/heads/<branch>")>.<tip>.<base>.<workflow>`

h is the first 26 characters of the lowercase base32 SHA-256. Record keys sort as strings, so a prefix can be listed with one listRecords scan: all of one runner’s checks of a repository share <h(owner did)>.<repo key>, and a pull’s, a queue’s or a ref’s pushes share a narrower prefix, as do a ref’s checks of one commit: knotless checks --ref scans only those, as deploy.yml’s wait does. The owner’s hash keeps one owner from naming another’s repository with a key of their own. A pull run against a newer base is a check of its own.

The record’s fields are in the dev.knotless.check lexicon. Two points it understates: baseCommit is set for a batch’s run as well as a pull’s, and cancelled covers every reason a run is cancelled, not only a newer commit.

The runner retries a write while the PDS fails with a 5xx, a 429 or no status at all, three times, waiting 0.5, 1 and 2 seconds. A write given up would leave a check that never ends, or, for the log uploaded as a run ends, file a run that passed as an error. So a 429 that says when its rate limit lets writes in again (RateLimit-Reset) is waited out until then instead, for up to ten minutes in all for the one write, while the run keeps its job.

Output as it is printed

A check’s record is written as the run starts and as it ends, and its log is a blob uploaded at the end, so nothing between shows in the record. With an appview (KNOTLESS_APPVIEW), the runner also sends it what the run prints as it prints it (dev.knotless.putRunOutput), for the check’s page to show while the run goes (serve). It reads the log file as the steps write it, every second while the appview says a page follows the run and every five seconds otherwise, and at once as a step starts or ends, with the steps ended so far, the name of the one going and when the run started, on every send, so a page names the step going even when the appview never had its ==> line (it started before the appview did, or before the runner could reach it). Only whole lines go (a line ends at a newline or a carriage return), each masked with the run’s secrets as the log is (see Secrets), so a secret printed in pieces is never sent split where masking would miss it; the masked text is then cut into pieces of at most 16,000 bytes. A line still unended past a megabyte is let go unsent. A send that fails stops that round, and what it held goes again in the next, so a page sees what the run printed while the appview was out of reach; past a megabyte held, as much as the appview keeps, the oldest is let go. The last send, with the rest of the log and saying the run ended, goes before the check is filed, given five seconds at most, and is not sent again if it fails: the log is what is kept, and a run never waits on the appview. A private repository’s runs are sent too: the appview shows them only to its members, as it does their logs. A runner logged in with OAuth before these methods were in knotless’s scope gets no token for them until it logs in again: until then its runs’ output is not sent, and its private repositories’ spaces are asked every --poll seconds, as with no appview.

Pruning

Nothing else deletes a check, and most are soon of no use: on 2026-10-25 our account held 4,041 checks of knotless, three quarters of them pull runs of heads or bases since moved, and batches. So the runner deletes its own, at the end of a pass, at most once an hour per repository, in applyWrites of 200 deletes. A check still running, or finished less than a day ago, stays. Of the rest, it deletes:

a merge-queue batch’s run;
a pull’s run, once the pull is closed or merged, or a newer run of the same pull and workflow was filed;
a run on a ref other than the default branch, once the ref is gone or a newer run of the same ref and workflow was filed;
a run on the default branch older than --keep-checks days (30), unless it is the newest of its workflow there.

A push that takes a pull’s or a batch’s pass copies that run’s log (see Reusing a pull’s pass), so the default branch’s checks read whole once the run they took is gone. Blobs go with their records: the PDS reaps a blob no record references. --keep-checks 0 prunes nothing, and neither does a runner with no appview (KNOTLESS_APPVIEW set empty) for a public repository, whose open pulls it cannot then know.

Whose checks count

Anyone can write a check record about any repository, so the repository record lists, in runners, the accounts whose checks readers trust, and readers show only theirs. Readers read runners when they show checks, so a runner added later shows what it reported before. The owner sets the list with knotless runners.

Private repositories

A check says which commits a repository has, what its workflows are called and what their steps printed, so a private repository’s checks must be as private as its code. The runner reads a private repository in its space, as its own account, a member there, and files each check with the space’s putRecord into its own repo in the space. Logs and images are blobs those records reference, which only members can read. knotless create --private, for the runners it names, and knotless runners <url> add make the runner a member, as reading the space and writing to it both need; remove leaves it one, since it may be a member for itself too.

The appview indexes no private repository, and Jetstream carries nothing written in a space, so a runner would never find one by itself. Both commands also leave the runner an invite at the appview (dev.knotless.inviteRunner, with the owner’s service token), and a runner with --discover reads its own (dev.knotless.listRunnerInvites, with its token) every --sweep seconds, keeping where the last page ended. An invite only says where to look: anyone can leave one, and the runner watches the repository only if it can read its record in the space, as a member, and the record names it. An owner whose invite could not be left, with no appview or a login from before the methods were in knotless’s scope, is warned (<did> was not told of it: …); knotless runners <url> add invites it again once they log in again, and naming the repository on the runner’s command line works as ever.

Readers read a private repository’s checks from each runner’s repo in the space, with a space credential, which even the owner needs to read another member’s repo there. Reruns need nothing new: the owner writes them where all of their records about the repository are, in the space.

Nothing written in a space reaches Jetstream. The runner asks the space’s authority instead, every --poll seconds, with com.atproto.space.listRepos from where it last looked: the members’ repos written since, if any, whatever was written. One that answers rings the door, and the pass reads what moved as it reads a public repository’s. Reading the space takes a credential, so a runner of a private repository logs in with OAuth.

Asking every space every ten seconds is slow to hear of a push and costs a request per space whether anything was written or not. So, with an appview, the runner also asks each space’s authority, as a member, to tell the appview of every write there (com.atproto.space.registerNotify, naming the appview’s #knotless_appview service), every twelve hours, as the authority keeps the ask for a day. The appview passes each write on to the runners the space’s owner invited (dev.knotless.awaitSpaceWrites): the runner asks from where it last heard, and is answered as soon as one of its spaces is written to, or with none after 25 seconds. A write passed on rings that repository’s door; an answer saying the runner may have missed some, as the first does and one after the appview restarted, rings every private repository’s. The runner registers rather than being told directly because a service a space’s authority calls needs a DID document and an address it can reach, which a runner on a laptop or behind NAT has not; the appview has both. A word passed on is only a reason to look, as an invite is.

While the appview answers, each space is asked itself only once a minute, for a write the appview missed. With no appview, or one that does not answer, it is asked every --poll seconds as before, and so is one whose owner invited no runner (one named only on the runner’s command line).

A private repository that is deleted, or no longer names the runner, is left, and its space is asked no more: one runner asked a deleted one every few seconds for hours.

A repository found deleted, its owner holding no repository record with its key (one renamed still does), is no longer watched, and the runner clears what it kept of it: the checks it filed of it, with their logs and outputs, the grants the secrets file gives its label, so that a repository made under that name later does not inherit them, and its caches’ bases. A cache is named for its repository, workflow and path, each hashed, so a repository’s are found by name; one named as before that, by a single hash, is removed at start.

Reruns and runs by name

The owner asks for a run with a dev.knotless.rerun record in their own account, keyed by the repository’s key and a TID, written by knotless rerun, the MCP rerun tool, or the pages: Re-run on a pull’s checks and on a check’s page, and Run workflow on the checks page, which names a ref, one of the workflows knotless workflows says may be asked for there, and its variables. On every pass the runner reads the owner’s rerun records from the owner’s PDS. Only the owner’s count, and a forged frame cannot fake one. The appview does not index them.

A request does one of four things:

Naming a ref: the runner forgets the ref’s last generation and dropped runs, so the next pass runs all its workflows at its current target.
Naming a ref and a workflow: the next pass runs that one workflow at the ref, even if the ref did not move.
Naming a pull: the runner forgets where the pull last ran, and runs all its workflows where it is now, even if it is queued or untrusted. A pull run dropped at a stop runs again on the same terms.
Naming a pull and a workflow: the next pass runs only that workflow where the pull is, as a run dropped at a stop runs only what was dropped, leaving its other checks standing. It is for one check that failed alone, or that reads the pull’s body, such as a screenshot check reading which pages may move. A pull asked for in full, or moved, runs all of its workflows anyway. Variables go only to a run at a ref.

A request naming neither a ref nor a pull does nothing. Without --pulls, requests for pulls are recorded but not acted on.

By name

A run asked for by name lets a costly workflow, such as a long scenario suite, run only when someone wants it. A workflow may run by name at a ref if it runs on a push to that ref or its on.asked.refs matches. A workflow with only on.asked runs in no other way. Asking for one that is not there, or may not run at that ref, files an error check saying so.

The request may carry variables, which replace the workflow’s env entries of the same name. The runner drops any named as one of the workflow’s secrets, and writes the variables at the top of the public log. These runs rank last, may exceed --step-timeout with a step’s own timeout, and are cancelled at once by a stop and run again on the next start. Workflows wanted at a ref that no longer exists are forgotten.

One whose on.asked names commit globs writes back. Once it passes, the runner reads git status in its checkout, commits the files changed among those globs on top of the commit it ran at, fetches the commit into its mirror and pushes it to the branch, as with a landing. It refuses the default branch, a protected one, and a branch that moved while it ran, failing the check with not written back: <why>. See Writing back.

The merge gate

In a repository that names runners, knotless merge and the MCP merge tool merge a pull only once each workflow that runs on it has a success check at its head and its base as it is now. The workflows are those of the head merged with the base, plus any that fail to load, and the commit is the head for a fast-forward, else the same merge commit the runner checks.

For each workflow the newest check at that head and base decides. A status other than success, queued, running or cancelled fails the gate, and that wins. Otherwise a queued or running check means the checks are still running. Otherwise a missing or cancelled check means the pull is not checked yet. Otherwise it passes, including a success skipped by paths.

With --wait, merge polls while checks are running or missing. When the owner waits on a pull that is not checked, merge writes one rerun request for it, since a runner started after the pull last moved has not run it. --force, the owner’s alone, merges whatever the checks say. A repository naming no runners merges without a gate.

The gateway, through which collaborators push to the owner’s account, holds a collaborator’s merge to the gate as well, reading the pull’s checks as merge --check does when the push arrives: a push whose commit ends with a pull’s AT-URI, as a merge’s does, and is that pull’s merge, must be one that may merge (gateway.md). A branch whose rule asks for reviews holds a merge to those too, read before the checks (required reviews); the runner landing the queue ejects a queued pull that lacks them before it goes into a batch. knotless merge --check, MCP merge with check, and a pull page’s merge box read the same gate without merging: passed, failed, running, or pending for a workflow with no check there yet.

The merge queue

The merge queue keeps a merge from retesting every open pull: queued pulls are checked together, as a batch, and land together. A runner lands a repository’s queue only with --queue, and only if its own account writes to the repository, or the repository names it and no runner that does: then it lands through the gateway (merge-queue.md). merge-queue.md has the design.

The runner stacks up to --batch queued pulls on the branch tip and runs the batch once at its tip, in the Main class, without secrets. If every run passes, it lands the batch in one push, which refuses if the branch is anywhere but where the batch was built. If not, it halves the batch to find the pull that broke it. With --depth 2, the default, it runs the next batch ahead, built on the one being checked, in the Later class. A pull goes into a batch only once its pre-flights, the workflows that say queue: first, have passed at its head; with none passed or running, the runner runs them on the pull itself, as a pull’s run.

While a runner lands a repository’s queue, or runs its pulls alone with --pulls-only, a queued pull does not run on its own: its batch is its run. Pulls not in the queue rank Later and run again only when their head moves, not their base. A trusted batch’s passes are kept like a trusted pull’s, so the push that lands it reuses them and their cache layers.