Workflow files

What knotless-run reads from a repository, and what it gives each step. Workflows shows how to use them.

Files

Workflows are .knotless/workflows/*.yml, *.yaml or *.json, read at the commit being run, in file name order. Only files directly in that directory count, and a commit without it has no workflows. Keys not listed here are refused, and on is a plain key (YAML 1.2), not true.

A file that does not parse is filed as an error check with the parse message, under its name, whether or not it would have run. Two files with the same name both fail to run, and one error check says more than one workflow is named <name>, <file> among them. A name is 1 to 64 letters, digits, _ or -, since it ends the check’s record key; a file whose name is not is filed under the name invalid.

name: ci                      # defaults to the file name
on:
  push:
    refs: [refs/heads/main, refs/tags/v*]   # default: refs/heads/*
    paths: [src/**, Cargo.*]  # default: any change
  pull:
    base: [refs/heads/main]   # default: refs/heads/*; only with --pulls
    paths: [src/**, Cargo.*]  # default: any change
  asked:
    refs: [refs/heads/main]   # default: refs/heads/*
    commit: [shots/**]        # default: none; files written back
image: rust:1                 # docker executor only; default: --image
cache: [target, /usr/local/cargo/registry]   # docker executor only
env: { RUSTFLAGS: -Dwarnings }
secrets: [DEPLOY_TOKEN]
runners: [deploy.alice.example.com]   # default: every runner named
superseded: cancel            # or finish
queue: batch                  # or first
steps:
  - name: test
    run: cargo test
    timeout: 30               # minutes
  - name: deploy
    run: ./deploy.sh
    on: {refs: [refs/heads/main, refs/tags/v*]}   # default: wherever the workflow runs

name

Default the file name, without its extension

Meaning the workflow’s name, in its checks

on

Default push to refs/heads/*

Meaning the triggers below; with no on, every push to a branch, and with on: {}, nothing

image

Default the runner’s --image (debian:stable-slim)

Meaning the container image the steps run in

cache

Default none

Meaning paths kept from run to run; relative ones are in the checkout, under /work; see Caches

env

Default none

Meaning variables every step sees

secrets

Default none

Meaning the names of secrets the steps need, given only where the runner’s secrets file or the owner grants them (Secrets file)

runners

Default every runner the repository names

Meaning the runners that run it, by DID or handle, of those the repository names; any other files no check of it, so a workflow naming none of them never runs, and a pull waiting on it never merges

superseded

Default cancel

Meaning what a newer commit’s run on the same ref or pull does to this one: cancel it, or, with finish, wait for it once it holds the workflow’s turn, which it takes as soon as it is filed if no run holds it; anything else fails to parse

queue

Default batch

Meaning where it runs in the merge queue, if it runs on pulls: batch, on each batch, or first, on each batch and first on each pull queued, which goes into a batch only once it passes; anything else fails to parse

steps

Default required

Meaning the steps, run in order until one fails

Triggers

Trigger	Runs on	Keys
`push`	a ref moving	`refs`, `paths`
`pull`	an open pull’s head merged with its base, under a runner started with `--pulls`	`base` (the branches the pull merges into), `paths`
`asked`	the owner asking for the workflow by name with `knotless rerun --workflow`	`refs`, `commit`

refs and base are full ref names. Each matches exactly or, if it ends in *, as a prefix. That is not a glob: refs/heads/* matches refs/heads/a/b, and refs/tags/v* matches refs/tags/v1.2. The owner can ask by name for a workflow at any ref its push or asked trigger names. A workflow with only asked runs only when asked for, which suits one too costly for every push.

Writing back

asked.commit names globs, matched as paths are, of files a run asked for by name may write back. Once such a run passes, the runner commits what its steps changed among them, added, changed or deleted, onto the branch it ran at, and pushes that commit as its own account, author knotless-run. Nothing else the steps changed goes in. It writes only to a branch that is neither the repository’s default branch nor protected, only from the commit the run was at, and only if its account may push there. A run that cannot write back fails, saying why.

This makes files that must come from one place, such as screenshots taken in the image they are checked in:

on: {asked: {refs: [refs/heads/*], commit: [shots/linux/**]}}

Asked for at a pull’s branch, the commit moves the pull’s head, so its checks run again on what was written.

Paths

paths are globs matched against a changed file’s whole path from the repository’s root:

* matches any run of characters within one part of the path, between /s.
? matches one character within a part.
**, as a part of its own, matches any number of parts, none included.
Anything else matches itself.

So web/** is everything under web, web included; *.md matches only at the root, and **/*.md anywhere. A workflow runs if any glob matches any changed file. What counts as changed:

for a pull, what its head merged with its base changes from the base’s tip;
for a push, git diff --name-only --no-renames from where the runner’s mirror had the ref to the commit it moved to, so a rename is a delete and an add;
for a new ref, a ref only asked for by name, a ref another pass already synced, a conflicting pull, or a git error, everything.

A workflow no changed file matches is filed at once as success, with its steps skipped and the note skipped: nothing changed in <paths>, cut to 1000 characters. A run asked for by name is never skipped this way. A merge-queue batch picks its workflows by pull.base but skips by the push trigger’s paths.

The merge queue

A merge queue’s batch runs every workflow whose pull.base names its branch. One that also says queue: first is a pre-flight: a pull queued goes into a batch only once its run of it has passed at the pull’s head, against any tip of the branch, as any runner the repository names filed it. Until then it waits, and those queued behind it that are ready go ahead. A run of it that fails against the branch’s tip ejects the pull without a batch. With none passed or running, the runner landing the queue runs it on the pull itself.

# check.yml: fmt and clippy, minutes where the tests take many more.
queue: first
on: {pull: {base: [refs/heads/main]}}

Make a pre-flight quick and likely to catch what would fail a batch, such as a build: each pull that fails it in a batch costs a halving of the batch to find. The pull’s run of it carries queue: first in its check record, which is how readers tell that a pull is waiting for one.

Steps

name: the step’s name, in its check; default step <n>, counting from 1, or for a step with uses, the shared step’s name
run: a shell script, run with sh -ec, so any command in it that fails fails the step; a step has this or uses
uses: a shared step, <owner>/<repo>/<step>@<version>, put in this step’s place
with: with uses only: the shared step’s inputs, by name
secrets: with uses only: the workflow’s secrets the shared step is given, as a list of names, each passed under its own, or a map from the name the step file asks for to the workflow’s ({SSH_KEY: DEPLOY_SSH_KEY}); none if absent
timeout: minutes; can only shorten the runner’s --step-timeout, except in a run asked for by name, where it may go past it
on: {refs: [...]}, the refs the step runs at; see Steps for some refs

Under the docker executor the checkout is mounted at /work, and every step runs in one container, so a step sees what earlier steps installed. The image needs sh and tail. Under shell, a step runs in the checkout as the runner’s user.

A step passes on exit 0. One killed, for time or by a cancel, has no exit code in its check; one ended by a signal otherwise has -1.

Steps for some refs

A step with on runs only at the refs its refs names, full ref names matched as on.push.refs are: exactly, or as a prefix before a final *. refs is required and may not be empty, and on takes no other key.

The ref matched is the one a push moved, or the one a run asked for by name is at. A pull’s run and a merge-queue batch’s run are at no ref of the repository’s, so a step with on runs in neither, whatever KNOTLESS_RUN_REF holds there (a pull’s head branch, which may be a fork’s main; a batch’s base branch, not moved yet).

A step left out is not run: its check lists it as skipped, it counts as passed, and the log says why under its ==> <step name> line:

==> skipped, as refs/heads/x is not refs/heads/main or refs/tags/v*
==> skipped, as a pull's run is at no ref of the repository's
==> skipped, as a merge queue batch's run is at no ref until it lands

A run that left a step out is not taken by the push that lands its pull or batch, as other runs that pass are (merge queue): the push runs the workflow itself, with the steps for its ref.

On a step with uses, on holds for every step the shared step comes to, and a step in the step file may name its own on too: it runs only where both match, and so on down nested shared steps.

steps:
  - uses: alice.example.com/ci/publish@1
    on: {refs: [refs/heads/main]}

Shared steps

A step with uses takes the steps of a file in another repository, .knotless/steps/<step>.yml in knotless://<owner>/<repo>, where <owner> is a handle or a DID. Why it works this way is in Shared steps.

# alice.example.com/steps: .knotless/steps/wait-for-checks.yml
description: Waits for this commit's checks to pass.
inputs:
  workflows: {required: true, description: the workflows that must pass}
  minutes: {default: "25"}
needs: [curl]          # commands the image must have
secrets: [TOKEN]       # the secrets it may be given
steps:
  - name: wait
    run: |
      for wf in $INPUT_WORKFLOWS; do
        knotless checks "knotless://$KNOTLESS_RUN_REPO" --commit "$KNOTLESS_RUN_SHA" \
          --workflow "$wf" --wait $((INPUT_MINUTES * 60)) | grep -q '^success'
      done
    timeout: 30

# a workflow using it
secrets: [TOKEN]
steps:
  - uses: alice.example.com/steps/wait-for-checks@1
    with: {workflows: test check}
    secrets: [TOKEN]

# a workflow passing its own secret under the name the step file asks for
secrets: [CI_TOKEN]
steps:
  - uses: alice.example.com/steps/wait-for-checks@1
    with: {workflows: test check}
    secrets: {TOKEN: CI_TOKEN}   # seen as TOKEN, and not as CI_TOKEN

description: what it does
inputs: its inputs, by name, each with description, required (default false) and default
needs: command names, each letters, digits, ., _, + or -, checked with command -v before its steps
secrets: the names of the secrets a step using it may give it
steps: required: steps as a workflow’s are, with run, or with uses and with and secrets, passing on only secrets its own secrets names, and each with on if it likes

The version after @ takes itself and the versions it begins, part by part: 1 takes 1.0.0 and 1.4.2 but not 10.0.0, 1.2 takes 1.2.7, and 1.2.0 only itself.

A step is released with knotless publish as the package @step/<step>, its one file the step file, and --commit the commit it is at. The repository using shared steps pins each in .knotless/uses.lock, keyed by the uses exactly as its workflows and step files write it, which knotless uses update writes (CLI):

alice.example.com/steps/wait-for-checks@1:
  version: 1.2.0
  commit: 3f2a91c04be1d5e8a7b6c9d0e1f2a3b4c5d6e7f8   # in full
  release: at://did:plc:…/dev.knotless.release/…   # optional, for readers

The runner reads the lock at the commit it runs, and the step file at the commit the lock names, from its mirror of the step’s repository. Each step of the file runs in the step’s place, in the same container, named <step> / <its step>, after a <step> / needs step when it names any. A step of the file with uses is put in place the same way, named <step> / <its step> / <their step>, at most four deep. Each sees:

the inputs as INPUT_<NAME>, the name upper-cased with - as _: the value with gives, else the default, else empty;
of the run’s secrets, only those the step passes, each under the name the step file asks for it by and not the workflow’s, and in a nested step, only those of them the step using it passes on, its secrets mapping from that step file’s names;
KNOTLESS_STEP_DIR, the step’s repository at the locked commit, checked out beside the checkout, read-only under docker at /steps/<n>.

Every other step sees all the run is granted, each step given them as its own process starts rather than the container holding them. In a workflow that uses shared steps, what a step given secrets leaves running is ended once it ends, under docker, logging ==> ended <n> of what <step> left running, as it was given secrets.

A pull, or a merge-queue batch, whose lock names an account the base branch’s lock does not, and that the repository does not vouch for, runs untrusted, as an agent’s pull does (runner), and its log says first ==> runs as code no one vouches for, as its lock adds shared steps from <accounts>.

A workflow does not parse when a step has both run and uses or neither, has with or secrets without uses, passes a secret its secrets does not name, passes one name twice, has an on naming no refs, or uses is not of the form above. A run is filed as an error check, before any step runs, saying:

<workflow> uses shared steps, but there is no .knotless/uses.lock to pin them
.knotless/uses.lock does not pin <uses>
.knotless/uses.lock pins <uses> at <version>, which @<spec> does not take
.knotless/steps/<step>.yml at <commit> of knotless://<owner>/<repo>, with why it cannot be read
<uses> has no input <name>, <uses> needs the input <name>, or <uses> does not ask for the secret <name>
<uses> uses itself, through <uses>, …, or <uses> is past 4 shared steps deep, through <uses>, …

A step from the file that finds a command it needs missing fails with <step> needs <command> in the image.

Caches

Each cache path is kept in a base the run sees through an overlay of its own, under docker only (design). A new cache shows the image’s own files at that path only where its base has nothing. A path is refused, with <file>: cannot cache "<path>": a path in the container, not the checkout itself, `..` or `,`, when it is empty, / or /work itself, has an empty, . or .. part, or holds a ,. A trailing / is dropped.

Environment

A step sees the workflow’s env, with the variables given with rerun -e in place of any of the same name, the secrets granted to it, and the variables below. Under shell it also sees the runner’s PATH, HOME, LANG, TMPDIR, USER, DOCKER_HOST and DOCKER_CONFIG. Under docker those reach only the docker client, and the container has its image’s own environment instead.

KNOTLESS_RUN: 1
KNOTLESS_RUN_REPO: the repository, as the runner’s command line names it
KNOTLESS_RUN_REF: the ref, a pull’s head branch, or a batch’s base branch
KNOTLESS_RUN_SHA: the commit being run; a pull’s head, not the merge commit checked out
KNOTLESS_RUN_PULL: the pull’s AT-URI, in a pull’s run
KNOTLESS_RUN_BASE: in a pull’s run, the commit its head is merged with, the tip of the branch it is into; in a merge-queue batch’s, the tip the batch is built on
KNOTLESS_RUN_OUTPUTS: a directory for outputs, beside the checkout (/outputs under docker)

Nothing else of the runner’s environment reaches a step.

Outputs

When the steps end, passed or failed (not on an error), each image in KNOTLESS_RUN_OUTPUTS becomes one of the check’s outputs: a regular file ending png, jpg, jpeg, webp or gif in any case, at most 1,000,000 bytes, the first 32 by file name. Each is a blob, its alt the file name without its extension, cut to 300 characters; files with the same bytes and type share one. Only the kept are uploaded, four at a time. Anything else gets a line in the log: ==> <name>: not kept, as only images are outputs, being over 1000000 bytes, or past 32 outputs. The shots workflow’s harness/shots/outputs.mjs writes each page’s before and after as lossy WebP (quality 75) and the difference as lossless WebP, at twice the size, shrinks any over 1 MB further, and keeps as PNG what will not be WebP.

The log

The log, which is public, has a ==> <step name> line before each step, and the runner’s own lines start ==> too:

==> killed after <secs>s, for a step out of time, or ==> killed as the run's <secs>s were up, for one cut off by --run-timeout;
==> cancelled <why>, or ... before it started, where <why> is for a newer commit's run, as the pull is no longer at that head and base, as the merge queue dropped its batch, as the runner stopped: asked for again when it starts, or as the runner stopped, past what a stop waits for;
==> asked for by name NAME=value ..., first in a run asked for by name. Its values show to whoever reads the log, as they do in the dev.knotless.rerun record that asked: pass no secret with -e;
==> wrote back <n> files to <branch> as <sha12>, or ==> nothing to write back in <globs>, at the end of a run asked for whose workflow names asked.commit, or ==> not written back: <why>, when the run fails for it: only a run at a branch writes back, <branch> is <url>'s default branch, <branch> is protected, <branch> has moved on from <sha12>, or the push’s refusal;
`==> passed: the same tree, , as pull at merged with , whose run of passed (); not run again` (or `as the merge queue's batch at on `), for a push that takes an earlier pass, then `==> caches:` saying what that run wrote to which paths is kept, is not kept because its base has moved on, or that none was kept, and then that run's log, as much of its end as fits.

A check filed without running says why in error: asked for, but there is no workflow named <name> at <sha12>; asked for, but it runs neither on a push to <ref> nor when asked there; `the head, , conflicts with at in : merge into it, or rebase it onto

`; `reading the runner's secrets, `; ` did not start in s`, ` did not start: see the log`, or `the overlay at was not made: see the log`; `paused by this runner's operator: `, for every workflow at the commit of an owner the runner's `--paused` file names; and, for a check left `running` by a runner that stopped, `the runner stopped before this run finished`, or `... before this run started` for one left `queued`.

Secrets file

knotless-run --secrets <file> names a JSON file of secrets, keyed by each repository exactly as the runner’s command line names it, less any knotless://. The runner reads it again for each run that needs a secret, and not for one that needs none. A secret is a string, granted to every run of the repository’s workflows that names it, or an object with refs, granted only to pushes to those refs:

{
  "alice.example.com/my-project": {
    "API_KEY": "…",
    "DEPLOY_TOKEN": { "value": "…", "refs": ["refs/heads/main"] }
  }
}

refs match as a trigger’s do. The file grants a name before the appview: a repository’s owner can also give its runners secrets with knotless secret <url> set <name>, each sealed to the runner’s key and held at the appview, read for each run that needs a secret the file does not grant (see Secrets). An appview that does not answer then makes the check error, as a file that cannot be read does. A secret not granted is absent, and the log says so before the steps: ==> NAME is not granted to <repo> at <ref>. A pull’s or merge-queue batch’s run gets no secrets, and its log says that of each the workflow lists. In the log, which is public, each granted secret’s value is replaced by ***, and so is each trimmed line of a value with several lines, such as a PEM key, the longest first. Values shorter than four bytes are not masked.