# Workflow files What `knotless-run` reads from a repository, and what it gives each step. [Workflows](../guide/workflows.md) shows how to use them. ## Files Workflows are `.knotless/workflows/*.yml`, `*.yaml` or `*.json`, read at the commit being run, in file name order. Only files directly in that directory count, and a commit without it has no workflows. Keys not listed here are refused, and `on` is a plain key (YAML 1.2), not `true`. A file that does not parse is filed as an `error` check with the parse message, under its name, whether or not it would have run. Two files with the same `name` both fail to run, and one `error` check says `more than one workflow is named , among them`. A name is 1 to 64 letters, digits, `_` or `-`, since it ends the check's record key; a file whose name is not is filed under the name `invalid`. ```yaml name: ci # defaults to the file name on: push: refs: [refs/heads/main, refs/tags/v*] # default: refs/heads/* paths: [src/**, Cargo.*] # default: any change pull: base: [refs/heads/main] # default: refs/heads/*; only with --pulls paths: [src/**, Cargo.*] # default: any change asked: refs: [refs/heads/main] # default: refs/heads/* commit: [shots/**] # default: none; files written back image: rust:1 # docker executor only; default: --image cache: [target, /usr/local/cargo/registry] # docker executor only env: { RUSTFLAGS: -Dwarnings } secrets: [DEPLOY_TOKEN] runners: [deploy.alice.example.com] # default: every runner named superseded: cancel # or finish queue: batch # or first steps: - name: test run: cargo test timeout: 30 # minutes - name: deploy run: ./deploy.sh on: {refs: [refs/heads/main, refs/tags/v*]} # default: wherever the workflow runs ``` | Key | Default | Meaning | | --- | --- | --- | | `name` | the file name, without its extension | the workflow's name, in its checks | | `on` | `push` to `refs/heads/*` | the triggers below; with no `on`, every push to a branch, and with `on: {}`, nothing | | `image` | the runner's `--image` (`debian:stable-slim`) | the container image the steps run in | | `cache` | none | paths kept from run to run; relative ones are in the checkout, under `/work`; see [Caches](#caches) | | `env` | none | variables every step sees | | `secrets` | none | the names of secrets the steps need, given only where the runner's secrets file or the owner grants them ([Secrets file](#secrets-file)) | | `runners` | every runner the repository names | the runners that run it, by DID or handle, of those the repository names; any other files no check of it, so a workflow naming none of them never runs, and a pull waiting on it never merges | | `superseded` | `cancel` | what a newer commit's run on the same ref or pull does to this one: `cancel` it, or, with `finish`, wait for it once it holds the workflow's turn, which it takes as soon as it is filed if no run holds it; anything else fails to parse | | `queue` | `batch` | where it runs in the [merge queue](#the-merge-queue), if it runs on pulls: `batch`, on each batch, or `first`, on each batch and first on each pull queued, which goes into a batch only once it passes; anything else fails to parse | | `steps` | required | the steps, run in order until one fails | ## Triggers | Trigger | Runs on | Keys | | --- | --- | --- | | `push` | a ref moving | `refs`, `paths` | | `pull` | an open pull's head merged with its base, under a runner started with `--pulls` | `base` (the branches the pull merges into), `paths` | | `asked` | the owner asking for the workflow by name with `knotless rerun --workflow` | `refs`, `commit` | `refs` and `base` are full ref names. Each matches exactly or, if it ends in `*`, as a prefix. That is not a glob: `refs/heads/*` matches `refs/heads/a/b`, and `refs/tags/v*` matches `refs/tags/v1.2`. The owner can ask by name for a workflow at any ref its `push` or `asked` trigger names. A workflow with only `asked` runs only when asked for, which suits one too costly for every push. ### Writing back `asked.commit` names globs, matched as [`paths`](#paths) are, of files a run asked for by name may write back. Once such a run passes, the runner commits what its steps changed among them, added, changed or deleted, onto the branch it ran at, and pushes that commit as its own account, author `knotless-run`. Nothing else the steps changed goes in. It writes only to a branch that is neither the repository's default branch nor protected, only from the commit the run was at, and only if its account may push there. A run that cannot write back fails, saying why. This makes files that must come from one place, such as screenshots taken in the image they are checked in: ```yaml on: {asked: {refs: [refs/heads/*], commit: [shots/linux/**]}} ``` Asked for at a pull's branch, the commit moves the pull's head, so its checks run again on what was written. ### Paths `paths` are globs matched against a changed file's whole path from the repository's root: - `*` matches any run of characters within one part of the path, between `/`s. - `?` matches one character within a part. - `**`, as a part of its own, matches any number of parts, none included. - Anything else matches itself. So `web/**` is everything under `web`, `web` included; `*.md` matches only at the root, and `**/*.md` anywhere. A workflow runs if any glob matches any changed file. What counts as changed: - for a pull, what its head merged with its base changes from the base's tip; - for a push, `git diff --name-only --no-renames` from where the runner's mirror had the ref to the commit it moved to, so a rename is a delete and an add; - for a new ref, a ref only asked for by name, a ref another pass already synced, a conflicting pull, or a git error, everything. A workflow no changed file matches is filed at once as `success`, with its steps `skipped` and the note `skipped: nothing changed in `, cut to 1000 characters. A run asked for by name is never skipped this way. A merge-queue batch picks its workflows by `pull.base` but skips by the `push` trigger's `paths`. ### The merge queue A [merge queue](../design/merge-queue.md#pre-flights)'s batch runs every workflow whose `pull.base` names its branch. One that also says `queue: first` is a pre-flight: a pull queued goes into a batch only once its run of it has passed at the pull's head, against any tip of the branch, as any runner the repository names filed it. Until then it waits, and those queued behind it that are ready go ahead. A run of it that fails against the branch's tip ejects the pull without a batch. With none passed or running, the runner landing the queue runs it on the pull itself. ```yaml # check.yml: fmt and clippy, minutes where the tests take many more. queue: first on: {pull: {base: [refs/heads/main]}} ``` Make a pre-flight quick and likely to catch what would fail a batch, such as a build: each pull that fails it in a batch costs a halving of the batch to find. The pull's run of it carries `queue: first` in its check record, which is how readers tell that a pull is waiting for one. ## Steps | Key | Meaning | | --- | --- | | `name` | the step's name, in its check; default `step `, counting from 1, or for a step with `uses`, the shared step's name | | `run` | a shell script, run with `sh -ec`, so any command in it that fails fails the step; a step has this or `uses` | | `uses` | a [shared step](#shared-steps), `//@`, put in this step's place | | `with` | with `uses` only: the shared step's inputs, by name | | `secrets` | with `uses` only: the workflow's `secrets` the shared step is given, as a list of names, each passed under its own, or a map from the name the step file asks for to the workflow's (`{SSH_KEY: DEPLOY_SSH_KEY}`); none if absent | | `timeout` | minutes; can only shorten the runner's `--step-timeout`, except in a run asked for by name, where it may go past it | | `on` | `{refs: [...]}`, the refs the step runs at; see [Steps for some refs](#steps-for-some-refs) | Under the `docker` executor the checkout is mounted at `/work`, and every step runs in one container, so a step sees what earlier steps installed. The image needs `sh` and `tail`. Under `shell`, a step runs in the checkout as the runner's user. A step passes on exit 0. One killed, for time or by a cancel, has no exit code in its check; one ended by a signal otherwise has `-1`. ### Steps for some refs A step with `on` runs only at the refs its `refs` names, full ref names matched as `on.push.refs` are: exactly, or as a prefix before a final `*`. `refs` is required and may not be empty, and `on` takes no other key. The ref matched is the one a push moved, or the one a run asked for by name is at. A pull's run and a merge-queue batch's run are at no ref of the repository's, so a step with `on` runs in neither, whatever `KNOTLESS_RUN_REF` holds there (a pull's head branch, which may be a fork's `main`; a batch's base branch, not moved yet). A step left out is not run: its check lists it as `skipped`, it counts as passed, and the log says why under its `==> ` line: - `==> skipped, as refs/heads/x is not refs/heads/main or refs/tags/v*` - `==> skipped, as a pull's run is at no ref of the repository's` - `==> skipped, as a merge queue batch's run is at no ref until it lands` A run that left a step out is not taken by the push that lands its pull or batch, as other runs that pass are ([merge queue](#the-merge-queue)): the push runs the workflow itself, with the steps for its ref. On a step with `uses`, `on` holds for every step the shared step comes to, and a step in the step file may name its own `on` too: it runs only where both match, and so on down nested shared steps. ```yaml steps: - uses: alice.example.com/ci/publish@1 on: {refs: [refs/heads/main]} ``` ### Shared steps A step with `uses` takes the steps of a file in another repository, `.knotless/steps/.yml` in `knotless:///`, where `` is a handle or a DID. Why it works this way is in [Shared steps](../design/reusable-workflows.md). ```yaml # alice.example.com/steps: .knotless/steps/wait-for-checks.yml description: Waits for this commit's checks to pass. inputs: workflows: {required: true, description: the workflows that must pass} minutes: {default: "25"} needs: [curl] # commands the image must have secrets: [TOKEN] # the secrets it may be given steps: - name: wait run: | for wf in $INPUT_WORKFLOWS; do knotless checks "knotless://$KNOTLESS_RUN_REPO" --commit "$KNOTLESS_RUN_SHA" \ --workflow "$wf" --wait $((INPUT_MINUTES * 60)) | grep -q '^success' done timeout: 30 ``` ```yaml # a workflow using it secrets: [TOKEN] steps: - uses: alice.example.com/steps/wait-for-checks@1 with: {workflows: test check} secrets: [TOKEN] ``` ```yaml # a workflow passing its own secret under the name the step file asks for secrets: [CI_TOKEN] steps: - uses: alice.example.com/steps/wait-for-checks@1 with: {workflows: test check} secrets: {TOKEN: CI_TOKEN} # seen as TOKEN, and not as CI_TOKEN ``` | Step file key | Meaning | | --- | --- | | `description` | what it does | | `inputs` | its inputs, by name, each with `description`, `required` (default `false`) and `default` | | `needs` | command names, each letters, digits, `.`, `_`, `+` or `-`, checked with `command -v` before its steps | | `secrets` | the names of the secrets a step using it may give it | | `steps` | required: steps as a workflow's are, with `run`, or with `uses` and `with` and `secrets`, passing on only secrets its own `secrets` names, and each with `on` if it likes | The version after `@` takes itself and the versions it begins, part by part: `1` takes `1.0.0` and `1.4.2` but not `10.0.0`, `1.2` takes `1.2.7`, and `1.2.0` only itself. A step is released with `knotless publish` as the package `@step/`, its one file the step file, and `--commit` the commit it is at. The repository using shared steps pins each in `.knotless/uses.lock`, keyed by the `uses` exactly as its workflows and step files write it, which `knotless uses update` writes ([CLI](cli.md)): ```yaml alice.example.com/steps/wait-for-checks@1: version: 1.2.0 commit: 3f2a91c04be1d5e8a7b6c9d0e1f2a3b4c5d6e7f8 # in full release: at://did:plc:…/dev.knotless.release/… # optional, for readers ``` The runner reads the lock at the commit it runs, and the step file at the commit the lock names, from its mirror of the step's repository. Each step of the file runs in the step's place, in the same container, named ` / `, after a ` / needs` step when it names any. A step of the file with `uses` is put in place the same way, named ` / / `, at most four deep. Each sees: - the inputs as `INPUT_`, the name upper-cased with `-` as `_`: the value `with` gives, else the default, else empty; - of the run's secrets, only those the step passes, each under the name the step file asks for it by and not the workflow's, and in a nested step, only those of them the step using it passes on, its `secrets` mapping from that step file's names; - `KNOTLESS_STEP_DIR`, the step's repository at the locked commit, checked out beside the checkout, read-only under docker at `/steps/`. Every other step sees all the run is granted, each step given them as its own process starts rather than the container holding them. In a workflow that uses shared steps, what a step given secrets leaves running is ended once it ends, under docker, logging `==> ended of what left running, as it was given secrets`. A pull, or a merge-queue batch, whose lock names an account the base branch's lock does not, and that the repository does not vouch for, runs untrusted, as an agent's pull does ([runner](../design/runner.md#whose-code-runs)), and its log says first `==> runs as code no one vouches for, as its lock adds shared steps from `. A workflow does not parse when a step has both `run` and `uses` or neither, has `with` or `secrets` without `uses`, passes a secret its `secrets` does not name, passes one name twice, has an `on` naming no refs, or `uses` is not of the form above. A run is filed as an `error` check, before any step runs, saying: - ` uses shared steps, but there is no .knotless/uses.lock to pin them` - `.knotless/uses.lock does not pin ` - `.knotless/uses.lock pins at , which @ does not take` - `.knotless/steps/.yml at of knotless:///`, with why it cannot be read - ` has no input `, ` needs the input `, or ` does not ask for the secret ` - ` uses itself, through , …`, or ` is past 4 shared steps deep, through , …` A step from the file that finds a command it `needs` missing fails with ` needs in the image`. ### Caches Each `cache` path is kept in a base the run sees through an overlay of its own, under docker only ([design](../design/runner.md#caches)). A new cache shows the image's own files at that path only where its base has nothing. A path is refused, with `` : cannot cache "": a path in the container, not the checkout itself, `..` or `,` ``, when it is empty, `/` or `/work` itself, has an empty, `.` or `..` part, or holds a `,`. A trailing `/` is dropped. ### Environment A step sees the workflow's `env`, with the variables given with `rerun -e` in place of any of the same name, the secrets granted to it, and the variables below. Under `shell` it also sees the runner's `PATH`, `HOME`, `LANG`, `TMPDIR`, `USER`, `DOCKER_HOST` and `DOCKER_CONFIG`. Under docker those reach only the docker client, and the container has its image's own environment instead. | Variable | Value | | --- | --- | | `KNOTLESS_RUN` | `1` | | `KNOTLESS_RUN_REPO` | the repository, as the runner's command line names it | | `KNOTLESS_RUN_REF` | the ref, a pull's head branch, or a batch's base branch | | `KNOTLESS_RUN_SHA` | the commit being run; a pull's head, not the merge commit checked out | | `KNOTLESS_RUN_PULL` | the pull's AT-URI, in a pull's run | | `KNOTLESS_RUN_BASE` | in a pull's run, the commit its head is merged with, the tip of the branch it is into; in a merge-queue batch's, the tip the batch is built on | | `KNOTLESS_RUN_OUTPUTS` | a directory for [outputs](#outputs), beside the checkout (`/outputs` under docker) | Nothing else of the runner's environment reaches a step. ### Outputs When the steps end, passed or failed (not on an error), each image in `KNOTLESS_RUN_OUTPUTS` becomes one of the check's `outputs`: a regular file ending `png`, `jpg`, `jpeg`, `webp` or `gif` in any case, at most 1,000,000 bytes, the first 32 by file name. Each is a blob, its `alt` the file name without its extension, cut to 300 characters; files with the same bytes and type share one. Only the kept are uploaded, four at a time. Anything else gets a line in the log: `==> : not kept, as only images are outputs`, `being over 1000000 bytes`, or `past 32 outputs`. The `shots` workflow's `harness/shots/outputs.mjs` writes each page's before and after as lossy WebP (quality 75) and the difference as lossless WebP, at twice the size, shrinks any over 1 MB further, and keeps as PNG what will not be WebP. ### The log The log, which is public, has a `==> ` line before each step, and the runner's own lines start `==>` too: - `==> killed after s`, for a step out of time, or `==> killed as the run's s were up`, for one cut off by `--run-timeout`; - `==> cancelled `, or `... before it started`, where `` is `for a newer commit's run`, `as the pull is no longer at that head and base`, `as the merge queue dropped its batch`, `as the runner stopped: asked for again when it starts`, or `as the runner stopped, past what a stop waits for`; - `==> asked for by name NAME=value ...`, first in a run asked for by name. Its values show to whoever reads the log, as they do in the `dev.knotless.rerun` record that asked: pass no secret with `-e`; - `==> wrote back files to as `, or `==> nothing to write back in `, at the end of a run asked for whose workflow names `asked.commit`, or `==> not written back: `, when the run fails for it: `only a run at a branch writes back`, ` is 's default branch`, ` is protected`, ` has moved on from `, or the push's refusal; - `==> passed: the same tree, , as pull at merged with , whose run of passed (); not run again` (or `as the merge queue's batch at on `), for a push that takes an earlier pass, then `==> caches:` saying what that run wrote to which paths is kept, is not kept because its base has moved on, or that none was kept, and then that run's log, as much of its end as fits. A check filed without running says why in `error`: `asked for, but there is no workflow named at `; `asked for, but it runs neither on a push to nor when asked there`; `the head, , conflicts with at in : merge into it, or rebase it onto `; `reading the runner's secrets, `; ` did not start in s`, ` did not start: see the log`, or `the overlay at was not made: see the log`; `paused by this runner's operator: `, for every workflow at the commit of an owner the runner's `--paused` file names; and, for a check left `running` by a runner that stopped, `the runner stopped before this run finished`, or `... before this run started` for one left `queued`. ## Secrets file `knotless-run --secrets ` names a JSON file of secrets, keyed by each repository exactly as the runner's command line names it, less any `knotless://`. The runner reads it again for each run that needs a secret, and not for one that needs none. A secret is a string, granted to every run of the repository's workflows that names it, or an object with `refs`, granted only to pushes to those refs: ```json { "alice.example.com/my-project": { "API_KEY": "…", "DEPLOY_TOKEN": { "value": "…", "refs": ["refs/heads/main"] } } } ``` `refs` match as a trigger's do. The file grants a name before the appview: a repository's owner can also give its runners secrets with `knotless secret set `, each sealed to the runner's key and held at the appview, read for each run that needs a secret the file does not grant (see [Secrets](../design/runner.md#secrets)). An appview that does not answer then makes the check `error`, as a file that cannot be read does. A secret not granted is absent, and the log says so before the steps: `==> NAME is not granted to at `. A pull's or merge-queue batch's run gets no secrets, and its log says that of each the workflow lists. In the log, which is public, each granted secret's value is replaced by `***`, and so is each trimmed line of a value with several lines, such as a PEM key, the longest first. Values shorter than four bytes are not masked.