# Deploying knotless-serve One Docker host, a reverse proxy in front of it for TLS, and a domain. | | | | --- | --- | | `../Dockerfile` | one image, `knotless:current`: knotless-serve, knotless-run, knotless-triage, knotless-agent, knotless-mcp with Claude Code, the knotless CLI, git, the docker client and the built pages | | `docker-compose.yml` | `/etc/knotless/docker-compose.yml`: `serve` (knotless-serve, on :2590), `pages` (the SvelteKit app, on :2591) and, if asked, `run` (knotless-run), `triage` (knotless-triage), `review` (knotless-agent review) and `keep` (knotless-agent keep) | | `deploy.sh` | builds the image on the host over SSH, writes the compose file and its `.env`, and restarts what changed; or one of those steps, `build`, `up` or `drop` | | `rollout.sh` | run on the host by `deploy.sh`: moves `pages` and `serve` to a new container each, with no gap | | `handover.sh` | run on the host by `deploy.sh`: moves `run` to a new container, the old one handing over to it as it finishes its runs | | `rollout-test.sh` | tries `rollout.sh` on a local docker behind a Caddy, counting failed requests | | `release.sh` | builds the client for macOS and Linux here and puts the builds on the host, for `/install.sh` | | `setup.sh` | makes a fresh Ubuntu 24.04 host ready for `deploy.sh`: docker, a Caddy of its own for the domain, a firewall, swap, upgrades and pruning | ```sh deploy/setup.sh root@host git.example.com # once, on a fresh host just deploy root@host git.example.com # build on the host and restart; or put both in deploy/local.env ``` The domain needs an A (and AAAA) record for the host, and an `_atproto` TXT record for each of the service's own accounts with a handle under it. knotless.dev's are kept by the `stacks/edge` Terraform in the infra repository, beside `stacks/knotless`, which makes the host. A worker that serves an account's handle at its own name, as the reviewer's pds.js does at `review.knotless.dev`, holds that name as a custom domain in its `wrangler.toml`, which makes the record: the infra repository keeps only its `_atproto` TXT. ### From a push to main to the site running it `.knotless/workflows/deploy.yml` deploys each push to main once its seven checks (check, test, web, lexicons, shots, docs and e2e) pass at it. It names them in `after`, so it waits for them holding none of the runner's jobs, and the checks it waits for are never kept from one. One that fails files the deploy failed without running. Then: 1. **build**: `deploy.sh build` builds `knotless:build-`, a candidate, which nothing runs: `knotless:current` and the containers stay as they are. It removes any candidate an earlier deploy left. 2. **deploy**: if main is still at this commit, `deploy.sh up` tags the candidate `knotless:current`, removes the candidate's tag, and rolls out as below, restarting `run` only if what it runs changed ([When a deploy restarts the runner](#when-a-deploy-restarts-the-runner)), then removes the untagged images no container runs, the ones earlier deploys replaced. Until it goes up, a deploy gives way: when main moves past its commit, asked every 30s through the build, it stops (a build is cancelled with its client, its cargo cache kept for the next), removes its candidate and passes, and the newer commit's deploy, which waits for it (`superseded: finish`), starts. Once it goes up it runs to its end. So a push goes live a few minutes after its checks pass: the build, two minutes or so for a change to our crates in the `deploy` profile, then the rollout, about 30s. The site comes first on the host for CPU, then a deploy and main's checks, then pulls' runs. serve and pages have `cpu_shares` of 4096 (a `cpu.weight` of about 157); the build runs in the cgroup `/system.slice/knotless-build`, which `deploy.sh build` makes with a `cpu.weight` of 100, as main's runs have, where docker drives cgroups (v2) with systemd; and knotless-run starts a pull's run with `--cpu-shares 256`, a weight of 10, so a burst of pulls takes only what the rest leave. Before, the build had 20 against the pulls' 100, and a merge that ran every open pull again held the deploy back for minutes. Docker's buildkit puts each build step in a cgroup of its own, and by default beside the containers', where `nice` inside it would count for nothing against them. `KNOTLESS_BUILD_CGROUP` names another parent, or set empty, none. The image's binaries are built in the `deploy` profile (`Cargo.toml`): release's, but in 16 codegen units without LTO across crates, so the crates a push rebuilds, knotless-serve the longest, use every core. The cargo registry and `target/` are cache mounts on the host's builder, so a deploy rebuilds only our four crates, not their dependencies; a fresh checkout's files are all new to cargo, so all four, whichever changed. `docker buildx du --filter type=exec.cachemount` on the host shows them. Each build then prunes buildkit's cache to the 20 GB used last, which keeps those mounts; uncapped, it held 49 GB when the disk filled. By hand, `just deploy` does all three steps at once, as before. The proxy joins the `knotless` docker network (`KNOTLESS_PROXY_NETWORK`) and sends the domain to `knotless-serve:2590`, passing `X-Forwarded-Proto`. With Caddy: ```caddy git.example.com { reverse_proxy knotless-serve:2590 { # Holds a request that meets no listening serve, for a moment # within a deploy, and tries again rather than answering 502. lb_try_duration 20s lb_try_interval 250ms } } ``` ### Push notifications The iOS app's notifications go through Apple's push service, with a key from the Apple Developer account of the team that signs the app. Put it on the host once, where the server can read it: ```sh install -o 1000 -g 1000 -m 600 AuthKey_XXXXXXXXXX.p8 "$KNOTLESS_DATA/apns.p8" ``` Then set `KNOTLESS_APNS_KEY_ID` and `KNOTLESS_APNS_TEAM` and deploy. `deploy.sh` gives serve `--apns-key` only when the file is there; without it, serve runs as ever and sends nothing. Anyone holding the key can send to any app of that team, so it stays on this host and in the encrypted secrets it came from. ### Deploys without downtime `deploy.sh` moves `pages` and then `serve` with `rollout.sh`: a second container starts beside the running one, with the same names on the same networks, and once its healthcheck passes the old one is stopped, finishing the requests it holds (SIGTERM, then `stop_grace_period`), and removed. A new one that is not healthy in 180s (`KNOTLESS_ROLLOUT_TIMEOUT`) is removed and the old one keeps serving; the deploy fails. `run` is restarted last, and only if what it runs changed ([When a deploy restarts the runner](#when-a-deploy-restarts-the-runner)). While both run, docker's DNS gives `knotless-serve`, `serve` and `pages` as every running container's address. The proxy, serve and the pages each dial every address in turn, so one not listening yet, or no longer, is refused and the next is tried. `lb_try_duration` is the proxy's belt and braces over that: it holds a request that found none and tries again. Both serves share `KNOTLESS_DATA` for those seconds. The index is SQLite in WAL mode, each process waiting up to 5s for the other's writes; mirrors are written under lock files and renames, as concurrent processes do already; both follow the streams and write the cursor, and a record indexed twice is the same row. The one unsafe overlap is a build that changes the index's tables: it drops and rebuilds them under the old one. So `rollout.sh` first asks the new build (`knotless-serve --check-index`), and if it would rebuild the index, serve is stopped and then started instead: down until the new one listens, a second or two, which the proxy's `lb_try_duration` holds requests through. `deploy/rollout-test.sh` tries all of this on a local docker, behind a Caddy, counting failed requests across the rollouts. #### The proxy knotless.dev has a host of its own (Hetzner `knotless`, a cx43, made by the `stacks/knotless` Terraform in the infra repository), apart from the sites it once shared one with: its runs and builds took the cores they needed. `setup.sh` gives it Caddy in `/etc/knotless-proxy`, which dials `knotless-serve:2590` by name over the `knotless` network and holds a request through a rollout's second or two with no serve: ```caddy knotless.dev { request_body { max_size 100MB } reverse_proxy knotless-serve:2590 { lb_try_duration 20s lb_try_interval 250ms } encode gzip zstd } ``` With `KNOTLESS_PAGES_DOMAIN` (`knotless.page`), the proxy answers repositories' sites too ([../docs/design/pages.md](../docs/design/pages.md)): the domain and every owner's label under it, and each private site's label under `private.` there, on wildcard certificates from the DNS challenge. The zone's `*` record covers `*.private` too, as nothing is at `private` itself. That takes Caddy built with the `caddy-dns/cloudflare` module, which `setup.sh` builds from a Dockerfile beside the compose file, and the zone's own Cloudflare token, Zone:Read and DNS:Edit on that zone alone. Give the token once, as `KNOTLESS_PAGES_DNS_TOKEN` in `deploy/local.env`, and run `setup.sh`: it keeps it in `/etc/knotless-proxy/cloudflare.env`, and a run without it keeps that one. A run with nothing changed rebuilds and recreates nothing. ```caddy knotless.page, *.knotless.page, *.private.knotless.page { tls { dns cloudflare {env.CLOUDFLARE_API_TOKEN} } reverse_proxy knotless-serve:2590 { lb_try_duration 20s lb_try_interval 250ms } encode gzip zstd } ``` `deploy.yml` sets `KNOTLESS_PAGES_DOMAIN` for serve's `--pages-domain`. The zone's DNS (its apex and `*`, DNS-only, at this host) is in the infra repository's `stacks/edge`. On a host whose proxy is another's, join that proxy's network, named `knotless` (`KNOTLESS_PROXY_NETWORK`), and give the domain that block in its Caddyfile instead. knotless-serve is the appview (`dev.knotless.listRepos`, `getForge`, `getFeed`, indexed from Jetstream), the gateway, as `did:web:`, and the moderation service. Its state is in `KNOTLESS_DATA` (`/var/lib/knotless`): - `index.db`, the index: a cache. A version that changes its tables rebuilds it, and it refills itself at start from the accounts the relay (`KNOTLESS_RELAY`, relay1.us-east.bsky.network) lists as holding knotless records, and again every ten minutes, for any event Jetstream missed. - `cache/`, mirrors and identities: a cache. - `usage.db`, how it is used (see [Usage](#usage)): counts to keep, but nothing breaks without them. - `config/`, the gateway's OAuth client key and the sessions owners delegated to it. Losing it breaks every delegation, which owners then have to make again: back it up. - `bots/`, the sessions run, triage and review act with, once they log in at `/oauth/bot` (see [knotless-run](#knotless-run)): lost, they log in again. Keep the records DNS-only: PDSes fetch the gateway's DID document and OAuth client metadata from the origin, and the proxy holds the certificate. ## Usage `knotless admin stats` (`--days `, 14 by default; `--json`) shows the operator how the service is used, from `dev.knotless.getUsage`, to the accounts `KNOTLESS_ADMIN_DIDS` names (set in `.knotless/workflows/deploy.yml`), signed in as one of them: ```sh KNOTLESS_APPVIEW=https://knotless.dev knotless admin stats ``` - **Each day**: the people who visited the pages and the pages they looked at; the requests serve took and how many it failed; and, from the index, what the network wrote: repositories, pushes, pulls, issues, comments and checks, and how many accounts wrote them. - **The network**: every account that has made a repository, pushed or written in a thread, and those that did in the last 7 and 30 days. - **Routes**: requests per route as axum matched it (the pattern, never the path), by surface (`pages`, `api`, `git`, `xrpc`, `mcp`, `install`, `other`), with failures and times. The time is to the answer's head, so a long clone counts its start. - **Pages, referrers and clients**: the routes people looked at, the sites that sent them, and requests by kind of client (`browser`, `git`, the `cli`, `bot`, `other`, from the `User-Agent`), and `pages`, the queries the pages ask as they draw a page (`knotless-pages`), which are not a client of their own. - **Events**, what the handlers saw, with how often each failed, its mean time and the accounts that did it, in the days asked but at most the last 30: `mcp.tool` by tool, `mcp.resource` by the tool that answers the read, and `mcp.signin`; `web.login`; `git.fetch` by repository, once per clone or fetch, as asked for (`/`); `cli.version` by the CLI's version; `gateway.delegated`, `gateway.push`, `gateway.commit`, and `gateway.served`, by the owner the gateway pushed or committed for. - **Funnels**: of the accounts that delegated to the gateway in the days asked, not having in the 30 days before, how many the gateway has written for then or since, by anyone's push or commit that moved a ref; of those that signed in to MCP so, how many have called a tool. A day older than 30 is counted as it stood once 30 days had passed. - **Hours**: each hour's requests, how many failed and how many took over two seconds, to know when to read the [logs](#logs) of. The lines show only the hours with either; `--json` has every hour. - **Statements** (`sql`): the index's statements that took 100 ms or more, as the `sql` log line has them (the SQL as written, no value it was given), with how often, their mean and their longest, the most time first: the fifty that cost most. - **Failures**: why an event failed, as it answered, most often first: so far an MCP tool's error. A DID, an `at://` URI, a hash and a number in it are put as ``, ``, `` and ``, so like failures count together. A handle or a repository's name in it is kept as written. - **Agents**: requests by the `User-Agent` of each crawler or script, the `bot` client, most first: never a browser's. Statements, failures and agents keep at most 500 of each a hour apart; past that, the new ones count together as `(others)`. What is kept, in `usage.db`, is counts by the hour, and no address; and, for the pages' logins, MCP and the gateway, where people are signed in already, which accounts (by DID) did what each day, for the events' accounts and the funnels; and the statements, failures and crawlers' agents above, by the hour. Those are kept 30 days, today the last: on the 30th day after, serve folds each day into its funnels' counts, how many started and how many went on, with no one in them, and deletes them, and the hours of the rest. A page, once drawn in a browser, sends `/_hit` its route and the site the visitor came from. Each visitor is counted once a day by a hash of their address, less its last byte (its last 80 bits for IPv6), and their `User-Agent`, salted with a key made for the day. Two serves run at once for a moment in each deploy, so the key is kept in the database for both, and deleted with the day's hashes once the day (UTC) is over, leaving the day's count: then nothing links a visitor to an address, or to the next day. Crawlers, by their `User-Agent` or by not running the page, and browsers that ask not to be followed (`Sec-GPC: 1`), are not counted. Anyone can read the days and the network, but not the requests, routes, events or funnels: the `/stats` page shows them, from `dev.knotless.getStats`. Requests are counted in memory and written every 15 seconds and as serve stops; one killed loses those seconds. It reads like this (`--json` gives the same as [`knotless::usage::Report`](../crates/knotless/src/usage.rs)): ```text network 212 accounts, 340 repositories, 41 authors this week, 97 this month day visitors views requests errors repos pushes pulls issues comments checks authors 2026-10-22 118 431 20544 3 4 96 7 5 22 61 19 2026-10-23 131 502 22310 0 2 104 9 3 31 70 23 route git POST /{authority}/{repo}/git-upload-pack 1840 requests, 0 failed, 210ms mean, 96 over 0.5s, 4 over 2s route xrpc GET /xrpc/dev.knotless.getRepo 1203 requests, 2 failed, 48ms mean, 11 over 0.5s, 0 over 2s event cli.version 0.1.0 2210 calls, 0 failed, 0ms mean, 0 accounts event git.fetch alice.test/proj 140 calls, 0 failed, 0ms mean, 0 accounts event mcp.tool repo_read 612 calls, 9 failed, 85ms mean, 6 accounts hour 2026-10-23T14 1022 requests, 3 failed, 2 over 2s funnel gateway.delegated -> gateway.served 4 started, 3 reached funnel mcp.signin -> mcp.tool 7 started, 6 reached page /[authority]/[repo] 388 referrer bsky.app 52 client browser 9120 client bot 7310 ``` So it answers, among others: - **Growth**: how many accounts write, and how many did this week and month; repositories, pushes, pulls, issues and checks a day, and whether a day's pushes are many people or a few (`authors` beside `pushes`). From the index, so the days before `--usage` too. - **The pages**: visitors and views a day, which pages they read, and which sites sent them. - **The surfaces**: requests by surface and by kind of client, so how much is git, the CLI, MCP, the installer, or crawlers. - **MCP**: which tools are called, how often they fail, by how many accounts, and how many who sign in go on to use it. - **Repositories and releases**: which repositories are cloned and fetched most, and which CLI versions are still about. - **The gateway**: how many who delegate to it go on to push. - **Health**: which routes fail or are slow, and a deploy that made either worse, day against day; and the hours it happened in. A long poll is never counted slow. What it does not count: - Anyone across days: a visitor's hash is new each day, so there are visitors a day but none a week or returning; weekly and monthly figures are of accounts that write, from the index. - Any account past 30 days: an event's accounts over more days are of the last 30, and a funnel cannot tell an account that comes back after a month from a new one. - What a route was asked of, past the events: routes are counted by pattern, so the API's reads of every repository are one row. - Installs: the installer and the releases are counted as requests, by no one; the funnels start at a delegation or a sign-in. ## Releases The server hands out the client, so installing it needs no clone and no Rust: `curl -fsSL https://knotless.dev/install.sh | sh` installs `knotless`, `git-remote-knotless`, `knotless-mcp`, `knotless-run`, `knotless-triage`, `knotless-agent` and `knotless-claude-review` into `~/.local/bin` (`KNOTLESS_INSTALL_DIR`), the latest release or `KNOTLESS_VERSION`'s, checked against its sha256. ```sh just release root@host # or with KNOTLESS_HOST in deploy/local.env ``` builds the workspace's version, from `Cargo.toml`, for macOS and Linux (aarch64 and x86_64; Linux static, with musl) on a Mac, with `deploy/build-client.sh`, puts them in `releases/` under `KNOTLESS_DATA`, and makes it the latest. The Linux builds need `brew install zig` and `cargo install cargo-zigbuild`. Bump the version before a release: one already out is replaced in place. Or, as the releases of the repository's `knotless` package ([packages](../docs/design/packages.md)): the `release` workflow publishes each new version once main's checks pass at it. It publishes as the owner, with `RELEASE_PASSWORD`, an app password of theirs, which the runner's secrets file grants to pushes to main: ```json { "chadtmiller.com/knotless": { "RELEASE_PASSWORD": { "value": "…", "refs": ["refs/heads/main"] } } } ``` The deploy publishes the image it runs the same way, as the release of the `knotless-server` package at that version, once it is up, and needs `RELEASE_PASSWORD` for it too; a version with that release already is left alone. The `publish-lexicons` workflow publishes the lexicons once main's checks pass at a commit that changes them, with the shared `publish-lexicons` step from `chadtmiller.com/steps` (its README lists its inputs): a `_lexicon.` TXT record naming `lexicons.knotless.dev` for each authority (`_lexicon.knotless.dev` for `dev.knotless.*`), and a `com.atproto.lexicon.schema` record in that account for each lexicon. It writes what is new or changed and deletes what main no longer has. `lexicons.knotless.dev` is an account of its own, on a pds.js worker as the runner's is, so what CI holds can write lexicons and nothing else. It writes as `lexicons.knotless.dev` with `LEX_PASSWORD`, an app password of that account's, which the secrets file grants to pushes to main. It reads the DNS as resolvers see it, and changes it only with `CLOUDFLARE_API_TOKEN`, a token with Zone:DNS:Edit on the zone allowed from this host's address. That is needed only when an authority is added or dropped, and is not granted now: such a run writes the schemas, then fails, naming the record to make. Every run first plans, saying what it would change, and `knotless run rerun --ref refs/heads/ --workflow publish-lexicons` plans on a branch. One image runs every service, each by its command, as docker-compose.yml starts them: ```sh docker pull knotless.dev/chadtmiller.com/knotless/knotless-server ``` | Service | Command | | --- | --- | | `serve` | `knotless-serve --listen 0.0.0.0:2590 …`, the image's default | | `pages` | `node /app/build` | | `run` | `knotless-run --secrets … --state … --work … --caches …` | | `triage` | `knotless-triage --decide --jetstream ` | | `review` | `knotless-agent review --command knotless-claude-review --jetstream `, where the repositories may be `--owner ` | | `keep` | `knotless-agent keep --jetstream `, where the repositories may be `--owner ` | knotless.dev hands those out, with `--releases knotless://chadtmiller.com/knotless` in docker-compose.yml: each file is read from the owner's PDS once, checked against its record, and kept. A host of your own takes `--releases /data/releases` there instead, for the directory `just release` fills. ## knotless-run Other people's code runs on a host of its own, under gVisor: see [the runner host](runner-host/README.md). What follows is the runner beside serve, for the repositories whose writers you trust. knotless.dev runs none: its checks run on the runner host, and its workflows that hold a secret on [the trusted runner](runner-host/README.md#the-trusted-runner). Set `KNOTLESS_RUN_REPOS` in `deploy/local.env` to the repositories to watch, and on the first deploy `KNOTLESS_RUN_IDENTIFIER` and `KNOTLESS_RUN_PASSWORD`, the account it writes checks in and an app password for it, which are kept on the host in `/etc/knotless/run.env`. The deploy then starts `run` too. The repositories' owners name that account with `knotless runner add ` for their checks to show. An app password reads no private repositories. To watch one, set `KNOTLESS_BOT_DIDS` to the runner's DID (and the triager's, comma apart), deploy, and open `https:///oauth/bot?account=` signed in to that account's PDS, to approve. serve keeps the session in `bots/` under `KNOTLESS_DATA`, which run and triage take as their config directory, and refreshes it there: it lasts two years if used every three months. The password stays in `run.env` but logs in only when there is no session, or one can no longer be refreshed; take it out to make that a failure instead, and log in again then. Steps run in containers on the host's docker daemon, through its socket, so `run` is root on the host in all but name. A step sees only its checkout and the secrets granted to it, but a container is a thin wall: watch only repositories whose writers you trust. The step containers have no limits of their own beyond `--step-timeout` unless `KNOTLESS_RUN_ARGS` gives them (`--cpus`, `--memory`, `--pids`, `--network`, `--run-timeout`: see [the CLI reference](../docs/reference/cli.md)). Give them beside serve: one coverage `test` of knotless's uncapped took about 9 GiB and every core, and two at once stalled the host it shared with serve. A restart, a deploy's among them, waits only for the runs whose steps have started, a deploy's included, up to `--stop-wait` (an hour, so that no run is cut off before its own limits: a release's build of the client takes about 17 minutes), when it cancels those still going, and within `stop_grace_period` (65 minutes) in any case: each run still waiting its turn is dropped, its check filed as cancelled (not deleted: see `drop_run` in knotless-run), and the runner started next runs it, at once, as it was. The checks of a deploy going's own commit, which it waits for, are kept until it ends. Killed or crashed instead, the runner leaves its checks running and its step containers up; the next one ends those checks as errors, for `knotless run rerun` to run again, and removes the containers labeled as its own (`dev.knotless.runner=`), or mounting a path under its `--work` directory, as one from before runs were labeled does, never another's; and every `--sweep` after, those of its own whose run it no longer has. A deploy restarts it only if what it runs changed ([When a deploy restarts the runner](#when-a-deploy-restarts-the-runner)), with `docker compose up -d run` from the host, in the background, logged to `/var/log/knotless-run-restart.log`: a recreate cut short leaves a container named `_knotless-run-1` that is never started, on whose name the next fails, having sent the runner its SIGTERM but never its SIGKILL, so each such leftover is removed first and `up` is tried again, three times in all. It reads every repository again every `--sweep` (600 seconds), for pushes and pulls Jetstream did not ring for. - `KNOTLESS_RUN_DATA` (`/var/lib/knotless-run`): its session, the refs it last ran, checkouts while they run, and the workflows' caches in `caches/`, at the same path inside as on the host, for the step containers to mount. - The caches (`cache:` in a workflow): each path is a base directory, `caches/bases//`, which a run sees through an overlay of its own, a `local` docker volume of type `overlay` mounted with `userxattr`: the base read only below, `caches/runs//` taking what the run writes. So runs of one workflow go side by side, waiting only for one of the `--jobs` (4). Only a push to the default branch that passes keeps what it wrote: its layer goes into a new base, of hard links to the old one's files where it left them, which the runs after it start from, and the old base goes once no run reads it. A push that takes a pull's checks rather than running them (see `KNOTLESS_RUN_ARGS`) takes that run's layers too: an owner's or collaborator's pull run that passes keeps them, in `caches/kept//`, and the push promotes them as its own run's would be, onto the base each was written over. One over a base that has moved on since, as when main ran in between, is not applied, its whiteouts being of what that base held, and the next run of main warms it; the push's log says which. Kept layers go once taken, once their pull is at another head or base, or a pass after it closed, and past three days, sixteen runs, or `--cache-max` GiB for them all, the oldest first; a start removes those of runs no push could take. Any other run's layer, an untrusted pull's always, goes when it ends, and what a stopped runner left goes when it starts again. An untrusted pull of a workflow that asks for secrets starts from an empty base of its own instead, as the default branch's may hold what those secrets left. A base over `--cache-max` GiB (16) once made is dropped, for the next run to start afresh. Disk, then: each base once, each run going's own writes on top (a test run's build, a few GB), and up to `--cache-max` GiB of kept layers. Below `--disk-floor` GiB free (20), a run sheds the bases no run reads and the kept layers before it starts, and waits for room if that is not enough, so the site on the same disk keeps room to write. The host needs overlayfs over ext4 with user xattrs, as Linux 5.11 and up has. The docker daemon mounts the overlays by the host's paths, one more reason `KNOTLESS_RUN_DATA` is the same path inside and out. The first start after the switch from volumes removes the old `knotless-cache-*` volumes, and runs start cold until each workflow's first run on the default branch passes. - `KNOTLESS_RUN_ARGS`: more of its command line, such as `--pulls`, to run open pulls too, which it finds through `serve`: each at its head merged with its base, those by the owner or a collaborator as they move, anyone else's once the owner asks with `knotless run rerun --pull`. A push to the default branch at the tree an owner's or collaborator's pull passed at, as merging it leaves the branch, takes those checks as passed rather than running them again, its log saying whose they were; a workflow that asks for secrets, or whose runs finish, as deploy.yml, runs as ever. `--queue` also lands the merge queue (`knotless pull merge --queue`): up to `--batch` (8) queued pulls at a time, each squashed onto the branch as its own commit, checked once at the batch's tip. While one is checked the next is checked on its tip (`--depth 1` turns that off): it runs as a pull's run does, after main's runs and with a pull's CPU share, is dropped if the batch it is on fails or changes, and lands only on the commit that batch landed as. It reads the queue records from the repos of the owner and collaborators, not only from `serve`'s index, and looks at them every `--notice` (30) seconds besides Jetstream, so a pull queued starts within that. It pushes as the runner's account, as the owner or a collaborator who may write (`knotless repo share --write`), or, as a runner the repository names when none it names may write, through the gateway, once the owner delegates writes there ([Who lands a queue](../docs/design/merge-queue.md#who-lands-a-queue)). While it lands the queue, open pulls not in it run again only as their heads move, after the queue's batches. - `/etc/knotless/run-secrets.json`: the secrets it grants, by `/`, as `{"alice.example.com/tally": {"DEPLOY_TOKEN": "…"}}`, or `{"DEPLOY_TOKEN": {"value": "…", "refs": ["refs/heads/main"]}}` for pushes to `main` alone. It starts empty; edit it on the host, and the next run reads it. ### When a deploy restarts the runner A restart costs little since the runner hands over ([How the runner hands over](#how-the-runner-hands-over)), but the new runner still waits for the jobs the old one's runs hold, so a deploy restarts `run` only when what it runs changed. `deploy.sh up` compares two things, and leaves `run` running only when it knows both are the same: - **Its build.** The image keeps a fingerprint of what the runner is made of, in `/etc/knotless-run.fingerprint`: the `knotless-run` binary, the docker client its steps run through, and the packages installed (`/var/lib/dpkg/status`: git and the libraries). `up` reads it from the running container and from `knotless:current`, the image just built. The commit the binaries log is set as they run (`ENV KNOTLESS_COMMIT`, the image's last layer) rather than built in, so a commit that changes only serve, the pages, another binary or the docs leaves the runner's binary, and so its fingerprint, as it was. - **Its settings.** A hash of the `run` service as `docker compose config run` resolves it from the compose file and the `.env` this deploy just wrote (its command line, `KNOTLESS_RUN_ARGS` and `KNOTLESS_RUN_REPOS` among it, its environment and mounts), and of `run.env`. Each restart removes `/etc/knotless/run-deployed` before it starts and, once the new container is up, writes ` ` there. A deploy whose hash, or whose running container, is not the one recorded restarts it. `run-secrets.json` is not in it: the runner reads that at each run. Anything it cannot tell, a container not running, no fingerprint, a record missing, restarts `run` as every deploy once did; so the first deploy with this does, and so does one while a restart is still under way. Either way the deploy log says which, and why: ```text 2026-10-27T04:02:11Z 1a2b3c4d5e6f → run: its build, 5d41402abc4b, and settings are the same, not restarted 2026-10-27T05:13:40Z 7d8e9f0a1b2c → run: restarting: its build changed, 5d41402abc4b to 7d793037a076 ``` With `KNOTLESS_RUN_REPOS` empty, a deploy that finds `run` running, as an earlier deploy with repositories to watch left it, stops and removes it, once its own run is filed if it runs under it, as a restart would, and starts no other: `run: removing: no KNOTLESS_RUN_REPOS` in the deploy log. A runner left running keeps logging the commit it was built from at its start, as it should. To restart it anyway, deploy with `KNOTLESS_RUN_RESTART=1`, or on the host `cd /etc/knotless && docker compose up -d --force-recreate run`, after which the next deploy, finding no record for that container, restarts it once more. To ask the first question by hand, on the host: `docker exec cat /etc/knotless-run.fingerprint` beside `docker run --rm --entrypoint cat knotless:current /etc/knotless-run.fingerprint`. ### How the runner hands over A restart does not stop the runner and then start the new one. `handover.sh` sends the runner leading SIGTERM, and stops docker restarting it. The runner drops the runs waiting that it will not run, writes its state file a last time, and from then on writes a note of its own under `state.handover/` beside it. It goes on finishing the runs it has going: those started, a merge-queue batch's, and those at a going deploy's commit. Once the note is there, `handover.sh` starts the new runner beside it (`docker compose up --scale`). The new one reads the state file, leaves the old one's runs, checks and caches alone, and takes no job the old one's runs hold. The old one exits once its runs end, `--stop-wait` (an hour) after its stop at most, and is removed then. It holds only the jobs the new one lends it meanwhile. The deploy log says how it went: ```text 2026-11-03T18:02:11Z → run: handed over in 2s; starting the new one beside 4f1c2a9b0d3e 2026-11-03T18:02:14Z → run: started 9a8b7c6d5e4f ``` A runner that writes no note within `KNOTLESS_HANDOVER_WAIT` seconds (60), one built before runners handed over, is stopped first instead, and the new one started once it has, as every deploy once did. While one is still finishing, `docker compose ps run` lists two: the one docker restarts (`docker inspect -f '{{.HostConfig.RestartPolicy.Name}}'` says `always`) leads, and a deploy compares only that one. ## knotless-triage Set `KNOTLESS_TRIAGE_REPOS` to the repositories to triage and `KNOTLESS_TRIAGE_ARGS` to the rest of its command line (the deploy workflow gives `--model clef --scopes kind,area --discover`: with `--discover` it also triages every repository that names its account a triager, as the runner host runs those naming it, so an owner turns it on with `knotless triager add kind area` alone). It starts once `/etc/knotless/triage.env` is on the host, which a deploy from one's own checkout writes, once, from `deploy/local.env`: - `KNOTLESS_TRIAGE_IDENTIFIER` and `KNOTLESS_TRIAGE_PASSWORD`: the account its labels are written in, and an app password for it. Not the owner's: an owner's labels are a person's, which outrank a triager's and count at once. For private repositories it logs in at `/oauth/bot` as the runner does. - `KNOTLESS_DECIDE_URL`: the decision model's endpoint, for Clef `https://api.cloudflare.com/client/v4/accounts//ai/run/@cf/cloudflare/clef`. - `KNOTLESS_DECISION_TOKEN`: its token, for Clef a Cloudflare API token that may use Workers AI. Until then a deploy leaves it out and says so. Its labels count for nothing until the owner names its account a triager of some scopes, with `knotless triager add kind area`; `knotless triager list eval ` says first how often they agree with people's. It logs each issue it judged, or failed to, as `judged` or `judging failed`. ## The keeper `knotless-agent keep` brings the pulls the merge queue turned away for a conflict up to date and queues them again ([the merge queue](../docs/design/merge-queue.md#keeping-turned-away-pulls-moving)). Set `KNOTLESS_KEEP_REPOS` to the repositories it keeps, or `--owner ` for all of an account's, and `KNOTLESS_KEEP_ARGS` to the rest of its command line (`--attempts 2`, say). It starts once `/etc/knotless/keep.env` is on the host, which a deploy from one's own checkout writes, once, from `KNOTLESS_KEEP_IDENTIFIER` and `KNOTLESS_KEEP_PASSWORD` in `deploy/local.env`: its account, and an app password for it. The account is its own, not the owner's, and each repository it keeps names it a collaborator (`knotless repo share --write`), as it pushes to their pulls' branches and queues them. Until then a deploy leaves it out and says so. It logs each pull it kept, or failed to, as `kept` or `keeping failed`. ## Monitoring `/healthz` says only that serve answers, and Docker restarts it when it does not. `/_status` says whether it is doing its job, for a monitor off the host to poll: 200 when it is, 503 and what failed when not. - **Feeds**: each one followed (Jetstream, the firehose) is connected and brought an event in the last two minutes (`--stale ` to change it). The firehose brings hundreds a second; Jetstream, asked only for `dev.knotless.*`, still brings every account's identity and account events, one every few seconds, eleven at the longest gap measured. With both followed, one bringing events is enough, and each is still shown. - **Index**: it answers a read. - **A mirror**, with `?repo=/`: its default branch's commit and tree are read as its code page reads them, so a mirror whose packs can no longer be read (as when one outgrew its open handle's pack slots) fails here, and its pack count shows. - **Disk**: the one serve's cache, and with it the mirrors, is on (`KNOTLESS_CACHE_DIR`, `/data/cache`). Under 15% free it is `low`, a problem told of while the answer stays 200; under 5% it is `full` and the answer is 503. On 2026-10-31 that disk filled and every page that needed a mirror answered 502 (`No space left on device`); 15% of its 150 GB is a day or more of deploys and runs to clear it in. A keyword monitor (step 4 below) shows a `low` disk before it fails. - **Code**, with `--code-index`: Zoekt's webserver answers, and the repositories, shards and bytes its index holds show. A webserver that died fails here, as every code search then fails. It takes milliseconds, so polling it every minute costs nothing. A dead upstream feed is not a reason to restart serve, so the compose healthcheck stays on `/healthz`. ```sh knotless status https://knotless.dev --repo chadtmiller.com/knotless # or --json curl -s 'https://knotless.dev/_status?repo=chadtmiller.com/knotless' | jq ``` ```json { "ok": true, "problems": ["jetstream: not connected"], "uptimeSeconds": 5321, "feeds": [ {"name": "firehose", "url": "wss://relay1.us-east.bsky.network", "ok": true, "connected": true, "silentSeconds": 0, "staleAfterSeconds": 120, "events": 2103344, "seq": 26329004819}, {"name": "jetstream", "url": "wss://jetstream.us-east.bsky.network", "ok": false, "connected": false, "silentSeconds": 7310, "staleAfterSeconds": 120, "events": 1650, "seq": 26328850636} ], "index": {"ok": true, "error": null}, "mirror": {"repo": "chadtmiller.com/knotless", "ok": true, "error": null, "commit": "b1fbf45…", "packs": 3}, "disk": {"path": "/data/cache", "state": "ok", "ok": true, "freeBytes": 41203998720, "totalBytes": 160982630400, "error": null} } ``` ### An external monitor The check has to run somewhere other than the host, or the host going down goes unreported. An external uptime service is the least to run: nothing to keep alive, and it pages a phone. UptimeRobot's free plan is enough (Better Stack or an Uptime Kuma elsewhere do the same): 1. Sign up at uptimerobot.com and install its app, or add an email or Slack contact under *Integrations & API*. 2. *New monitor*: type HTTP(s), URL `https://knotless.dev/_status?repo=chadtmiller.com/knotless`, interval 5 minutes (the shortest the free plan has), timeout 30 seconds. Any status but 2xx is down, so a 503 alerts. Name it "knotless status". 3. *New monitor*: HTTP(s), `https://knotless.dev/`, 5 minutes: the pages, which `/_status` does not draw. 4. Optionally a keyword monitor on the status URL for `"ok":true`, which shows the body's `problems` in the alert. 5. Pick the contact under each monitor's *Alert contacts*, and *Notify after* 1 check, or 2 to ride out a deploy's restart. A 503 then reaches the phone within the interval plus two minutes of a feed going silent, or at the first poll after a mirror stops reading. ## Logs serve and the runner log JSON lines on stderr, one an event, at the level `RUST_LOG` sets (`KNOTLESS_LOG` in `.env`, `info` unset); the pages log what node prints. Docker writes each container's to the host's journal (the `journald` driver, `x-logging` in docker-compose.yml), so they outlive the container: a rollout removes the old one, and its lines stay, for 30 days: setup.sh sets journald's `MaxRetentionSec`, which the privacy policy promises (journald's own cap, a tenth of the disk and at most 4 GB, can drop the oldest sooner). Each line names its project and service (`COM_DOCKER_COMPOSE_PROJECT`, `COM_DOCKER_COMPOSE_SERVICE`) and its container (`CONTAINER_TAG`). deploy.sh and rollout.sh add a line with the time for each step of a deploy (built, started beside the old one, healthy, the old one stopped) to `/var/log/knotless-deploy.log` on the host, to line a freeze up against. [docs/observability.md](../docs/observability.md) says what was chosen, and why. What serve logs: - `request`, one a request: `http.route` (its template), `url.path` (no query), `http.request.method`, `user_agent.original` (the client as it names itself), `http.response.status_code`, `http.response.body.size` when known, `duration_ms` and `trace_id`, at info; at warn when over `KNOTLESS_SLOW_MS` (1000), unless it is a long poll (`awaitSpaceWrites`, `getDelegation`), which waits on purpose; at error for a 5xx with its `error`, and `/healthz` at debug. The answer carries the trace as `x-trace-id`; the pages hand it back on their XRPC calls, so a page and the data it drew share one. - `outgoing`, each call made, to a PDS, the PLC directory, Bluesky's CDN or a handle's `/.well-known`: `server.address`, `url.path`, `rpc.method` for XRPC, `http.response.status_code`, `error.type`, `http.request.resend_count`, `duration_ms`, `trace_id`. At debug, or warn when it failed, answered 5xx or was slow. It sends `traceparent`. - `sql`, each of the index's statements that took a tenth of `KNOTLESS_SLOW_MS` (100 ms) or more, at warn: `db.query.text` (the SQL as written, its whitespace closed up, cut at 500 characters, with no value it was given) and `duration_ms`, with the `trace_id` of the request it ran in. The slow request whose line shares that trace is the one it made slow. - `summary: requests`, `summary: outgoing` (by host), `summary: cache` (`hits`, `misses`, `failed` fills, by cache) and `summary: feed` (lag, events) each minute, for what saw any. - `mcp tool failed` at warn, one a tool call that failed: `gen_ai.tool.name`, `did`, `error` (what the tool answered, as the client saw it), `duration_ms` and `trace_id`; `mcp resource failed` the same, of a resource read, named for the tool that answers it. - `mirror sync`, one each sync of a repository's mirror: `repo`, `duration_ms`, and `records_ms`, the part spent reading its records. At info, or warn when slow. Reads of the repository wait on a sync under way, so it explains a burst of slow requests for one repository that end together. - `cache fill failed` at warn, with why: an avatar, claim or picture that could not be read. - `knotless-serve starting` (`version`, `commit`), `index opened` (`duration_ms`, `rebuilt`), `listening`, `stopping`, `stopped`; `feed connected` and `feed disconnected`; `index sweep started` and `ended`; `event dropped: not indexed`; `account not backfilled`; `panicked`. What the runner logs: `run queued`, `run started`, `run ended` (`status`, `duration_ms`), `run cancelled before it started`, each with `repo`, `workflow`, `ref_name`, `commit` and `pull`; `step started` and `step ended`; `container started` and `stopped`; `batch started`, `landed`, `dropped` and `pull ejected` (`reason`); `queue noticed` (debug); `pds write retried`; `feed connected` or `down`; `failed` for anything that went wrong. None logs a header, a token or a body. A request line logs its query (`url.query`), each value a secret could be in replaced by `_`, and none for `/oauth`; no call made logs one. DIDs are logged; handles only as a path or a `/.well-known` host names them; client IPs never. `deploy/logs.sh` (`just logs`) reads them from here, every container of a service at once, those gone too, through jq: ```sh just logs serve 'select(.duration_ms > 1000)' # slow requests and calls just logs serve --since 2h 'select(.level == "WARN" or .level == "ERROR")' just logs serve 'select(.message == "outgoing" and .level != "DEBUG")' # calls that failed just logs serve 'select(.message | startswith("summary"))' just logs serve 'select(.trace_id == "4bf92f3577b34da6a3ce929d0e0e4736")' # one request, and the pages' calls for it just logs run --follow 'select(.message | startswith("run"))' just logs runner --since 1h 'select(.message == "batch started")' # the runner host's, which lands the queue just logs deploys # the deploy steps, with times ``` `run` is a runner on this host. `runner` reads the runner host's (deploy/runner-host) instead, at `KNOTLESS_RUNNER_HOST` (runner.knotless.dev unset), by its container's tag, as that host's lines name no compose project. `KNOTLESS_RUNNER_ENV=deploy/ci.env just logs runner` reads another runner, such as the trusted one, from its own settings, as its deploy does. Or on the host itself, with journalctl, by the service's field; `-o cat` gives each line as it was written: ```sh ssh root@knotless.dev 'journalctl -o cat --since -30m COM_DOCKER_COMPOSE_PROJECT=knotless COM_DOCKER_COMPOSE_SERVICE=serve' | jq -Rc 'fromjson? | select(.duration_ms > 1000)' ssh root@knotless.dev 'journalctl -o cat --since -1h COM_DOCKER_COMPOSE_SERVICE=serve' | jq -Rc 'fromjson? | select(.message == "outgoing") | {t: .timestamp, host: ."server.address", path: ."url.path", status: ."http.response.status_code", error: ."error.type", ms: .duration_ms}' ssh root@knotless.dev tail -n 50 /var/log/knotless-deploy.log ``` Each request's line names its trace; a page's own line comes after those of the XRPC calls it made, since a line is written when its request ends. Calls are logged only at warn by default; to see every one for a while without a restart, set the level through `--metrics`' port, inside the container: ```sh ssh root@knotless.dev 'for c in $(docker ps -q -f label=com.docker.compose.project=knotless -f label=com.docker.compose.service=serve); do docker exec $c node -e "fetch(\"http://127.0.0.1:9464/log\", {method: \"PUT\", body: \"info,knotless::outgoing=debug\"}).then(r => r.text()).then(console.log)"; done' ssh root@knotless.dev 'docker exec $(docker ps -q -f label=com.docker.compose.project=knotless -f label=com.docker.compose.service=serve | head -1) node -e "fetch(\"http://127.0.0.1:9464/metrics\").then(r => r.text()).then(console.log)"' | grep -v '^#' ``` `PUT` it `info` again after; a restart goes back to `KNOTLESS_LOG`. ### Without SSH serve keeps each line it logs in `/data/logs.db` too (`--logs`), as it writes it to stderr, a second at a time. Every serve writes there, both during a rollout, so a deploy loses none. An operator reads them with `knotless admin logs` or MCP's `logs` tool, from `dev.knotless.getLogs`, without SSH: ```sh knotless admin logs --trace 4bf92f3577b34da6a3ce929d0e0e4736 # one request, and the calls it made knotless admin logs --message sql --limit 20 # the last slow statements knotless admin logs --level warn --min-ms 2000 # what was slow, and warned ``` A read needs one of `--trace`, `--message`, `--level` or `--min-ms`, answers at most 500 lines, and is itself logged as `logs read`, with who asked and for what. Lines at info and below, the most of them, are kept to the last 500,000 (`--logs-keep`); every line for 30 days at most, as the journal keeps them. They hold what the journal does, no more. `/metrics` gives what was counted since the start, in Prometheus's format: `http_server_request_duration_seconds` by route, method and status, `http_client_request_duration_seconds` by host and outcome, `knotless_cache_lookups_total` by cache and result, `knotless_cache_build_duration_seconds` by cache and how a miss was made good (`patched` or `rebuilt`), and each feed's `knotless_feed_connected`, `knotless_feed_events_total` and `knotless_feed_lag_seconds`.