Deploying knotless-serve

One Docker host, a reverse proxy in front of it for TLS, and a domain.

../Dockerfile	one image, knotless:current: knotless-serve, knotless-run, knotless-triage, knotless-agent, knotless-mcp with Claude Code, the knotless CLI, git, the docker client and the built pages
docker-compose.yml	/etc/knotless/docker-compose.yml: serve (knotless-serve, on :2590), pages (the SvelteKit app, on :2591) and, if asked, run (knotless-run), triage (knotless-triage), review (knotless-agent review) and keep (knotless-agent keep)
deploy.sh	builds the image on the host over SSH, writes the compose file and its .env, and restarts what changed; or one of those steps, build, up or drop
rollout.sh	run on the host by deploy.sh: moves pages and serve to a new container each, with no gap
handover.sh	run on the host by deploy.sh: moves run to a new container, the old one handing over to it as it finishes its runs
rollout-test.sh	tries rollout.sh on a local docker behind a Caddy, counting failed requests
release.sh	builds the client for macOS and Linux here and puts the builds on the host, for /install.sh
setup.sh	makes a fresh Ubuntu 24.04 host ready for deploy.sh: docker, a Caddy of its own for the domain, a firewall, swap, upgrades and pruning

deploy/setup.sh root@host git.example.com  # once, on a fresh host
just deploy root@host git.example.com   # build on the host and restart; or put both in deploy/local.env

The domain needs an A (and AAAA) record for the host, and an _atproto TXT record for each of the service's own accounts with a handle under it. knotless.dev's are kept by the stacks/edge Terraform in the infra repository, beside stacks/knotless, which makes the host. A worker that serves an account's handle at its own name, as the reviewer's pds.js does at review.knotless.dev, holds that name as a custom domain in its wrangler.toml, which makes the record: the infra repository keeps only its _atproto TXT.

From a push to main to the site running it

.knotless/workflows/deploy.yml deploys each push to main once its seven checks (check, test, web, lexicons, shots, docs and e2e) pass at it. It names them in after, so it waits for them holding none of the runner's jobs, and the checks it waits for are never kept from one. One that fails files the deploy failed without running. Then:

1. build: deploy.sh build builds knotless:build-<commit>, a candidate, which nothing runs: knotless:current and the containers stay as they are. It removes any candidate an earlier deploy left.
2. deploy: if main is still at this commit, deploy.sh up tags the candidate knotless:current, removes the candidate's tag, and rolls out as below, restarting run only if what it runs changed (When a deploy restarts the runner), then removes the untagged images no container runs, the ones earlier deploys replaced.

Until it goes up, a deploy gives way: when main moves past its commit, asked every 30s through the build, it stops (a build is cancelled with its client, its cargo cache kept for the next), removes its candidate and passes, and the newer commit's deploy, which waits for it (superseded: finish), starts. Once it goes up it runs to its end.

So a push goes live a few minutes after its checks pass: the build, two minutes or so for a change to our crates in the deploy profile, then the rollout, about 30s.

The site comes first on the host for CPU, then a deploy and main's checks, then pulls' runs. serve and pages have cpu_shares of 4096 (a cpu.weight of about 157); the build runs in the cgroup /system.slice/knotless-build, which deploy.sh build makes with a cpu.weight of 100, as main's runs have, where docker drives cgroups (v2) with systemd; and knotless-run starts a pull's run with --cpu-shares 256, a weight of 10, so a burst of pulls takes only what the rest leave. Before, the build had 20 against the pulls' 100, and a merge that ran every open pull again held the deploy back for minutes. Docker's buildkit puts each build step in a cgroup of its own, and by default beside the containers', where nice inside it would count for nothing against them. KNOTLESS_BUILD_CGROUP names another parent, or set empty, none.

The image's binaries are built in the deploy profile (Cargo.toml): release's, but in 16 codegen units without LTO across crates, so the crates a push rebuilds, knotless-serve the longest, use every core. The cargo registry and target/ are cache mounts on the host's builder, so a deploy rebuilds only our four crates, not their dependencies; a fresh checkout's files are all new to cargo, so all four, whichever changed. docker buildx du --filter type=exec.cachemount on the host shows them. Each build then prunes buildkit's cache to the 20 GB used last, which keeps those mounts; uncapped, it held 49 GB when the disk filled.

By hand, just deploy does all three steps at once, as before.

The proxy joins the knotless docker network (KNOTLESS_PROXY_NETWORK) and sends the domain to knotless-serve:2590, passing X-Forwarded-Proto. With Caddy:

git.example.com {
	reverse_proxy knotless-serve:2590 {
		# Holds a request that meets no listening serve, for a moment
		# within a deploy, and tries again rather than answering 502.
		lb_try_duration 20s
		lb_try_interval 250ms
	}
}

Push notifications

The iOS app's notifications go through Apple's push service, with a key from the Apple Developer account of the team that signs the app. Put it on the host once, where the server can read it:

install -o 1000 -g 1000 -m 600 AuthKey_XXXXXXXXXX.p8 "$KNOTLESS_DATA/apns.p8"

Then set KNOTLESS_APNS_KEY_ID and KNOTLESS_APNS_TEAM and deploy. deploy.sh gives serve --apns-key only when the file is there; without it, serve runs as ever and sends nothing. Anyone holding the key can send to any app of that team, so it stays on this host and in the encrypted secrets it came from.

Deploys without downtime

deploy.sh moves pages and then serve with rollout.sh: a second container starts beside the running one, with the same names on the same networks, and once its healthcheck passes the old one is stopped, finishing the requests it holds (SIGTERM, then stop_grace_period), and removed. A new one that is not healthy in 180s (KNOTLESS_ROLLOUT_TIMEOUT) is removed and the old one keeps serving; the deploy fails. run is restarted last, and only if what it runs changed (When a deploy restarts the runner).

While both run, docker's DNS gives knotless-serve, serve and pages as every running container's address. The proxy, serve and the pages each dial every address in turn, so one not listening yet, or no longer, is refused and the next is tried. lb_try_duration is the proxy's belt and braces over that: it holds a request that found none and tries again.

Both serves share KNOTLESS_DATA for those seconds. The index is SQLite in WAL mode, each process waiting up to 5s for the other's writes; mirrors are written under lock files and renames, as concurrent processes do already; both follow the streams and write the cursor, and a record indexed twice is the same row. The one unsafe overlap is a build that changes the index's tables: it drops and rebuilds them under the old one. So rollout.sh first asks the new build (knotless-serve --check-index), and if it would rebuild the index, serve is stopped and then started instead: down until the new one listens, a second or two, which the proxy's lb_try_duration holds requests through.

deploy/rollout-test.sh tries all of this on a local docker, behind a Caddy, counting failed requests across the rollouts.

The proxy

knotless.dev has a host of its own (Hetzner knotless, a cx43, made by the stacks/knotless Terraform in the infra repository), apart from the sites it once shared one with: its runs and builds took the cores they needed. setup.sh gives it Caddy in /etc/knotless-proxy, which dials knotless-serve:2590 by name over the knotless network and holds a request through a rollout's second or two with no serve:

knotless.dev {
	request_body {
		max_size 100MB
	}

	reverse_proxy knotless-serve:2590 {
		lb_try_duration 20s
		lb_try_interval 250ms
	}

	encode gzip zstd
}

With KNOTLESS_PAGES_DOMAIN (knotless.page), the proxy answers repositories' sites too (../docs/design/pages.md): the domain and every owner's label under it, and each private site's label under private. there, on wildcard certificates from the DNS challenge. The zone's * record covers *.private too, as nothing is at private itself. That takes Caddy built with the caddy-dns/cloudflare module, which setup.sh builds from a Dockerfile beside the compose file, and the zone's own Cloudflare token, Zone:Read and DNS:Edit on that zone alone. Give the token once, as KNOTLESS_PAGES_DNS_TOKEN in deploy/local.env, and run setup.sh: it keeps it in /etc/knotless-proxy/cloudflare.env, and a run without it keeps that one. A run with nothing changed rebuilds and recreates nothing.

knotless.page, *.knotless.page, *.private.knotless.page {
	tls {
		dns cloudflare {env.CLOUDFLARE_API_TOKEN}
	}

	reverse_proxy knotless-serve:2590 {
		lb_try_duration 20s
		lb_try_interval 250ms
	}

	encode gzip zstd
}

deploy.yml sets KNOTLESS_PAGES_DOMAIN for serve's --pages-domain. The zone's DNS (its apex and *, DNS-only, at this host) is in the infra repository's stacks/edge.

On a host whose proxy is another's, join that proxy's network, named knotless (KNOTLESS_PROXY_NETWORK), and give the domain that block in its Caddyfile instead.

knotless-serve is the appview (dev.knotless.listRepos, getForge, getFeed, indexed from Jetstream), the gateway, as did:web:<domain>, and the moderation service. Its state is in KNOTLESS_DATA (/var/lib/knotless):

• index.db, the index: a cache. A version that changes its tables rebuilds it, and it refills itself at start from the accounts the relay (KNOTLESS_RELAY, relay1.us-east.bsky.network) lists as holding knotless records, and again every ten minutes, for any event Jetstream missed.
• cache/, mirrors and identities: a cache.
• usage.db, how it is used (see Usage): counts to keep, but nothing breaks without them.
• config/, the gateway's OAuth client key and the sessions owners delegated to it. Losing it breaks every delegation, which owners then have to make again: back it up.
• bots/, the sessions run, triage and review act with, once they log in at /oauth/bot (see knotless-run): lost, they log in again.

Keep the records DNS-only: PDSes fetch the gateway's DID document and OAuth client metadata from the origin, and the proxy holds the certificate.

Usage

knotless admin stats (--days <n>, 14 by default; --json) shows the operator how the service is used, from dev.knotless.getUsage, to the accounts KNOTLESS_ADMIN_DIDS names (set in .knotless/workflows/deploy.yml), signed in as one of them:

KNOTLESS_APPVIEW=https://knotless.dev knotless admin stats

• Each day: the people who visited the pages and the pages they looked at; the requests serve took and how many it failed; and, from the index, what the network wrote: repositories, pushes, pulls, issues, comments and checks, and how many accounts wrote them.
• The network: every account that has made a repository, pushed or written in a thread, and those that did in the last 7 and 30 days.
• Routes: requests per route as axum matched it (the pattern, never the path), by surface (pages, api, git, xrpc, mcp, install, other), with failures and times. The time is to the answer's head, so a long clone counts its start.
• Pages, referrers and clients: the routes people looked at, the sites that sent them, and requests by kind of client (browser, git, the cli, bot, other, from the User-Agent), and pages, the queries the pages ask as they draw a page (knotless-pages), which are not a client of their own.
• Events, what the handlers saw, with how often each failed, its mean time and the accounts that did it, in the days asked but at most the last 30: mcp.tool by tool, mcp.resource by the tool that answers the read, and mcp.signin; web.login; git.fetch by repository, once per clone or fetch, as asked for (<handle or did>/<name>); cli.version by the CLI's version; gateway.delegated, gateway.push, gateway.commit, and gateway.served, by the owner the gateway pushed or committed for.
• Funnels: of the accounts that delegated to the gateway in the days asked, not having in the 30 days before, how many the gateway has written for then or since, by anyone's push or commit that moved a ref; of those that signed in to MCP so, how many have called a tool. A day older than 30 is counted as it stood once 30 days had passed.
• Hours: each hour's requests, how many failed and how many took over two seconds, to know when to read the logs of. The lines show only the hours with either; --json has every hour.
• Statements (sql): the index's statements that took 100 ms or more, as the sql log line has them (the SQL as written, no value it was given), with how often, their mean and their longest, the most time first: the fifty that cost most.
• Failures: why an event failed, as it answered, most often first: so far an MCP tool's error. A DID, an at:// URI, a hash and a number in it are put as <did>, <uri>, <hex> and <n>, so like failures count together. A handle or a repository's name in it is kept as written.
• Agents: requests by the User-Agent of each crawler or script, the bot client, most first: never a browser's.

Statements, failures and agents keep at most 500 of each a hour apart; past that, the new ones count together as (others).

What is kept, in usage.db, is counts by the hour, and no address; and, for the pages' logins, MCP and the gateway, where people are signed in already, which accounts (by DID) did what each day, for the events' accounts and the funnels; and the statements, failures and crawlers' agents above, by the hour. Those are kept 30 days, today the last: on the 30th day after, serve folds each day into its funnels' counts, how many started and how many went on, with no one in them, and deletes them, and the hours of the rest. A page, once drawn in a browser, sends /_hit its route and the site the visitor came from. Each visitor is counted once a day by a hash of their address, less its last byte (its last 80 bits for IPv6), and their User-Agent, salted with a key made for the day. Two serves run at once for a moment in each deploy, so the key is kept in the database for both, and deleted with the day's hashes once the day (UTC) is over, leaving the day's count: then nothing links a visitor to an address, or to the next day. Crawlers, by their User-Agent or by not running the page, and browsers that ask not to be followed (Sec-GPC: 1), are not counted.

Anyone can read the days and the network, but not the requests, routes, events or funnels: the /stats page shows them, from dev.knotless.getStats.

Requests are counted in memory and written every 15 seconds and as serve stops; one killed loses those seconds.

It reads like this (--json gives the same as knotless::usage::Report):

network	212 accounts, 340 repositories, 41 authors this week, 97 this month
day	visitors	views	requests	errors	repos	pushes	pulls	issues	comments	checks	authors
2026-10-22	118	431	20544	3	4	96	7	5	22	61	19
2026-10-23	131	502	22310	0	2	104	9	3	31	70	23
route	git	POST /{authority}/{repo}/git-upload-pack	1840 requests, 0 failed, 210ms mean, 96 over 0.5s, 4 over 2s
route	xrpc	GET /xrpc/dev.knotless.getRepo	1203 requests, 2 failed, 48ms mean, 11 over 0.5s, 0 over 2s
event	cli.version	0.1.0	2210 calls, 0 failed, 0ms mean, 0 accounts
event	git.fetch	alice.test/proj	140 calls, 0 failed, 0ms mean, 0 accounts
event	mcp.tool	repo_read	612 calls, 9 failed, 85ms mean, 6 accounts
hour	2026-10-23T14	1022 requests, 3 failed, 2 over 2s
funnel	gateway.delegated -> gateway.served	4 started, 3 reached
funnel	mcp.signin -> mcp.tool	7 started, 6 reached
page	/[authority]/[repo]	388
referrer	bsky.app	52
client	browser	9120
client	bot	7310

So it answers, among others:

• Growth: how many accounts write, and how many did this week and month; repositories, pushes, pulls, issues and checks a day, and whether a day's pushes are many people or a few (authors beside pushes). From the index, so the days before --usage too.
• The pages: visitors and views a day, which pages they read, and which sites sent them.
• The surfaces: requests by surface and by kind of client, so how much is git, the CLI, MCP, the installer, or crawlers.
• MCP: which tools are called, how often they fail, by how many accounts, and how many who sign in go on to use it.
• Repositories and releases: which repositories are cloned and fetched most, and which CLI versions are still about.
• The gateway: how many who delegate to it go on to push.
• Health: which routes fail or are slow, and a deploy that made either worse, day against day; and the hours it happened in. A long poll is never counted slow.

What it does not count:

• Anyone across days: a visitor's hash is new each day, so there are visitors a day but none a week or returning; weekly and monthly figures are of accounts that write, from the index.
• Any account past 30 days: an event's accounts over more days are of the last 30, and a funnel cannot tell an account that comes back after a month from a new one.
• What a route was asked of, past the events: routes are counted by pattern, so the API's reads of every repository are one row.
• Installs: the installer and the releases are counted as requests, by no one; the funnels start at a delegation or a sign-in.

Releases

The server hands out the client, so installing it needs no clone and no Rust: curl -fsSL https://knotless.dev/install.sh | sh installs knotless, git-remote-knotless, knotless-mcp, knotless-run, knotless-triage, knotless-agent and knotless-claude-review into ~/.local/bin (KNOTLESS_INSTALL_DIR), the latest release or KNOTLESS_VERSION's, checked against its sha256.

just release root@host        # or with KNOTLESS_HOST in deploy/local.env

builds the workspace's version, from Cargo.toml, for macOS and Linux (aarch64 and x86_64; Linux static, with musl) on a Mac, with deploy/build-client.sh, puts them in releases/ under KNOTLESS_DATA, and makes it the latest. The Linux builds need brew install zig and cargo install cargo-zigbuild. Bump the version before a release: one already out is replaced in place.

Or, as the releases of the repository's knotless package (packages): the release workflow publishes each new version once main's checks pass at it. It publishes as the owner, with RELEASE_PASSWORD, an app password of theirs, which the runner's secrets file grants to pushes to main:

{
  "chadtmiller.com/knotless": {
    "RELEASE_PASSWORD": { "value": "…", "refs": ["refs/heads/main"] }
  }
}

The deploy publishes the image it runs the same way, as the release of the knotless-server package at that version, once it is up, and needs RELEASE_PASSWORD for it too; a version with that release already is left alone.

The publish-lexicons workflow publishes the lexicons once main's checks pass at a commit that changes them, with the shared publish-lexicons step from chadtmiller.com/steps (its README lists its inputs): a _lexicon.<authority> TXT record naming lexicons.knotless.dev for each authority (_lexicon.knotless.dev for dev.knotless.*), and a com.atproto.lexicon.schema record in that account for each lexicon. It writes what is new or changed and deletes what main no longer has. lexicons.knotless.dev is an account of its own, on a pds.js worker as the runner's is, so what CI holds can write lexicons and nothing else. It writes as lexicons.knotless.dev with LEX_PASSWORD, an app password of that account's, which the secrets file grants to pushes to main. It reads the DNS as resolvers see it, and changes it only with CLOUDFLARE_API_TOKEN, a token with Zone:DNS:Edit on the zone allowed from this host's address. That is needed only when an authority is added or dropped, and is not granted now: such a run writes the schemas, then fails, naming the record to make. Every run first plans, saying what it would change, and knotless run rerun <repo> --ref refs/heads/<branch> --workflow publish-lexicons plans on a branch.

One image runs every service, each by its command, as docker-compose.yml starts them:

docker pull knotless.dev/chadtmiller.com/knotless/knotless-server

Service	Command
serve	knotless-serve --listen 0.0.0.0:2590 …, the image's default
pages	node /app/build
run	knotless-run --secrets … --state … --work … --caches …
triage	knotless-triage --decide <url> --jetstream <url> <repos>
review	knotless-agent review --command knotless-claude-review --jetstream <url> <repos>, where the repositories may be --owner <handle>
keep	knotless-agent keep --jetstream <url> <repos>, where the repositories may be --owner <handle>

knotless.dev hands those out, with --releases knotless://chadtmiller.com/knotless in docker-compose.yml: each file is read from the owner's PDS once, checked against its record, and kept. A host of your own takes --releases /data/releases there instead, for the directory just release fills.

knotless-run

Other people's code runs on a host of its own, under gVisor: see the runner host. What follows is the runner beside serve, for the repositories whose writers you trust. knotless.dev runs none: its checks run on the runner host, and its workflows that hold a secret on the trusted runner.

Set KNOTLESS_RUN_REPOS in deploy/local.env to the repositories to watch, and on the first deploy KNOTLESS_RUN_IDENTIFIER and KNOTLESS_RUN_PASSWORD, the account it writes checks in and an app password for it, which are kept on the host in /etc/knotless/run.env. The deploy then starts run too. The repositories' owners name that account with knotless runner add <url> <handle> for their checks to show.

An app password reads no private repositories. To watch one, set KNOTLESS_BOT_DIDS to the runner's DID (and the triager's, comma apart), deploy, and open https://<domain>/oauth/bot?account=<the runner's handle> signed in to that account's PDS, to approve. serve keeps the session in bots/ under KNOTLESS_DATA, which run and triage take as their config directory, and refreshes it there: it lasts two years if used every three months. The password stays in run.env but logs in only when there is no session, or one can no longer be refreshed; take it out to make that a failure instead, and log in again then.

Steps run in containers on the host's docker daemon, through its socket, so run is root on the host in all but name. A step sees only its checkout and the secrets granted to it, but a container is a thin wall: watch only repositories whose writers you trust. The step containers have no limits of their own beyond --step-timeout unless KNOTLESS_RUN_ARGS gives them (--cpus, --memory, --pids, --network, --run-timeout: see the CLI reference). Give them beside serve: one coverage test of knotless's uncapped took about 9 GiB and every core, and two at once stalled the host it shared with serve.

A restart, a deploy's among them, waits only for the runs whose steps have started, a deploy's included, up to --stop-wait (an hour, so that no run is cut off before its own limits: a release's build of the client takes about 17 minutes), when it cancels those still going, and within stop_grace_period (65 minutes) in any case: each run still waiting its turn is dropped, its check filed as cancelled (not deleted: see drop_run in knotless-run), and the runner started next runs it, at once, as it was. The checks of a deploy going's own commit, which it waits for, are kept until it ends. Killed or crashed instead, the runner leaves its checks running and its step containers up; the next one ends those checks as errors, for knotless run rerun to run again, and removes the containers labeled as its own (dev.knotless.runner=<did>), or mounting a path under its --work directory, as one from before runs were labeled does, never another's; and every --sweep after, those of its own whose run it no longer has. A deploy restarts it only if what it runs changed (When a deploy restarts the runner), with docker compose up -d run from the host, in the background, logged to /var/log/knotless-run-restart.log: a recreate cut short leaves a container named <id>_knotless-run-1 that is never started, on whose name the next fails, having sent the runner its SIGTERM but never its SIGKILL, so each such leftover is removed first and up is tried again, three times in all. It reads every repository again every --sweep (600 seconds), for pushes and pulls Jetstream did not ring for.

• KNOTLESS_RUN_DATA (/var/lib/knotless-run): its session, the refs it last ran, checkouts while they run, and the workflows' caches in caches/, at the same path inside as on the host, for the step containers to mount.
• The caches (cache: in a workflow): each path is a base directory, caches/bases/<cache>/<generation>, which a run sees through an overlay of its own, a local docker volume of type overlay mounted with userxattr: the base read only below, caches/runs/<run>/ taking what the run writes. So runs of one workflow go side by side, waiting only for one of the --jobs (4). Only a push to the default branch that passes keeps what it wrote: its layer goes into a new base, of hard links to the old one's files where it left them, which the runs after it start from, and the old base goes once no run reads it. A push that takes a pull's checks rather than running them (see KNOTLESS_RUN_ARGS) takes that run's layers too: an owner's or collaborator's pull run that passes keeps them, in caches/kept/<run>/, and the push promotes them as its own run's would be, onto the base each was written over. One over a base that has moved on since, as when main ran in between, is not applied, its whiteouts being of what that base held, and the next run of main warms it; the push's log says which. Kept layers go once taken, once their pull is at another head or base, or a pass after it closed, and past three days, sixteen runs, or --cache-max GiB for them all, the oldest first; a start removes those of runs no push could take. Any other run's layer, an untrusted pull's always, goes when it ends, and what a stopped runner left goes when it starts again. An untrusted pull of a workflow that asks for secrets starts from an empty base of its own instead, as the default branch's may hold what those secrets left. A base over --cache-max GiB (16) once made is dropped, for the next run to start afresh. Disk, then: each base once, each run going's own writes on top (a test run's build, a few GB), and up to --cache-max GiB of kept layers. Below --disk-floor GiB free (20), a run sheds the bases no run reads and the kept layers before it starts, and waits for room if that is not enough, so the site on the same disk keeps room to write. The host needs overlayfs over ext4 with user xattrs, as Linux 5.11 and up has. The docker daemon mounts the overlays by the host's paths, one more reason KNOTLESS_RUN_DATA is the same path inside and out. The first start after the switch from volumes removes the old knotless-cache-* volumes, and runs start cold until each workflow's first run on the default branch passes.
• KNOTLESS_RUN_ARGS: more of its command line, such as --pulls, to run open pulls too, which it finds through serve: each at its head merged with its base, those by the owner or a collaborator as they move, anyone else's once the owner asks with knotless run rerun --pull. A push to the default branch at the tree an owner's or collaborator's pull passed at, as merging it leaves the branch, takes those checks as passed rather than running them again, its log saying whose they were; a workflow that asks for secrets, or whose runs finish, as deploy.yml, runs as ever. --queue also lands the merge queue (knotless pull merge --queue): up to --batch (8) queued pulls at a time, each squashed onto the branch as its own commit, checked once at the batch's tip. While one is checked the next is checked on its tip (--depth 1 turns that off): it runs as a pull's run does, after main's runs and with a pull's CPU share, is dropped if the batch it is on fails or changes, and lands only on the commit that batch landed as. It reads the queue records from the repos of the owner and collaborators, not only from serve's index, and looks at them every --notice (30) seconds besides Jetstream, so a pull queued starts within that. It pushes as the runner's account, as the owner or a collaborator who may write (knotless repo share <url> <handle> --write), or, as a runner the repository names when none it names may write, through the gateway, once the owner delegates writes there (Who lands a queue). While it lands the queue, open pulls not in it run again only as their heads move, after the queue's batches.
• /etc/knotless/run-secrets.json: the secrets it grants, by <handle>/<name>, as {"alice.example.com/tally": {"DEPLOY_TOKEN": "…"}}, or {"DEPLOY_TOKEN": {"value": "…", "refs": ["refs/heads/main"]}} for pushes to main alone. It starts empty; edit it on the host, and the next run reads it.

When a deploy restarts the runner

A restart costs little since the runner hands over (How the runner hands over), but the new runner still waits for the jobs the old one's runs hold, so a deploy restarts run only when what it runs changed. deploy.sh up compares two things, and leaves run running only when it knows both are the same:

• Its build. The image keeps a fingerprint of what the runner is made of, in /etc/knotless-run.fingerprint: the knotless-run binary, the docker client its steps run through, and the packages installed (/var/lib/dpkg/status: git and the libraries). up reads it from the running container and from knotless:current, the image just built. The commit the binaries log is set as they run (ENV KNOTLESS_COMMIT, the image's last layer) rather than built in, so a commit that changes only serve, the pages, another binary or the docs leaves the runner's binary, and so its fingerprint, as it was.
• Its settings. A hash of the run service as docker compose config run resolves it from the compose file and the .env this deploy just wrote (its command line, KNOTLESS_RUN_ARGS and KNOTLESS_RUN_REPOS among it, its environment and mounts), and of run.env. Each restart removes /etc/knotless/run-deployed before it starts and, once the new container is up, writes <container id> <hash> there. A deploy whose hash, or whose running container, is not the one recorded restarts it. run-secrets.json is not in it: the runner reads that at each run.

Anything it cannot tell, a container not running, no fingerprint, a record missing, restarts run as every deploy once did; so the first deploy with this does, and so does one while a restart is still under way. Either way the deploy log says which, and why:

2026-10-27T04:02:11Z 1a2b3c4d5e6f → run: its build, 5d41402abc4b, and settings are the same, not restarted
2026-10-27T05:13:40Z 7d8e9f0a1b2c → run: restarting: its build changed, 5d41402abc4b to 7d793037a076

With KNOTLESS_RUN_REPOS empty, a deploy that finds run running, as an earlier deploy with repositories to watch left it, stops and removes it, once its own run is filed if it runs under it, as a restart would, and starts no other: run: removing: no KNOTLESS_RUN_REPOS in the deploy log.

A runner left running keeps logging the commit it was built from at its start, as it should. To restart it anyway, deploy with KNOTLESS_RUN_RESTART=1, or on the host cd /etc/knotless && docker compose up -d --force-recreate run, after which the next deploy, finding no record for that container, restarts it once more. To ask the first question by hand, on the host: docker exec <container> cat /etc/knotless-run.fingerprint beside docker run --rm --entrypoint cat knotless:current /etc/knotless-run.fingerprint.

How the runner hands over

A restart does not stop the runner and then start the new one. handover.sh sends the runner leading SIGTERM, and stops docker restarting it. The runner drops the runs waiting that it will not run, writes its state file a last time, and from then on writes a note of its own under state.handover/ beside it. It goes on finishing the runs it has going: those started, a merge-queue batch's, and those at a going deploy's commit. Once the note is there, handover.sh starts the new runner beside it (docker compose up --scale). The new one reads the state file, leaves the old one's runs, checks and caches alone, and takes no job the old one's runs hold. The old one exits once its runs end, --stop-wait (an hour) after its stop at most, and is removed then. It holds only the jobs the new one lends it meanwhile.

The deploy log says how it went:

2026-11-03T18:02:11Z → run: handed over in 2s; starting the new one beside 4f1c2a9b0d3e
2026-11-03T18:02:14Z → run: started 9a8b7c6d5e4f

A runner that writes no note within KNOTLESS_HANDOVER_WAIT seconds (60), one built before runners handed over, is stopped first instead, and the new one started once it has, as every deploy once did. While one is still finishing, docker compose ps run lists two: the one docker restarts (docker inspect -f '{{.HostConfig.RestartPolicy.Name}}' says always) leads, and a deploy compares only that one.

knotless-triage

Set KNOTLESS_TRIAGE_REPOS to the repositories to triage and KNOTLESS_TRIAGE_ARGS to the rest of its command line (the deploy workflow gives --model clef --scopes kind,area --discover: with --discover it also triages every repository that names its account a triager, as the runner host runs those naming it, so an owner turns it on with knotless triager add <url> <handle> kind area alone). It starts once /etc/knotless/triage.env is on the host, which a deploy from one's own checkout writes, once, from deploy/local.env:

• KNOTLESS_TRIAGE_IDENTIFIER and KNOTLESS_TRIAGE_PASSWORD: the account its labels are written in, and an app password for it. Not the owner's: an owner's labels are a person's, which outrank a triager's and count at once. For private repositories it logs in at /oauth/bot as the runner does.
• KNOTLESS_DECIDE_URL: the decision model's endpoint, for Clef https://api.cloudflare.com/client/v4/accounts/<account id>/ai/run/@cf/cloudflare/clef.
• KNOTLESS_DECISION_TOKEN: its token, for Clef a Cloudflare API token that may use Workers AI.

Until then a deploy leaves it out and says so. Its labels count for nothing until the owner names its account a triager of some scopes, with knotless triager add <url> <handle> kind area; knotless triager list <url> eval <handle> says first how often they agree with people's. It logs each issue it judged, or failed to, as judged or judging failed.

The keeper

knotless-agent keep brings the pulls the merge queue turned away for a conflict up to date and queues them again (the merge queue). Set KNOTLESS_KEEP_REPOS to the repositories it keeps, or --owner <handle> for all of an account's, and KNOTLESS_KEEP_ARGS to the rest of its command line (--attempts 2, say). It starts once /etc/knotless/keep.env is on the host, which a deploy from one's own checkout writes, once, from KNOTLESS_KEEP_IDENTIFIER and KNOTLESS_KEEP_PASSWORD in deploy/local.env: its account, and an app password for it. The account is its own, not the owner's, and each repository it keeps names it a collaborator (knotless repo share <url> <handle> --write), as it pushes to their pulls' branches and queues them. Until then a deploy leaves it out and says so. It logs each pull it kept, or failed to, as kept or keeping failed.

Monitoring

/healthz says only that serve answers, and Docker restarts it when it does not. /_status says whether it is doing its job, for a monitor off the host to poll: 200 when it is, 503 and what failed when not.

• Feeds: each one followed (Jetstream, the firehose) is connected and brought an event in the last two minutes (--stale <secs> to change it). The firehose brings hundreds a second; Jetstream, asked only for dev.knotless.*, still brings every account's identity and account events, one every few seconds, eleven at the longest gap measured. With both followed, one bringing events is enough, and each is still shown.
• Index: it answers a read.
• A mirror, with ?repo=<handle>/<name>: its default branch's commit and tree are read as its code page reads them, so a mirror whose packs can no longer be read (as when one outgrew its open handle's pack slots) fails here, and its pack count shows.
• Disk: the one serve's cache, and with it the mirrors, is on (KNOTLESS_CACHE_DIR, /data/cache). Under 15% free it is low, a problem told of while the answer stays 200; under 5% it is full and the answer is 503. On 2026-10-31 that disk filled and every page that needed a mirror answered 502 (No space left on device); 15% of its 150 GB is a day or more of deploys and runs to clear it in. A keyword monitor (step 4 below) shows a low disk before it fails.
• Code, with --code-index: Zoekt's webserver answers, and the repositories, shards and bytes its index holds show. A webserver that died fails here, as every code search then fails.

It takes milliseconds, so polling it every minute costs nothing. A dead upstream feed is not a reason to restart serve, so the compose healthcheck stays on /healthz.

knotless status https://knotless.dev --repo chadtmiller.com/knotless   # or --json
curl -s 'https://knotless.dev/_status?repo=chadtmiller.com/knotless' | jq

{
  "ok": true,
  "problems": ["jetstream: not connected"],
  "uptimeSeconds": 5321,
  "feeds": [
    {"name": "firehose", "url": "wss://relay1.us-east.bsky.network", "ok": true, "connected": true,
     "silentSeconds": 0, "staleAfterSeconds": 120, "events": 2103344, "seq": 26329004819},
    {"name": "jetstream", "url": "wss://jetstream.us-east.bsky.network", "ok": false, "connected": false,
     "silentSeconds": 7310, "staleAfterSeconds": 120, "events": 1650, "seq": 26328850636}
  ],
  "index": {"ok": true, "error": null},
  "mirror": {"repo": "chadtmiller.com/knotless", "ok": true, "error": null,
             "commit": "b1fbf45…", "packs": 3},
  "disk": {"path": "/data/cache", "state": "ok", "ok": true, "freeBytes": 41203998720,
           "totalBytes": 160982630400, "error": null}
}

An external monitor

The check has to run somewhere other than the host, or the host going down goes unreported. An external uptime service is the least to run: nothing to keep alive, and it pages a phone. UptimeRobot's free plan is enough (Better Stack or an Uptime Kuma elsewhere do the same):

1. Sign up at uptimerobot.com and install its app, or add an email or Slack contact under Integrations & API.
2. New monitor: type HTTP(s), URL https://knotless.dev/_status?repo=chadtmiller.com/knotless, interval 5 minutes (the shortest the free plan has), timeout 30 seconds. Any status but 2xx is down, so a 503 alerts. Name it "knotless status".
3. New monitor: HTTP(s), https://knotless.dev/, 5 minutes: the pages, which /_status does not draw.
4. Optionally a keyword monitor on the status URL for "ok":true, which shows the body's problems in the alert.
5. Pick the contact under each monitor's Alert contacts, and Notify after 1 check, or 2 to ride out a deploy's restart.

A 503 then reaches the phone within the interval plus two minutes of a feed going silent, or at the first poll after a mirror stops reading.

Logs

serve and the runner log JSON lines on stderr, one an event, at the level RUST_LOG sets (KNOTLESS_LOG in .env, info unset); the pages log what node prints. Docker writes each container's to the host's journal (the journald driver, x-logging in docker-compose.yml), so they outlive the container: a rollout removes the old one, and its lines stay, for 30 days: setup.sh sets journald's MaxRetentionSec, which the privacy policy promises (journald's own cap, a tenth of the disk and at most 4 GB, can drop the oldest sooner). Each line names its project and service (COM_DOCKER_COMPOSE_PROJECT, COM_DOCKER_COMPOSE_SERVICE) and its container (CONTAINER_TAG). deploy.sh and rollout.sh add a line with the time for each step of a deploy (built, started beside the old one, healthy, the old one stopped) to /var/log/knotless-deploy.log on the host, to line a freeze up against. docs/observability.md says what was chosen, and why.

What serve logs:

• request, one a request: http.route (its template), url.path (no query), http.request.method, user_agent.original (the client as it names itself), http.response.status_code, http.response.body.size when known, duration_ms and trace_id, at info; at warn when over KNOTLESS_SLOW_MS (1000), unless it is a long poll (awaitSpaceWrites, getDelegation), which waits on purpose; at error for a 5xx with its error, and /healthz at debug. The answer carries the trace as x-trace-id; the pages hand it back on their XRPC calls, so a page and the data it drew share one.
• outgoing, each call made, to a PDS, the PLC directory, Bluesky's CDN or a handle's /.well-known: server.address, url.path, rpc.method for XRPC, http.response.status_code, error.type, http.request.resend_count, duration_ms, trace_id. At debug, or warn when it failed, answered 5xx or was slow. It sends traceparent.
• sql, each of the index's statements that took a tenth of KNOTLESS_SLOW_MS (100 ms) or more, at warn: db.query.text (the SQL as written, its whitespace closed up, cut at 500 characters, with no value it was given) and duration_ms, with the trace_id of the request it ran in. The slow request whose line shares that trace is the one it made slow.
• summary: requests, summary: outgoing (by host), summary: cache (hits, misses, failed fills, by cache) and summary: feed (lag, events) each minute, for what saw any.
• mcp tool failed at warn, one a tool call that failed: gen_ai.tool.name, did, error (what the tool answered, as the client saw it), duration_ms and trace_id; mcp resource failed the same, of a resource read, named for the tool that answers it.
• mirror sync, one each sync of a repository's mirror: repo, duration_ms, and records_ms, the part spent reading its records. At info, or warn when slow. Reads of the repository wait on a sync under way, so it explains a burst of slow requests for one repository that end together.
• cache fill failed at warn, with why: an avatar, claim or picture that could not be read.
• knotless-serve starting (version, commit), index opened (duration_ms, rebuilt), listening, stopping, stopped; feed connected and feed disconnected; index sweep started and ended; event dropped: not indexed; account not backfilled; panicked.

What the runner logs: run queued, run started, run ended (status, duration_ms), run cancelled before it started, each with repo, workflow, ref_name, commit and pull; step started and step ended; container started and stopped; batch started, landed, dropped and pull ejected (reason); queue noticed (debug); pds write retried; feed connected or down; failed for anything that went wrong.

None logs a header, a token or a body. A request line logs its query (url.query), each value a secret could be in replaced by _, and none for /oauth; no call made logs one. DIDs are logged; handles only as a path or a /.well-known host names them; client IPs never.

deploy/logs.sh (just logs) reads them from here, every container of a service at once, those gone too, through jq:

just logs serve 'select(.duration_ms > 1000)'                 # slow requests and calls
just logs serve --since 2h 'select(.level == "WARN" or .level == "ERROR")'
just logs serve 'select(.message == "outgoing" and .level != "DEBUG")'   # calls that failed
just logs serve 'select(.message | startswith("summary"))'
just logs serve 'select(.trace_id == "4bf92f3577b34da6a3ce929d0e0e4736")'  # one request, and the pages' calls for it
just logs run --follow 'select(.message | startswith("run"))'
just logs runner --since 1h 'select(.message == "batch started")'   # the runner host's, which lands the queue
just logs deploys                                             # the deploy steps, with times

run is a runner on this host. runner reads the runner host's (deploy/runner-host) instead, at KNOTLESS_RUNNER_HOST (runner.knotless.dev unset), by its container's tag, as that host's lines name no compose project. KNOTLESS_RUNNER_ENV=deploy/ci.env just logs runner reads another runner, such as the trusted one, from its own settings, as its deploy does.

Or on the host itself, with journalctl, by the service's field; -o cat gives each line as it was written:

ssh root@knotless.dev 'journalctl -o cat --since -30m COM_DOCKER_COMPOSE_PROJECT=knotless COM_DOCKER_COMPOSE_SERVICE=serve' | jq -Rc 'fromjson? | select(.duration_ms > 1000)'
ssh root@knotless.dev 'journalctl -o cat --since -1h COM_DOCKER_COMPOSE_SERVICE=serve' |
  jq -Rc 'fromjson? | select(.message == "outgoing") | {t: .timestamp, host: ."server.address", path: ."url.path", status: ."http.response.status_code", error: ."error.type", ms: .duration_ms}'
ssh root@knotless.dev tail -n 50 /var/log/knotless-deploy.log

Each request's line names its trace; a page's own line comes after those of the XRPC calls it made, since a line is written when its request ends. Calls are logged only at warn by default; to see every one for a while without a restart, set the level through --metrics' port, inside the container:

ssh root@knotless.dev 'for c in $(docker ps -q -f label=com.docker.compose.project=knotless -f label=com.docker.compose.service=serve); do docker exec $c node -e "fetch(\"http://127.0.0.1:9464/log\", {method: \"PUT\", body: \"info,knotless::outgoing=debug\"}).then(r => r.text()).then(console.log)"; done'
ssh root@knotless.dev 'docker exec $(docker ps -q -f label=com.docker.compose.project=knotless -f label=com.docker.compose.service=serve | head -1) node -e "fetch(\"http://127.0.0.1:9464/metrics\").then(r => r.text()).then(console.log)"' | grep -v '^#'

PUT it info again after; a restart goes back to KNOTLESS_LOG.

Without SSH

serve keeps each line it logs in /data/logs.db too (--logs), as it writes it to stderr, a second at a time. Every serve writes there, both during a rollout, so a deploy loses none. An operator reads them with knotless admin logs or MCP's logs tool, from dev.knotless.getLogs, without SSH:

knotless admin logs --trace 4bf92f3577b34da6a3ce929d0e0e4736   # one request, and the calls it made
knotless admin logs --message sql --limit 20                    # the last slow statements
knotless admin logs --level warn --min-ms 2000                  # what was slow, and warned

A read needs one of --trace, --message, --level or --min-ms, answers at most 500 lines, and is itself logged as logs read, with who asked and for what. Lines at info and below, the most of them, are kept to the last 500,000 (--logs-keep); every line for 30 days at most, as the journal keeps them. They hold what the journal does, no more. /metrics gives what was counted since the start, in Prometheus's format: http_server_request_duration_seconds by route, method and status, http_client_request_duration_seconds by host and outcome, knotless_cache_lookups_total by cache and result, knotless_cache_build_duration_seconds by cache and how a miss was made good (patched or rebuilt), and each feed's knotless_feed_connected, knotless_feed_events_total and knotless_feed_lag_seconds.