johnandersen777.bsky.social/knotless
git that lives in your Atmosphere account
Forked from chadtmiller.com/knotless
0
0
# Deploying knotless-serve
One Docker host, a reverse proxy in front of it for TLS, and a domain.
| | |
| --- | --- |
| `../Dockerfile` | one image, `knotless:current`: knotless-serve, knotless-run, knotless-triage, knotless-agent, knotless-mcp with Claude Code, the knotless CLI, git, the docker client and the built pages |
| `docker-compose.yml` | `/etc/knotless/docker-compose.yml`: `serve` (knotless-serve, on :2590), `pages` (the SvelteKit app, on :2591) and, if asked, `run` (knotless-run), `triage` (knotless-triage), `review` (knotless-agent review) and `keep` (knotless-agent keep) |
| `deploy.sh` | builds the image on the host over SSH, writes the compose file and its `.env`, and restarts what changed; or one of those steps, `build`, `up` or `drop` |
| `rollout.sh` | run on the host by `deploy.sh`: moves `pages` and `serve` to a new container each, with no gap |
| `handover.sh` | run on the host by `deploy.sh`: moves `run` to a new container, the old one handing over to it as it finishes its runs |
| `rollout-test.sh` | tries `rollout.sh` on a local docker behind a Caddy, counting failed requests |
| `release.sh` | builds the client for macOS and Linux here and puts the builds on the host, for `/install.sh` |
| `setup.sh` | makes a fresh Ubuntu 24.04 host ready for `deploy.sh`: docker, a Caddy of its own for the domain, a firewall, swap, upgrades and pruning |
```sh
deploy/setup.sh root@host git.example.com # once, on a fresh host
just deploy root@host git.example.com # build on the host and restart; or put both in deploy/local.env
```
The domain needs an A (and AAAA) record for the host, and an `_atproto`
TXT record for each of the service's own accounts with a handle under it.
knotless.dev's are kept by the `stacks/edge` Terraform in the infra
repository, beside `stacks/knotless`, which makes the host. A worker that
serves an account's handle at its own name, as the reviewer's pds.js does
at `review.knotless.dev`, holds that name as a custom domain in its
`wrangler.toml`, which makes the record: the infra repository keeps only
its `_atproto` TXT.
### From a push to main to the site running it
`.knotless/workflows/deploy.yml` deploys each push to main once its
seven checks (check, test, web, lexicons, shots, docs and e2e) pass at
it. It names them in `after`, so it waits for them holding none of the
runner's jobs, and the checks it waits for are never kept from one. One
that fails files the deploy failed without running. Then:
1. **build**: `deploy.sh build` builds `knotless:build-<commit>`, a
candidate, which nothing runs: `knotless:current` and the containers
stay as they are. It removes any candidate an earlier deploy left.
2. **deploy**: if main is still at this commit, `deploy.sh up` tags the
candidate `knotless:current`, removes the candidate's tag, and rolls
out as below, restarting `run` only if what it runs changed
([When a deploy restarts the runner](#when-a-deploy-restarts-the-runner)),
then removes the untagged images no container runs, the ones earlier
deploys replaced.
Until it goes up, a deploy gives way: when main moves past its commit,
asked every 30s through the build, it stops (a build is
cancelled with its client, its cargo cache kept for the next), removes
its candidate and passes, and the newer commit's deploy, which waits for
it (`superseded: finish`), starts. Once it goes up it runs to its end.
So a push goes live a few minutes after its checks pass: the build, two
minutes or so for a change to our crates in the `deploy` profile, then
the rollout, about 30s.
The site comes first on the host for CPU, then a deploy and main's
checks, then pulls' runs. serve and pages have `cpu_shares` of 4096 (a
`cpu.weight` of about 157); the build runs in the cgroup
`/system.slice/knotless-build`, which `deploy.sh build` makes with a
`cpu.weight` of 100, as main's runs have, where docker drives cgroups
(v2) with systemd; and knotless-run starts a pull's run with
`--cpu-shares 256`, a weight of 10, so a burst of pulls takes only what
the rest leave. Before, the build had 20 against the pulls' 100, and a
merge that ran every open pull again held the deploy back for minutes. Docker's buildkit puts each build step in a
cgroup of its own, and by default beside the containers', where `nice`
inside it would count for nothing against them. `KNOTLESS_BUILD_CGROUP`
names another parent, or set empty, none.
The image's binaries are built in the `deploy` profile (`Cargo.toml`):
release's, but in 16 codegen units without LTO across crates, so the
crates a push rebuilds, knotless-serve the longest, use every core. The
cargo registry and `target/` are cache mounts on the host's builder, so
a deploy rebuilds only our four crates, not their dependencies; a fresh
checkout's files are all new to cargo, so all four, whichever changed.
`docker buildx du --filter type=exec.cachemount` on the host shows them.
Each build then prunes buildkit's cache to the 20 GB used last, which
keeps those mounts; uncapped, it held 49 GB when the disk filled.
By hand, `just deploy` does all three steps at once, as before.
The proxy joins the `knotless` docker network (`KNOTLESS_PROXY_NETWORK`)
and sends the domain to `knotless-serve:2590`, passing `X-Forwarded-Proto`.
With Caddy:
```caddy
git.example.com {
reverse_proxy knotless-serve:2590 {
# Holds a request that meets no listening serve, for a moment
# within a deploy, and tries again rather than answering 502.
lb_try_duration 20s
lb_try_interval 250ms
}
}
```
### Push notifications
The iOS app's notifications go through Apple's push service, with a key
from the Apple Developer account of the team that signs the app. Put it on
the host once, where the server can read it:
```sh
install -o 1000 -g 1000 -m 600 AuthKey_XXXXXXXXXX.p8 "$KNOTLESS_DATA/apns.p8"
```
Then set `KNOTLESS_APNS_KEY_ID` and `KNOTLESS_APNS_TEAM` and deploy.
`deploy.sh` gives serve `--apns-key` only when the file is there; without
it, serve runs as ever and sends nothing. Anyone holding the key can send
to any app of that team, so it stays on this host and in the encrypted
secrets it came from.
### Deploys without downtime
`deploy.sh` moves `pages` and then `serve` with `rollout.sh`: a second
container starts beside the running one, with the same names on the same
networks, and once its healthcheck passes the old one is stopped, finishing
the requests it holds (SIGTERM, then `stop_grace_period`), and removed. A
new one that is not healthy in 180s (`KNOTLESS_ROLLOUT_TIMEOUT`) is removed
and the old one keeps serving; the deploy fails. `run` is restarted last,
and only if what it runs changed
([When a deploy restarts the runner](#when-a-deploy-restarts-the-runner)).
While both run, docker's DNS gives `knotless-serve`, `serve` and `pages`
as every running container's address. The proxy, serve and the pages each
dial every address in turn, so one not listening yet, or no longer, is
refused and the next is tried. `lb_try_duration` is the proxy's belt and
braces over that: it holds a request that found none and tries again.
Both serves share `KNOTLESS_DATA` for those seconds. The index is SQLite
in WAL mode, each process waiting up to 5s for the other's writes; mirrors
are written under lock files and renames, as concurrent processes do
already; both follow the streams and write the cursor, and a record
indexed twice is the same row. The one unsafe overlap is a build that
changes the index's tables: it drops and rebuilds them under the old one.
So `rollout.sh` first asks the new build (`knotless-serve --check-index`),
and if it would rebuild the index, serve is stopped and then started
instead: down until the new one listens, a second or two, which the
proxy's `lb_try_duration` holds requests through.
`deploy/rollout-test.sh` tries all of this on a local docker, behind a
Caddy, counting failed requests across the rollouts.
#### The proxy
knotless.dev has a host of its own (Hetzner `knotless`, a cx43, made by
the `stacks/knotless` Terraform in the infra repository), apart from the
sites it once shared one with: its runs and builds took the cores they
needed. `setup.sh` gives it Caddy in `/etc/knotless-proxy`, which dials
`knotless-serve:2590` by name over the `knotless` network and holds a
request through a rollout's second or two with no serve:
```caddy
knotless.dev {
request_body {
max_size 100MB
}
reverse_proxy knotless-serve:2590 {
lb_try_duration 20s
lb_try_interval 250ms
}
encode gzip zstd
}
```
With `KNOTLESS_PAGES_DOMAIN` (`knotless.page`), the proxy answers
repositories' sites too ([../docs/design/pages.md](../docs/design/pages.md)):
the domain and every owner's label under it, and each private site's
label under `private.` there, on wildcard certificates from the DNS
challenge. The zone's `*` record covers `*.private` too, as nothing is
at `private` itself. That takes Caddy built with the `caddy-dns/cloudflare`
module, which `setup.sh` builds from a Dockerfile beside the compose file,
and the zone's own Cloudflare token, Zone:Read and DNS:Edit on that zone
alone. Give the token once, as `KNOTLESS_PAGES_DNS_TOKEN` in
`deploy/local.env`, and run `setup.sh`: it keeps it in
`/etc/knotless-proxy/cloudflare.env`, and a run without it keeps that one.
A run with nothing changed rebuilds and recreates nothing.
```caddy
knotless.page, *.knotless.page, *.private.knotless.page {
tls {
dns cloudflare {env.CLOUDFLARE_API_TOKEN}
}
reverse_proxy knotless-serve:2590 {
lb_try_duration 20s
lb_try_interval 250ms
}
encode gzip zstd
}
```
`deploy.yml` sets `KNOTLESS_PAGES_DOMAIN` for serve's `--pages-domain`.
The zone's DNS (its apex and `*`, DNS-only, at this host) is in the
infra repository's `stacks/edge`.
On a host whose proxy is another's, join that proxy's network, named
`knotless` (`KNOTLESS_PROXY_NETWORK`), and give the domain that block in
its Caddyfile instead.
knotless-serve is the appview (`dev.knotless.listRepos`, `getForge`,
`getFeed`, indexed from Jetstream), the gateway, as `did:web:<domain>`, and
the moderation service. Its state is in `KNOTLESS_DATA`
(`/var/lib/knotless`):
- `index.db`, the index: a cache. A version that changes its tables
rebuilds it, and it refills itself at start from the accounts the relay
(`KNOTLESS_RELAY`, relay1.us-east.bsky.network) lists as holding knotless
records, and again every ten minutes, for any event Jetstream missed.
- `cache/`, mirrors and identities: a cache.
- `usage.db`, how it is used (see [Usage](#usage)): counts to keep, but
nothing breaks without them.
- `config/`, the gateway's OAuth client key and the sessions owners
delegated to it. Losing it breaks every delegation, which owners then
have to make again: back it up.
- `bots/`, the sessions run, triage and review act with, once they log in at
`/oauth/bot` (see [knotless-run](#knotless-run)): lost, they log in
again.
Keep the records DNS-only: PDSes fetch the gateway's DID document and OAuth
client metadata from the origin, and the proxy holds the certificate.
## Usage
`knotless admin stats` (`--days <n>`, 14 by default; `--json`) shows the
operator how the service is used, from `dev.knotless.getUsage`, to the
accounts `KNOTLESS_ADMIN_DIDS` names (set in `.knotless/workflows/deploy.yml`),
signed in as one of them:
```sh
KNOTLESS_APPVIEW=https://knotless.dev knotless admin stats
```
- **Each day**: the people who visited the pages and the pages they
looked at; the requests serve took and how many it failed; and, from the
index, what the network wrote: repositories, pushes, pulls, issues,
comments and checks, and how many accounts wrote them.
- **The network**: every account that has made a repository, pushed or
written in a thread, and those that did in the last 7 and 30 days.
- **Routes**: requests per route as axum matched it (the pattern, never
the path), by surface (`pages`, `api`, `git`, `xrpc`, `mcp`,
`install`, `other`), with failures and times. The time is to the
answer's head, so a long clone counts its start.
- **Pages, referrers and clients**: the routes people looked at, the
sites that sent them, and requests by kind of client (`browser`, `git`,
the `cli`, `bot`, `other`, from the `User-Agent`), and `pages`, the
queries the pages ask as they draw a page (`knotless-pages`), which are
not a client of their own.
- **Events**, what the handlers saw, with how often each failed, its mean
time and the accounts that did it, in the days asked but at most the
last 30: `mcp.tool` by tool, `mcp.resource` by the tool that answers
the read, and `mcp.signin`; `web.login`;
`git.fetch` by repository, once per clone or fetch, as asked for
(`<handle or did>/<name>`); `cli.version` by the CLI's version;
`gateway.delegated`, `gateway.push`, `gateway.commit`, and
`gateway.served`, by the owner the gateway pushed or committed for.
- **Funnels**: of the accounts that delegated to the gateway in the days
asked, not having in the 30 days before, how many the gateway has
written for then or since, by anyone's push or commit that moved a
ref; of those that signed in to MCP so, how many have called a tool.
A day older than 30 is counted as it stood once 30 days had passed.
- **Hours**: each hour's requests, how many failed and how many took over
two seconds, to know when to read the [logs](#logs) of. The lines show
only the hours with either; `--json` has every hour.
- **Statements** (`sql`): the index's statements that took 100 ms or
more, as the `sql` log line has them (the SQL as written, no value it
was given), with how often, their mean and their longest, the most
time first: the fifty that cost most.
- **Failures**: why an event failed, as it answered, most often first:
so far an MCP tool's error. A DID, an `at://` URI, a hash and a number
in it are put as `<did>`, `<uri>`, `<hex>` and `<n>`, so like failures
count together. A handle or a repository's name in it is kept as
written.
- **Agents**: requests by the `User-Agent` of each crawler or script,
the `bot` client, most first: never a browser's.
Statements, failures and agents keep at most 500 of each a hour apart;
past that, the new ones count together as `(others)`.
What is kept, in `usage.db`, is counts by the hour, and no address; and,
for the pages' logins, MCP and the gateway, where people are signed in
already, which accounts (by DID) did what each day, for the events'
accounts and the funnels; and the statements, failures and crawlers'
agents above, by the hour. Those are kept 30 days, today the last: on the
30th day after, serve folds each day into its funnels' counts, how many
started and how many went on, with no one in them, and deletes them, and
the hours of the rest. A
page, once drawn in a browser, sends `/_hit` its route and the site the
visitor came from. Each visitor is counted once a day by a hash of their
address, less its last byte (its last 80 bits for IPv6), and their
`User-Agent`, salted with a key made for the day. Two serves run at once
for a moment in each deploy, so the key is kept in the database for both,
and deleted with the day's hashes once the day (UTC) is over, leaving the
day's count: then nothing links a visitor to an address, or to the next
day. Crawlers, by their `User-Agent` or by not running the page, and
browsers that ask not to be followed (`Sec-GPC: 1`), are not counted.
Anyone can read the days and the network, but not the requests, routes,
events or funnels: the `/stats` page shows them, from
`dev.knotless.getStats`.
Requests are counted in memory and written every 15 seconds and as serve
stops; one killed loses those seconds.
It reads like this (`--json` gives the same as [`knotless::usage::Report`](../crates/knotless/src/usage.rs)):
```text
network 212 accounts, 340 repositories, 41 authors this week, 97 this month
day visitors views requests errors repos pushes pulls issues comments checks authors
2026-10-22 118 431 20544 3 4 96 7 5 22 61 19
2026-10-23 131 502 22310 0 2 104 9 3 31 70 23
route git POST /{authority}/{repo}/git-upload-pack 1840 requests, 0 failed, 210ms mean, 96 over 0.5s, 4 over 2s
route xrpc GET /xrpc/dev.knotless.getRepo 1203 requests, 2 failed, 48ms mean, 11 over 0.5s, 0 over 2s
event cli.version 0.1.0 2210 calls, 0 failed, 0ms mean, 0 accounts
event git.fetch alice.test/proj 140 calls, 0 failed, 0ms mean, 0 accounts
event mcp.tool repo_read 612 calls, 9 failed, 85ms mean, 6 accounts
hour 2026-10-23T14 1022 requests, 3 failed, 2 over 2s
funnel gateway.delegated -> gateway.served 4 started, 3 reached
funnel mcp.signin -> mcp.tool 7 started, 6 reached
page /[authority]/[repo] 388
referrer bsky.app 52
client browser 9120
client bot 7310
```
So it answers, among others:
- **Growth**: how many accounts write, and how many did this week and
month; repositories, pushes, pulls, issues and checks a day, and
whether a day's pushes are many people or a few (`authors` beside
`pushes`). From the index, so the days before `--usage` too.
- **The pages**: visitors and views a day, which pages they read, and
which sites sent them.
- **The surfaces**: requests by surface and by kind of client, so how
much is git, the CLI, MCP, the installer, or crawlers.
- **MCP**: which tools are called, how often they fail, by how many
accounts, and how many who sign in go on to use it.
- **Repositories and releases**: which repositories are cloned and
fetched most, and which CLI versions are still about.
- **The gateway**: how many who delegate to it go on to push.
- **Health**: which routes fail or are slow, and a deploy that made
either worse, day against day; and the hours it happened in. A long
poll is never counted slow.
What it does not count:
- Anyone across days: a visitor's hash is new each day, so there are
visitors a day but none a week or returning; weekly and monthly
figures are of accounts that write, from the index.
- Any account past 30 days: an event's accounts over more days are of
the last 30, and a funnel cannot tell an account that comes back after
a month from a new one.
- What a route was asked of, past the events: routes are counted by
pattern, so the API's reads of every repository are one row.
- Installs: the installer and the releases are counted as requests, by
no one; the funnels start at a delegation or a sign-in.
## Releases
The server hands out the client, so installing it needs no clone and no
Rust: `curl -fsSL https://knotless.dev/install.sh | sh` installs
`knotless`, `git-remote-knotless`, `knotless-mcp`, `knotless-run`,
`knotless-triage`, `knotless-agent` and `knotless-claude-review` into `~/.local/bin`
(`KNOTLESS_INSTALL_DIR`), the latest release or `KNOTLESS_VERSION`'s,
checked against its sha256.
```sh
just release root@host # or with KNOTLESS_HOST in deploy/local.env
```
builds the workspace's version, from `Cargo.toml`, for macOS and Linux
(aarch64 and x86_64; Linux static, with musl) on a Mac, with
`deploy/build-client.sh`, puts them in `releases/` under `KNOTLESS_DATA`,
and makes it the latest. The Linux builds need `brew install zig` and
`cargo install cargo-zigbuild`. Bump the version before a release: one
already out is replaced in place.
Or, as the releases of the repository's `knotless` package
([packages](../docs/design/packages.md)): the `release` workflow
publishes each new version once main's checks pass at it. It publishes
as the owner, with `RELEASE_PASSWORD`, an app password of theirs, which
the runner's secrets file grants to pushes to main:
```json
{
"chadtmiller.com/knotless": {
"RELEASE_PASSWORD": { "value": "…", "refs": ["refs/heads/main"] }
}
}
```
The deploy publishes the image it runs the same way, as the release of
the `knotless-server` package at that version, once it is up, and needs
`RELEASE_PASSWORD` for it too; a version with that release already is
left alone.
The `publish-lexicons` workflow publishes the lexicons once main's checks
pass at a commit that changes them, with the shared `publish-lexicons`
step from `chadtmiller.com/steps` (its README lists its inputs): a
`_lexicon.<authority>` TXT record naming `lexicons.knotless.dev` for
each authority (`_lexicon.knotless.dev` for `dev.knotless.*`), and a
`com.atproto.lexicon.schema` record in that account for each lexicon. It
writes what is new or changed and deletes what main no longer has.
`lexicons.knotless.dev` is an account of its own, on a pds.js worker as
the runner's is, so what CI holds can write lexicons and nothing else. It
writes as `lexicons.knotless.dev` with `LEX_PASSWORD`, an app password
of that account's, which the secrets file grants to pushes to main. It
reads the DNS as resolvers see it, and changes it only with
`CLOUDFLARE_API_TOKEN`, a token with Zone:DNS:Edit on the zone allowed
from this host's address. That is needed only when an authority is added
or dropped, and is not granted now: such a run writes the schemas, then
fails, naming the record to make. Every run first plans, saying what it
would change, and `knotless run rerun <repo> --ref refs/heads/<branch>
--workflow publish-lexicons` plans on a branch.
One image runs every service, each by its command, as docker-compose.yml
starts them:
```sh
docker pull knotless.dev/chadtmiller.com/knotless/knotless-server
```
| Service | Command |
| --- | --- |
| `serve` | `knotless-serve --listen 0.0.0.0:2590 …`, the image's default |
| `pages` | `node /app/build` |
| `run` | `knotless-run --secrets … --state … --work … --caches …` |
| `triage` | `knotless-triage --decide <url> --jetstream <url> <repos>` |
| `review` | `knotless-agent review --command knotless-claude-review --jetstream <url> <repos>`, where the repositories may be `--owner <handle>` |
| `keep` | `knotless-agent keep --jetstream <url> <repos>`, where the repositories may be `--owner <handle>` |
knotless.dev hands those out, with `--releases
knotless://chadtmiller.com/knotless` in docker-compose.yml: each file is
read from the owner's PDS once, checked against its record, and kept. A
host of your own takes `--releases /data/releases` there instead, for the
directory `just release` fills.
## knotless-run
Other people's code runs on a host of its own, under gVisor: see
[the runner host](runner-host/README.md). What follows is the runner beside
serve, for the repositories whose writers you trust. knotless.dev runs
none: its checks run on the runner host, and its workflows that hold a
secret on [the trusted runner](runner-host/README.md#the-trusted-runner).
Set `KNOTLESS_RUN_REPOS` in `deploy/local.env` to the repositories to
watch, and on the first deploy `KNOTLESS_RUN_IDENTIFIER` and
`KNOTLESS_RUN_PASSWORD`, the account it writes checks in and an app
password for it, which are kept on the host in `/etc/knotless/run.env`.
The deploy then starts `run` too. The repositories' owners name that
account with `knotless runner add <url> <handle>` for their checks to
show.
An app password reads no private repositories. To watch one, set
`KNOTLESS_BOT_DIDS` to the runner's DID (and the triager's, comma
apart), deploy, and open
`https://<domain>/oauth/bot?account=<the runner's handle>` signed in to
that account's PDS, to approve. serve keeps the session in `bots/` under
`KNOTLESS_DATA`, which run and triage take as their config directory,
and refreshes it there: it lasts two years if used every three months.
The password stays in `run.env` but logs in only when there is no
session, or one can no longer be refreshed; take it out to make that a
failure instead, and log in again then.
Steps run in containers on the host's docker daemon, through its socket,
so `run` is root on the host in all but name. A step sees only its
checkout and the secrets granted to it, but a container is a thin wall:
watch only repositories whose writers you trust. The step containers have
no limits of their own beyond `--step-timeout` unless `KNOTLESS_RUN_ARGS`
gives them (`--cpus`, `--memory`, `--pids`, `--network`, `--run-timeout`:
see [the CLI reference](../docs/reference/cli.md)). Give them beside
serve: one coverage `test` of knotless's uncapped took about 9 GiB and
every core, and two at once stalled the host it shared with serve.
A restart, a deploy's among them, waits only for the runs whose steps have
started, a deploy's included, up to `--stop-wait` (an hour, so that no
run is cut off before its own limits: a release's build of the client
takes about 17 minutes), when it cancels those still going, and within
`stop_grace_period` (65 minutes) in any case:
each run still waiting its turn is dropped, its check filed as cancelled
(not deleted: see `drop_run` in knotless-run), and the
runner started next runs it, at once, as it was. The checks of a deploy
going's own commit, which it waits for, are kept until it ends. Killed or crashed
instead, the runner leaves its checks running and its step containers
up; the next one ends those checks as errors, for `knotless run rerun` to
run again, and removes the containers labeled as its own
(`dev.knotless.runner=<did>`), or mounting a path under its `--work`
directory, as one from before runs were labeled does, never another's;
and every `--sweep` after, those of its own whose run it no longer has. A deploy restarts it only if what it runs changed
([When a deploy restarts the runner](#when-a-deploy-restarts-the-runner)), with
`docker compose up -d run` from the host, in the background, logged to
`/var/log/knotless-run-restart.log`: a recreate cut short leaves a
container named `<id>_knotless-run-1` that is never started, on whose
name the next fails, having sent the runner its SIGTERM but never its
SIGKILL, so each such leftover is removed first and `up` is tried again,
three times in all. It reads every repository again every `--sweep`
(600 seconds), for pushes and pulls Jetstream did not ring for.
- `KNOTLESS_RUN_DATA` (`/var/lib/knotless-run`): its session, the refs it
last ran, checkouts while they run, and the workflows' caches in
`caches/`, at the same path inside as on the host, for the step
containers to mount.
- The caches (`cache:` in a workflow): each path is a base directory,
`caches/bases/<cache>/<generation>`, which a run sees through an overlay
of its own, a `local` docker volume of type `overlay` mounted with
`userxattr`: the base read only below, `caches/runs/<run>/` taking what
the run writes. So runs of one workflow go side by side, waiting only
for one of the `--jobs` (4). Only a push to the default branch that
passes keeps what it wrote: its layer goes into a new base, of hard
links to the old one's files where it left them, which the runs after
it start from, and the old base goes once no run reads it. A push that
takes a pull's checks rather than running them (see `KNOTLESS_RUN_ARGS`)
takes that run's layers too: an owner's or collaborator's pull run that
passes keeps them, in `caches/kept/<run>/`, and the push promotes them
as its own run's would be, onto the base each was written over. One over
a base that has moved on since, as when main ran in between, is not
applied, its whiteouts being of what that base held, and the next run
of main warms it; the push's log says which. Kept layers go once taken,
once their pull is at another head or base, or a pass after it closed,
and past three days, sixteen runs, or `--cache-max` GiB for them all,
the oldest first; a start removes those of runs no push could take. Any
other run's layer, an untrusted pull's always, goes when it ends, and
what a stopped runner left goes when it starts again. An untrusted
pull of a workflow that asks for secrets starts from an empty base of
its own instead, as the default branch's may hold what those secrets
left. A base over `--cache-max` GiB (16) once made is dropped, for the
next run to start afresh. Disk, then: each base once, each run going's
own writes on top (a test run's build, a few GB), and up to
`--cache-max` GiB of kept layers. Below `--disk-floor` GiB free (20),
a run sheds the bases no run reads and the kept layers before it starts,
and waits for room if that is not enough, so the site on the same disk
keeps room to write. The host needs overlayfs over ext4
with user xattrs, as Linux 5.11 and up has. The docker daemon mounts
the overlays by the host's paths, one more reason `KNOTLESS_RUN_DATA` is
the same path inside and out. The first start after the switch from
volumes removes the old `knotless-cache-*` volumes, and runs start cold
until each workflow's first run on the default branch passes.
- `KNOTLESS_RUN_ARGS`: more of its command line, such as `--pulls`, to
run open pulls too, which it finds through `serve`: each at its head
merged with its base, those by the owner or a collaborator as they
move, anyone else's once the owner asks with `knotless run rerun --pull`.
A push to the default branch at the tree an owner's or collaborator's
pull passed at, as merging it leaves the branch, takes those checks as
passed rather than running them again, its log saying whose they were;
a workflow that asks for secrets, or whose runs finish, as deploy.yml,
runs as ever.
`--queue` also lands the merge queue (`knotless pull merge --queue`): up to
`--batch` (8) queued pulls at a time, each squashed onto the branch as
its own commit, checked once at the batch's tip. While one is checked
the next is checked on its tip (`--depth 1` turns that off): it runs as
a pull's run does, after main's runs and with a pull's CPU share, is
dropped if the batch it is on fails or changes, and lands only on the
commit that batch landed as. It reads the queue records from the repos
of the owner and collaborators, not only from `serve`'s index, and
looks at them every `--notice` (30) seconds besides Jetstream, so a
pull queued starts within that. It pushes as the runner's account,
as the owner or a collaborator who may write (`knotless repo share <url>
<handle> --write`), or, as a runner the repository names when none it
names may write, through the gateway, once the owner delegates writes
there ([Who lands a
queue](../docs/design/merge-queue.md#who-lands-a-queue)). While it lands the queue, open pulls not
in it run again only as their heads move, after the queue's batches.
- `/etc/knotless/run-secrets.json`: the secrets it grants, by
`<handle>/<name>`, as `{"alice.example.com/tally": {"DEPLOY_TOKEN":
"…"}}`, or `{"DEPLOY_TOKEN": {"value": "…", "refs":
["refs/heads/main"]}}` for pushes to `main` alone. It starts empty; edit
it on the host, and the next run reads it.
### When a deploy restarts the runner
A restart costs little since the runner hands over ([How the runner
hands over](#how-the-runner-hands-over)), but the new runner still waits
for the jobs the old one's runs hold, so a deploy restarts `run` only
when what it runs changed. `deploy.sh up`
compares two things, and leaves `run` running only when it knows both
are the same:
- **Its build.** The image keeps a fingerprint of what the runner is
made of, in `/etc/knotless-run.fingerprint`: the `knotless-run`
binary, the docker client its steps run through, and the packages
installed (`/var/lib/dpkg/status`: git and the libraries). `up` reads it
from the running container and from `knotless:current`, the image just
built. The commit the binaries log is set as they run (`ENV
KNOTLESS_COMMIT`, the image's last layer) rather than built in, so a
commit that changes only serve, the pages, another binary or the docs
leaves the runner's binary, and so its fingerprint, as it was.
- **Its settings.** A hash of the `run` service as `docker compose config
run` resolves it from the compose file and the `.env` this deploy just
wrote (its command line, `KNOTLESS_RUN_ARGS` and `KNOTLESS_RUN_REPOS`
among it, its environment and mounts), and of `run.env`. Each restart
removes `/etc/knotless/run-deployed` before it starts and, once the new
container is up, writes `<container id> <hash>` there. A deploy whose
hash, or whose running container, is not the one recorded restarts it.
`run-secrets.json` is not in it: the runner reads that at each run.
Anything it cannot tell, a container not running, no fingerprint, a
record missing, restarts `run` as every deploy once did;
so the first deploy with this does, and so does one while a restart is
still under way. Either way the deploy log says which, and why:
```text
2026-10-27T04:02:11Z 1a2b3c4d5e6f → run: its build, 5d41402abc4b, and settings are the same, not restarted
2026-10-27T05:13:40Z 7d8e9f0a1b2c → run: restarting: its build changed, 5d41402abc4b to 7d793037a076
```
With `KNOTLESS_RUN_REPOS` empty, a deploy that finds `run` running, as
an earlier deploy with repositories to watch left it, stops and removes
it, once its own run is filed if it runs under it, as a restart would,
and starts no other: `run: removing: no KNOTLESS_RUN_REPOS` in the
deploy log.
A runner left running keeps logging the commit it was built from at its
start, as it should. To restart it anyway, deploy with
`KNOTLESS_RUN_RESTART=1`, or on the host `cd /etc/knotless && docker
compose up -d --force-recreate run`, after which the next deploy, finding
no record for that container, restarts it once more. To ask the first
question by hand, on the host: `docker exec <container> cat
/etc/knotless-run.fingerprint` beside `docker run --rm --entrypoint cat
knotless:current /etc/knotless-run.fingerprint`.
### How the runner hands over
A restart does not stop the runner and then start the new one.
`handover.sh` sends the runner leading SIGTERM, and stops docker
restarting it. The runner drops the runs waiting that it will not run,
writes its state file a last time, and from then on writes a note of its
own under `state.handover/` beside it. It goes on finishing the runs it
has going: those started, a merge-queue batch's, and those at a going
deploy's commit. Once the note is there, `handover.sh` starts the new
runner beside it (`docker compose up --scale`). The new one reads the
state file, leaves the old one's runs, checks and caches alone, and
takes no job the old one's runs hold. The old one exits once its runs
end, `--stop-wait` (an hour) after its stop at most, and is removed
then. It holds only the jobs the new one lends it meanwhile.
The deploy log says how it went:
```text
2026-11-03T18:02:11Z → run: handed over in 2s; starting the new one beside 4f1c2a9b0d3e
2026-11-03T18:02:14Z → run: started 9a8b7c6d5e4f
```
A runner that writes no note within `KNOTLESS_HANDOVER_WAIT` seconds
(60), one built before runners handed over, is stopped first instead,
and the new one started once it has, as every deploy once did. While one
is still finishing, `docker compose ps run` lists two: the one docker
restarts (`docker inspect -f '{{.HostConfig.RestartPolicy.Name}}'` says
`always`) leads, and a deploy compares only that one.
## knotless-triage
Set `KNOTLESS_TRIAGE_REPOS` to the repositories to triage and
`KNOTLESS_TRIAGE_ARGS` to the rest of its command line (the deploy
workflow gives `--model clef --scopes kind,area --discover`: with
`--discover` it also triages every repository that names its account a
triager, as the runner host runs those naming it, so an owner turns it on
with `knotless triager add <url> <handle> kind area` alone). It starts once
`/etc/knotless/triage.env` is on the host, which a deploy from one's own
checkout writes, once, from `deploy/local.env`:
- `KNOTLESS_TRIAGE_IDENTIFIER` and `KNOTLESS_TRIAGE_PASSWORD`: the
account its labels are written in, and an app password for it. Not the
owner's: an owner's labels are a person's, which outrank a triager's
and count at once. For private repositories it logs in at `/oauth/bot`
as the runner does.
- `KNOTLESS_DECIDE_URL`: the decision model's endpoint, for Clef
`https://api.cloudflare.com/client/v4/accounts/<account id>/ai/run/@cf/cloudflare/clef`.
- `KNOTLESS_DECISION_TOKEN`: its token, for Clef a Cloudflare API token
that may use Workers AI.
Until then a deploy leaves it out and says so. Its labels count for
nothing until the owner names its account a triager of some scopes, with
`knotless triager add <url> <handle> kind area`; `knotless triager list <url>
eval <handle>` says first how often they agree with people's. It logs each
issue it judged, or failed to, as `judged` or `judging failed`.
## The keeper
`knotless-agent keep` brings the pulls the merge queue turned away for a
conflict up to date and queues them again
([the merge queue](../docs/design/merge-queue.md#keeping-turned-away-pulls-moving)).
Set `KNOTLESS_KEEP_REPOS` to the repositories it keeps, or `--owner
<handle>` for all of an account's, and `KNOTLESS_KEEP_ARGS` to the rest
of its command line (`--attempts 2`, say). It starts once
`/etc/knotless/keep.env` is on the host, which a deploy from one's own
checkout writes, once, from `KNOTLESS_KEEP_IDENTIFIER` and
`KNOTLESS_KEEP_PASSWORD` in `deploy/local.env`: its account, and an app
password for it. The account is its own, not the owner's, and each
repository it keeps names it a collaborator (`knotless repo share <url>
<handle> --write`), as it pushes to their pulls' branches and queues
them. Until then a deploy
leaves it out and says so. It logs each pull it kept, or failed to, as
`kept` or `keeping failed`.
## Monitoring
`/healthz` says only that serve answers, and Docker restarts it when it
does not. `/_status` says whether it is doing its job, for a monitor off
the host to poll: 200 when it is, 503 and what failed when not.
- **Feeds**: each one followed (Jetstream, the firehose) is connected and
brought an event in the last two minutes (`--stale <secs>` to change
it). The firehose brings hundreds a second; Jetstream, asked only for
`dev.knotless.*`, still brings every account's identity and account
events, one every few seconds, eleven at the longest gap measured. With
both followed, one bringing events is enough, and each is still shown.
- **Index**: it answers a read.
- **A mirror**, with `?repo=<handle>/<name>`: its default branch's commit
and tree are read as its code page reads them, so a mirror whose packs
can no longer be read (as when one outgrew its open handle's pack slots)
fails here, and its pack count shows.
- **Disk**: the one serve's cache, and with it the mirrors, is on
(`KNOTLESS_CACHE_DIR`, `/data/cache`). Under 15% free it is `low`, a
problem told of while the answer stays 200; under 5% it is `full` and
the answer is 503. On 2026-10-31 that disk filled and every page that
needed a mirror answered 502 (`No space left on device`); 15% of its
150 GB is a day or more of deploys and runs to clear it in. A keyword
monitor (step 4 below) shows a `low` disk before it fails.
- **Code**, with `--code-index`: Zoekt's webserver answers, and the
repositories, shards and bytes its index holds show. A webserver that
died fails here, as every code search then fails.
It takes milliseconds, so polling it every minute costs nothing. A dead
upstream feed is not a reason to restart serve, so the compose healthcheck
stays on `/healthz`.
```sh
knotless status https://knotless.dev --repo chadtmiller.com/knotless # or --json
curl -s 'https://knotless.dev/_status?repo=chadtmiller.com/knotless' | jq
```
```json
{
"ok": true,
"problems": ["jetstream: not connected"],
"uptimeSeconds": 5321,
"feeds": [
{"name": "firehose", "url": "wss://relay1.us-east.bsky.network", "ok": true, "connected": true,
"silentSeconds": 0, "staleAfterSeconds": 120, "events": 2103344, "seq": 26329004819},
{"name": "jetstream", "url": "wss://jetstream.us-east.bsky.network", "ok": false, "connected": false,
"silentSeconds": 7310, "staleAfterSeconds": 120, "events": 1650, "seq": 26328850636}
],
"index": {"ok": true, "error": null},
"mirror": {"repo": "chadtmiller.com/knotless", "ok": true, "error": null,
"commit": "b1fbf45…", "packs": 3},
"disk": {"path": "/data/cache", "state": "ok", "ok": true, "freeBytes": 41203998720,
"totalBytes": 160982630400, "error": null}
}
```
### An external monitor
The check has to run somewhere other than the host, or the host going
down goes unreported. An external uptime service is the least to run:
nothing to keep alive, and it pages a phone. UptimeRobot's free plan is
enough (Better Stack or an Uptime Kuma elsewhere do the same):
1. Sign up at uptimerobot.com and install its app, or add an email or
Slack contact under *Integrations & API*.
2. *New monitor*: type HTTP(s), URL
`https://knotless.dev/_status?repo=chadtmiller.com/knotless`, interval
5 minutes (the shortest the free plan has), timeout 30 seconds. Any
status but 2xx is down, so a 503 alerts. Name it "knotless status".
3. *New monitor*: HTTP(s), `https://knotless.dev/`, 5 minutes: the pages,
which `/_status` does not draw.
4. Optionally a keyword monitor on the status URL for `"ok":true`, which
shows the body's `problems` in the alert.
5. Pick the contact under each monitor's *Alert contacts*, and *Notify
after* 1 check, or 2 to ride out a deploy's restart.
A 503 then reaches the phone within the interval plus two minutes of a
feed going silent, or at the first poll after a mirror stops reading.
## Logs
serve and the runner log JSON lines on stderr, one an event, at the level
`RUST_LOG` sets (`KNOTLESS_LOG` in `.env`, `info` unset); the pages log
what node prints. Docker writes each container's to the host's journal
(the `journald` driver, `x-logging` in docker-compose.yml), so they
outlive the container: a rollout removes the old one, and its lines stay,
for 30 days: setup.sh sets journald's `MaxRetentionSec`, which the
privacy policy promises (journald's own cap, a tenth of the disk and at
most 4 GB, can drop the oldest sooner). Each line names its
project and service (`COM_DOCKER_COMPOSE_PROJECT`,
`COM_DOCKER_COMPOSE_SERVICE`) and its container (`CONTAINER_TAG`). deploy.sh and rollout.sh add a line with the time for each
step of a deploy (built, started beside the old one, healthy, the old one
stopped) to `/var/log/knotless-deploy.log` on the host, to line a freeze
up against. [docs/observability.md](../docs/observability.md) says what
was chosen, and why.
What serve logs:
- `request`, one a request: `http.route` (its template), `url.path` (no
query), `http.request.method`, `user_agent.original` (the client as it
names itself), `http.response.status_code`,
`http.response.body.size` when known, `duration_ms` and `trace_id`, at
info; at warn when over `KNOTLESS_SLOW_MS` (1000), unless it is a long
poll (`awaitSpaceWrites`, `getDelegation`), which waits on purpose; at
error for a 5xx with its `error`, and `/healthz` at debug. The answer carries the trace
as `x-trace-id`; the pages hand it back on their XRPC calls, so a
page and the data it drew share one.
- `outgoing`, each call made, to a PDS, the PLC directory, Bluesky's CDN
or a handle's `/.well-known`: `server.address`, `url.path`,
`rpc.method` for XRPC, `http.response.status_code`, `error.type`,
`http.request.resend_count`, `duration_ms`, `trace_id`. At debug, or
warn when it failed, answered 5xx or was slow. It sends `traceparent`.
- `sql`, each of the index's statements that took a tenth of
`KNOTLESS_SLOW_MS` (100 ms) or more, at warn: `db.query.text` (the SQL
as written, its whitespace closed up, cut at 500 characters, with no
value it was given) and `duration_ms`, with the `trace_id` of the
request it ran in. The slow request whose line shares that trace is
the one it made slow.
- `summary: requests`, `summary: outgoing` (by host), `summary: cache`
(`hits`, `misses`, `failed` fills, by cache) and `summary: feed` (lag,
events) each minute, for what saw any.
- `mcp tool failed` at warn, one a tool call that failed: `gen_ai.tool.name`,
`did`, `error` (what the tool answered, as the client saw it),
`duration_ms` and `trace_id`; `mcp resource failed` the same, of a
resource read, named for the tool that answers it.
- `mirror sync`, one each sync of a repository's mirror: `repo`,
`duration_ms`, and `records_ms`, the part spent reading its records. At
info, or warn when slow. Reads of the repository wait on a sync under
way, so it explains a burst of slow requests for one repository that
end together.
- `cache fill failed` at warn, with why: an avatar, claim or picture that
could not be read.
- `knotless-serve starting` (`version`, `commit`), `index opened`
(`duration_ms`, `rebuilt`), `listening`, `stopping`, `stopped`;
`feed connected` and `feed disconnected`; `index sweep started` and
`ended`; `event dropped: not indexed`; `account not backfilled`; `panicked`.
What the runner logs: `run queued`, `run started`, `run ended` (`status`,
`duration_ms`), `run cancelled before it started`, each with `repo`,
`workflow`, `ref_name`, `commit` and `pull`; `step started` and `step
ended`; `container started` and `stopped`; `batch started`, `landed`,
`dropped` and `pull ejected` (`reason`); `queue noticed` (debug); `pds
write retried`; `feed connected` or `down`; `failed` for anything that
went wrong.
None logs a header, a token or a body. A request line logs its query
(`url.query`), each value a secret could be in replaced by `_`, and none
for `/oauth`; no call made logs one. DIDs are logged;
handles only as a path or a `/.well-known` host names them; client IPs
never.
`deploy/logs.sh` (`just logs`) reads them from here, every container of a
service at once, those gone too, through jq:
```sh
just logs serve 'select(.duration_ms > 1000)' # slow requests and calls
just logs serve --since 2h 'select(.level == "WARN" or .level == "ERROR")'
just logs serve 'select(.message == "outgoing" and .level != "DEBUG")' # calls that failed
just logs serve 'select(.message | startswith("summary"))'
just logs serve 'select(.trace_id == "4bf92f3577b34da6a3ce929d0e0e4736")' # one request, and the pages' calls for it
just logs run --follow 'select(.message | startswith("run"))'
just logs runner --since 1h 'select(.message == "batch started")' # the runner host's, which lands the queue
just logs deploys # the deploy steps, with times
```
`run` is a runner on this host. `runner` reads the runner host's
(deploy/runner-host) instead, at `KNOTLESS_RUNNER_HOST`
(runner.knotless.dev unset), by its container's tag, as that host's
lines name no compose project. `KNOTLESS_RUNNER_ENV=deploy/ci.env just
logs runner` reads another runner, such as the trusted one, from its
own settings, as its deploy does.
Or on the host itself, with journalctl, by the service's field; `-o cat`
gives each line as it was written:
```sh
ssh root@knotless.dev 'journalctl -o cat --since -30m COM_DOCKER_COMPOSE_PROJECT=knotless COM_DOCKER_COMPOSE_SERVICE=serve' | jq -Rc 'fromjson? | select(.duration_ms > 1000)'
ssh root@knotless.dev 'journalctl -o cat --since -1h COM_DOCKER_COMPOSE_SERVICE=serve' |
jq -Rc 'fromjson? | select(.message == "outgoing") | {t: .timestamp, host: ."server.address", path: ."url.path", status: ."http.response.status_code", error: ."error.type", ms: .duration_ms}'
ssh root@knotless.dev tail -n 50 /var/log/knotless-deploy.log
```
Each request's line names its trace; a page's own line comes after those
of the XRPC calls it made, since a line is written when its request
ends. Calls are logged only at warn by default; to see every one for a
while without a restart, set the level through `--metrics`' port, inside
the container:
```sh
ssh root@knotless.dev 'for c in $(docker ps -q -f label=com.docker.compose.project=knotless -f label=com.docker.compose.service=serve); do docker exec $c node -e "fetch(\"http://127.0.0.1:9464/log\", {method: \"PUT\", body: \"info,knotless::outgoing=debug\"}).then(r => r.text()).then(console.log)"; done'
ssh root@knotless.dev 'docker exec $(docker ps -q -f label=com.docker.compose.project=knotless -f label=com.docker.compose.service=serve | head -1) node -e "fetch(\"http://127.0.0.1:9464/metrics\").then(r => r.text()).then(console.log)"' | grep -v '^#'
```
`PUT` it `info` again after; a restart goes back to `KNOTLESS_LOG`.
### Without SSH
serve keeps each line it logs in `/data/logs.db` too (`--logs`), as it
writes it to stderr, a second at a time. Every serve writes there, both
during a rollout, so a deploy loses none. An operator reads them with
`knotless admin logs` or MCP's `logs` tool, from `dev.knotless.getLogs`,
without SSH:
```sh
knotless admin logs --trace 4bf92f3577b34da6a3ce929d0e0e4736 # one request, and the calls it made
knotless admin logs --message sql --limit 20 # the last slow statements
knotless admin logs --level warn --min-ms 2000 # what was slow, and warned
```
A read needs one of `--trace`, `--message`, `--level` or `--min-ms`,
answers at most 500 lines, and is itself logged as `logs read`, with
who asked and for what. Lines at info and below, the most of them, are
kept to the last 500,000 (`--logs-keep`); every line for 30 days at most,
as the journal keeps them. They hold what the journal does, no more.
`/metrics` gives what was counted since the start, in Prometheus's format:
`http_server_request_duration_seconds` by route, method and status,
`http_client_request_duration_seconds` by host and outcome,
`knotless_cache_lookups_total` by cache and result,
`knotless_cache_build_duration_seconds` by cache and how a miss was made
good (`patched` or `rebuilt`), and each feed's
`knotless_feed_connected`, `knotless_feed_events_total` and
`knotless_feed_lag_seconds`.