chadtmiller.com/knotless
git that lives in your Atmosphere account
7
# Working on knotless
For every agent working here: CLAUDE.md is this file, imported.
Use `rustup run stable cargo` (the Justfile's `cargo`), not a system cargo.
Node is the version `.node-version` names, CI's: the harness's PDS loads a
better-sqlite3 built for it, and refuses another major.
Tests run against the npm reference PDS through `harness/`, never pds.js.
## Running tests
Build and test on the runner, not on this machine: several agents
building here at once have brought it down. Open the pull early, push to
its branch as you go, and read what its checks say with `knotless checks
<repo> --pull <uri> --wait <secs> --log`. Each push runs warm from
`main`'s caches (see What CI checks): `check` (fmt and clippy) in a minute
or so, `test` in a few, running only the test binaries the push changed.
`cargo fmt` costs nothing, so run it before pushing.
Build here only for a loop the runner is too slow for, such as one test
being fixed, and then in one checkout at a time, with
`CARGO_BUILD_JOBS=2`, and `just pds` stopped once done. Never run `just
cov`, `just lint`'s clippy over the workspace, `shots-linux`,
`shots-check` or `shots-live` here: their checks run them. What follows
is for that loop.
- `just pds` in the background first: test binaries then share that PDS
instead of each starting one.
- `just changed` names the files changed since tests last passed and the
tests that cover them; `just t-changed` runs those and marks the pass.
`just t <cargo test args>` runs any selection on the shared PDS.
- The web pages are the SvelteKit app in `web/`; knotless-serve answers
what they show as XRPC queries, each with its lexicon under
`lexicons/` and the views it shares in `dev.knotless.defs`, as
atproto's appviews do. The pages' TypeScript types are made from those
lexicons by @atproto/lex (`npm run lex` in `web/`, which `dev`, `build`
and `check` run first), into `web/src/lib/lexicons`, not kept in git.
`just lex` checks the answers the screenshots recorded against their
lexicons, and the `test` check every answer the serve tests receive.
- Before changing a page, load the `ui` skill (`.claude/skills/ui`):
its house rules (sentence case, empty states, space), and `just
look`, which draws any page in any state from the recording as
pictures to open: empty, signed in, a menu open, a long title. Look
at every state you touch before pushing.
- For the web pages: the pull's `shots` check shows what moved, with
pictures. No screenshot is kept anywhere: the pages are shot as main
draws them and as the change does, in one run. To look before pushing,
`just shots`, which builds `web/` and main's, shoots only the pages
whose HTML differs, drawn from `harness/shots/fixtures.json` with no
PDS, and leaves each that moved under `target/shots/results/` (its
expected, actual and diff). After a change to what a page's answer
holds, `just shots-up` in the background once, then `just
shots-record`. Each checkout's shots have three ports of their own,
from its path; `SHOTS_PORT` sets another base for all of these.
- `shots-live` shoots this checkout's pages from the recording and again
through knotless-serve over a world: one that moved means the
recording is out of date. Keep `just shots-up` running in the
background while working on pages: `shots-live` then reuses its world
rather than seeding one, which is most of its time.
- The social cards are drawn by knotless-serve: `just shots-cards` takes
them from a world into `target/shots/cards` to look at.
- The components in `web/src/lib` are layered, atomic design's way:
`atoms` (one element: an icon, an avatar), `molecules` (a few atoms
that mean something together: a person, a commit row), `organisms` (a
part of a page: a check list, a feed), `templates` (a page's layout,
without its data). A component uses its own layer and those below it,
never one above; `just lint` checks it. Pages under `web/src/routes`
fill a template with what the queries answer.
- The pages' styles are `web/src/app.css`, and every colour, size,
space, radius and weight in it is a token from `web/src/tokens.css`:
take one by name rather than writing a value, and add one there when
none fits. A colour for what it means is its role, `--danger`,
`--success`, `--attention`, `--done` or `--neutral`, never the hue
under it (`--red`), so a theme can draw success in blue. Every colour that is text meets WCAG AA (4.5:1) on what
it is drawn on, in every theme: `harness/contrast.mjs` checks it, in
`web` and `just lint`, and a new colour that is text goes in
`web/src/lib/contrast.json`, which the theme editor, and knotless's
own check of a theme made anywhere (`crates/knotless/src/theme.rs`),
hold people's own themes to too. A menu that opens is a `<details class="popover …">`.
- In Svelte markup, keep a run of inline text one expression or one line:
a block or component beside it splits the text into nodes, which shapes
differently from the same text whole and moves pixels.
- Done: all seven of the pull's checks pass. When one fails in a way its
log and pictures do not explain, run that one thing here, alone
(`COV_CLEAN=1` rebuilds everything first, for a cov miss that makes no
sense).
## Lexicons
A new or changed schema under `lexicons/` follows the Lexicon Style Guide
(https://atproto.com/guides/lexinomicon) and passes `just lex`: goat's
linter, less two warnings kept on purpose (`pack`'s inline `bytes` and
`listReports`' `id`), then `harness/lexicons.mjs`. That checks what the
linter does not see: `knownValues` rather than `enum`; objects in new
arrays rather than bare values; open unions; a description on every
`main`; a `maxLength` on strings without a format in what people write
(records and procedure input); accounts by DID, not handle. And that every
schema only grows from main's, as the branch left main: a schema already
published is evolved, not changed, with new fields optional and nothing removed, renamed, retyped,
made required or narrowed. Fix anything either says, or say why not.
Still by hand: no `required` a reader could do without, since it can never
be dropped; a description on any field whose name does not say what it
holds; `maxGraphemes` beside `maxLength` for text people read; reuse of
`dev.knotless.defs` and `com.atproto.repo.strongRef` over new shapes.
`cargo fmt` can put a `?` on a line of its own, which coverage then counts
as a line never run: bind the value first, or use a named function for a
row mapper, rather than lowering the gate.
## Queries
A query of the index (`crates/knotless-serve/src/index.rs`) is planned
as the tests run it, against tables said to be prod's size
(`crates/knotless-serve/src/plans.rs`), and `test` fails on any that
scans a table that grows with the network: `forge`, `check_run`,
`comment` and the rest of `GROWING`, or sorts many of its rows (a range
leading its index, or what a search finds for each row of a scan, as a
repository's pushes for every repository). Tests on empty tables pass a query
that takes seconds on prod, so this is what holds them. Its log names
the query and its plan. Give it an index, or a query that uses the one
there: an expression index needs the expression as written, and a
column compared to it marked `+` (`json_extract(c.value, '$.pull') =
+p.uri`), as the column's affinity otherwise keeps the index unused.
A scan meant to be, of a few rows an operator reads now and then, says
so in its SQL with why, `-- scans: forge, as only the operator asks`,
where review sees it; a sort meant to be, `-- sorts: forge, …`. On prod,
a statement slower than 100 ms logs `sql`, with the request's trace. To see the plans while fixing one, run its tests
with `KNOTLESS_PLANS=<dir>` and `node harness/plans.mjs <dir>`.
## Docs
`docs/` is a small wiki: design pages explain why, the guide shows how,
the reference lists what. A change that alters what a page says updates
that page in the same pull:
- storage records, push, claims, compaction: `docs/design/storage.md`
- gateway, delegation, `pushRefs`, `createCommit`: `docs/design/gateway.md`
- pulls, issues, states, merging, moderation: `docs/design/collaboration.md`
- the merge queue: `docs/design/merge-queue.md`
- knotless-run: `docs/design/runner.md`, `docs/reference/runner-internals.md`
- knotless-triage and `triage.rs`: `docs/design/collaboration.md` (Labels),
`docs/reference/cli.md`
- knotless-serve, its cache, appview and pages: `docs/design/serve.md`,
and its methods in `docs/reference/api.md`
- names, redirects, transfers, commit authors: `docs/design/identity.md`
- spaces and private repositories: `docs/design/access.md`
- releases, `publish`, versioning: `docs/design/packages.md`
- a flag, environment variable or command's output: `docs/reference/cli.md`
- an MCP tool: `docs/reference/mcp.md`
- a workflow key: `docs/reference/workflows.md`
- who can do what: `docs/design/security.md`
To write or rewrite a page, or to check one against the code, use the
`docs` skill (`.claude/skills/docs/SKILL.md`): facts from the code first,
then the page from the reader's questions. `just docs` checks links,
anchors and diagrams (`npm ci` in `web/` first, for Pele).
The pages also serve the docs as a site at `/docs`: `docs-site/`, a
Starlight site that reads `docs/` as it is, each page's `#` heading its
title, its links made the site's, and its sidebar in `astro.config.mjs`.
A page added to `docs/` goes in that sidebar too. `just docs-site` shows
it as you write.
## Versioning
A pull that changes what those who use knotless see (a command, a flag,
what the client does) adds a change file under `.changeset/`, as
`knope document-change` writes one (https://knope.tech): `default:
patch`, `minor` or `major` in its front matter, then a heading and a
line for them. `knope release` turns them into the version and
CHANGELOG.md. Commit titles stay prose: no conventional commits. The
`check` check fails a pull that changes `crates/knotless/src` or
`crates/knotless-mcp/src` and adds no change file, unless its body says,
on a line of its own, `Changeset: none`: for a change no one using
knotless sees.
## What CI checks
The workflows in `.knotless/workflows/` run on pushes to `main` and on
pulls into it or into a `feature/` branch (see Landing a change), each
pull at its head merged with its base as it is now, and again whenever
either moves; a push to any other branch runs nothing.
On a pull, `web`, `lexicons`, `shots`, `docs` and `e2e` run only when it
changes what they read (each file's `paths`); otherwise each passes at
once, noted as skipped. A push to `main` runs all seven, and `check` and `test` run
on every pull. `main` deploys only once they all pass:
- `check`: `cargo fmt --check` and clippy, warnings denied; on a pull,
a change file for a change to the client (`harness/changes.sh`).
- `test`: `just cov`'s 100% gate, over the spaces alpha, and the stock
PDS for the few test binaries that reach what a PDS without spaces does
(`COV_STOCK` in harness/cov.mjs),
and that every XRPC answer the serve tests received, kept with a
stored run or fresh, is as its lexicon says (`harness/answers.mjs`
over what `harness/cov.mjs` gathers), and that no query the index
ran scans a table that grows with the network, as SQLite would plan
it at prod's size, or sorts many of its rows (`harness/plans.mjs`; see Queries). A test binary that
passed before is not run again: when it, the workspace binaries it ran
and the harness are the same, its stored profile counts instead
(`COV_STORE` in harness/cov.mjs), so a change runs only the binaries it
changes. A pull's run, its pre-flight, goes against both nets and
holds the gate, as a batch's does, so a line nothing reaches fails
the pull rather than a batch. A binary that fails is run once more; one that then passes is
logged `FLAKY: <net> <binary> <test>` with its first failure: fix it.
One that fails again only in tests `harness/quarantine.json` names,
until the day it gives, passes too, logged `QUARANTINED: <net> <binary>
<test>`, and is run alone until it passes, three times at most, so
the lines only it reaches still count: a test that fails now and then
is put there while it is fixed, and taken out with the fix.
- `web`: the component layers and svelte-check.
- `lexicons`: `just lex`, against main's schemas where the branch left main.
- `shots`: `just shots-check` in Playwright's Linux image: each page whose
HTML differs from the base's, shot as the base draws it and as the
change does. Its pictures (before, after, and where they differ) of
each that moved, or is new, are the check's outputs, and one the pull's
body does not name on a `Shots-moved:` line fails it.
- `docs`: `just docs`: every relative link and anchor in the docs lands,
and every Mermaid block draws with Pele; and the site at `/docs`
builds.
- `e2e`: `just e2e`: the pages used in Chromium over a world seeded for
the run as the shots' is, served as a gateway behind the harness's
https front: logging in through the PDS's approval page, then every
write the pages make, from starring and commenting to merging,
settings, organizations, transfers and deleting an account's data,
as two accounts, each read back on the page and from the PDS
(`harness/e2e/flows.spec.mjs`); then a private repository's, over
the spaces alpha, each read back from its space and as not there to
anyone else (`private.spec.mjs`, `just e2e-private`). A flow that
fails leaves its page among the check's outputs. A new control that
writes gets a flow there, and in the private flows too when a private
repository reads it back differently.
A push to `main` at the same tree as a pull's run that passed, as
`knotless merge` leaves it, does not run these again: the runner files
them as passed at once, each log naming the pull's run. deploy.yml still
runs, and waits for them as ever.
`scenarios.yml` is not one of the seven: it runs the scenario tracks on
main only when the owner asks for it, as `knotless rerun <repo> --ref
refs/heads/main --workflow scenarios -e TRACKS="team"`, and costs model
usage, so ask before starting one. Its report is the end of its log.
`look.yml` is too, at any branch: `just look` for an agent with no
checkout, its pictures the check's outputs (the `ui` skill says how).
`docs-audit.yml` is the same kind: Claude Code sessions checking each
docs page against the code (`-e PAGES="design/storage.md"` for some),
whose report says what each page gets wrong or leaves out.
What `just lint`, `just cov` and `just lex` pass here passes there. A
check's result and log: `knotless checks <repo> --pull <pull uri>`
(`--wait <seconds>` waits for it; `--log` adds each log's end; `--json`
gives each step), or the `checks` tool over MCP. Its pictures:
`--images <dir>` saves them to open, and over MCP `images: true` returns
them to look at. A pull that moves pages on purpose names them in its
body, on a line of their own, `Shots-moved: home, pull-open` (the names
its check gives; `all` for every page), then runs that check again
(`knotless rerun <repo> --pull <pull uri> --workflow shots`): the body is read as the check
runs, and editing it moves nothing. A batch reads the line from the
squashed pulls' bodies. One moved by mistake is a fix to make.
A change to a page in a state the check does not draw, as a menu or
dialog open, an edit box, a filter that empties a list or a button
busy, shows that state in the pull: `just look`'s pictures of it,
desktop light and phone dark, each given to `knotless pull` (or
`knotless edit`) as an `--image`. Only the states the change touches:
the check already shows what moved of the pages it records.
Prod's health: `knotless status https://knotless.dev --repo
chadtmiller.com/knotless` (`--json` for all of it): its feeds, index and a
mirror, from `/_status`; deploy/README.md says what each means.
## Landing a change
Open a pull rather than committing to `main`: push the work to a branch
named for it (`web/mermaid`, `serve/jetstream`), then `knotless pull
knotless://chadtmiller.com/knotless main knotless://chadtmiller.com/knotless
<branch> "<title>" -m - < body.md`, the title and body written from
`.knotless/pull.md` (`knotless template <repo> pull`); `-m -` keeps the
shell off the Markdown. It is merged into `main` and deployed from there. A body saying `Fixes <issue>` (or Closes, Resolves), the issue by
its knotless.dev page or `at://` URI, closes that issue as it merges. Once its own checks pass, land it with `knotless merge <repo>
<pull uri> --queue`, which returns at once: the runner lands queued
pulls in batches, main with each squashed onto it as a commit of its
own, checked once at the batch's tip, so many landing together cost one
run. It takes a pull queued in within half a minute, and checks the next
batch on top of the one being checked, so a pull queued behind a batch
does not wait for that one to land. `knotless queue <repo>` lists the queue and what each is doing, and
`--leave <pull uri>` takes one out. A pull that conflicts with those
ahead of it, or breaks the batch (found by halving it), is ejected, and
`knotless show` says why: fix it and queue it again. Plain `knotless
merge` merges at once, and only once all seven checks have passed at its
head merged with `main` as it is now, and otherwise says which failed,
are running, or are of an older head or base: `--wait <secs>` waits for
them, and `--force` is the owner's alone. A pull that conflicts with
`main` fails each check without running: merge `main` into it, or
rebase it onto `main`. The runner checks a pull as it moves only when its
author and its head repository's owner are the owner or a collaborator;
anyone else's is not checked until the owner runs it with `knotless rerun
<repo> --pull <pull uri>`. Every run starts warm from the caches `main`'s
last passing run left, each over a copy-on-write layer of its own, so runs
of one workflow go side by side, and only `main`'s keep what they wrote:
a pull's writes go when it ends. (Anyone else's pull of a workflow that
asks for secrets starts from empty caches instead.)
While the runner lands the queue, a pull not in it is checked again only
as its head moves, not as `main` does, and after the queue's batches:
the queue checks it against `main` once it is queued.
Queue a pull at once, once pushed: the queue holds it until its
pre-flights, `check` and `test` (`queue: first` in their workflows),
pass at its head, and ejects it if either fails, so one that does not
build, fails a test or misses a line never fails a batch. A pass against
an older `main` counts. Pulls queued behind it that are ready go ahead
meanwhile, and `knotless queue` says `pre-flight check` or `pre-flight
test` while it waits. Queueing turns away a pull that conflicts with `main` or whose
checks failed, saying why. One ejected: read why with `knotless show`,
fix it, and queue it again; one that conflicts with a pull ahead of it
can take that pull's branch merged in, or wait for it to land and take
`main`. Ejected three times for the same reason, stop and say so rather
than queue it again. When several agents land pulls at once, one of them
lands them all, and the rest hand it their pulls' URIs rather than each
watching its own. A deploy restarts the runner. A batch whose checks
pass while it stops still lands; one it cuts off is built and checked
again, with nothing to do.
Work that should not reach `main` yet goes on a feature branch,
`feature/<name>`, cut from `main`. Its parts are pulls into it, made as
pulls into `main` are but with `feature/<name>` as the base: the seven
checks run on them at their heads merged with the feature branch, and
`knotless merge`, queued or not, lands them there. Nothing deploys from
a feature branch, and a push to one runs nothing. Keep one pull open
from it into `main`, unqueued, as the check that the whole of it still
passes on `main` as it is now; merge `main` into the feature branch as
`main` moves, since a pull into `main` that conflicts fails without
running. When it is ready, that pull is queued as any other.