chadtmiller.com/knotless
git that lives in your Atmosphere account
7
# knotless-serve
A knotless repository lives as records in its owner's atproto account, on the
server that holds that account, its PDS (see https://atproto.com). Every push
writes records there, and nothing else decides where a ref points.
`knotless-serve` is an optional server that reads those records and makes them
faster and easier to use. It keeps a cache of them, never the source of truth:
whatever it holds can be thrown away and read again from the PDSes. A client
that talks to the PDS directly needs no server at all. [storage.md](storage.md)
describes the records themselves.
## The problems it answers
A PDS is a poor place to read a repository from, and a poor place to
collaborate in.
- It is far away, and hands out only records and blobs: no partial clone, no
shallow fetch, no file at a commit.
- It knows only its own accounts. Pulls, issues and comments are records in
their authors' accounts, spread over many PDSes, and none can list them.
- It lets only its owner write, so a collaborator cannot push there.
- It offers no place to report content, or to take it down.
People also want web pages, which need all of the above. The server answers
each need with a job of its own. It has to stay cheap, so anyone can run one,
and safe, since it shows content strangers wrote.
## One binary, several jobs
`knotless-serve` is one binary. Every job past the read cache is turned on by
its own setting, so a small install runs only what it needs.
| Job | Turned on by | What it adds |
| --- | --- | --- |
| Read cache | always | git over HTTP and an object API from local mirrors |
| Appview | `--index <file>` | an index of pulls, issues, checks and repositories |
| Gateway | `--public-url` | writes for collaborators, as the owner |
| Moderation | index and gateway | reports, takedowns, labels |
| Web pages | `--web <app>` | the SvelteKit app, fed from XRPC queries |
| MCP | `--public-url` | assistants over HTTP, for any account |
| Releases | `--releases <dir or repository>` | client builds and `/install.sh` |
| Images | always | pulls at `/v2/` of the images repositories publish as releases |
Its state is a directory of bare git mirrors and one SQLite file, so the whole
thing fits on one small VPS. Most of it is cache. What it keeps for good is
takedowns, reports, the labels it has given out, and the OAuth sessions owners
grant the gateway, and those of people logged in to the pages; what it holds
from an account goes when the account is deleted at its host
([Accounts that go](#accounts-that-go)). Git is read
only, and the pages write only as someone logged in to them: they star a
repository, follow an account, react to what is said, create and fork repositories, and open and
talk on pulls and issues, through the same code as the CLI and agents over
MCP; everything else is written with git, the CLI and agents.
```mermaid
flowchart LR
PDS[(owners' and authors' PDSes)]
Feed[Jetstream or firehose]
subgraph serve[knotless-serve]
Mirrors[(bare mirrors)]
Index[(SQLite index)]
end
App[SvelteKit app]
Git[git clients]
Browser[browsers]
PDS -- records and packs --> Mirrors
PDS -- changes --> Feed
Feed -- events --> serve
PDS -- records --> Index
Mirrors --> Git
Mirrors -- XRPC --> App
Index -- XRPC --> App
App --> Browser
```
[../../deploy/README.md](../../deploy/README.md) covers running one, and
[../reference/cli.md](../reference/cli.md) lists every flag and message.
## Mirrors and freshness
Each repository the server is asked for gets a bare git mirror in its cache
directory, kept in sync from the owner's records. The mirror is named by the
owner's DID (an account's permanent ID) and the repository's key, so a rename
or a transfer reuses it. Clones and the API follow a renamed or moved
repository without a redirect, because syncing follows the move.
Every object the server hands out is named by its hash, so a stale cache still
cannot lie about an object.
Asking the PDS on every read would make the server as slow as the PDS, so the
question is when to sync:
- While a feed is connected, a mirror is fresh until the feed reports a change
in its owner's account. Reads between changes cost no round trip.
- While no feed is connected, a mirror is fresh for `--ttl` seconds after its
last sync, 5 by default.
- A private repository's mirror, which no feed tells of, is fresh until the
sync of its space ([Private repositories](#private-repositories)) finds
something written there: at once as its authority tells of a write, and
every thirty seconds besides. A server not syncing its spaces falls back
to the TTL.
- A repository found not to be there is taken as not there for the TTL, or
until a feed reports a change in its owner's account.
When the last feed disconnects, every mirror drops back to the TTL, so the
server never relies on a feed that is not there.
## Feeds
A feed is a public stream of record changes across the network. The server
follows Jetstream, the firehose, or both.
Jetstream (`--jetstream`) is a relay that sends JSON. The server asks it for
`dev.knotless.*` collections, and Bluesky's `app.bsky.actor.profile` and
`app.bsky.actor.link`, which profiles are made of, so it carries little
traffic, and each event
carries its record decoded inline. Its frames are unsigned: a reader cannot
prove an event is real. The server gives Jetstream the same trust it gives any
relay it chose, and that trust is bounded. The index is a cache, and mirrors
still sync from the PDS, so a false event can at worst cause a needless sync or
a wrong index row.
The firehose (`--firehose`) is `com.atproto.sync.subscribeRepos`, from a relay
or a single PDS. It sends every commit on the network as CBOR, and an operation
carries no decoded record, so the index has to fetch each one from its author's
PDS. It costs more to follow, and it does not depend on a Jetstream operator.
Profile edits come from every account on the network. The server keeps
those of the accounts whose profiles it has shown and drops the rest as they
arrive, without counting them toward any freshness.
Either way, an event counts toward freshness only when it can change a
repository: a write to the repository's own collections (`repo`, `ref`,
`claim`, `pack`, `redirect`), or an identity, account or sync event. A new
comment does not mark a mirror stale. A counted event refreshes that account's
known mirrors in the background, so the next read is often already warm.
With an index, the server also indexes each event. Records that arrive inline
are written at once, in order, so a later write cannot land before an earlier
one. Otherwise each is fetched and indexed as its PDS now has it.
The index also keeps each handle the pages have shown, so handles outlast a
restart and search can find owners by them. A handle is kept only when it
resolves back to its DID, and an identity event for an account whose handle
is kept looks it up again at once: the event names the new handle, but is not
trusted for it. One kept a day is still shown, and looked up again behind
the page showing it. Without an index, handles are kept in memory.
### Accounts that go
An account event says whether the account is active at its host and, if
not, why: `deactivated`, `takendown`, `suspended` or `deleted` (a relay
adds states of its own, such as `desynchronized`). The server takes each
as follows:
| What the host says | What the server does |
| --- | --- |
| `active` | shows the account again, with what was kept of it |
| `deleted` | drops everything it holds from the account, for good |
| anything else | hides the account, keeping what it indexed |
Only deletion is final at the host, so only deletion drops anything. A
hidden account is left out of every read, as one taken down here is: its
repositories, records, stars, follows and reactions, its mirrors' reads,
and its sign-ins to the pages and MCP. Its records stay in the index, so
it comes back whole once its host says it is active, with no backfill.
Which accounts are hidden is kept across rebuilds, since the records of
one deactivated cannot be read back from its host to find out.
A deleted account loses what both indexes hold from it: its
repositories and their pushes, every record it wrote (pulls, issues,
comments, checks, stars, follows, reactions, labels, blocks, releases),
the spaces it owned and the records in them, what it was shown of
others' spaces, its profile, handle, notification state and invites.
The mirrors of its repositories go, this server's and those read for
each account here, as do the pages' and MCP's sessions for it and the
session it granted the gateway. What others wrote about it stays, as
their records, and so does what the operator holds about it: its
takedowns, the labels given out for them and reports. The usage
counts are kept apart from all of this.
Before dropping anything, the server asks the host the account's
identity names now whether it still has it
(`com.atproto.sync.getRepoStatus`). An account that moved to another host
is deleted from the old one, and a server following that host's
firehose directly hears it; the new host still answers for it, so it
stays. Only an answer that says the account is gone drops anything:
| What is asked | Answers | The account |
| --- | --- | --- |
| its DID document, from PLC or its `did:web` host | `404` or `410` (a tombstone) | gone |
| `getRepoStatus` at the host the document names | `RepoNotFound`, or `active: false` with `status: deleted` | gone |
| `getRepoStatus` | `active: true`, or inactive for any other reason (deactivated, suspended, taken down) | kept |
| either | anything else: no answer, a timeout, a `5xx`, no host named, no `active` | asked again |
One asked again waits 30 s, then twice as long each time up to 10
minutes, six asks in all (about half an hour, long enough for PLC or a
host to come back), and is kept if none of them says.
Details:
- A connection silent for 60 seconds gets a ping, and silent another 60 is
given up, because a half-open connection never ends. A Jetstream
connection that ends is made again in 500 ms, the wait doubling after
each failed attempt up to 60 s. knotless-run and knotless-triage follow
Jetstream through the same loop, `knotless::jetstream::follow`, without
a cursor. The firehose is connected again 500 ms after each end or
failed attempt, with no doubling.
- With an index, the Jetstream cursor is saved per host, so a restart misses
nothing Jetstream still keeps. The firehose's cursor is kept only in
memory: a reconnect resumes from it, and a restart starts at the relay's
present. A cursor means something only on the host that
issued it.
## Git and the object API
Git clients clone over smart HTTP: `info/refs` and `git-upload-pack` under
`/<authority>/<repo>`. The server runs `git upload-pack` against the mirror and
passes the client's `Git-Protocol` header through, so protocol v2, `--filter`
and `--depth` work, which a PDS cannot offer. A push is refused: it goes to the
PDS or the gateway.
The object API under `/api/<authority>/<repo>` answers what a PDS cannot: refs,
a tree, a blob or a log at a revision, a three-dot diff, or one raw object by
its hash. A tree or blob read at a full commit hash cannot change, so it is
cached as immutable. One read by branch name is `no-cache`. [The API reference](../reference/api.md#object-api) lists
the routes.
Details:
- A revision that is empty, starts with `-`, or holds a NUL, newline or space
is refused, so git can never read it as an option.
- Pages read mirrors in process through gix, since starting a git process costs
more than most reads. A mirror's handle is reopened when its packs change,
because an open repository has room for only so many packs.
- A directory's page shows the commit that last changed each entry, asked
apart from the directory (`dev.knotless.getLastCommits`), since finding them
can walk a long history. One walk serves every entry: it passes over each
commit that left the directory as a parent had it, and stops once every
entry has its commit. The answer is kept in memory by commit and path, as a
commit's history never changes; a push on top of a kept answer applies only
its own commits' changes. A page drawn on the server waits a quarter of a
second for it, and otherwise fills it in once it comes.
- At the root, the same answer says how many commits the ref's history holds,
which the History button shows: a walk of the whole history, read from the
mirror's commit-graph rather than the packs (see
[storage](storage.md)), and kept in memory by its tip, up to 32,768 of
them, as that history never changes. A folder's button says only History,
as counting the commits that touch one path would compare every commit's
tree with its parent's. There is no Commits tab: the history pages sit
under Code, and the History button is the way to them.
- A file's blame (`dev.knotless.getBlame`) runs `git blame --porcelain` on
the mirror, as gix has no blame of its own. Blame reads every version of a
file back to where each line began, so it is bounded: a binary file is
refused, as is one over 256 KiB, and at most four run at once, the rest
waiting their turn. A blame at a commit never changes, so each is kept in
memory by commit and path, the last 256 to 512 of them. The page draws it
beside the highlighted text: each run of lines with its commit, and a link
to blame the file at the commit's parent, as it was before.
- A revision's files download as one archive, `.tar.gz` or `.zip`, from the
Code menu at the revision shown and beside each tag. `git archive` writes
it from the mirror, streamed as it goes, in a folder named for the
repository and the revision. Each archive reads the whole tree and
compresses it, so at most four are written at once, the rest waiting their
turn. One still writing after ten minutes, as for a client that stalls,
is cut short, so that slow clients cannot keep the rest waiting. One at
a commit's hash never changes, so it is cached for good.
## The appview index
With `--index`, the server is an appview: a service that indexes records across
accounts. The SQLite index holds every public repository, and every pull,
issue, comment, state change, hide, block, queue record, check, release and
push claim, each runner's `dev.knotless.runnerKey`, which the pages seal
secrets to in the browser, and each organization's public `dev.knotless.org` and
`dev.knotless.orgMember` records and its members' public
`dev.knotless.orgAcceptance` ones. A member in public of the
organization that owns a repository counts as one of its collaborators
wherever the index decides who may push: whose push it is, who may merge or
close, whose email to believe. For a private repository the server's own
index of its space adds the members its organization's records there name,
hidden ones too: those who are members still, where it keeps an index of
the organization's space, else every one named, as a reader outside the
organization counts them. `dev.knotless.getOrg` lists an organization's
members, and to the organization and its members its invitations and hidden
members, from its space, which the server keeps an index of, synced as the
organization when it holds the organization's delegation, else as the first
member to look at it or at one of its private repositories. As a gateway it writes each
change it makes to an organization into both indexes before answering, so
the index never trails what it did.
Private repositories are on no public stream, so they never reach it.
The index knows a repository by its owner's DID and key, since names change. A
pull written against a repository's old account still shows after a transfer,
when both owners agree on the move.
It answers these queries over XRPC ([parameters and
answers](../reference/api.md#appview)), and the web pages read the same answers:
- `dev.knotless.getForge`: every record naming a repository, less what
moderation removes.
- `dev.knotless.listRepos`: repositories by name or description, newest first,
or those naming a runner, for it to find them, or a topic.
- `dev.knotless.searchOwners`: the accounts owning those repositories, by
handle, for search to find people as well as projects.
- `dev.knotless.listChecks`: check runs by ref, commit or pull. Only the
runners the repository record names now count, so naming or dropping a runner
takes effect on the next read. [runner.md](runner.md) describes checks.
- `dev.knotless.getFeed`: the front page's recent activity, everyone's or what
the viewer and those it follows did, or one account's. Everyone's and the
followed hold nothing a repository's bots said or pushed on it: the
runners and triagers its record names, but its owner and collaborators,
who are people a bot may act as; and nothing an account said or pushed
anywhere whose `dev.knotless.profile` says it is a bot (`bot`), which the
index keeps of every account. What a bot says restates what people did
and would crowd them out. A pull a runner merged landing the merge queue
shows as merged by whoever queued it. A bot's own feed holds it all. The
page puts it together further: one account's pushes to a repository within half an hour
are one line, a merged pull takes in the push that landed it, and stars in
a row of one repository, or by one account, are one line.
- `dev.knotless.getActorActivity`: how much one account did each day, and when
each of its repositories was last pushed to: its profile's year of weeks,
and the order and sparklines of its repositories.
Without an index, these answer 404. `dev.knotless.getProfile` needs none: it
reads the account's PDS. With an index, the first read of an account's
profile keeps its records there (its knotless profile, its Bluesky profile
and the links that names), and the feed keeps them current from then on.
While the server follows a feed, the profile is answered from the index,
asking the PDS nothing; without one it is read from the PDS every time, as a
kept profile could have changed unseen. `dev.knotless.getActorRepos` does the
same with an account's repositories, which the index has already.
`dev.knotless.describeServer` needs none either: it says which runner
(`--default-runner`) a client names in a new public repository.
Push claims are the reflog, and the PDS may compact them away, so the index
keeps each as a row of its own: the history of the reflog. A new ref has no
claim, so its record's first version, at generation 1, is its row, under
the key its claim would have had.
### Languages
Lists of repositories (an account's, its stars) show what each is mostly
written in, as the file listing counts it at the default branch. Counting
it needs the repository's mirror, so a list never does: it reads what the
index keeps for each public repository, one row by owner and key. One with
no row yet is shown without a language and asked for, and a single task in
the background syncs its mirror and counts it, one repository at a time,
so the next read has it. A row, once kept, is counted again whenever one of
the repository's refs or its record changes in a feed, which a push or a
new default branch does. Only repositories that were counted are counted
again: a feed carries every push on the network, and syncing a mirror of
each would mirror the network. A private repository, which the index never
sees, is counted from its mirror as its list is read; such lists are short.
### Shared steps
The index keeps a directory of what repositories publish for others to
use. Shared steps are the one kind it holds yet; its tables name each
thing by a kind, so themes or other kinds can join it later.
A step is a release of the package `@step/<name>`
([shared steps](reusable-workflows.md)). As such a release is indexed, a
row is kept for it, to be read, and a single task in the background
reads its file from the release's blobs on its owner's PDS, checks it
against the release's size and hash, and keeps it, as a release's files
never change. A release whose one file is a step file, and which names
the commit it was published at, is kept as what that file says: its
description, inputs, the commands it needs and the secrets it asks for.
Any other is kept as not a step and never shown. A file that cannot be
read now, its owner's PDS away, is read again when the server next
starts.
Who uses a step comes from `.knotless/uses.lock` on each public
repository's default branch. It is read from the same tree, at the same
time, as what the repository is written in is [counted](#languages): when
a list first shows the repository, and again as its refs move. Each
`uses` it pins is kept as one row for the repository and the step, so a
repository that pins a step twice, at two versions, uses it once. A
step's repository is found by its owner's handle or DID and its name; one
the index does not have yet is passed by until the next count.
`dev.knotless.listSteps` lists the steps by name, each at its latest
version (the highest that is not a pre-release), and `dev.knotless.getStep`
gives one with its versions and the repositories using it. A step whose
account, repository or releases are taken down is left out, and so is a
repository taken down from those using one.
### Pulls and issues kept
Every page of a repository shows its open pulls and issues, and a pull's
pages show its thread. Those are worked out from every forge record on
the repository: each pull, issue, comment, state, queue entry and label,
with what moderation keeps out. A busy repository has thousands, and
reading and assembling them takes tens of milliseconds, so the index
keeps what it assembled for each repository, in memory, and pages read
that.
What it keeps stands until something it was made from changes. Each
write stamps only what it is about: a record about a repository stamps
that repository, and a block stamps its owner's repositories. So the
network writing elsewhere, as it always is, leaves the rest kept. A
change whose reach is wide stamps everything at once: a takedown, a
label, a move, an account forgotten, or an account that wrote here going
inactive or coming back. Most accounts the network reports never wrote
here, and those stamp nothing. The record of the repository is compared
as well, since it says who may set a state or a label. And what is kept
lasts a minute at most, for a label that runs out with time.
The cache holds up to 50,000 records in all, across repositories, and
drops the oldest assembled first. It is per process, so a restart
empties it.
### A repository deleted
A repository whose record is deleted drops out of every read at once:
the feed, notifications and the stats count only what was said on a
repository that stands. Once the owner's PDS lists no repository record
with its key, the server drops what it kept of it: the pulls, issues
and comments on it, whoever wrote them, the reactions to those, its
checks, releases, stars, pushes and language, and its mirrors, this server's and
those read for each account logged in here. A rename deletes the record
and makes another with the same key, so the PDS is asked rather than the
event believed. A repository another account took up keeps what was said
on it, which that account's repository shows.
### An account deleting its data
An account deletes its knotless data ([storage.md](storage.md#deleting-your-knotless-data))
through `dev.knotless.deleteAccountData`. Asked by the pages, the server
deletes the records itself, with the session it keeps for the account,
up to a limit a call, so the page can show progress and a run cut short
carries on. The command line and MCP delete them with their own session
and ask with a service token only to be forgotten, which the server does
only once the account's PDS lists no knotless collection. Forgetting is
what an account deleted at its PDS gets ([Accounts that
go](#accounts-that-go)): the index's rows, the mirrors and the sessions,
the pages' and MCP's, revoked at the PDS first. Takedowns and reports
stay, and every cookie signed for the account before is refused, should
it log in again.
### Filling and rebuilding
The index records its table version. A build with another version drops the
cache tables and rebuilds them from scratch, keeping only takedowns, reports,
emitted labels, pushes, repositories' [languages](#languages), which accounts are hidden ([Accounts that
go](#accounts-that-go)), and, in the index of private repositories, which
members were proven to read which space. Pushes are kept because nothing can bring them
back: each is read from a claim, compaction deletes claims a day old, and the
index keeps the push as a repository's history and the activity of whoever
pushed it: its owner, or the collaborator or runner its claim names. A
column added to the push table is added in place. They
go only when their repository or their owner's account is deleted ([A
repository deleted](#a-repository-deleted)). A
rebuilt index otherwise starts empty, and a feed only brings what happens
next, so it needs a backfill.
A backfill reads the knotless records of every account that holds any. A relay
can name those accounts by collection; a PDS lists all of its own. `--backfill`
runs one at startup. `--relay` runs one when the index was rebuilt, then sweeps
every 10 minutes, since a record whose event a feed never delivered would
otherwise be missing for good.
Each account is read whole, as one CAR from its PDS (`com.atproto.sync.getRepo`),
a few accounts at a time, and up to 256 MiB each. Nothing in it is believed
until it checks out: the commit must be the account's and signed by the key its
DID document names, and every block on the way from the commit to a record must
hash to its CID. Only the indexed collections' records are kept. An account is
first asked for its latest commit (`com.atproto.sync.getLatestCommit`), and one
whose commit is the one last read is passed over, so a sweep reads only the
accounts that changed. An account that fails is logged as `account not
backfilled` and tried again by a later backfill, a minute later for each time it
has failed, up to an hour. It does not stop the others.
A backfill reads while the feed goes on, so it must not put back what the feed
has since changed. Each write the index takes from a feed keeps the revision of
the commit that made it, and a deletion is remembered with its revision too.
Each record a backfill reads is as of the commit it read. A record whose indexed
write came at a later revision is left as the feed has it. Once the account is
read, the deletions at or before that revision are forgotten, as what was read
no longer holds them.
A rollout runs a new build beside the old one, both holding the index open.
That is safe only if the new build would not rebuild it, so the deploy script
asks first with `--check-index`.
A stream can miss the write that finished a check, as when it lands during a
rollout. So a check still `queued` or `running` a minute after it was filed so is read again
from its runner's PDS when a page shows it, at most once a minute.
## Live
A page shows what the index held as it was loaded, and a check's record
says nothing of a run until it ends. Two streams keep a page current, as
server-sent events from the server the page came from. Each is followed
only by a viewer allowed to read what it is about, checked as the query
for the same thing checks, and lasts ten minutes at most, ending too as
the server stops: the browser then asks again by itself, and is checked
again, so a member removed meanwhile stops hearing.
- Each page of a repository follows the repository (`/_live/repo`). As
the index takes in a record, or lets one go, it tells which repositories
the record is about: its own, the one it names, the one a star is of,
and for a ref that moved, its repository and those it is the head of a
pull into, so a push to a fork's branch reaches the pull's page. It asks
nothing while no page follows any. The page is told only that something
changed, a quarter of a second after the first of a burst, as a push or
a run writes several records, and asks its answers again: the same reads
a page load makes, so nothing new is drawn from, and a missed event
costs nothing. A private repository's are told by the server's own index
of its spaces, kept current by `notifyWrite` (see
[Private repositories](#private-repositories)).
An event sent while the connection was down is not heard: the page asks
its answers again once the connection is back, or once it is looked at,
if it was out of sight then.
- A check's page follows the check while it runs (`/_live/check`). Its
runner sends the server what the run prints as it prints it, in whole
lines masked with the run's secrets, and its steps as each starts and
ends (`dev.knotless.putRunOutput`; see
[runner.md](runner.md#output-as-it-is-printed)). Only the runner that
files a check may send its output, and only a runner the repository
names is shown, as everywhere checks show. The server keeps the last
megabyte of each run, as much as its log holds, in memory and never in
the index, at most 1,024 runs and 128 of one runner's; it lets go of one
nobody follows ten minutes after its runner last sent. A run is kept by
its check, and each send says when its run started, so a rerun at the
same commit, which files the same check, starts from nothing: the pages
following the last run are told it ended, and look again. It tells the
runner whether anyone follows, so the runner sends every second then
and every five seconds otherwise. Each piece sent to a page is named by
the byte where it ends, so a browser that reconnects says where it was
(`Last-Event-ID`) and is sent only what it missed. The running step's
log stays scrolled to its end as output comes, unless the reader has
scrolled up from it. Once the run ends the
record is filed, the repository's stream says so, and the page draws the
record's log.
Any page left out of sight a minute or more asks its answers again as it
is looked at, and any page asks again as the browser is back online, so a
tab left open does not go on showing what was.
Writing the output to the PDS as it comes was the other way, and the
record would then hold it. But a blob cannot be appended to, so each
update would upload the whole tail again; and the runner's account would
reach its PDS's write limits in a merge queue's bursts.
## Moderation
Owners moderate their own repositories with `block` and `hide` records, which
the index reads like any other. A block naming a repository keeps its subject
out of that one. A block naming none keeps it out of all the blocker's
repositories. What was said about a hidden item goes with it.
[collaboration.md](collaboration.md) describes these records.
The server operator moderates the whole server. This takes both an index and a
gateway URL, since reports need somewhere to be stored and labels need a DID to
come from. It speaks atproto's moderation methods, so existing tools work.
Anyone with a service token (a short-lived token their PDS signs) can file a
report with `com.atproto.moderation.createReport`, and the operator reads them
with `dev.knotless.listReports`. `com.atproto.admin.updateSubjectStatus` takes
an account or record down, or restores it. The operators are accounts, by
DID (`KNOTLESS_ADMIN_DIDS`), and each of their calls carries a service token
for that one method, so there is no shared secret to keep or share. An
operator logged in to the pages moderates from `/admin` instead, with
the session cookie: the reports, a day-by-day table of use
(`dev.knotless.getUsage`), and a form that looks a subject up and takes it
down or puts it back. `getViewer` says `admin` for an operator, which is
how the pages know to offer the page; the methods check again. A
takedown leaves the index, the pages, the API and clones at once.
Each takedown is also published as a `!takedown` label: a statement about an
account or record, which is how atproto carries moderation between services.
`com.atproto.label.queryLabels` and `subscribeLabels` publish them, and undoing
a takedown publishes its negation. Another server can follow this one with
`--labeler <did>`, and this one can follow others. Following a labeler takes an
index. A followed labeler's own `!takedown` and `!hide` labels hold here as if
made here, until they expire or are negated. Labels go out unsigned: a follower
trusts the endpoint the labeler's DID document names.
## The gateway
A collaborator cannot write to the owner's PDS. With `--public-url`, the server
is a gateway the owner delegates writes to, by OAuth, so it can push as the
owner on behalf of the collaborators the repository record lists. Its identity
is `did:web:<host>`. Every call carries a service token, so the gateway knows
who is asking without accounts of its own. `pushRefs` takes a pack, and `createCommit` builds commits
on the server, batching those to one branch into one write.
[gateway.md](gateway.md) describes delegation, both paths, and protected
branches.
## Web pages
The pages are a SvelteKit app in `web/`, run separately. The server hands it
every request it does not answer itself, and the app asks the server for each
page's data, then draws the HTML. It asks `dev.knotless.*` XRPC queries, as
atproto's appviews answer theirs: each with a lexicon under `lexicons/`,
sharing the views of `dev.knotless.defs` (a repository, a commit, a check, a
diff, a pull or an issue). Pictures a page shows, a check's or an account's own,
are queries too, sent as they are. The server decides
what a page shows, from Markdown to highlighting, and the app draws it. The
app's TypeScript types are made from the lexicons, by @atproto/lex, and every
answer the server's tests receive is checked against them, so the two sides
cannot drift; `just lex` checks the answers the screenshots recorded too.
The app also serves these docs, at `/docs`: a static site built from
`docs/` by `docs-site/`, which the image puts among the app's own files
([Development](../development.md#the-docs-site)). No query feeds it.
### Fresh answers
Every answer a page reads is `Cache-Control: no-cache`, even at a commit hash. A
page shows the repository as it is now, its authors' accounts as they are now,
and a shape a later release may change. The page takes its answer's caching, so
a browser revalidates instead of showing stale HTML. Pages stay fast anyway:
reads are local, and a link's data loads while it is hovered.
[../reference/performance.md](../reference/performance.md) has numbers.
A repository asked for by an old name answers under its new name, and the page
redirects with a 307 to the same path under the new one, as a raw file or a card
asked for by the old name does. The redirect is temporary, because the old name can be
given to another repository.
### Crawlers
Each file at each branch and commit, and each file's history and blame,
is a page of its own, so a crawler that follows every link never runs
out. `web/static/robots.txt` keeps crawlers to a repository's code on
`main`, its pulls and issues, and the pages around them. A crawler that
calls itself a browser may not read it, and each request's log line names
its client (`user_agent.original`), so they can be told apart.
### Who asks
A query can know who asks it, as the Bluesky appview knows: an app signed
in to someone's PDS asks that PDS for a service token for this server's
DID and the one query, or has the PDS proxy the query to the
`#knotless_appview` service in this server's DID document, which signs one
itself. The server checks the token against the key the asker's DID
document names, and hands the query the asker's DID. No token is fine,
and the query answers as for anyone; a bad one is refused, so an app
never mistakes an anonymous answer for its user's.
With no token, the pages' session cookie says who asks
([Logging in](#logging-in)); a cookie that no longer holds asks as no one,
so a page is drawn rather than failed. A repository's answer says whether
its asker starred it, and this is where answers that say more, such as
what the asker may do to a repository or a private repository from the
hosted index, take who asks from. Every answer varies by the token and
the cookie, and one for an asker is private to them, so no cache hands one
person's answer to another.
### Logging in
Someone reading the pages logs in with their Atmosphere account: they give
a handle, approve on their own PDS, and land back on the page they started
from. The server is the OAuth client, a confidential one signing with the
gateway's key, so it needs `--public-url`; a server without one offers no
login. The browser never holds a token: the server keeps the session, and
the browser a cookie, `knotless_session`, naming the account, signed with
the same key, HttpOnly and SameSite=Lax, for 90 days. The session asks for
every scope the CLI and MCP use, so what the pages come to write later
needs no second approval.
When knotless comes to ask for more, a login from before falls behind,
and the first write it was not granted would fail. So the server keeps the
scope each session was asked for, and `dev.knotless.getViewer` says
`reauth` for one that asked for less than the pages ask for now, or was
kept before its scope was. Every page then says so above what it shows,
with a button that logs in again as the same account and comes back to
the same page. A gateway's grant that asked for less than the gateway asks
for now is said the same way, `redelegate`, with the button that delegates
again, once the login is up to date, as delegating asks with it. Comparing
what was asked rather than what was granted keeps a PDS that drops a scope
it does not know, such as spaces, from asking forever. A write a PDS
refuses for want of scope all the same is answered in a sentence that says
how to grant it, not in the PDS's words: log in again, or, when it was the
gateway's grant that fell short, delegate again.
The problem is keeping a login for the pages from becoming something else.
A session kept as the gateway's would let the gateway push as that
account, so the pages' are kept apart, one directory per account under the
config directory's `web/`, and grant the gateway nothing
([gateway.md](gateway.md)). A login is finished only in the browser that
started it, by a cookie holding its OAuth state, so no one can log another
browser in as themselves. What the pages write, and logging out, is
refused unless it comes from the site itself, by its `Origin`. Logging out
revokes the session and refuses every cookie signed for the account before
then, in every browser. An account the index has taken down, here or by a
labeler the server follows, cannot log in, and its cookie names no one, so
it reads as no one and writes nothing. Restored, it is logged in again
wherever the cookie still holds.
Someone with no account makes one on the way. The login starts as a
sign-up instead: no handle, and the request goes to the PDS the server
sends people to (`--sign-up-pds`, selfhosted.social unless set) with
`prompt=create`, which has its authorization server show its sign-up
page. Who the account is is learned only from the tokens, so the server
takes it only once its DID resolves to a PDS whose authorization server
is the one the request went to, as atproto's OAuth asks: else any server
could name any account. From there it is a login like any other, and an
account taken down is refused all the same.
A browser can be logged in to several accounts, to switch between them
from the account menu without approving again. Logging in to another
account while logged in keeps the one before in a second cookie,
`knotless_accounts`, which holds the other accounts' session cookies as
they were signed, up to four, the one used last first. Switching swaps one
of those for the one in use; nothing is signed or kept anew, so a cookie in
that list is held to the same checks as the one in use, and one whose
account has logged out since is dropped. Logging out of the account in use
goes on as the other used last, if there is one. `getViewer` names the
others as `accounts`.
### Stars
A star is a `dev.knotless.star` record in the starring account's PDS,
naming the repository, as a Bluesky like names a post: it is public, so
a private repository cannot be starred. The index counts each account once,
leaving out accounts taken down, and answers how many starred a repository
and which repositories an account starred. A star the pages write is
indexed at once, so the count they show next is the new one. The pages,
`knotless star` and MCP's `star` share the code that writes it: one star
per account and repository, starring twice keeps the first.
### Follows
A follow is a `dev.knotless.follow` record in the following account,
naming the account it follows by DID. It is knotless's own, not Bluesky's
`app.bsky.graph.follow`, so following someone's code does not fill a
Bluesky timeline with their posts, nor the other way about. Whom an
account follows is in its own account, but who follows it is spread over
everyone else's, so only an index can say: the index counts each account
once, leaving out accounts taken down, and pages through followers and
follows the latest first. As with stars, a follow the pages write is
indexed at once, the pages, `knotless follow` and MCP's `follow` share
the code that writes it, and no account follows itself.
### Badges
A profile shows the badges its account earned, each an enamel pin: its
metal the tier, from steel through bronze and silver to gold, and its
shape its group. A badge earned once has one tier, drawn in steel, and
the limited one is cobalt with a dashed rim. No badge is a record. An
account could write any record it liked about itself, so a badge it
wrote would prove nothing. The index counts each from the records
accounts already keep, as `dev.knotless.getProfile` is answered, so a
badge comes back whole whenever the index is rebuilt and refilled: one
that could not be counted again from those records is not made.
A pull merged here is merged as the repository's owner, a collaborator
or its runner said, as everywhere else. What the merge queue did is
read from its runner's records only: the `batch` on its `merged` state,
and the reason on each `ejected` one, whose words for a conflict and for
a pull halving found are kept beside each other in
`knotless::records::queue`, so the runner and the count cannot drift.
| Kind | Group | Counts | Tiers from |
| --- | --- | --- | --- |
| `upstream` | collaboration | pulls it opened that merged into other accounts' repositories | 2, 16, 128, 1024 |
| `second-pair` | collaboration | pulls of other accounts it first commented on before anyone else said they merged | 1, 10, 50, 250 |
| `peacemaker` | collaboration | pulls it opened that merged after the queue ejected them for a conflict | 1, 5, 25, 100 |
| `forked-and-fixed` | collaboration | pulls other accounts sent its repositories from forks of them that merged | once |
| `good-company` | merge queue | pulls it opened that the queue landed in a batch of five or more | 1, 10, 50, 200 |
| `smooth-landing` | merge queue | pulls it opened that the queue landed and never ejected | 1, 10, 50, 250 |
| `bisected` | merge queue | pulls it opened that halving a failed batch found broke it | once |
| `shipped` | packages | releases it published | 1, 10, 50, 200 |
| `stargazer` | stars | stars on its most starred repository, each account once | 16, 128, 512, 4096 |
| `carry-your-own` | atproto | its PDS lists five accounts or fewer, its own among them | once |
| `moved-house` | atproto | its DID's history at the PLC directory names more than one PDS | once |
| `first-light` | limited | here with a repository before knotless 1.0 | once |
Two are not records but where the account lives: whether it moved, and
whether its PDS is its own. knotless-serve looks both up when a profile
is first asked for, from the PLC directory's audit log (a did:web has
none) and the PDS's `com.atproto.sync.listRepos`, keeps what it found in
the index's `home` table, and looks again in the background once that is
a day old. Like the handles it keeps, the table is dropped and found
again on a rebuild.
`first-light` is the one badge that will stop being earned. Until 1.0,
every account with a repository has it. At 1.0 the accounts that have
it then are written, a DID a line, to
`crates/knotless-serve/src/web/first_light.txt`, from the index
(`SELECT DISTINCT did FROM repo`, less any taken down), and from then on
only that list earns it, whatever a rebuild finds.
A badge's kind is never reused for another meaning, so a reader that
knows a kind knows what it counts. Its name, its picture and the words
the pages say of it are the pages' own, and change as they do.
### Reactions
A reaction is a `dev.knotless.reaction` record in the reacting account: a
strong ref (URI and CID) to a pull, an issue or a comment, and one of GitHub's eight contents, `+1`, `-1`, `laugh`,
`hooray`, `confused`, `heart`, `rocket` and `eyes`. Like follows, they are
spread over everyone's accounts, so only the index counts them: each
account once per subject and content, leaving out accounts taken down and
those the repository's owner blocked, from all of their repositories or
from this one. `getThread` carries the counts on the item and each
comment, naming the asker's own reaction where there is an asker, and
`getReactions` answers the same for `knotless show` and MCP's
`show_item`. The pages react only to what the repository's pages show,
never to what its owner hid, and a reaction the pages write is indexed at
once. The pages, `knotless react` and MCP's `react` share the code that
writes it: reacting the same way twice keeps the first, and taking it
back deletes every record of it. A reaction notifies no one.
On a private repository, a reaction is written in the reacting member's
repo in its space, as a comment is, so it names nothing to anyone else.
No appview index has it: the server counts it from its own index of the
space, where the owner lets it keep one, and otherwise from the space as
each look reads it, with the pulls and issues. `knotless show` and MCP's
`show_item` count them from the space too.
### Landing and checks
A pull's page ends in a merge box. An open pull's
checks show there, not in the timeline: how many are in each state, then
a row for each, those that failed or are running open and the rest
folded, each linked to its run's page, which has its log and the
pictures it made. A closed or merged pull's checks stay in the timeline,
pictures and all. A pull the merge queue landed ran only its
[pre-flights](merge-queue.md#pre-flights) at its head, so its page shows
the checks it landed on instead: the runs on its base at the tip of its
batch, which its `merged` state names as `batch`. Then the box asks the server, once the page is drawn
and only for someone logged in, whether the pull can merge now
(`dev.knotless.getMergeStatus`), and says whether it conflicts with its
base: the files that do, each linked to its place in the pull's diff,
and the commands that bring the base into the branch to resolve them. The server decides through
the same code as `knotless merge --check`, from what it holds rather
than what a merge reads afresh: the repository as its mirror last synced,
both sides in its mirrors, and the pull, its reviews and its checks as its
index has them. So the box says what a merge would, without a merge's
writes, and in a tenth of the time; a review or a check written a moment
ago shows once the index hears of it, while a merge reads them at once.
It is asked only of someone logged in, as it merges the two sides, and
only they can act on it. It reads
the repository as the pages read it for that viewer: a public one as the
server, a private one as the member asking, so it works for a private
repository's members. A pull into a repository whose runners have no
workflow that runs on it waits for no checks (`none`), as one with no
runners does.
The box's buttons, and the re-runs and runners on the checks and settings
pages, write as the account logged in, with the session the server keeps
for it, through the code `knotless merge`, `queue --leave`, `retarget`,
`rerun` and `runners` run. Each is enabled for whoever the answer's `may`
says may do it, from the repository's record: the owner and collaborators
merge and queue, the owner alone runs checks again and names runners, and
anyone who may close a pull retargets its base. The server enforces none
of that itself. The code it calls refuses what the account may not do, as
it does for the command line, and the PDS, the gateway and the runners
honour only what their own rules allow. A merge from the pages waits for
no checks: it merges what is ready, and the merge queue is there for the
rest. So the box leads with whichever lands the pull next: Add to merge
queue is its primary button while the pull's checks are running, or
passed only against a base that has since moved, as the queue checks it
against the base as it is now without being asked; Merge pull once it
can merge; neither while its checks have failed or it conflicts. Above the buttons it says what a merge waits for and how to
get there, so a pull held by its checks is never a disabled button with
no reason beside it. A merge, a queued pull or a retargeted one is indexed as soon as it
is written, from the owner's and collaborators' newest queue and state
records, so the page drawn after shows it.
### Repositories
Someone logged in creates a repository at `/new`, the header's `new repo`:
a name, a description and a default branch, as `knotless create` takes
them, written by the same code, so it names the appview's default runners
too. It is public: the pages cannot show a private repository, which only
a server run with `--private` on its owner's machine reads. A repository's
header has a Fork button beside Star, which copies it, every branch and
tag, into the account logged in, under its own name or another, as
`knotless fork` does. Either answers once the records are written, and the
new repository record is indexed at once and the account's mirrors
refreshed, so the page it goes on to, whose empty state says how to push,
and the account's profile show it.
A repository's owner sees a Settings tab, at `settings`, with a side nav
of its sections. General sets its description, links, topics
and default branch (`knotless describe`), renames it, shows what it takes up and
compacts it, and has a Danger zone to offer it to another account or
withdraw the offer, and to delete it once its full name is typed.
Branches makes, renames and deletes branches and protects them, with or
without merge commits (`knotless branch`, `protect`); Labels defines,
changes and takes away labels (`knotless labels`); Access adds and removes
collaborators (`knotless share --write`) and triagers, by scope
(`knotless triagers`), and for a private repository its readers
(`knotless share`, `members`), whom only its owner sees; Actions adds and
removes runners (`knotless runners`) and secrets (`knotless secret`),
sealing each value in the browser to the keys the runners publish, which
`listRunners` answers from the index. Collaborators
push through a gateway, so until the owner has delegated their writes the
tab offers to delegate them to this server, as `knotless delegate` does
(`delegateWrites`, then `awaitDelegation`): the owner approves the
gateway's own grant at their PDS, in a new tab, while the page waits. The
pages' session is never handed to the gateway, whose grant is narrower
and lasts longer ([gateway.md](gateway.md#delegation)). What
the page shows is `getRepoSettings`, read from the repository's record as
its PDS has it now, not the index, so a change shows at once. Each change
is the owner's alone, which the server checks, not only the pages: anyone
else is refused 403, whatever they send. The account a repository is
offered to sees the offer on its pages, and takes it up there, as
`knotless accept` does.
What a repository is about and who keeps it show under its description:
its topics, each linked to search for the others naming it, then a line
with its first link, its website, and a count of any others that opens
the list of them, then the faces of the first of its collaborators, as
its record names them, with a count past those that opens the list by
handle. Its organization's members are not among them: its
organization's page names those. What it publishes shows on its pages
from the index, on that line too: the release to show and its packages,
a Releases tab
with how many releases it has, and a releases page listing them
(`dev.knotless.listReleases`), each file linked to download from this
server, read from the owner's PDS once and checked against its release.
[packages.md](packages.md#on-the-pages) says what each shows.
### Pulls and issues
Logged in, someone opens an issue from the Issues list's New issue, its
body started from the repository's issue template, and a pull from the
Pulls list's New pull, or Compare & pull on a branch's files: it chooses the branch to merge into, then the branch to
bring in, from the repository or a fork of it, shows the commits and diff
it would bring (`getCompare`), with a button that unfolds the form to open
it, its body started from the pull template at that base (`getTemplate`).
Compare & pull links with `expand=1`, which shows the form at once.
Beside the form, a sidebar
chooses the labels to put on it, those the repository's viewer state says
the account may (`getRepo`'s `viewer.labels`), and for a pull the
reviewers to ask, from the owner and collaborators. Opening (`createIssue`,
`createPull`) goes through the same code as `knotless issue`, `knotless
pull` and `knotless label`, so a body that follows none of the template is
refused there too, and a label the account may not put on is refused
before anything is written.
A pull's or an issue's page is written on as the account logged in:
a comment at the end of the thread, and on someone else's
open pull a review, Comment, Approve or Request changes. Each author edits
and deletes their own comments, and edits the title and body of what they
opened. Each write is a procedure on the server (`createComment`,
`editItem`, `deleteComment`, `setItemState`, `labelItem`, `hideItem`,
`createReport`), which calls the same code as the CLI's `comment`, `edit`,
`delete-comment`, `close` and `reopen`, `label` and `unlabel`, `hide` and
`unhide`, and `report`, so the pages allow and refuse what they do, in
their words. A write is indexed at once, read back from its author's PDS,
so the page after shows it; a private repository's go to its space, and
show once the space is read again.
`getThread` answers who asks with what they may do past saying something
(`viewer`), and the page shows only those controls: closing and reopening
to who opened it, the owner and collaborators; the labels each may set, to
the owner, collaborators and triagers of their scopes; hiding to the owner.
Reporting is in each comment's "…" menu, for anyone. Logged out, the
controls there open the login dialog. A hidden comment leaves the page
at once, with an Undo; a hidden pull or issue leaves with its page.
Each box written in Markdown (a new issue's or pull's body, a comment, a
review, an edit of a body or a comment) has Write and Preview tabs.
Preview asks `renderMarkdown`, which draws the text with the
renderer the thread's comments are drawn with, so it shows what will be
shown once posted. It needs no login, reads nothing and fetches nothing,
and takes no more than a body may hold. It is a reading aid for the box
it previews, so the CLI and MCP, which post what they are given as it is,
have no form of it. The box stays, hidden, while the preview shows, so
what was written and its undo survive switching. A picture pasted or
dropped into a box is uploaded at once (`uploadImage`) and linked where it
went, and the write attaches it
([collaboration.md](collaboration.md#pictures)); the preview draws it from
what was pasted, as its PDS serves it only once attached. The thread's
pictures show from `getAttachment`.
Line comments on a pull's diff wait for a comment record that can name a
path and a line, which `dev.knotless.comment` cannot yet.
### Notifications
An account hears of what others did that concerns it: a star of one of
its repositories, a follow of it, a pull or an issue opened on a
repository it looks after, and a comment on one of its pulls or issues, or on one it
commented on, from after it did; a pull, an issue or a comment that mentions it; and an
invitation to an organization. It hears too of its own pulls' checks,
from the runners its repository names:
`checks-failed` when checks fail at a pull's latest run, and `ejected`
when the merge queue takes the pull out ([merge-queue.md](merge-queue.md)).
A
comment that reviews a pull says what it found (`verdict`), so it reads
as approving it, or asking for changes. A pull that asks the account to
review it, once it is not a draft, is a `review-requested`
notification, as of when it asked, or was marked ready; on the
account's own repository it is that, not a `pull` one, and when it
mentions the account, not a `mention` one.
The repositories an account looks after are its own, those whose records
name it a collaborator, and every repository of an organization it has
joined, in public or hidden.
The index keeps who each repository's record names in a table, so that
asking does not read every record; the organizations an account joined
it reads from both indexes, as a hidden membership is only in the
organization's space. A member hears of a private repository of the
organization as its owner does, being one who may read it.
Each of those is a record in its doer's own account, so only an index can gather them,
and the server keeps no notifications of its own: it works them out from
the index whenever an account asks, the latest first, from tables kept for
it (comments by what they are on and by who wrote them, whom each pull asks to review it, as well as stars and follows), leaving out the
account's own doings and the accounts it blocked or that are taken down.
A private repository's come from the server's own index of its space,
where its owner lets it keep one, each only while the asker can still
read the repository, and by the asker's choices, blocks and follows as
the index keeps them; of one it keeps no index of, no one hears.
What it does keep is when each account last saw them, in a table kept
across rebuilds, as takedowns are, so the pages, `knotless
notifications` and MCP's `notifications` agree on what is new. Asking
takes an asker: a service token, which the CLI and MCP get from their
PDS, or the pages' session cookie. The pages show a bell with how many
are new, and the notifications page, once shown, says they were seen up
to the newest.
The page reads as Bluesky's does. Each notification leads with its
author's picture, marked with what kind it is, as the feed's lines are;
a runner's, being no one's doing, is the mark alone. What several did
together shows as one, by the first and how many others: stars of one
repository, follows, and plain comments on one thread, new ones apart
from those seen. Under each is the start of what it says (`text`, as
Markdown, at most 1,000 characters), so a comment can be read without
opening it. Each about a pull or an issue says what state it is in now
(`subjectState`), worked out as its page does, so a pull that has
merged shows so. Invitations, and reviews asked of the account that it
has yet to give (`reviewed`) on pulls still open, come first, as waiting
on it; the rest by how
long ago. Chips narrow the page to one `kind`, which the server filters
rather than the page, so each page of them is full: mentions, reviews
(asked, and given on the account's pulls), comments, pulls and issues,
checks, or stars and follows. A listing of one kind, on the pages, the
CLI or MCP, marks nothing seen, as that would mark seen those of the
other kinds before it.
A mention is heard of too: a pull, an issue or a comment whose facets name
the account ([collaboration.md](collaboration.md#mentions)), which the
index keeps as it indexes the record, in a table of its own. Not by its
author, nor when the repository's owner hid the record or what it is on,
or blocked its author. A record is heard of once, for one reason: a
comment that mentions the author of the issue it is on is heard of as a
comment, unless comments are turned off, and says so (`mentioned`), and a
pull that mentions a collaborator is heard of as the pull. A
review is heard of with the number of line comments sent with it
(`lineComments`), not one notification each. Each names what it is about
(`reasonSubject`): the pull or issue a comment is on, else the pull or
issue itself, or for a star the repository's record. An account an edit names first
hears of it as of the edit (its `updatedAt`), so it does not land among
what was seen already; one named before hears nothing more of the edit.
The index keeps when it first saw each account named; one it reads anew,
as when it is built again, is dated by its record's `createdAt`. On
a private repository, a mention of someone who cannot read it is never
heard of, as nothing else on it is: they learn nothing of it, not even
that it is there.
An invitation to join an organization is heard of as well, from the
organization, with the role it offers, for as long as it stands: until
the account joins or declines, the organization withdraws it, or it
lapses after 7 days. It is the one notification about no repository, and
is answered where it is heard of: the pages' notification joins in
public, joins hidden or declines, as the organization's Members tab does.
One kept only in the organization's space comes from the server's index
of that space, as a private repository's notifications do.
Checks are heard of by the pull's author alone, and only from a runner
the repository names, as its checks are shown. A pull's failed checks
are one notification for each head and base they ran at, naming each
workflow that failed (`checks`), as of the first to end, rather than one
for each workflow. Only the latest run of each workflow on the pull
counts, by whichever runner the repository names ran it: one that failed
and then passed when run again, on that runner or another, or that a
newer head's run replaced, is no longer heard of. Nor is either, once
the pull has merged or closed: there is nothing left to do about it. An ejection is the runner's
`dev.knotless.queue` record turning the pull away, with the checks that
failed when it was for them. Neither is any account's doing, so hearing
only of accounts one follows leaves them in. The repository's owner
hears of neither for others' pulls: only the author does.
A browser the pages' settings turned it on in shows each new
notification as the system's own while a tab is open but out of sight:
the tab asks `getUnreadCount` every minute, and `listNotifications` for
the new ones. The browser keeps that it was turned on, beside its
permission, rather than the server: it is that browser's, not the
account's. Nothing is sent once every tab is closed; the server pushes
nothing.
Which notifications an account hears of is its to choose, and the server
keeps that too, among the account's preferences, rather than in a record:
as with Bluesky's preferences, it is how one reader wants this appview to
read for it, nothing anyone else needs, and a record would publish it to
the network. Each reason `listNotifications` gives can
be turned off, and an account can hear only of what accounts it follows
did; `listNotifications` and `getUnreadCount` leave out the rest. The
reasons come from that lexicon's `knownValues`, so a reason added there can
be turned off at once. The pages' settings, `knotless notifications
settings` and MCP's `notification_settings` change them, through
`getPreferences` and `putPreferences` ([Preferences](#preferences)), with
the cookie or a service token.
### Preferences
An account's preferences are the server's to keep, in one table of the
index, `pref`, a row for each kind an account has set, which a rebuilt
index keeps and forgetting the account drops. `getPreferences` answers them
as a list, each told apart by `$type`, as Bluesky's are; `putPreferences`
sets those given, each taking the place of its kind whole, and leaves the
rest. Three kinds so far: `notificationPref`, which reasons it hears of and
whether only from accounts it follows, given back always, every reason
listed; `appearancePref`, the theme the pages are drawn in, with the
palette of one of the account's own, and the one for night if it chose
two; and `themesPref`, the themes people made that it added to its own
([Settings](#settings)). The
server keeps only what a kind has, as its lexicon bounds it, and refuses a
kind it does not know: a new one is a new def in `getPreferences`, added to
both unions. Notification preferences had methods of their own until
these; they were folded in, and their lexicons retired, while knotless had
few accounts.
### Settings
`/settings` is the logged-in account's, with its
sections down the side: its public profile, the addresses its commits show
as its by, its appearance, its notifications, its delegated writes, and
the accounts it blocks. A section of more than one page is a group:
Appearance's name opens it, to Themes and Make a theme, and
it stays open while one of them is the page; on a phone, where the
sections are a row of tabs, it is one tab. The profile is
its `dev.knotless.profile`, field by field over the Bluesky profile, with
a preview beside the form and the profile page after saving. A box there
says the account is a bot, which its profile page then marks, and the
feed leaves out (`dev.knotless.getFeed` above). The picture
is uploaded first, as the body of `uploadAvatar`. Emails can only be
counted, added and taken back by name, since the record keeps hashes.
Each write goes through the same `knotless` code as the command line, as
the account, with the session kept for it, and a change to the profile, or
a block from all repositories, is indexed at once. A profile's "…" menu
blocks it, or unblocks it, from all the viewer's repositories.
Delegated writes says whether this server's gateway writes for the
account (`getViewer`'s `delegated`), and until it does offers to delegate
to it, as the Access tab does. Delegating is per account, not per
repository: through it, collaborators push to all the account's
repositories, and runners land their merge queues when none they name may
push ([merge-queue.md](merge-queue.md#who-lands-a-queue)). A pull whose
queue would land through this gateway before the owner delegates says so
in its merge box, `queueBlocked` in `getMergeStatus`, linking the owner
here, and the pages refuse to queue it.
Appearance chooses the colours the pages are drawn in: the system's light
or dark, as with no choice, light or dark always, or a theme after an
editor's (Catppuccin, Dracula) or an app's of the Atmosphere (pckt,
plyr.fm). Or two, Sync with system: a day theme
while the system is light and a night theme while it is dark, each
chosen in its own tab over the one list of themes, which marks each in
use with a sun or a moon. The browser
keeps it in a cookie, `theme`, which the pages' server reads and writes on
`<html>` as `data-theme`, so a page comes in its theme from the first
paint, without asking for anything; and the account keeps it as its
`appearancePref` ([Preferences](#preferences)), which the cookie takes
whenever the browser comes to be in use as the account (logging in,
switching to it, or going on as it after logging out of another), so
another browser takes it at once; an account that chose none clears the
cookie, for the system's light or dark rather than the last account's. The cookie, unlike the
session's, is not `HttpOnly`: the pages' script changes it as one chooses.
A theme is named in `a-z`, `0-9` and `-`, which the server holds it to, as
it goes in a cookie as it is. Each theme is a block of colour tokens in
`web/src/tokens.css`. A pair keeps its night half in `theme-night` and
`palette-night`, and the server writes both halves on `<html>`, as
`data-day` and `data-night`, with the palettes of either in a `<style>`
of their own; `/theme.js`, before the page is drawn, takes the half the
system's light or dark says, as `data-theme` and `data-half`, and again
as it changes, so the pages flip with the system without a reload. Its
own file, as the pages allow no script inline.
Anyone can make a theme of their own colours, and publish it for anyone to
choose. A `dev.knotless.theme` record holds one: a name, and a palette of
colours named for what they colour (`background`, `text`, `textMuted`,
`primary`, `accent`; the states' `success`, `danger`, `attention`, `done`
and `neutral`; and code's), each `#rrggbb`.
They are named by purpose, never by hue: a theme may draw success in blue,
for those who cannot tell red from green, and `green` would then lie. The
hue names are the pages' own tokens, which the palette is mapped onto
(`web/src/lib/theme.ts`). Only four are
required: the rest fall back on light's or dark's, as its `colorScheme`
says, or for the surfaces and texts, are mixed from its background and
text (`[data-theme^='custom-']` in `tokens.css`), so a theme written by
hand needs few. knotless's own shape rather than standard.site's
`site.standard.theme.basic`: that one has four colours, and the pages are
drawn in twenty-odd. The index keeps every theme record (`theme`), and
`listThemes` answers them, the latest first, with only the colours that
are `#rrggbb`, or with `q` those whose name starts so (a lower-cased
`name` column, indexed, read as a range). Finding a theme and choosing
one are apart, as Obsidian's community themes are: Browse themes opens a
dialog of every one, each card drawn in its colours, a page of 48 at a
time, with a box that finds them by the start of their name, where each
is added to the viewer's own, or taken out again; and Appearance chooses
only among its own. Your themes there are those the viewer published
(`listThemes` with `actor`, 100 at most), each to delete from its menu,
and those it added, each to remove: copies, in its `themesPref`, 50 at
most, so one its maker deletes stays. Each says who made it, those added
as `listThemes` with `uris` names their makers now, and one since deleted
from its copy, with no maker. They are one list, the latest
first, by when each was made or added. Appearance lists none of anyone
else's otherwise, so it stays short however many there are. A palette in
use that is none of these, one used without publishing, shows as a card
of its own, its colours being the copy kept in preferences. It
links too to the editor at
`/settings/appearance/new`, which starts from a theme there is, draws the
theme as each colour is picked, and says each text that falls short of
AA on what it is drawn on, as `harness/contrast.mjs` holds the pages' own
themes to (both read `web/src/lib/contrast.ts`), and will not use or
publish one that does. Publishing is `createTheme`, written as the
account with its pages' session, and taking back, `deleteTheme`.
`createTheme` checks the palette again, with knotless's own `theme`
(`crates/knotless/src/theme.rs`): the same measure in Rust, reading the
same `tokens.css` and the same pairs (`web/src/lib/contrast.json`), so a
theme made by the command line or an agent, which write the record
themselves, is held to what one made on the pages is. A palette someone
writes by hand needs only its four colours to pass: what the pages mix
for it from them (`[data-theme^='custom-']`) is mixed to read.
Choosing one copies its palette into the account's `appearancePref`, with
`theme` `custom` and `source` the record it came from: its maker
changing or deleting it later leaves those who chose it as they were, and
a theme made only for oneself needs no record. The browser keeps it as
`theme` `custom-light` or `custom-dark`, and its colours in a second
cookie, `palette` (`dark.background-101820.text-f2f2f2…`), which the
pages' server writes on `<html>` as custom properties in its `style`, over
light's or dark's; logging in writes both. Only colours as `#rrggbb` get
there: the server holds a palette to its lexicon as it is put, and the
pages' server reads only those from the cookie, so nothing else reaches
the page. Contrast is the pages' to check, not the server's: a palette
put through the API that reads badly only draws its own account's pages
so.
### Safe to show
A repository holds whatever its owner pushed, and a pull holds whatever a
stranger wrote. Each way a page shows that content is built to keep it from
running script on the site.
- Pages carry a strict content security policy: scripts only from the site, and
no other site may frame them.
- Markdown keeps raw HTML as text. Links keep only http, https and mailto.
Relative links resolve inside the repository, never above its root.
- Mermaid blocks arrive as code. The browser swaps them for diagrams drawn by
Pele, loaded only on a page that has one. A block that fails shows its source
again. Pele escapes text, drops script callbacks, blocks `javascript:` and
`data:` URLs and caps input size, and no icon or math resolvers are given
it. Links in a diagram get `rel="noopener nofollow ugc"`, as anyone with a
repository writes diagrams.
- Raw files are served with a sandbox policy, `nosniff`, and a type chosen from
a short list by extension: a few image types, plain text, or octet-stream. A
raw HTML or SVG file never runs as a page of the site.
- A release's files are served as octet-stream, as an attachment to save,
with the same sandbox and `nosniff`.
- Check images, avatars and the pictures pulls, issues and comments attach
are served only for a CID the record names, with the same sandbox.
A commit's author is shown as an account only when someone allowed claimed that
email: the owner or a collaborator anywhere, or a merged pull's author within
the commits that merge brought in. The owner and collaborators come first, so
no pull's author can take their addresses. [identity.md](identity.md) describes
email claims. A handle is shown only when it resolves back to the DID.
### Highlighting and diffs
Syntax highlighting sends ranges, not markup. The text goes to the browser
once, with a compact list of offsets per kind, and the browser paints them with
the CSS Custom Highlight API. The HTML stays small, and a browser without the
API shows plain text. Grammars are tree-sitter's. Markdown's fenced code,
in a comment, a pull or issue, or a file, is highlighted the same way, by
the language its fence names: its block carries its ranges as
`data-ranges`, and a fence naming no language known here is left plain.
A diff is highlighted against the whole file on each side where it can be, so a
line inside a multi-line string or comment still reads right. A removed line
paired with an added one has its changed middle marked, when the two lines
share enough to make that useful. A pull's page reads the whole of each file
it changes, both sides, every time it is shown, and parsing a long file takes
a good part of a second, so each text's highlighting is kept by its language
and SHA-256, for the last 128 texts.
A pull's diff is on a page of its own, its Files changed tab, and that page
does not wait for all of it. Both of a pull's pages ask `getThread` with
`filesOnly`, which lists the changed files and their line counts from the
patch alone, reading and highlighting nothing; the conversation needs only
their number. The files page adds `firstRows`, and the first few files come
with their rows in the same answer, up to 10 files or 1500 lines, from the
same patch. A second request would wait on the first and repeat its work.
The browser fetches the rest with `getPullDiff`, ten at a time, as each file
nears the screen. A file changing more than
400 lines waits for the reader to ask for it. Beside the diffs, a tree of the
files links to each, with how many comments the threads on a file's lines
hold. A button hides or shows the tree, and a `tree` cookie
keeps the choice, so the server draws the next pull the same way; with no
cookie, the tree shows beside the diffs and is hidden on a phone.
An open pull's changes compare the base branch in the target's mirror with the
head branch in the source repository's mirror, read together. A pull's checks
show only at its current head, because checks of an older head would read as
passing what never ran.
A check's page reads its record from the runner's PDS every time, since a
running check changes and Jetstream's frames are not signed. Its log and
pictures are blobs named by CID, which never change, so each is kept once
read: the log's shown tail, or the picture, for the last 64. One in a space
is read each time, as whoever asks may read it.
A blob over 1 MiB, binary, or not UTF-8 is not shown inline, though its raw
link works. A diff over 2000 rows is not shown.
## Social cards
Shared links show a card: a PNG at a page's path plus `/card.png`, for the
front page, each repository and each pull and issue. The server writes an SVG
and rasterizes it with resvg, using fonts compiled into the binary, so drawing
needs no browser, no font files and no network. It reads only the account's
picture, fetched with a 2 second limit.
A pull's or an issue's card shows who opened it and the day they did. A pull's
card also counts what GitHub's do: its comments, reviews and files, the
lines it adds and removes with five squares of their share, and its commits.
They are read from its mirror with `git rev-list --count` and `git diff
--shortstat` over the range its diff shows. When that range cannot be read,
as when its head's branch is gone, the card leaves the files, lines and
commits out.
A card's URL carries `?v=`, a hash of its SVG, so a card at its current version
is immutable and changes as soon as its page does. At most one draws per core.
A card drawn without its picture is not kept, so the next request tries again.
## Status and MCP
`/healthz` answers "ok" while the process runs, since a dead upstream feed is
no reason to restart.
`/_status` says whether the server is doing its job: each feed is connected and
has brought an event recently, the index answers a read, and with `?repo=` that
repository's mirror reads as its code page would, and the disk its cache is on
has room: under 15% free it is told of, under 5% it fails. It answers 200 when
all hold and 503 otherwise. One feed bringing events is enough, though dead feeds are
still listed.
[../../deploy/README.md#monitoring](../../deploy/README.md#monitoring) says
what each part means.
`/mcp` serves knotless's MCP tools over HTTP to any account, as the pages let
anyone log in. The server is its own OAuth authorization server: a user signs
in with atproto OAuth and gets a token, unless the index has taken the account
down, here or by a labeler it follows. Registrations and tokens are signed
rather than stored, so taking an account down stops its tokens at their next
use. Each
account has a share of the server: 8 tool calls running at once, 2 of them
with a `wait`, and 120 started a minute, past which a call answers as a tool
error saying to wait. At most 1000 sign-ins wait on PDSes at once; past that, a
sign-in is asked to come back in a few minutes. Public repositories are read
from this server's own mirrors, the ones its pages read, however many accounts
read them. Each account's session and mirrors of private repositories have
their own directory, so no one reads as anyone else. An account no token has
been made for in 90 days, so whose refresh tokens have all run out, is
forgotten: its directory is removed, looked for at most once an hour as tokens
are made. It needs `--public-url`,
the address assistants reach. Pulls and issues on public repositories come from
this server's own index when it keeps one (`--index`), asked over this
machine's address so the reads stay off the proxy, unless `KNOTLESS_APPVIEW`
names another appview; without an index they come from `KNOTLESS_APPVIEW`,
`https://knotless.dev` when it is unset.
[../reference/mcp.md](../reference/mcp.md) lists the tools.
## Private repositories
Private repositories live in a space, a part of an account only its members may
read. A server people log in to shows one to each member logged in, read with
that member's own session and kept for them alone, its answers cached by no
one; anyone else gets a 404, as for a repository never made.
[access.md](access.md#browsing) explains how, and [who can see
one](access.md#who-can-see-a-private-repository).
Every private repository read there is indexed there: the server syncs its
space into an index of its own, `serve/private.sqlite`, apart from the
appview's, with the session of the member who last read it there, and reads its
pulls, issues and checks from that, still only for members. One made with
`knotless create --private` is in the index before anyone looks, its owner
logged in there, since the command asks the owner's PDS to tell the server of
each write in its space; an owner's others are found as they log in or look at
their profile, and those shared with a member are asked after as they log in or
search. Their profile lists them from the index, never from their PDS as
anyone looks, to the owner and to each member proven so, and each space is synced as its authority tells of writes there. [An index of every one read
here](access.md#an-index-of-every-one-read-here) explains how it is synced and
when it is forgotten.
`--private`, which read every space as whoever was logged in on the machine,
for pages on loopback only, is gone: the pages above do the same for each
member, logged in to any server. Given, it is refused with what to do instead.