chadtmiller.com/knotless

Switch repository

Loading…

# Running knotless-serve

knotless works without any server. `knotless-serve` makes reads faster,
lists pulls and issues, lets collaborators push, and draws the web pages.
It is one binary, and each of its jobs is optional. Its state is a
directory of bare mirrors and one SQLite file, so everything fits on one
small VPS.

[knotless-serve](../design/serve.md) explains how each part works, and
[deploy/README.md](../../deploy/README.md) shows how knotless.dev runs it.
The [command reference](../reference/cli.md#knotless-serve) lists every
flag.

## Choose what it does

| Job | Turned on by | What it does |
| --- | --- | --- |
| Read cache | always | git smart HTTP (protocol v0 and v2, partial and shallow clones) and an object API, read only, from cached mirrors |
| Appview | `--index` | `dev.knotless.listRepos`, `getForge`, `getFeed` and `listChecks` from a SQLite index of the feed, and `getProfile` from the account's PDS |
| Gateway | `--public-url` | `dev.knotless.pushRefs`, `createCommit`, `putDelegation` and `getDelegation`, for collaborators |
| Moderation service | `--index` and `--public-url` | `com.atproto.moderation.createReport`, takedowns, and labels on `subscribeLabels` |
| MCP over HTTP | `--public-url` | `/mcp` for [remote agents](agents.md#connect-a-remote-client) |
| Web pages | `--web` | the SvelteKit app in [`web/`](../../web) |
| Status | always | `/_status` |
| Usage counts | `--usage` | requests, page views and events for the operator, as `dev.knotless.getUsage` |

A server with all of them:

```sh
knotless-serve --listen 0.0.0.0:2590 \
  --jetstream wss://jetstream.us-east.bsky.network \
  --index /var/lib/knotless/index.db \
  --backfill https://bsky.network \
  --public-url https://git.example.com \
  --labeler did:plc:… \
  --web http://127.0.0.1:2591
```

Here `--jetstream` keeps mirrors warm and feeds the index, and `--backfill`
indexes what was there before the server started. `--public-url` makes the
server a gateway, as `did:web:git.example.com`. `--labeler` honours another
labeler's takedowns.

## Who can use it

With `--public-url`, the server is open to every atproto account. Anyone
can log in to its pages, push through its gateway to a repository whose
owner delegated to it, and connect an assistant to `/mcp`. There is no
list of accounts to keep, and no flag that turns one of these off on its
own.

A takedown is what turns an account away, and takedowns need `--index`.
The server's operators, the accounts `KNOTLESS_ADMIN_DIDS` names, take an
account down with `knotless admin takedown <handle>`, and a labeler named
with `--labeler` can too. A taken-down account cannot log in to the pages
or connect an assistant to `/mcp`, and the cookie and tokens it already
has stop working at their next use. Its pushes through the gateway are
not refused. Without
`--index`, the server keeps no takedowns and refuses no one: run it that
way only where you trust everyone who can reach it.

## Pick a feed

The server learns of changes from a Jetstream v2 host (`--jetstream`), or
from a relay or PDS directly (`--firehose wss://…`). Jetstream is the
cheaper of the two. It sends only `dev.knotless.*` commits, plus identity
and account changes, as small JSON frames that carry each record. The server
then neither decodes every commit on the network nor fetches each record
from its author's PDS.

Jetstream's frames are unsigned, so the index trusts the Jetstream operator
for what a record says. Mirrors still sync from the PDS, and `--backfill`
rebuilds the index.

## Read through it

Clone over HTTP, or read single objects through the API:

```sh
git clone --filter=blob:none http://localhost:2590/alice.test/proj
curl 'localhost:2590/api/alice.test/proj/blob?rev=main&path=src/lib.rs'
curl 'localhost:2590/api/alice.test/proj/tree?rev=main&path=src'
curl 'localhost:2590/api/alice.test/proj/log?rev=main&path=src&limit=20'
curl 'localhost:2590/api/alice.test/proj/diff?base=main&head=feature'
curl 'localhost:2590/api/alice.test/proj/object/<sha>'   # immutable, cacheable
```

## Serve the web pages

The pages are read only. They show what knotless is and how to connect,
what happened lately, each account's profile and public repositories,
and each repository's trees, files, history, commits, pulls, issues and
checks. A repository's page is at its clone URL. Commits show each file's
diff with old and new line numbers, syntax highlighting, and the changed
part of each edited line. Markdown is rendered, Mermaid diagrams included.
Every page's header searches repositories by name or description, and
their owners by handle: `/` or ⌘K opens a palette over the page, showing
matches as they are typed, and `/search?q=` lists them all, or every
repository, newest first, without `q`. Logged in, the private repositories
they may read come first, marked so.
The front page, repositories, pulls and issues unfurl into social cards
that the server draws itself.

The pages are a SvelteKit app in `web/`, run with Node. `--web` names where
it listens: the server hands it every request it does not answer itself,
and the app draws each page from the server's XRPC queries. Without
`--web`, the server has no pages.

```sh
just web                                   # build web/build
just pages                                 # serve it on :2591
just serve --index index.db --web http://127.0.0.1:2591
just web-dev                               # or redraw as the source changes
```

## Check its health

`/_status` answers 200 while the server's feeds bring events, its index
answers and, with `?repo=<handle>/<name>`, that repository's mirror reads.
Otherwise it answers 503 and says what failed. `knotless status` reads it:

```sh
knotless status https://knotless.dev --repo chadtmiller.com/knotless
```

[deploy/README.md](../../deploy/README.md#monitoring) says what each part
means.

## See how it is used

With `--usage <file>`, the server counts its requests by surface, route,
status, time taken and kind of client, and the pages drawn in a browser,
each visitor once a day by a salted hash it forgets once the day is over,
and the events of signed-in accounts (logins, MCP, the gateway), keeping
who did each for 30 days and only counts after.
`knotless admin stats` shows those beside what the index says the network
did, to the accounts `KNOTLESS_ADMIN_DIDS` names on the server:

```sh
KNOTLESS_ADMIN_DIDS=did:plc:… knotless-serve --usage usage.db …   # on the server
knotless admin stats --days 30                                     # signed in as one of them
```

[deploy/README.md](../../deploy/README.md#usage) says what is counted and
what is kept.