chadtmiller.com/knotless

# Design

## Constraints

knotless has to run against the reference PDS (`@atproto/pds`) as shipped,
so public repositories use only `com.atproto.repo.*`,
`com.atproto.sync.getBlob` and `com.atproto.server.*`. Private repositories
add `com.atproto.space.*` and `com.atproto.simplespace.*` from the
permissioned-data alpha of the same PDS. What that PDS allows shapes every
choice below.

| Limit (reference PDS defaults) | Consequence |
| --- | --- |
| Blob upload ≤ 5 MB (`PDS_BLOB_UPLOAD_LIMIT`) | Large packs are split into 4 MB parts; a part refused as too large is retried at half the size. |
| JSON request body ≈ 1 MB; the firehose carries every record | Packs ride inline only up to 192 KiB (`KNOTLESS_INLINE_MAX`). |
| `applyWrites` ≤ 200 writes | A push of more than ~99 refs spans batches. Each ref's claim travels in the same batch as the ref. |
| Rate limits, when enabled (as on bsky.social): 5000 points/hour per account (create 3, update 2, delete 1); `uploadBlob` 1000/day | A small push costs 8 points (pack 3, claim 3, ref 2), so about 600 pushes an hour per account. Inline packs keep small pushes off the blob quota. |
| `applyWrites` has no per-write swap, only `swapCommit` | Compare-and-swap comes from create-only claim keys (below), not from `swapCommit`, which would conflict with every other write to the account. |
| A create on an existing key fails the whole batch, with a 500 | A refused batch is followed by a reread of the refs, to tell a lost race from any other failure. |
| Only an account writes to its own repo | Collaborators push through a gateway the owner has delegated to (below). |
| `listRecords` filters by collection, not by prefix | Keys share the repository's prefix, and scans start from the cursor `<key>/`. The reference PDS orders by rkey and pages with `rkey < cursor`, so the scan stops at the first key outside the prefix. |

## Records

```
dev.knotless.repo     <name>                        key, name, description?, defaultBranch?,
                                                    collaborators?, gateway?, forkOf?,
                                                    transfer?, transferredFrom?
dev.knotless.ref      <key>.<h(ref)>                name, target?, generation
dev.knotless.claim    <key>.<h(ref)>.<generation>   name, generation, from?, to?, pack?
dev.knotless.pack     <key>.<tid>                   tips, bases, size, objects, data | parts, supersedes?
dev.knotless.redirect <old name>                    to, space?

dev.knotless.pull     <tid>                         repo, base, head {repo, branch}, title, body?
dev.knotless.issue    <tid>                         repo, title, body?
dev.knotless.comment  <tid>                         repo, subject, body, verdict?, commit?
dev.knotless.state    <tid>                         repo, subject, state?, commit?, base?
dev.knotless.hide     <tid>                         repo, subject
dev.knotless.block    <tid>                         subject, repo?
```

`h(ref)` is the first 26 characters of the lowercase base32 SHA-256 of the
ref name, so any ref git allows becomes a valid record key. `key` is a TID
minted when the repository is created. Every record carries it, so a rename
rewrites only the `repo` record.

The first five live in the owner's repo. The pull, issue, comment and
state records live in each author's repo, and hides and blocks in the owner's; they
are described under [Pulls and issues](#pulls-and-issues).

### Identity

A repository is its owner's DID and its `key`. The name is for people: it
can change, and once a repository is deleted or renamed away, another can
take it. So a record that names a repository (a pull, issue, comment or
state, a pull's head, a fork's `forkOf`) holds a `repoRef`: the `at://` URI
of the `repo` record as it was named when written, and the key. Readers
match on the URI's owner and the key, never the name, so those records
follow a rename and never attach to a later repository of the same name.

A rename, in one `applyWrites`, creates the `repo` record under the new
name, deletes the old one, and leaves a `redirect` under the old name. It
also repoints every redirect already leading to the repository, so an old
name is always one hop from the current one, and deletes a redirect under
the new name if there was one. A lookup by name that finds no `repo` record
reads the `redirect` under that name and follows it, so clone URLs,
remotes and links made with the old name keep working; `git` is told the
new name. A redirect is followed only within the account and only to a
record with the key it names. Creating a repository under a name that holds
a redirect replaces it; deleting a repository deletes its redirects.

A private repository's records stay in the space named for its first name,
its home. A rename creates a space under the new name, with the same
members, holding only a redirect whose `space` names the home; a member who
opens the repository by its new name finds the redirect there and joins the
home. Sharing and unsharing change every such space, and deleting the
repository deletes them all. Members' cached credentials are bound to a
space's address, so a space's name is never taken by another repository:
creating one where a renamed repository's space is refuses.

A repository moves to another account in two steps, since neither account
can write to the other. The owner offers it, setting `transfer` on the
`repo` record to a `repoRef` in the new account. The recipient accepts:
the repository is copied into their account with the same key and
`transferredFrom` naming the old place. A move holds only when both sides
say so: a reader following the old name goes to the new place only if the
new `repo` record names the old one, and records about the old repository
count for the new one only if the old owner's offer, or the redirect they
leave when they delete the old copy, leads there. State changes the old
owner made before the new `repo` record was created still count; later
ones do not. When the old copy is deleted, its name and redirects become
redirects into the new account.

A `knotless://handle/name` remote resolves the handle on every use. The
helper records the DID the remote first resolved to
(`remote.<name>.knotlessdid`) and refuses to push or fetch if the handle
later names another account.

### Push

1. Read the `repo` record, the repository's refs and its newest page of
   packs, in parallel.
2. For each ref: refuse a non-fast-forward unless forced, and refuse an old
   target that is not available locally ("fetch first").
3. `git pack-objects --revs` of the new tips, excluding every remote ref
   target the local repository has. The pack is self-contained: no thin
   deltas, so it can be indexed alone.
4. One `applyWrites`: create the `repo` record if this is the first push,
   create the pack, and for each ref create
   `claim(<key>.<h>.<generation>)` and put
   `ref(<key>.<h>, generation + 1)`.

Two writers that read generation *n* both create `claim(...n)`, and the PDS
accepts only the first. The ref update travels in the same batch, so it
lands only with its claim. A deleted ref keeps its record, without a
target, so its generation keeps counting and an old claim can never be
replayed.

### Fetch

Every wanted object present already: done. Otherwise, walk the pack pages
newest first. Download, in parallel, each pack with a tip that is missing
locally, and index it. Add the pack's `bases` to the wanted set, and stop
once everything wanted is present. If a walk ends with something still
missing, walk once more from the top: a compaction may have replaced packs
between two pages. If it is still missing after that, report it.

### Compaction

Repack every object the live refs reach into one pack, written with
`supersedes` listing the packs it replaces, and delete those in the same
`applyWrites`. Claims older than a day go in the same writes: a claim
guards only the generation it names while a writer may still hold a read
of it, and a push refuses to write from a read older than half that long. Only packs older than the grace period (ten minutes) are
replaced. A reader that listed them before the delete either finishes
inside that window or restarts its walk once, as above. Objects that no
ref reaches are dropped here, and their blobs are dereferenced and removed
by the PDS.

### Commits without a checkout

`knotless commit` syncs a bare mirror in the cache directory. It builds
the commit there through a temporary index (`read-tree`, `hash-object`,
`update-index --index-info`, `write-tree`, `commit-tree`) and pushes it.
Put and delete name paths, so replaying them onto a tip that moved cannot
conflict. A lost race backs off with exponential jitter and replays. The
claim pins the parent's generation, so these pushes skip git's
fast-forward check. A commit to a branch that does not exist yet starts
it from the commit the changes were made against, or else from the
default branch. Commits are authored by whoever makes them, by handle.
`knotless branch` makes and removes branches the same way, without a
commit.

Mirrors are shared by every process using the cache. Packs are indexed
without a lock, because each lands under its own name. The ref transaction
runs under a lock file, and a lock older than two minutes is treated as
abandoned.

## Logging in

`knotless login <handle>` logs in with OAuth as atproto profiles it.
knotless is a loopback client: its client ID is `http://localhost` with
its scope and redirect in the query, so there is no client metadata to
host. It finds the account's authorization server from its PDS
(`/.well-known/oauth-protected-resource`), pushes the request with PKCE,
opens the authorization page in a browser, and takes the code back on a
listener on `127.0.0.1`. Tokens are bound to a P-256 DPoP key kept with
the session; every request to the PDS carries a proof, and one refused
for want of the server's nonce is sent again with it. Refresh tokens are
used once, so processes refresh under a lock file, and one that finds the
session already refreshed takes it. Logging out revokes the tokens.

The scope is `transition:generic`, what an app password allows, plus a
`space` scope for private repositories, which `transition:generic` does
not reach. It names every space type: naming `dev.knotless.space` alone
makes the authorization server resolve its lexicon, which is not
published. A PDS without spaces drops the scope it does not know.

An app password (`--password-stdin`, or `KNOTLESS_PASSWORD`) still logs
in. A loopback client's session lasts two weeks at most, after which
`knotless login` again.

## Private repositories

A private repository is the space
`at://<owner>/space/dev.knotless.space/<name>`, created with open app access
and member-list read and write policies. Its records are the public ones,
written with the space's `applyWrites` into the owner's repo inside the
space. A create on an existing key fails there too, so the claim
compare-and-swap is unchanged.

The owner reads their own repo in the space with their session. Anyone else,
the owner included when reading another member's repo, exchanges a
delegation token from their own PDS for a space credential from the
owner's, signing both requests with a fresh P-256 key (RFC 9421, as the
alpha specifies). Reads then carry the credential and an audience naming
the repo being read. App passwords cannot mint delegation tokens, so
reading someone else's private repository takes an OAuth login. The
credential and its key are cached, readable only by the user, and reused
until a minute before they expire.

A repository is looked for publicly first, then in a space as whoever is
logged in; which one it was is cached with its key. A PDS without spaces
answers the space methods as unknown, and the repository is public.

Every member is a space writer: the alpha lists a member's repo in the
space only if their writes are allowed, and members write their issues and
comments there. Pushing is a separate grant, the `collaborators` of the
`repo` record. Deleting a private repository deletes the space.

`knotless-serve` reads public records only, so what it shows anyone may
see. With `--private` it looks in spaces too, as whoever is logged in. It
reads a private repository's pulls and issues from its members' repos in
the space, and lists the account's own private repositories on its profile
(`com.atproto.space.listSpaces`, which names only spaces the account has
written to, so not others' it may read). It then refuses to listen anywhere
but a loopback address, since its pages would show that person's private
repositories to whoever reached them.

## Gateway

Only an account can write to its own repo, so a collaborator cannot push
directly. A gateway is a `knotless-serve` with `--public-url`, identified as
`did:web` of its host and serving its own DID document with a
`#knotless_gateway` service.

- **Delegation.** The gateway is an OAuth client of its own, a
  confidential one: its client ID is `/oauth-client-metadata.json` on its
  public URL, and it authenticates to authorization servers with a P-256
  key kept in its config directory (`oauth-client-key`), published in that
  metadata. `knotless delegate` calls `dev.knotless.putDelegation` with a
  service token; the gateway pushes an authorization request for the owner
  and returns its URL, which the owner approves in a browser. The browser
  comes back to the gateway's `/oauth/callback`, and the gateway keeps the
  session, refreshing it as that client. `knotless delegate` waits on
  `dev.knotless.getDelegation`. The grant is narrow: the four knotless
  record types, blobs, and those collections in the owner's own spaces.
  A confidential client's session lasts up to two years if it is used
  every three months; the owner revokes it from their PDS's account
  settings. Sharing a repository for writing records the gateway in its
  `repo` record.
- **Authentication.** Every call carries a service token from
  `com.atproto.server.getServiceAuth`, with the gateway as audience and the
  method as `lxm`. The gateway checks it against the signing key in the
  issuer's DID document (Multikey, or the legacy secp256k1 type PLC still
  returns), then checks the issuer is the owner or a collaborator.
- **Push.** `dev.knotless.pushRefs` takes a JSON header line (the repository and
  ref updates) and then the pack. The gateway indexes it into its mirror and
  pushes as the owner, with the same compare-and-swap as any push.
- **Commit.** `dev.knotless.createCommit` takes path changes and an optional
  `base`. Commits to one branch queue; whichever request takes the branch
  turn drains the queue, builds the commits as a chain and lands them in
  one push. If the branch moved, the chain is rebuilt on the new tip, up to
  eight times. A change to a path that also changed since its `base` is a
  conflict (409) for that commit alone.

The gateway holds an OAuth session for the owner, so it is trusted with
writing knotless records to the owner's account: run your own, or use one
you trust.

## Pulls and issues

Items and everything said about them are records in their authors'
accounts, naming the repository as under [Identity](#identity). A pull names a target branch and a head repository and branch,
which may be a fork. A comment with a verdict is a review and records the
head commit it saw. An item's state is the latest `state` record, ordered
by key, from someone allowed to set it: the item's author, the owner, or a
collaborator. Only the owner and collaborators can set `merged`. A `state`
with a `base` and no `state` moves a pull to another target branch, as
after a branch is renamed, and leaves its state alone, so it cannot undo a
merge it raced; the pull's author changes its head branch by rewriting the
pull record.

To list them, a client needs every author's records about one repository.
For public repositories an appview indexes them from the firehose
(`dev.knotless.getForge`, a page at a time); for private ones the client enumerates the
space's member repos (`com.atproto.space.listRepos`) and reads each. A
comment reads the item it is about straight from its author's repo, so it
does not wait for an index.

`knotless merge` syncs the target's mirror, fetches the head from the
source's packs, and fast-forwards when it can. Otherwise it builds a merge
commit with `git merge-tree --write-tree`, and stops on conflicts. It pushes
directly as the owner or through the gateway as a collaborator, then writes
a `merged` state naming the commit. Comparing a pull does the same up to
the push: the commits it brings in, and whether it is merged already,
fast-forwards, merges cleanly or conflicts, and on which paths. The diff
of a pull is its head against where it left its target, wherever the head
lives. Renaming a branch makes the new one, retargets the open pulls on
the old one, then deletes it, so a rename cut short loses nothing.

A pull or issue just opened may not be indexed yet, so showing one the
appview lacks reads it from its author's repo if it is under a minute
old, and neither the appview's takedowns (asked of its `queryLabels`) nor
the owner's own hides and blocks keep it out. A fork copies every ref into a new
repository whose `repo` record names its source; a fork of a private
repository is private, and shared for reading with the source's owner so
they can take pulls from it.

### Moderation

Moderation is atproto's. A repository's owner keeps out what accounts they
block write about it, and hides pulls, issues and comments with a `hide`
record for each, as a postgate detaches quotes of one post: one record
each, so there is no list to outgrow. What was said about a hidden item
goes with it. A block is the forge's own record, shaped as `app.bsky.graph.block`:
blocking someone on Bluesky says nothing about their pulls, and a block
can name one repository, keeping its subject out of that one alone. A
block naming a private repository is kept in the space with it. Readers
apply both: the appview indexes owners' blocks and hides; in a space the
client reads the owner's blocks from their repo and the space.

Anyone can report an account, a repository or a record with
`com.atproto.moderation.createReport`, sent with a service token to the
moderation service the DID document of `KNOTLESS_MODERATOR` names
(`#atproto_labeler`), or by default to the appview's. The service's
operator reads reports with `dev.knotless.listReports` and takes subjects
down with `com.atproto.admin.updateSubjectStatus`, both with the admin
password as a PDS takes it.

## knotless-serve

A stateless cache in front of PDSes. Per repository it keeps a bare
mirror, synced on demand when older than `--ttl`. With `--firehose` it
also follows `subscribeRepos`: a commit touching a `dev.knotless.*`
collection marks that account's mirrors stale and refreshes them in the
background. While the subscription is up, a mirror with no new events for
its account is served without asking the PDS; while it is down, the TTL
applies.

With `--jetstream` the same watching runs over
`network.bsky.jetstream.subscribeEvents`, asking only for `dev.knotless.*`
commits (identity, account and sync events arrive for every account, since
Jetstream filters collections on commits alone). Jetstream re-encodes the
firehose as JSON and includes each record decoded, so the index takes the
record from the event and writes it in event order, with no `getRecord`; it
resumes from the sequence after the last one seen, as the cursor is
inclusive. Jetstream frames are unsigned: the record is what the Jetstream
operator says the commit held. That is the trust a server already gives the
relay it follows, and the index stays a cache, rebuildable by `--backfill`.
A cursor means something only on the host that issued it.

Clones run through `git upload-pack --stateless-rpc`, so protocol v2,
`--filter` and `--depth` work as they do anywhere. The API answers from
the same mirror. Revisions are rejected if git could read them as options,
and responses named by a full hash are marked immutable.

With `--index` it is also an appview. Firehose events for knotless
collections are indexed into SQLite by fetching the record as it now
stands, or removing it; `--backfill` walks `com.atproto.sync.listRepos`
on a PDS or relay once at start. The index answers
`dev.knotless.listRepos` (newest first, matched on name and description, paged by cursor)
and `dev.knotless.getForge`, which takes the repository's URI and key and
returns the records naming that owner and key under any name, paged by
URI. The index is
a cache: one built for another version of its tables is dropped and rebuilt
from the firehose and `--backfill`.

An appview with `--public-url` is also a moderation service. It keeps the
reports it is sent and its takedowns in the same file, outside the cache,
so a rebuild keeps them. A takedown of an account or record leaves the
index, the pages, the API and clones at once. Each goes out as a
`!takedown` label (or its negation) on `com.atproto.label.subscribeLabels`
and `queryLabels`, and with `--labeler <did>` an appview follows another
labeler's stream and honours its `!takedown` and `!hide` labels the same
way. Labels go out unsigned; a follower trusts the endpoint the labeler's
DID document names.

### Web interface

Read-only HTML over the same mirrors and index. Pages read objects in
process through `gix`, opened once per mirror, because starting a git
process costs more than the reads themselves; only a commit's diff runs
`git show`. Each page is one document with its styles inline and two fixed
scripts, allowed by hash in a `default-src 'none'` content security policy:

- **Highlighting.** Code is sent as escaped text. Tree-sitter grammars for
  the common languages produce ranges, which travel in a `data-hl`
  attribute as `name:gap,len,…` in UTF-16 units; the script turns them into
  `Range`s for the CSS Custom Highlight API, and marks the line a `#L12`
  fragment names. Files over 1 MB, and languages without a grammar, are
  shown plain.
- **Diffs.** A commit's patch is read into files and rows, each row a line
  with its old and new numbers, shown as a table. The ranges for a diff run
  over the rows' text as if joined by newlines, and the script cuts each
  range at row boundaries. Removed lines are highlighted as the old side of
  the hunks reads, the rest as the new side reads, so a comment or string
  that spans lines is still recognized. A removed line paired with an added
  one also gets the part between their common prefix and suffix marked, when
  that shared part is at least half the shorter line. A file with more than
  2000 rows is folded away.
- **Navigation.** Speculation rules prerender a same-site link as the
  pointer settles on it, and cross-document view transitions keep the
  header still between pages.
- **Caching.** Pages named by a commit hash are immutable; pages named by a
  branch are revalidated. HTML is compressed; git responses are not.
- **Social cards.** The front page, each repository's pages, and each pull
  and issue name a 1200×630 PNG in Open Graph tags, at the page's path
  plus `/card.png`. A card is an SVG written like the pages, drawn by
  `resvg` with fonts compiled in, so it is the same on any host: two Inter
  Display faces subset to Latin, and Noto Color Emoji (COLRv1, 5 MB) for
  emoji; other scripts show as boxes. Each run of emoji is placed on its
  own, at an x measured here: shaped together with text in another font,
  resvg drops emoji sequences, or panics on them. A card is written with
  a palette when it has fewer than 256 colors, as one without emoji does:
  exact, a third of the bytes to compress, and smaller with a fast
  compressor than RGB with a thorough one. The URL carries `?v=` and a hash of the SVG, so under
  that version it is immutable, and a page names a new URL when what its
  card shows changes, which is also what makes chat apps that cache
  previews by URL show the new one. Under any other version, it is the
  current card, revalidated by ETag. Cards are kept in memory by version,
  the oldest of 512 dropped, and drawn on blocking threads, one per core
  at most. The origin comes from `Host`, and `X-Forwarded-Proto: https`
  from a proxy in front.
- **Layout.** One stylesheet of custom properties, redefined for dark mode,
  with a single column under 640 px. Nothing is rendered lazily with
  `content-visibility`, so a full-page screenshot is the same every time.

An account's page, at `/<handle>`, lists its public repositories from its
own PDS rather than the index, so it is right without an appview, and shows
the display name and description of its Bluesky profile record if it has
one. Private repositories are in spaces and never listed there.

A page asked for by a repository's old name redirects to the same page
under its new name. The redirect is temporary, as the old name can be
given to another repository. Clones over HTTP follow the old name without
a redirect.

README markdown is rendered with raw HTML shown as text, relative links
pointed into the repository, and links with schemes other than http(s)
and mailto dropped. Raw files are served sandboxed, typed only as text,
an image, or bytes. A handle is shown for a DID only if it resolves back to
that DID.

## knotless-mcp

An MCP server over stdio, acting as the logged-in account (or
`KNOTLESS_IDENTIFIER` and `KNOTLESS_PASSWORD`). Its tools are the CLI's
operations with structured results: reads answer from the cache mirror,
writes go through the same code as `knotless commit`, `fork`, `pull` and
`merge`, and errors come back as tool errors for the agent to read.

## Integrity

Every git object is named by its hash, and `index-pack` verifies every pack
it reads. A PDS, or a `knotless-serve` in front of one, can withhold data
but cannot substitute it. Ref targets come from the owner's signed atproto
repo. `knotless` reads them with plain XRPC and does not check the commit
signature itself; `com.atproto.sync.getRecord` returns the proof a client
that wants to check it needs.

A gateway can write the knotless records its OAuth grant allows to an
owner's account; what it writes to a repository is still checked by every reader's
`index-pack`. Paths in commits are refused if they are empty, contain `.`,
`..` or `.git` segments, or contain NUL or newline characters.

## Prior art

- **pds.js `@pdsjs/git`** stored a repository as one record holding every
  ref plus a chain of git bundles. Any two pushes to the repository
  contended on that one record. knotless splits refs into records with
  their own compare-and-swap, and replaces bundles with self-contained
  packs that arrive inline with the listing.
- **Tangled** keeps git on separate servers ("knots") and the social
  records in the PDS. knotless keeps git in the PDS, and adds a server
  (`knotless-serve`) as an optional cache that anyone can run and nobody
  has to trust, and as a gateway that only the owners who delegate to it
  trust.
- **Anthropic's Scion** ("Redesigning source control at Anthropic",
  October 2026) replaced git internally once agents pushed merges past
  30,000 a day. Its lessons that carry over: clients are online, so fetch
  by immutable hash through tiers of caches (`knotless-serve`, the object
  API). Shared mutable state should be a compare-and-swap on one row (one
  record per ref). Refs are the scaling problem, so agents should work on
  their own branches. A working copy is optional (`knotless commit`).
- **Cursor Origin** keeps git's wire protocol and builds the agent-facing
  forge around it. `knotless-serve` makes the same bet for reads: plain git
  clients, with no special client required.