chadtmiller.com/knotless

Switch repository

Loading…

# Access

How knotless logs in, what it asks for, and how private repositories are
kept and read.

## Logging in

`knotless login <handle>` logs in with OAuth, following atproto's OAuth
profile.

```mermaid
sequenceDiagram
  participant K as knotless
  participant P as PDS
  participant AS as Authorization server
  participant B as Browser
  K->>P: /.well-known/oauth-protected-resource
  P-->>K: authorization server
  K->>AS: pushed authorization request (PKCE, DPoP)
  K->>B: open the authorization page
  B->>AS: approve
  AS->>K: redirect to 127.0.0.1 listener with code
  K->>AS: exchange code (DPoP proof)
  AS-->>K: tokens bound to the DPoP key
```

knotless is a loopback client. Its client ID is `http://localhost` with its
scope and redirect in the query, so there is no client metadata to host. It
finds the account's authorization server from the PDS
(`/.well-known/oauth-protected-resource`) and pushes the authorization
request with PKCE. It opens the authorization page in a browser
(`KNOTLESS_BROWSER` names the command), and takes the code back on a
listener on `127.0.0.1`.

The tokens are bound to a P-256 DPoP key kept with the session. Every
request to the PDS carries a DPoP proof. A request refused for lack of the
server's nonce is sent again with it.

Refresh tokens can be used only once, so processes refresh under a lock
file. A process that finds the session already refreshed uses the new one.

A loopback client's session lasts two weeks at most, after which you run
`knotless login` again. Logging out revokes the tokens.

An app password (`--password-stdin`, or `KNOTLESS_PASSWORD`) also logs in,
but cannot read other people's [private
repositories](#reading-a-private-repository).

### Scopes

knotless asks only for what it does, rather than for the whole account as
`transition:generic` would:

| Scope | For |
| --- | --- |
| `repo:` | each `dev.knotless.*` record it writes |
| `blob:` | packs, releases' files, pictures in profiles, pulls, issues and comments (PNG, JPEG, WebP or GIF), and a runner's checks' logs (plain text) and outputs (PNG, JPEG, WebP or GIF) |
| `rpc:` | the service tokens it gets to call gateways (`pushRefs`, `createCommit`, `putDelegation`, `getDelegation`, and for organizations `putOrgMember`, `removeOrgMember`, `joinOrg`, `leaveOrg` and `createOrgRepo`), to ask an appview who is in an organization, as one inside it (`getOrg`, `getProfile`), and moderation services (`createReport`, and for an operator `updateSubjectStatus`, `getSubjectStatus`, `listReports` and `getUsage`), to read and mark the account's notifications and choose which it hears of (`listNotifications`, `updateSeen`, `getNotificationPreferences`, `putNotificationPreferences`), to have an appview forget an account whose knotless data it deleted (`deleteAccountData`), to invite a private repository's runners or read a runner's invites (`inviteRunner`, `listRunnerInvites`), to give a repository's runners sealed secrets, or read them as a runner (`putSecret`, `deleteSecret`, `listSecrets`), and for a runner to send its runs' output and hear of writes in its private repositories' spaces (`putRunOutput`, `awaitSpaceWrites`), at any audience, since anyone can run a gateway |
| `space` | private repositories |

The scope is built from the same constants the records and calls use, so a
new record kind is asked for as soon as it is added. A session from before
then asks again at its next login.

The space scopes name knotless's own space types, each with the
collections its lexicon lists, so a knotless login reaches no other app's
spaces. An authorization server asked for a space type by name resolves
that type's lexicon as the account approves, and refuses the login if it
cannot. Each of knotless's space types has one, published with its other
lexicons: `dev.knotless.space`, a private repository's, and
`dev.knotless.orgSpace`, an organization's. A space of any other type,
as the `dev.knotless.org` spaces organizations kept before their type had
a lexicon, is out of a login's reach, and knotless reads none.
A PDS without spaces drops the scope it does not know.

## Private repositories

A private repository is the space
`at://<owner>/space/dev.knotless.space/<name>`, governed by the alpha's
memberspace policy: only its members read and write, with any app. Its records are the same as a
public repository's, written with the space's `applyWrites` into the
owner's repo inside the space. A create on an existing key fails there too,
so the claim [compare-and-swap](storage.md#compare-and-swap-through-claims) works
unchanged.

The calls differ from a public repository's only in their names and
arguments:

- A write is the same batch, sent to `com.atproto.space.applyWrites` as
  `{space, repo, writes}`, with each write's `$type` rewritten from
  `com.atproto.repo.` to `com.atproto.space.`.
- A pack's blobs are read with `com.atproto.space.getBlob`, carrying the
  credential, rather than `com.atproto.sync.getBlob`.
- A read with a credential carries it, and its `getRecord` also asks for a
  MAC header.
- A listing's cursor is the record's full URI,
  `<space uri>/<did>/<collection>/<rkey>`, not the bare record key.

A record in a space is named as atproto's permissioned data proposal
(0016) addresses it, every segment through its rkey:
`at://<owner>/space/dev.knotless.space/<name>/<author>/<collection>/<rkey>`.
That is the name knotless gives it everywhere: a pull's, issue's,
comment's or check's URI as a command prints it, as the pages and the
appview's answers carry it, as the appview's index keeps it, and as a
record written into the space names another there (a comment's `subject`,
a check's `pull`). A private repository's own record is named so too, in
the `repo` that records about it carry, though a reader matches a
repository by its owner and key.

The public form, `at://<author>/<collection>/<rkey>`, names it only among
one space's records. knotless reads either wherever it takes a record's
URI, from a command's arguments to a pull's `Fixes`, and gives the public
form the space of the repository it is about (`Target::named`). A key made
from one, as a label's or a pull's checks', is made from the public form,
so it is the same for both (`crates/knotless/src/aturi.rs`).

A record knotless writes into a space names what it refers to there in
full, and is read as written. What people typed, as a URI in a body, is
left as they wrote it, and read by `aturi::resolve`.

Spaces are in the reference PDS's permissioned-data alpha. A PDS without
them answers the space methods as unknown, and every repository on it is
public. Asked for a private one, knotless says the PDS can't hold one yet,
having made nothing; the pages' New repository shows Private as not yet
open for an account whose PDS has no spaces, as the server found listing
theirs when they logged in.

### Reading a private repository

```mermaid
sequenceDiagram
  participant M as Member's knotless
  participant MP as Member's PDS
  participant OP as Owner's PDS
  M->>MP: delegation token (with the member's session)
  MP-->>M: token
  M->>OP: exchange for a space credential bound to a fresh P-256 key
  OP-->>M: credential (cached until a minute before expiry)
  M->>OP: read records, with credential + audience naming the repo,<br/>signed with that key (RFC 9421)
```

The owner reads their own repo in the space with their session. Everyone
else exchanges a delegation token from their own PDS for a space credential
from the owner's, and so does the owner when reading another member's repo.
The credential is bound to a fresh P-256 key, named in the exchange. Reads
then carry the credential and an audience naming the repo being read,
signed with that key (RFC 9421, as the alpha specifies).

App passwords cannot mint delegation tokens, so reading someone else's
private repository takes an OAuth login. Bots on a server, such as a
runner, log in through its gateway ([Bots](gateway.md#bots)). The credential and its key are
cached, readable only by the user, and reused until a minute before they
expire. What holds a space open for longer than a credential lasts, as a
runner watching a private repository does, renews it then, as it renews
a session: from the cache, if another read renewed it already, else with
a fresh exchange. An owner reading with their session renews the session
the same way.

knotless looks for a repository publicly first, then in a space as whoever
is logged in. It caches which of the two it found, along with the key.

### Members and writers

A private repository is made with `knotless create --private`, the
`create_repo` tool's `private`, or the Private choice on the pages' New
repository. Its owner adds and removes members with `knotless share` and
`unshare`, the `share` tool, or the Readers list on the pages' Access tab,
and lists them with `knotless members`, the `members` tool, or that list,
which only the owner sees.

Every member is a space writer. The alpha lists a member's repo in the
space only if that member's writes are allowed, and members write their
issues and comments there, which keeps the conversation as private as the
code. Pushing is a separate grant: the `collaborators` of the `repo`
record, who push through the [gateway](gateway.md) as they do for public
repositories. Taking a collaborator out (`unshare`) takes their reading
too; sharing them for reading alone keeps them a member. Those shared to
read alone are named in the record's `readers`, as collaborators are in
`collaborators`: the owner's word, which an appview takes for whom it may
tell of what is said there ([Notifications](#notifications)).

A private repository of an [organization](gateway.md#organizations) has
its members in its space too, each added as they join and taken out as
they leave, which the organization's gateway does; and the organization
keeps a `dev.knotless.orgMember` record there for each of them, so its
readers know who of them pushes to it. A member shared it on their own as well, for
reading or writing, stays when they leave; unsharing a member leaves
them in while they are one. The organization's member list is a space of
its own, `at://<org>/space/dev.knotless.orgSpace/self`, which only it and
its members read.

A fork of a private repository is private, and is shared for reading with
the source's owner so that they can take pulls from it, made from the
pages' Fork button as from `knotless fork`. Deleting a private
repository deletes its space.

A runner named in a private repository's `runners`, the appview's default
one unless `--runner` or `--no-runner` says otherwise, is made a member, so
that it can read the repository and file its checks in its own repo in
the space, where they are as private as the code. `knotless create` says
which runners can read it. The
[runner](runner.md#private-repositories) explains how it finds the
repository, through an invite at the appview, and hears of pushes there,
which Jetstream never carries.

### Browsing

The pages show a private repository to each of its members logged in to
them, as a public one is shown, and to no one else. `knotless-serve` reads
it as the member asking, never as itself:

```mermaid
sequenceDiagram
  participant B as Member's browser
  participant S as knotless-serve
  participant MP as Member's PDS
  participant OP as Owner's PDS
  B->>S: a page, with the session cookie
  S->>S: not found publicly
  S->>MP: delegation token (the member's session kept here)
  MP-->>S: token
  S->>OP: exchange for a space credential
  OP-->>S: credential, to members only
  S->>OP: read the space as the member
  S-->>B: the page, Cache-Control: private, no-store
```

- Logging in to the pages ([serve.md](serve.md#logging-in)) leaves the
  server a session for the account, with the space scope. A repository
  that no one may read publicly is then looked for in its space with that
  session, just as `knotless` reads it on the member's own machine
  ([Reading a private repository](#reading-a-private-repository)): their
  PDS mints the delegation token, and the owner's PDS gives a credential
  only to a member.
- What is found is that member's alone: its mirror kept under their cache,
  and its entry under their DID, so nothing read for one viewer answers
  another. Its pulls, issues and checks are read from the server's own
  index of its space ([An index of every one read
  here](#an-index-of-every-one-read-here)), and a check's steps and log
  from the runner's repo in the space, as the member.
- Membership is asked again, with a fresh credential, once the last answer
  is a minute old; in between, the space is read as it was found then,
  with no round trip to look for it again. Its mirror is synced again only
  once the sync of its space finds it written to. Someone removed from a repository reads it here for at
  most that minute more. Logging out forgets what was shown to the
  account, its space credentials and its mirrors at once.
- A member who never logged in here, reading through an app that signed
  them in to their own PDS, proves it with a delegation token from there
  for the repository's space, one with each query: the server spends it
  once, exchanges it for a credential as it would one its session got,
  and keeps the credential under their DID for what reads the space as
  them after. It proves them a member of that repository alone. With no
  session here to sync its space with after, the space is synced with
  that credential at each such query, when no member's session syncs it.
- Anyone else, logged in or not, is answered 404 with the words a
  repository never made gets, so a private repository's existence does not
  show.
- An account's private repositories are listed on its profile from that
  index, never read from its PDS as anyone looks: all of them to it, and
  to each member those they were proven a member of. Its own profile
  shows it, apart, those of others it was proven a member of: "Shared
  with you" on the overview, and the Shared filter of its repositories.
- Search, the palette's and `/search`'s, finds the private repositories a
  viewer may read here, from that index, before the public ones: their
  own, and those they were proven a member of.

Only queries asked by someone read as them: the queries carrying the
session cookie, a service token or a delegation token, and the raw file
and archive routes. Cards, clones,
the object API, and refreshes the server starts on its own read as no one,
so they only ever find public repositories. Anything private never reaches
the appview index, so feeds, search, stars and activity never show it.
Notifications of one come only from the server's own index of it, and
only to members ([Notifications](#notifications)). A reaction on one is in
its space, as a comment is.

#### An index of every one read here

Read live, a private repository's pulls and issues would come from every
member's repo in the space at each look, and its owner's profile from a
listing of every space they own, which slows as members and repositories
grow. So a server people log in to keeps an index of every private
repository read there, in one of its own, `serve/private.sqlite` in its
cache directory, apart from the appview index. There is no switch: a
repository on the pages is a repository in that index.

- The first time a member reads the repository there, its space is
  followed and synced at once, before the page is answered, with the
  session of that member; after, with the session of the member who last
  read it there. There is no account of the server's own to sync as: a
  `did:web` holds no repo, and the alpha lets no one but the owner list a
  space's members. Which member each space is synced as is kept in the
  index, so syncing goes on across restarts.
- One made with `knotless create --private`, the `create_repo` tool or a
  fork is in the index before anyone looks, if its owner is logged in to
  the appview the command names (`KNOTLESS_APPVIEW`): the command asks the
  owner's PDS, the space's authority, to tell that appview of each write in
  the space (`com.atproto.space.registerNotify`), before it writes the
  repository's record there. That first write reaches the appview as
  `com.atproto.space.notifyWrite`, and it follows the space and syncs it as
  the owner. One made from the pages' New page is followed and synced as
  the page is answered.
- An owner's own are also found as they log in, and again as they look at
  their profile, at most once a minute and in the background: their
  spaces, as `com.atproto.space.listSpaces` names them, each followed and
  synced as them; one that fails to sync is logged and the rest are still
  looked at. One its authority said is deleted, or has no such space, is
  remembered, and not followed again this way, since a PDS can go on
  naming it. That finds any made some other way. Their profile lists
  their private repositories from the index, never from their PDS as they
  look.
- Which members were proven to read each one is kept in the index too:
  as each reads it, and, as each logs in or searches, at most once a
  minute and in the background, for every space of someone else's that
  they may be a member of. Those are the followed spaces whose repository
  names them a collaborator or where they wrote, and those they were
  proven to read before; each is asked of its authority as them, with a
  fresh credential, as a read is. Which members were proven to read which
  space outlives a rebuild of the index, which nothing else could bring
  back: a space so remembered that no one's session syncs any more is
  followed again and synced as the member it answers. So a repository shared with them
  for writing is in their search and on its owner's profile before they
  open it, and no longer once its authority refuses them; asking it, as
  them, can fail without an answer (the authority unreached, or their
  login not let ask), which leaves it listed, to ask again, and logs a
  warning, `space reader not checked`. One shared only to
  read names them only in `readers`, which the server takes for
  notifications alone, so is listed to them once they have read it here. Which they
  were is kept when they log out, listed to no one meanwhile, and asked
  again at their next login: for a space whose owner never logs in here,
  it is all that leads back to it. A space the server follows no one in is not asked after: the
  alpha has no way to ask an authority which of its spaces someone
  belongs to, and `listSpaces` names only those they hold a repo in.
- Its pulls, issues and checks are read from that index, as a public
  repository's are from the appview's, only for a viewer proven a member as
  above, and answered `private, no-store`. A check's steps, log and
  pictures are read from the runner's repo in the space, as the viewer.
- Each sync asks for a fresh credential as its member, which the owner's
  PDS gives only to a member still. A sync its authority refuses, as it
  does once that member is removed or the space deleted, forgets the space
  and every record it gave the index, until a member reads it there again.
  One that fails otherwise, as while the authority restarts, keeps it,
  synced again at the next. A member's logging out stops
  their syncing it.

Before, an owner chose whether a server kept an index, by naming it in
`appviews` in the repository's record. That field is no longer read, and
`knotless index`, the MCP tool `index_repo` and
`dev.knotless.setRepoIndexing` are gone.

Nothing written in a space reaches the firehose, so the server asks each
space's authority to tell it of writes there instead: once it follows a
space and has synced it, it calls `com.atproto.space.registerNotify` as the
member it syncs as, naming its own `#knotless_appview` service, and again
every twelve hours, since the authority keeps a registration for a day.
The authority then sends `com.atproto.space.notifyWrite` for each write,
signed as the owner, and the server syncs that space at once. A
notification is only a reason to look, never trusted for what it says. The
server also syncs every space it keeps every thirty seconds, for a
notification that went missing, and forgets a space whose authority sends
`com.atproto.space.notifySpaceDeleted`.

Runners of a private repository hear of its writes through the server
too. Each asks the space's authority, as a member, to tell the server of
writes there, as the server asks for itself, and the server passes each
`notifyWrite` on to the runners the space's owner invited
(`dev.knotless.awaitSpaceWrites`), whether or not it follows the space. A
runner it was not told of hears nothing of the space from it, so a write's
time reaches only those the owner chose
([runner.md](runner.md#private-repositories)).

Each sync asks the authority's `com.atproto.space.listRepos` which members'
repos were written to since the space revision it last synced to, then
reads each of those the way the alpha means a syncing service to, and
checks what it reads:

- A repo in a space keeps a set hash of its records (an LtHash: each
  record's path and CID expanded by BLAKE3 into 1024 16-bit lanes and
  summed), and a commit holding its SHA-256 digest at each revision. Its
  server vouches for the commit with an HMAC-SHA256, keyed by a secret
  agreed for that one request: the server sends a fresh P-256 key in
  `Atproto-Space-Want-Mac`, and the member's server derives the same secret
  from it with the member's `#atproto_kem` key (DHKEM(P-256, HKDF-SHA256),
  RFC 9180), and answers with the MAC in `Atproto-Space-Mac`.
- A repo synced before is read on with `com.atproto.space.listRepoOps` from
  the revision it was synced at, every page of it. Each write moves the
  set hash kept for it, taking out what it replaced and adding what it
  wrote, and the hash must come to the commit the last page carries, which
  must be vouched for. A write adds or replaces its record; a deletion (an
  op with no CID) takes it out.
- A repo never synced, or whose writes do not check out (its oplog no
  longer reaching back that far, say), is read whole with
  `com.atproto.space.getRepo`: a STAR archive, its commit then each
  record's DRISL encoding by path, in order, hashed afresh and checked the
  same way. Its records replace those it gave the index before, so one
  deleted meanwhile goes too.

The set hash and revision are kept per repo with the records, and a repo
is moved on only once what was read checks out. The index starts where it
left off at the next start, each space synced as the member it was synced
as before.

#### Notifications

What is said on a private repository is heard of from the server's own
index of its space, by the same rules as on a public one
([serve.md](serve.md#notifications)), and only by its members, though they
may never log in there, reading and writing only through `knotless` or
its MCP tools:

- What a member writes there from their client, a pull, an issue, a
  comment, a review or a state, and an owner's sharing it or taking it
  back, ends with the client showing the repository to the appview it
  names (`KNOTLESS_APPVIEW`), `dev.knotless.getRepo` with a delegation
  token for its space, as an app that signed them in elsewhere asks. The
  server syncs the space with the credential that gets it, when no
  session here syncs it, so what was written is in the index as the
  command returns. With no appview set, or one that cannot read it,
  nothing more is done: the write stands either way.
- The notifications the index finds in a space for an account are kept
  only for a repository they may read: their own; one whose record, as
  the index has it, names them in `collaborators` or `readers`, the
  owner's word; or one they prove they read, as a query of theirs would.
  So someone it mentions who is not a member hears nothing of it, nor that
  it is there, and one taken out of it, whom the record no longer names,
  hears nothing more.

### Who can see a private repository

What could show a private repository to someone who is not a member, and
what stops it on a server people log in to:

| Threat | What stops it |
| --- | --- |
| A shared cache keeps a member's answer and gives it to someone else | every answer that read anything private is `Cache-Control: private, no-store`, and every query answer `Vary: cookie`; the pages take their caching from their query's answer |
| The server's own caches give one viewer what another read | a private repository's mirror and entry are kept per viewer; the caches shared between viewers are keyed by content hash, which only a reader of that content has |
| A link's preview, the social card, or the page's meta tags show it | cards read as no one, so are 404 for it; the page itself is 404 to the crawler, as to anyone without a member's session |
| Feeds, search, stars, notifications, activity or a profile list it | none reads anything but the appview index, where nothing in a space ever lands, but notifications and search, which read the server's own index of it too, only for a member (notifications: its owner, one its record shares it with, or a member proven so; search: its owner, or a member proven so); a star of a private repository is refused, and a reaction is in its space; a profile lists private repositories only to their owner, and those a member was proven a member of to them, asking |
| A non-member learns that it exists | a 404 worded as for a repository never made, never a 403 |
| A removed member, or one logged out, keeps reading it | membership is asked again with a fresh credential once a minute, and a refusal takes it out of their search and profiles; logging out forgets what was shown, which they read, the credentials and the mirrors |
| The server reads with its own credentials, or one viewer's for another | every private read uses the asking viewer's own session, kept apart under their DID; what reads as no one finds only public repositories |
| The server keeps an index of it the owner did not ask for | nothing, by design: it keeps one of every private repository a member reads there, and of every one whose owner logs in there. An owner who wants none kept by a server keeps its members off that server's pages, and reads it with `knotless`, which keeps nothing anywhere but on their own machine |
| The index answers someone who is not a member | it is read only once the viewer is proven a member, as for a live read; a pull's head in a private repository shows its checks only to a viewer shown that repository |
| The index outlives a member's access | each sync asks the owner's PDS for a fresh credential as the member it syncs with; refused, the space and all it gave are forgotten |