~ knotless Search/
↑↓ move↵ open⇧↵ all results

chadtmiller.com/knotless

git that lives in your Atmosphere account

7
444 lines · 26.1 KB Raw History

Access

How knotless logs in, what it asks for, and how private repositories are kept and read.

Logging in

knotless login <handle> logs in with OAuth, following atproto's OAuth profile.

sequenceDiagram
  participant K as knotless
  participant P as PDS
  participant AS as Authorization server
  participant B as Browser
  K->>P: /.well-known/oauth-protected-resource
  P-->>K: authorization server
  K->>AS: pushed authorization request (PKCE, DPoP)
  K->>B: open the authorization page
  B->>AS: approve
  AS->>K: redirect to 127.0.0.1 listener with code
  K->>AS: exchange code (DPoP proof)
  AS-->>K: tokens bound to the DPoP key

knotless is a loopback client. Its client ID is http://localhost with its scope and redirect in the query, so there is no client metadata to host. It finds the account's authorization server from the PDS (/.well-known/oauth-protected-resource) and pushes the authorization request with PKCE. It opens the authorization page in a browser (KNOTLESS_BROWSER names the command), and takes the code back on a listener on 127.0.0.1.

The tokens are bound to a P-256 DPoP key kept with the session. Every request to the PDS carries a DPoP proof. A request refused for lack of the server's nonce is sent again with it.

Refresh tokens can be used only once, so processes refresh under a lock file. A process that finds the session already refreshed uses the new one.

A loopback client's session lasts two weeks at most, after which you run knotless login again. Logging out revokes the tokens.

An app password (--password-stdin, or KNOTLESS_PASSWORD) also logs in, but cannot read other people's private repositories.

Scopes

knotless asks only for what it does, rather than for the whole account as transition:generic would:

ScopeFor
repo:each dev.knotless.* record it writes
blob:packs, releases' files, pictures in profiles, pulls, issues and comments (PNG, JPEG, WebP or GIF), and a runner's checks' logs (plain text) and outputs (PNG, JPEG, WebP or GIF)
rpc:the service tokens it gets to call gateways (pushRefs, createCommit, putDelegation, getDelegation, and for organizations putOrgMember, removeOrgMember, joinOrg, leaveOrg and createOrgRepo), to ask an appview who is in an organization, as one inside it (getOrg, getProfile), and moderation services (createReport, and for an operator updateSubjectStatus, getSubjectStatus, listReports and getUsage), to read and mark the account's notifications and choose which it hears of (listNotifications, updateSeen, getNotificationPreferences, putNotificationPreferences), to have an appview forget an account whose knotless data it deleted (deleteAccountData), to invite a private repository's runners or read a runner's invites (inviteRunner, listRunnerInvites), to give a repository's runners sealed secrets, or read them as a runner (putSecret, deleteSecret, listSecrets), and for a runner to send its runs' output and hear of writes in its private repositories' spaces (putRunOutput, awaitSpaceWrites), at any audience, since anyone can run a gateway
spaceprivate repositories

The scope is built from the same constants the records and calls use, so a new record kind is asked for as soon as it is added. A session from before then asks again at its next login.

The space scopes name knotless's own space types, each with the collections its lexicon lists, so a knotless login reaches no other app's spaces. An authorization server asked for a space type by name resolves that type's lexicon as the account approves, and refuses the login if it cannot. Each of knotless's space types has one, published with its other lexicons: dev.knotless.space, a private repository's, and dev.knotless.orgSpace, an organization's. A space of any other type, as the dev.knotless.org spaces organizations kept before their type had a lexicon, is out of a login's reach, and knotless reads none. A PDS without spaces drops the scope it does not know.

Private repositories

A private repository is the space at://<owner>/space/dev.knotless.space/<name>, governed by the alpha's memberspace policy: only its members read and write, with any app. Its records are the same as a public repository's, written with the space's applyWrites into the owner's repo inside the space. A create on an existing key fails there too, so the claim compare-and-swap works unchanged.

The calls differ from a public repository's only in their names and arguments:

  • A write is the same batch, sent to com.atproto.space.applyWrites as {space, repo, writes}, with each write's $type rewritten from com.atproto.repo. to com.atproto.space..
  • A pack's blobs are read with com.atproto.space.getBlob, carrying the credential, rather than com.atproto.sync.getBlob.
  • A read with a credential carries it, and its getRecord also asks for a MAC header.
  • A listing's cursor is the record's full URI, <space uri>/<did>/<collection>/<rkey>, not the bare record key.

A record in a space is named as atproto's permissioned data proposal (0016) addresses it, every segment through its rkey: at://<owner>/space/dev.knotless.space/<name>/<author>/<collection>/<rkey>. That is the name knotless gives it everywhere: a pull's, issue's, comment's or check's URI as a command prints it, as the pages and the appview's answers carry it, as the appview's index keeps it, and as a record written into the space names another there (a comment's subject, a check's pull). A private repository's own record is named so too, in the repo that records about it carry, though a reader matches a repository by its owner and key.

The public form, at://<author>/<collection>/<rkey>, names it only among one space's records. knotless reads either wherever it takes a record's URI, from a command's arguments to a pull's Fixes, and gives the public form the space of the repository it is about (Target::named). A key made from one, as a label's or a pull's checks', is made from the public form, so it is the same for both (crates/knotless/src/aturi.rs).

A record knotless writes into a space names what it refers to there in full, and is read as written. What people typed, as a URI in a body, is left as they wrote it, and read by aturi::resolve.

Spaces are in the reference PDS's permissioned-data alpha. A PDS without them answers the space methods as unknown, and every repository on it is public. Asked for a private one, knotless says the PDS can't hold one yet, having made nothing; the pages' New repository shows Private as not yet open for an account whose PDS has no spaces, as the server found listing theirs when they logged in.

Reading a private repository

sequenceDiagram
  participant M as Member's knotless
  participant MP as Member's PDS
  participant OP as Owner's PDS
  M->>MP: delegation token (with the member's session)
  MP-->>M: token
  M->>OP: exchange for a space credential bound to a fresh P-256 key
  OP-->>M: credential (cached until a minute before expiry)
  M->>OP: read records, with credential + audience naming the repo,<br/>signed with that key (RFC 9421)

The owner reads their own repo in the space with their session. Everyone else exchanges a delegation token from their own PDS for a space credential from the owner's, and so does the owner when reading another member's repo. The credential is bound to a fresh P-256 key, named in the exchange. Reads then carry the credential and an audience naming the repo being read, signed with that key (RFC 9421, as the alpha specifies).

App passwords cannot mint delegation tokens, so reading someone else's private repository takes an OAuth login. Bots on a server, such as a runner, log in through its gateway (Bots). The credential and its key are cached, readable only by the user, and reused until a minute before they expire. What holds a space open for longer than a credential lasts, as a runner watching a private repository does, renews it then, as it renews a session: from the cache, if another read renewed it already, else with a fresh exchange. An owner reading with their session renews the session the same way.

knotless looks for a repository publicly first, then in a space as whoever is logged in. It caches which of the two it found, along with the key.

Members and writers

A private repository is made with knotless create --private, the create_repo tool's private, or the Private choice on the pages' New repository. Its owner adds and removes members with knotless share and unshare, the share tool, or the Readers list on the pages' Access tab, and lists them with knotless members, the members tool, or that list, which only the owner sees.

Every member is a space writer. The alpha lists a member's repo in the space only if that member's writes are allowed, and members write their issues and comments there, which keeps the conversation as private as the code. Pushing is a separate grant: the collaborators of the repo record, who push through the gateway as they do for public repositories. Taking a collaborator out (unshare) takes their reading too; sharing them for reading alone keeps them a member. Those shared to read alone are named in the record's readers, as collaborators are in collaborators: the owner's word, which an appview takes for whom it may tell of what is said there (Notifications).

A private repository of an organization has its members in its space too, each added as they join and taken out as they leave, which the organization's gateway does; and the organization keeps a dev.knotless.orgMember record there for each of them, so its readers know who of them pushes to it. A member shared it on their own as well, for reading or writing, stays when they leave; unsharing a member leaves them in while they are one. The organization's member list is a space of its own, at://<org>/space/dev.knotless.orgSpace/self, which only it and its members read.

A fork of a private repository is private, and is shared for reading with the source's owner so that they can take pulls from it, made from the pages' Fork button as from knotless fork. Deleting a private repository deletes its space.

A runner named in a private repository's runners, the appview's default one unless --runner or --no-runner says otherwise, is made a member, so that it can read the repository and file its checks in its own repo in the space, where they are as private as the code. knotless create says which runners can read it. The runner explains how it finds the repository, through an invite at the appview, and hears of pushes there, which Jetstream never carries.

Browsing

The pages show a private repository to each of its members logged in to them, as a public one is shown, and to no one else. knotless-serve reads it as the member asking, never as itself:

sequenceDiagram
  participant B as Member's browser
  participant S as knotless-serve
  participant MP as Member's PDS
  participant OP as Owner's PDS
  B->>S: a page, with the session cookie
  S->>S: not found publicly
  S->>MP: delegation token (the member's session kept here)
  MP-->>S: token
  S->>OP: exchange for a space credential
  OP-->>S: credential, to members only
  S->>OP: read the space as the member
  S-->>B: the page, Cache-Control: private, no-store
  • Logging in to the pages (serve.md) leaves the server a session for the account, with the space scope. A repository that no one may read publicly is then looked for in its space with that session, just as knotless reads it on the member's own machine (Reading a private repository): their PDS mints the delegation token, and the owner's PDS gives a credential only to a member.
  • What is found is that member's alone: its mirror kept under their cache, and its entry under their DID, so nothing read for one viewer answers another. Its pulls, issues and checks are read from the server's own index of its space (An index of every one read here), and a check's steps and log from the runner's repo in the space, as the member.
  • Membership is asked again, with a fresh credential, once the last answer is a minute old; in between, the space is read as it was found then, with no round trip to look for it again. Its mirror is synced again only once the sync of its space finds it written to. Someone removed from a repository reads it here for at most that minute more. Logging out forgets what was shown to the account, its space credentials and its mirrors at once.
  • A member who never logged in here, reading through an app that signed them in to their own PDS, proves it with a delegation token from there for the repository's space, one with each query: the server spends it once, exchanges it for a credential as it would one its session got, and keeps the credential under their DID for what reads the space as them after. It proves them a member of that repository alone. With no session here to sync its space with after, the space is synced with that credential at each such query, when no member's session syncs it.
  • Anyone else, logged in or not, is answered 404 with the words a repository never made gets, so a private repository's existence does not show.
  • An account's private repositories are listed on its profile from that index, never read from its PDS as anyone looks: all of them to it, and to each member those they were proven a member of. Its own profile shows it, apart, those of others it was proven a member of: "Shared with you" on the overview, and the Shared filter of its repositories.
  • Search, the palette's and /search's, finds the private repositories a viewer may read here, from that index, before the public ones: their own, and those they were proven a member of.

Only queries asked by someone read as them: the queries carrying the session cookie, a service token or a delegation token, and the raw file route. Cards, clones, the object API, and refreshes the server starts on its own read as no one, so they only ever find public repositories. Anything private never reaches the appview index, so feeds, search, stars and activity never show it. Notifications of one come only from the server's own index of it, and only to members (Notifications). A reaction on one is in its space, as a comment is.

An index of every one read here

Read live, a private repository's pulls and issues would come from every member's repo in the space at each look, and its owner's profile from a listing of every space they own, which slows as members and repositories grow. So a server people log in to keeps an index of every private repository read there, in one of its own, serve/private.sqlite in its cache directory, apart from the appview index. There is no switch: a repository on the pages is a repository in that index.

  • The first time a member reads the repository there, its space is followed and synced at once, before the page is answered, with the session of that member; after, with the session of the member who last read it there. There is no account of the server's own to sync as: a did:web holds no repo, and the alpha lets no one but the owner list a space's members. Which member each space is synced as is kept in the index, so syncing goes on across restarts.
  • One made with knotless create --private, the create_repo tool or a fork is in the index before anyone looks, if its owner is logged in to the appview the command names (KNOTLESS_APPVIEW): the command asks the owner's PDS, the space's authority, to tell that appview of each write in the space (com.atproto.space.registerNotify), before it writes the repository's record there. That first write reaches the appview as com.atproto.space.notifyWrite, and it follows the space and syncs it as the owner. One made from the pages' New page is followed and synced as the page is answered.
  • An owner's own are also found as they log in, and again as they look at their profile, at most once a minute and in the background: their spaces, as com.atproto.space.listSpaces names them, each followed and synced as them; one that fails to sync is logged and the rest are still looked at. One its authority said is deleted, or has no such space, is remembered, and not followed again this way, since a PDS can go on naming it. That finds any made some other way. Their profile lists their private repositories from the index, never from their PDS as they look.
  • Which members were proven to read each one is kept in the index too: as each reads it, and, as each logs in or searches, at most once a minute and in the background, for every space of someone else's that they may be a member of. Those are the followed spaces whose repository names them a collaborator or where they wrote, and those they were proven to read before; each is asked of its authority as them, with a fresh credential, as a read is. Which members were proven to read which space outlives a rebuild of the index, which nothing else could bring back: a space so remembered that no one's session syncs any more is followed again and synced as the member it answers. So a repository shared with them for writing is in their search and on its owner's profile before they open it, and no longer once its authority refuses them; asking it, as them, can fail without an answer (the authority unreached, or their login not let ask), which leaves it listed, to ask again, and logs a warning, space reader not checked. One shared only to read names them only in readers, which the server takes for notifications alone, so is listed to them once they have read it here. Which they were is kept when they log out, listed to no one meanwhile, and asked again at their next login: for a space whose owner never logs in here, it is all that leads back to it. A space the server follows no one in is not asked after: the alpha has no way to ask an authority which of its spaces someone belongs to, and listSpaces names only those they hold a repo in.
  • Its pulls, issues and checks are read from that index, as a public repository's are from the appview's, only for a viewer proven a member as above, and answered private, no-store. A check's steps, log and pictures are read from the runner's repo in the space, as the viewer.
  • Each sync asks for a fresh credential as its member, which the owner's PDS gives only to a member still. A sync its authority refuses, as it does once that member is removed or the space deleted, forgets the space and every record it gave the index, until a member reads it there again. One that fails otherwise, as while the authority restarts, keeps it, synced again at the next. A member's logging out stops their syncing it.

Before, an owner chose whether a server kept an index, by naming it in appviews in the repository's record. That field is no longer read, and knotless index, the MCP tool index_repo and dev.knotless.setRepoIndexing are gone.

Nothing written in a space reaches the firehose, so the server asks each space's authority to tell it of writes there instead: once it follows a space and has synced it, it calls com.atproto.space.registerNotify as the member it syncs as, naming its own #knotless_appview service, and again every twelve hours, since the authority keeps a registration for a day. The authority then sends com.atproto.space.notifyWrite for each write, signed as the owner, and the server syncs that space at once. A notification is only a reason to look, never trusted for what it says. The server also syncs every space it keeps every thirty seconds, for a notification that went missing, and forgets a space whose authority sends com.atproto.space.notifySpaceDeleted.

Runners of a private repository hear of its writes through the server too. Each asks the space's authority, as a member, to tell the server of writes there, as the server asks for itself, and the server passes each notifyWrite on to the runners the space's owner invited (dev.knotless.awaitSpaceWrites), whether or not it follows the space. A runner it was not told of hears nothing of the space from it, so a write's time reaches only those the owner chose (runner.md).

Each sync asks the authority's com.atproto.space.listRepos which members' repos were written to since the space revision it last synced to, then reads each of those the way the alpha means a syncing service to, and checks what it reads:

  • A repo in a space keeps a set hash of its records (an LtHash: each record's path and CID expanded by BLAKE3 into 1024 16-bit lanes and summed), and a commit holding its SHA-256 digest at each revision. Its server vouches for the commit with an HMAC-SHA256, keyed by a secret agreed for that one request: the server sends a fresh P-256 key in Atproto-Space-Want-Mac, and the member's server derives the same secret from it with the member's #atproto_kem key (DHKEM(P-256, HKDF-SHA256), RFC 9180), and answers with the MAC in Atproto-Space-Mac.
  • A repo synced before is read on with com.atproto.space.listRepoOps from the revision it was synced at, every page of it. Each write moves the set hash kept for it, taking out what it replaced and adding what it wrote, and the hash must come to the commit the last page carries, which must be vouched for. A write adds or replaces its record; a deletion (an op with no CID) takes it out.
  • A repo never synced, or whose writes do not check out (its oplog no longer reaching back that far, say), is read whole with com.atproto.space.getRepo: a STAR archive, its commit then each record's DRISL encoding by path, in order, hashed afresh and checked the same way. Its records replace those it gave the index before, so one deleted meanwhile goes too.

The set hash and revision are kept per repo with the records, and a repo is moved on only once what was read checks out. The index starts where it left off at the next start, each space synced as the member it was synced as before.

Notifications

What is said on a private repository is heard of from the server's own index of its space, by the same rules as on a public one (serve.md), and only by its members, though they may never log in there, reading and writing only through knotless or its MCP tools:

  • What a member writes there from their client, a pull, an issue, a comment, a review or a state, and an owner's sharing it or taking it back, ends with the client showing the repository to the appview it names (KNOTLESS_APPVIEW), dev.knotless.getRepo with a delegation token for its space, as an app that signed them in elsewhere asks. The server syncs the space with the credential that gets it, when no session here syncs it, so what was written is in the index as the command returns. With no appview set, or one that cannot read it, nothing more is done: the write stands either way.
  • The notifications the index finds in a space for an account are kept only for a repository they may read: their own; one whose record, as the index has it, names them in collaborators or readers, the owner's word; or one they prove they read, as a query of theirs would. So someone it mentions who is not a member hears nothing of it, nor that it is there, and one taken out of it, whom the record no longer names, hears nothing more.

Who can see a private repository

What could show a private repository to someone who is not a member, and what stops it on a server people log in to:

ThreatWhat stops it
A shared cache keeps a member's answer and gives it to someone elseevery answer that read anything private is Cache-Control: private, no-store, and every query answer Vary: cookie; the pages take their caching from their query's answer
The server's own caches give one viewer what another reada private repository's mirror and entry are kept per viewer; the caches shared between viewers are keyed by content hash, which only a reader of that content has
A link's preview, the social card, or the page's meta tags show itcards read as no one, so are 404 for it; the page itself is 404 to the crawler, as to anyone without a member's session
Feeds, search, stars, notifications, activity or a profile list itnone reads anything but the appview index, where nothing in a space ever lands, but notifications and search, which read the server's own index of it too, only for a member (notifications: its owner, one its record shares it with, or a member proven so; search: its owner, or a member proven so); a star of a private repository is refused, and a reaction is in its space; a profile lists private repositories only to their owner, and those a member was proven a member of to them, asking
A non-member learns that it existsa 404 worded as for a repository never made, never a 403
A removed member, or one logged out, keeps reading itmembership is asked again with a fresh credential once a minute, and a refusal takes it out of their search and profiles; logging out forgets what was shown, which they read, the credentials and the mirrors
The server reads with its own credentials, or one viewer's for anotherevery private read uses the asking viewer's own session, kept apart under their DID; what reads as no one finds only public repositories
The server keeps an index of it the owner did not ask fornothing, by design: it keeps one of every private repository a member reads there, and of every one whose owner logs in there. An owner who wants none kept by a server keeps its members off that server's pages, and reads it with knotless, which keeps nothing anywhere but on their own machine
The index answers someone who is not a memberit is read only once the viewer is proven a member, as for a live read; a pull's head in a private repository shows its checks only to a viewer shown that repository
The index outlives a member's accesseach sync asks the owner's PDS for a fresh credential as the member it syncs with; refused, the space and all it gave are forgotten