~ knotless Search/
↑↓ move↵ open⇧↵ all results

chadtmiller.com/knotless

git that lives in your Atmosphere account

7
357 lines · 19.8 KB Raw History

Gateway

Only an account can write to its own repo, so a collaborator cannot push to the owner's account directly. A gateway is a knotless-serve with --public-url that the owner has delegated to. It checks who is asking, then writes as the owner.

A gateway is identified as the did:web of its host (a port's : as %3A), and serves its own DID document at /.well-known/did.json, with a #knotless_gateway service, an #atproto_labeler one and a #knotless_appview one, all at its public URL. --public-url is http(s)://host[:port], with no path.

sequenceDiagram
  participant C as Collaborator's helper
  participant CP as Collaborator's PDS
  participant G as Gateway
  participant OP as Owner's PDS
  C->>CP: getServiceAuth(aud = gateway, lxm = pushRefs)
  CP-->>C: service token
  C->>G: dev.knotless.pushRefs (header line + pack), token
  G->>G: verify token against issuer's DID document
  G->>OP: read repo record: is the issuer owner or collaborator?
  G->>G: check branch rules
  G->>OP: applyWrites as owner (same compare-and-swap as any push)
  OP-->>G: ok
  G-->>C: ok

Delegation

The gateway is an OAuth client of its own, a confidential one. Its client ID is /oauth-client-metadata.json on its public URL. It authenticates to authorization servers with a P-256 key kept in its config directory (oauth-client-key) and published in that metadata.

The metadata names the redirect <public-url>/oauth/callback, private_key_jwt with ES256, DPoP-bound tokens, the grant types authorization_code and refresh_token, client_name knotless gateway and application_type web.

  1. knotless delegate calls dev.knotless.putDelegation with a service token.
  2. The gateway pushes an authorization request for the owner at their PDS, and answers {url, state}. The owner approves it in a browser. A delegation still waiting after 15 minutes is dropped when the next one starts.
  3. The browser comes back to the gateway's /oauth/callback. The gateway keeps the session, and refreshes it as that client. The page says The gateway now writes for you. You can close this tab., or The gateway was not delegated to: <error>; a state nobody is waiting on is 400 no delegation is waiting for this.
  4. Meanwhile knotless delegate calls dev.knotless.getDelegation?state=, which waits up to 25 seconds. It answers {done: false} to ask again, {done: true} once approved, 403 if the authorization failed, and 404 no such delegation for a state the caller did not start.

The grant is narrow: the four record types a push writes (repo, ref, claim and pack), blobs, and those collections in the owner's own spaces; and what an organization needs: its org record in public, its orgMember records in public and in its spaces, its own space for members, and making and changing the members of its spaces, never deleting one. Each space type is named, a private repository's and an organization's (Scopes), so the grant reaches no other app's spaces. redirect is left out, since a gateway never renames a repository. The scope is atproto repo:dev.knotless.repo repo:dev.knotless.ref repo:dev.knotless.claim repo:dev.knotless.pack repo:dev.knotless.org repo:dev.knotless.orgMember blob:*/* space:dev.knotless.space?authority=self&collection=dev.knotless.repo&collection=dev.knotless.ref&collection=dev.knotless.claim&collection=dev.knotless.pack&collection=dev.knotless.orgMember&manage=create&manage=update space:dev.knotless.orgSpace?authority=self&collection=dev.knotless.orgMember&collection=dev.knotless.orgAcceptance&manage=create&manage=update. A grant from before names every space type (space:*), which still covers all of this; the pages ask for it again all the same, as for any older grant. A delegation made before organizations lacks some of it: the organization's own space is refused rather than taken for a PDS without spaces, and knotless delegate again grants the rest. The gateway answers a write refused for want of scope 403, saying the grant is older than what it asks for now, and keeps the scope each delegation asked for, so the pages say so before any write is refused (serve.md). A confidential client's session lasts up to two years if it is used at least every three months. The owner revokes it from their PDS's account settings.

The pages' Access tab delegates the same way, for the account logged in to them, to the server's own gateway: dev.knotless.delegateWrites makes the first call with the session the server keeps for the account, and the page shows the approval page in a new tab while dev.knotless.awaitDelegation makes the last, until it is done. The gateway's grant is its own, apart from the pages' session.

Sharing a repository for writing records the gateway in its repo record. A runner landing the merge queue of a repository that records none goes through the appview's.

Bots

An operator's bots, knotless-run and knotless-triage, log in through the gateway too. An app password mints no delegation tokens, so a bot holding one cannot read private repositories, and a loopback OAuth login lasts only two weeks. Given --bots <dir>, the gateway is a second confidential client, /oauth/bot-client-metadata.json on its public URL, with the same key, the redirect <public-url>/oauth/bot-callback, client_name knotless bots, and all of knotless's scope, private repositories included.

  1. The operator opens <public-url>/oauth/bot?account=<handle or did> in a browser. An account KNOTLESS_BOT_DIDS does not name is 403 <did> is not one of this server's bots. The gateway pushes an authorization request for it at its PDS and redirects there (303).
  2. The operator approves, signed in as that account.
  3. The browser comes back to /oauth/bot-callback. The gateway keeps the session in <dir>/sessions/, and the client key beside it as <dir>/oauth-client-key, and says <did> is logged in for this server's bots. You can close this tab. A refused or failed login is 403 the bot was not logged in: <error>, and a state nobody is waiting on 400 no bot login is waiting for this. A login still waiting after 15 minutes is dropped when the next one starts.

The bots take <dir> as their config directory (KNOTLESS_CONFIG_DIR), and act and refresh as that client from there: the session lasts up to two years if used at least every three months. A stored session comes before KNOTLESS_PASSWORD, which only logs in when there is none, or when the session can no longer be refreshed. Without --bots, these routes are 404 this server keeps no bots' sessions.

Authentication

On a server without --public-url, every gateway route is 404 this server is not a gateway.

Every call to the gateway carries a service token from com.atproto.server.getServiceAuth, with the gateway as audience and the method as lxm. A call without one is 401 a service token is required, and one with a bad or expired token 401. The gateway checks the token against the signing key in the issuer's DID document (Multikey, or the legacy secp256k1 type that PLC still returns), read again when the copy it read in the last five minutes turns the signature away, as once the key rotated. A server that is its own gateway signs the tokens its pages send it on a logged-in account's behalf itself, with a key kept in its config directory, rather than ask the account's PDS: it holds that account's session already. Such a token says so in its header, lasts a minute, and passes only on servers sharing that key. Then it checks, in order, that the owner has delegated to it, else 403 <owner> has not delegated writes to this gateway; that the repository exists, else 404 <url>: no such repository; and that the issuer is the owner or a collaborator, else 403 <caller> is not a collaborator on <url>. A runner the repository names may push too, but only to land its merge queue (Runners landing a queue); it may not commit.

Push

dev.knotless.pushRefs takes a JSON header line, naming the repository and the ref updates, followed by the pack, as application/x-knotless-push, at most 512 MiB. Each update's src is a 40-character lowercase hex hash, or absent for a deletion, and its dst starts refs/; else the push is 400 bad update of <dst>.

Pushes to one repository take turns, under a lock in the process. The gateway syncs its mirror, indexes the pack into it, and refuses a tip the pack does not bring (400 the pack does not hold <sha>). It then pushes as the owner, with the same compare-and-swap as any push, naming the caller, from its verified token, as the pusher of each claim and ref it writes, unless the caller is the owner. It answers results, a {name, error?} per ref: each update lands or is refused on its own, and the client fails on the first one refused.

Runners landing a queue

A runner the repository names lands its merge queue through the gateway when no runner it names may push (merge-queue.md). The gateway takes a push from it only if that push is such a landing, else 403 <caller> runs <url>'s checks, and lands its merge queue here, nothing else: <why>. It checks, against the queue as the records say:

  • one branch moves, without force, and the push deletes branches at most (a runner lands a batch: one branch moved forward);
  • the branch exists (there is no branch <name> to land on);
  • each commit from the branch's tip to the new one has one parent, the one before (<commit> is not one commit on the one before);
  • each is the tree a pull queued into that branch makes merged onto the commit before it, as the runner squashes it, no pull twice (<commit> is no pull queued into <branch> squashed), and there is at least one (<commit> is no batch on <branch>);
  • each branch deleted is the head of one of those pulls, in this repository (<name> is not the branch of a pull landed).

So a runner, or whoever holds its account, can land what a maintainer queued, before or without its checks passing, and nothing more.

Commit

dev.knotless.createCommit takes path changes and an optional base, the commit they were made against. A change writes a file's data, deletes it, or, with edits, replaces text in it: each old, which must be in the file exactly once, by its new, in the file as the commit's parent holds it. An agent changing a line of a long file sends that line, not the file. One with no branch or message is 400 a commit needs a branch and a message. Whether the caller may write is checked before the commit is queued, so a batch holds only commits that may land.

Commits to one branch queue up, per owner, repository and branch. Whichever request takes the branch's turn, a lock of its own, drains the queue, builds the commits as a chain on the tip, and lands them as one forced ref update against the generation it read. Its pusher is the batch's caller when every commit in it came from one, and no one otherwise. If the branch moved, the chain is rebuilt on the new tip, up to eight times with no pause between, and then every commit in it fails 409 <ref>: <reason>. A change to a path that also changed since its base is a conflict (409) for that commit alone; the rest land. Edits are made in the file as the chain leaves it, so one rebuilt on a new tip keeps what landed meanwhile; an edit whose old is no longer there once is a 409 for its commit alone, edit <n> of <path>: …. A build error fails the whole batch: 400 for bad input, else 500.

Each answer is {sha, parent, changed, batched}, where batched is how many commits went out in the write that carried it. knotless commit names the committer as the author when none is given: their handle and <their did>@knotless.invalid. A request that names no author is authored with the name knotless and the owner's <did>@knotless.invalid. The client reports attempts as 1 for a commit through a gateway, as it pushed once.

This way many agents can commit to one branch without each losing races: the gateway turns many writers into one.

Protected branches

A rules entry on the repository record protects a branch. The gateway refuses a push to it that:

  • deletes the branch: protected branch: it is not deleted;
  • moves it anywhere but forward: protected branch: it only moves forward;
  • when the rule is linear (its history is anything but merges, or absent), brings it a merge commit: protected branch: no merge commits; squash or rebase. That is any commit with two parents between the branch's tip and the new one (or in all of its history, for a branch that does not exist yet), even if the commit is already on other branches;
  • when the rule says queue, moves it at all, from anyone but the owner and the runner that lands the merge queue: protected branch: pulls land on it only through the merge queue;
  • when the rule asks for reviews, moves it, from anyone but the owner and the queue's runner, other than by merging an open pull into it: protected branch: changes land on it only through reviewed pulls; push to another branch and open a pull;
  • by merging a pull that may not merge yet, its reviews or its checks short, as knotless merge --check would say: protected branch: <why>.

A merge of a pull is recognized as knotless merge builds one: the new tip is the pull's head, a fast-forward, or a commit on the branch's tip (and on the head, for a merge commit) of the tree the two merge to. The gateway then reads the pull's reviews and checks as merge --check does, as the repository stands when the push arrives. In a repository that names runners, the same holds a collaborator's push to any branch, protected or not, whose commit's message ends with a pull's AT-URI, as every merge's does (merge gate: <why> on an unprotected branch); a commit that says no such thing passes, as the collaborator could push it anyway.

Commits through createCommit build on the tip with one parent, so they pass the rest; into a queue branch they are refused from a collaborator before they are batched, 403 <branch> takes pulls only through the merge queue: commit to another branch and open a pull, and into a branch that asks for reviews, 403 <branch> takes changes only through reviewed pulls: commit to another branch and open a pull. Pulls merge into a linear branch squashed.

A history value this version does not know reads as linear, so a rule that a newer client wrote is held at least as strictly. A client rewriting the record keeps the fields it does not know. A field of a rule's reviews that this version does not know blocks merges into the branch, here as in the client.

The rules hold whoever pushes through the gateway, the owner too, but for queue, which the owner passes. The owner writes their own records without any gateway, so the rules bind the owner only as far as their own client keeps to them.

Organizations

Any account can be an organization: its members push to each of its repositories, as collaborators named on each would. Who they are is looked up where a repository is read, never copied into its record. The organization's public dev.knotless.org record says it is one. A dev.knotless.orgMember record for each account it invited, keyed by that account's DID, gives their roles, when they were invited, when the invitation lapses (expiresAt, 7 days on), and once they are in, joinedAt. A member's dev.knotless.orgAcceptance record, keyed by the organization's DID, says they joined. One record per member, as Bluesky keeps one list item per account, means a change touches only that member's record, and each has a URI of its own to point at. Membership has two sides in as many accounts, so it is looked up in the appview's index of both (dev.knotless.getOrg), not member by member. The gateway the organization delegated to (named in its dev.knotless.org record's gateway) makes every change, as the organization, even one the organization asks for itself:

MethodCallerWhat it does
dev.knotless.putOrgMemberthe organization, or an admin who joinedinvites member, or changes their role
dev.knotless.removeOrgMemberthe organization, or an admin who joinedtakes member out, or withdraws an invitation
dev.knotless.joinOrgthe inviteetakes them in, once they wrote their acceptance record and while the invitation stands (7 days); the visibility they ask for says which record counts while they move it
dev.knotless.leaveOrgthe member, or the inviteetakes them out of its spaces, after they deleted their acceptance record, or withdraws their invitation
dev.knotless.createOrgRepoan admin who joinedmakes a repository as the organization, as knotless create does

Each takes a service token for its method. The gateway makes one change to an organization at a time, and writes what it changed into its index before it answers, so a member pushes the moment they are taken in. A change goes by the organization's own records, read again for it: a space's records take no compare and swap, so changes are kept from crossing by making them one at a time, and each record is written over only the version read, so a change made elsewhere meanwhile is refused rather than lost. Taking someone in stamps joinedAt on their record: a acceptance record counts only after that, so an invitation that lapsed cannot be joined by writing the record alone. Leaving ends the membership as the member deletes their record, before the gateway is asked: from when the index hears of it, they push to nothing and no reader inside the organization counts them. If the gateway cannot be reached, knotless org leave warns that they may still read its private repositories, until it takes them out.

The pages make the same changes, as the account logged in to them, through the same code as knotless org: the appview asks the organization's gateway with the session it keeps for that account (Web data). An account that delegated to the appview's gateway from the command line begins an organization from the pages through it.

Each orgMember record is in one place. Invitations and hidden members are kept in the organization's space (type dev.knotless.orgSpace, key self), which only it and its members read; an invitee may write their acceptance record there but read nothing else. Those who joined with a public acceptance record have theirs in its public records, where anyone may see them. Moving between the two writes the new record before taking out the old, and while both are there the public one counts. A PDS without spaces keeps everything in public, and has no hidden members.

A push from a member, hidden or not, passes: the gateway keeps an index of the organization's space, synced with the organization's own session. Everyone else reads the roster as they can: public members on a public repository, so a hidden member's merge, close or label there would count for no reader, and knotless refuses them upfront. In each private repository's space the organization keeps an orgMember record for each member who pushes to it, written as it adds them to that space, which whoever reads the repository reads, so its collaborators and runners count hidden members too. A reader inside the organization counts only those of them still members; anyone else, every one it names. A member who was shared a private repository on their own (shared on that record, or a collaborator) stays in its space when they leave.

Trust

The gateway holds an OAuth session for the owner, so it is trusted with writing knotless records to the owner's account: run your own, or use one you trust. Every reader still checks what it writes to a repository with index-pack. See Integrity and trust.