chadtmiller.com/knotless
git that lives in your Atmosphere account
7
# Gateway
Only an account can write to its own repo, so a collaborator cannot push
to the owner's account directly. A gateway is a `knotless-serve` with
`--public-url` that the owner has delegated to. It checks who is asking,
then writes as the owner.
A gateway is identified as the `did:web` of its host (a port's `:` as
`%3A`), and serves its own DID document at `/.well-known/did.json`, with a
`#knotless_gateway` service, an `#atproto_labeler` one and a
`#knotless_appview` one, all at its public URL. `--public-url` is `http(s)://host[:port]`, with no path.
```mermaid
sequenceDiagram
participant C as Collaborator's helper
participant CP as Collaborator's PDS
participant G as Gateway
participant OP as Owner's PDS
C->>CP: getServiceAuth(aud = gateway, lxm = pushRefs)
CP-->>C: service token
C->>G: dev.knotless.pushRefs (header line + pack), token
G->>G: verify token against issuer's DID document
G->>OP: read repo record: is the issuer owner or collaborator?
G->>G: check branch rules
G->>OP: applyWrites as owner (same compare-and-swap as any push)
OP-->>G: ok
G-->>C: ok
```
## Delegation
The gateway is an OAuth client of its own, a confidential one. Its client
ID is `/oauth-client-metadata.json` on its public URL. It authenticates to
authorization servers with a P-256 key kept in its config directory
(`oauth-client-key`) and published in that metadata.
The metadata names the redirect `<public-url>/oauth/callback`,
`private_key_jwt` with `ES256`, DPoP-bound tokens, the grant types
`authorization_code` and `refresh_token`, `client_name` `knotless
gateway` and `application_type` `web`.
1. `knotless delegate` calls `dev.knotless.putDelegation` with a service
token.
2. The gateway pushes an authorization request for the owner at their
PDS, and answers `{url, state}`. The owner approves it in a browser. A
delegation still waiting after 15 minutes is dropped when the next one
starts.
3. The browser comes back to the gateway's `/oauth/callback`. The gateway
keeps the session, and refreshes it as that client. The page says `The
gateway now writes for you. You can close this tab.`, or `The gateway
was not delegated to: <error>`; a `state` nobody is waiting on is 400
`no delegation is waiting for this`.
4. Meanwhile `knotless delegate` calls `dev.knotless.getDelegation?state=`,
which waits up to 25 seconds. It answers `{done: false}` to ask again,
`{done: true}` once approved, 403 if the authorization failed, and 404
`no such delegation` for a state the caller did not start.
The grant is narrow: the four record types a push writes (`repo`, `ref`,
`claim` and `pack`), blobs, and those collections in the owner's own
spaces; and what an [organization](#organizations) needs: its `org`
record in public, its `orgMember` records in public and in its spaces,
its own space for members, and
making and changing the members of its spaces, never deleting one. Each
space type is named, a private repository's and an organization's
([Scopes](access.md#scopes)), so the grant reaches no other app's spaces.
`redirect` is left out, since a gateway never renames a repository. The
scope is `atproto repo:dev.knotless.repo repo:dev.knotless.ref
repo:dev.knotless.claim repo:dev.knotless.pack repo:dev.knotless.org
repo:dev.knotless.orgMember blob:*/*
space:dev.knotless.space?authority=self&collection=dev.knotless.repo&collection=dev.knotless.ref&collection=dev.knotless.claim&collection=dev.knotless.pack&collection=dev.knotless.orgMember&manage=create&manage=update
space:dev.knotless.orgSpace?authority=self&collection=dev.knotless.orgMember&collection=dev.knotless.orgAcceptance&manage=create&manage=update`.
A grant from before names every space type (`space:*`), which still
covers all of this; the pages ask for it again all the same, as for any
older grant.
A delegation made before organizations lacks some of it: the
organization's own space is refused rather than taken for a PDS without
spaces, and `knotless delegate` again grants the rest. The gateway answers a write
refused for want of scope `403`, saying the grant is older than what it
asks for now, and keeps the scope each delegation asked for, so the pages
say so before any write is refused ([serve.md](serve.md#logging-in)). A confidential client's session
lasts up to two years if it is used at least every three months. The owner
revokes it from their PDS's account settings.
The pages' Access tab delegates the same way, for the account logged in
to them, to the server's own gateway: `dev.knotless.delegateWrites` makes
the first call with the session the server keeps for the account, and
the page shows the approval page in a new tab while
`dev.knotless.awaitDelegation` makes the last, until it is done. The
gateway's grant is its own, apart from the pages' session.
Sharing a repository for writing records the gateway in its `repo` record.
A runner landing the merge queue of a repository that records none goes
through the appview's.
## Bots
An operator's bots, `knotless-run` and `knotless-triage`, log in through
the gateway too. An app password mints no delegation tokens, so a bot
holding one cannot read private repositories, and a loopback OAuth login
lasts only two weeks. Given `--bots <dir>`, the gateway is a second
confidential client, `/oauth/bot-client-metadata.json` on its public URL,
with the same key, the redirect `<public-url>/oauth/bot-callback`,
`client_name` `knotless bots`, and all of knotless's
[scope](access.md#scopes), private repositories included.
1. The operator opens `<public-url>/oauth/bot?account=<handle or did>` in
a browser. An account `KNOTLESS_BOT_DIDS` does not name is 403 `<did>
is not one of this server's bots`. The gateway pushes an authorization
request for it at its PDS and redirects there (303).
2. The operator approves, signed in as that account.
3. The browser comes back to `/oauth/bot-callback`. The gateway keeps
the session in `<dir>/sessions/`, and the client key beside it as
`<dir>/oauth-client-key`, and says `<did> is logged in for this
server's bots. You can close this tab.` A refused or failed login is
403 `the bot was not logged in: <error>`, and a `state` nobody is
waiting on 400 `no bot login is waiting for this`. A login still
waiting after 15 minutes is dropped when the next one starts.
The bots take `<dir>` as their config directory (`KNOTLESS_CONFIG_DIR`),
and act and refresh as that client from there: the session lasts up to
two years if used at least every three months. A stored session comes
before `KNOTLESS_PASSWORD`, which only logs in when there is none, or
when the session can no longer be refreshed. Without `--bots`, these
routes are 404 `this server keeps no bots' sessions`.
## Authentication
On a server without `--public-url`, every gateway route is 404 `this
server is not a gateway`.
Every call to the gateway carries a service token from
`com.atproto.server.getServiceAuth`, with the gateway as audience and the
method as `lxm`. A call without one is 401 `a service token is
required`, and one with a bad or expired token 401. The gateway checks the
token against the signing key in the issuer's DID document (Multikey, or
the legacy secp256k1 type that PLC still returns), read again when the
copy it read in the last five minutes turns the signature away, as once
the key rotated. A server that is its own gateway signs the tokens its
pages send it on a logged-in account's behalf itself, with a key kept in
its config directory, rather than ask the account's PDS: it holds that
account's session already. Such a token says so in its header, lasts a
minute, and passes only on servers sharing that key. Then it checks, in
order, that the owner has delegated to it, else 403 `<owner> has not
delegated writes to this gateway`; that the repository exists, else 404
`<url>: no such repository`; and that the issuer is the owner or a
collaborator, else 403 `<caller> is not a collaborator on <url>`. A runner the
repository names may push too, but only to land its merge queue
([Runners landing a queue](#runners-landing-a-queue)); it may not commit.
## Push
`dev.knotless.pushRefs` takes a JSON header line, naming the repository and
the ref updates, followed by the pack, as `application/x-knotless-push`, at
most 512 MiB. Each update's `src` is a 40-character lowercase hex hash, or
absent for a deletion, and its `dst` starts `refs/`; else the push is 400
`bad update of <dst>`.
Pushes to one repository take turns, under a lock in the process. The
gateway syncs its mirror, indexes the pack into it, and refuses a tip the
pack does not bring (400 `the pack does not hold <sha>`). It then pushes
as the owner, with the same
[compare-and-swap](storage.md#compare-and-swap-through-claims) as any push,
naming the caller, from its verified token, as the `pusher` of each claim and
ref it writes, unless the caller is the owner. It answers
`results`, a `{name, error?}` per ref: each update lands or is refused on
its own, and the client fails on the first one refused.
## Runners landing a queue
A runner the repository names lands its merge queue through the gateway
when no runner it names may push ([merge-queue.md](merge-queue.md#who-lands-a-queue)).
The gateway takes a push from it only if that push is such a landing,
else 403 `<caller> runs <url>'s checks, and lands its merge queue here,
nothing else: <why>`. It checks, against the queue as the records say:
- one branch moves, without force, and the push deletes branches at most
(`a runner lands a batch: one branch moved forward`);
- the branch exists (`there is no branch <name> to land on`);
- each commit from the branch's tip to the new one has one parent, the one
before (`<commit> is not one commit on the one before`);
- each is the tree a pull queued into that branch makes merged onto the
commit before it, as the runner squashes it, no pull twice
(`<commit> is no pull queued into <branch> squashed`), and there is at
least one (`<commit> is no batch on <branch>`);
- each branch deleted is the head of one of those pulls, in this
repository (`<name> is not the branch of a pull landed`).
So a runner, or whoever holds its account, can land what a maintainer
queued, before or without its checks passing, and nothing more.
## Commit
`dev.knotless.createCommit` takes path changes and an optional `base`, the
commit they were made against. A change writes a file's `data`, deletes
it, or, with `edits`, replaces text in it: each `old`, which must be in
the file exactly once, by its `new`, in the file as the commit's parent
holds it. An agent changing a line of a long file sends that line, not the
file. One with no branch or message is 400 `a
commit needs a branch and a message`. Whether the caller may write is
checked before the commit is queued, so a batch holds only commits that
may land.
Commits to one branch queue up, per owner, repository and branch.
Whichever request takes the branch's turn, a lock of its own, drains the
queue, builds the commits as a chain on the tip, and lands them as one
forced ref update against the generation it read. Its `pusher` is the
batch's caller when every commit in it came from one, and no one otherwise. If the branch moved, the
chain is rebuilt on the new tip, up to eight times with no pause between,
and then every commit in it fails 409 `<ref>: <reason>`. A change to a
path that also changed since its `base` is a conflict (409) for that
commit alone; the rest land. Edits are made in the file as the chain
leaves it, so one rebuilt on a new tip keeps what landed meanwhile; an
edit whose `old` is no longer there once is a 409 for its commit alone,
`edit <n> of <path>: …`. A build error fails the whole batch: 400 for
bad input, else 500.
Each answer is `{sha, parent, changed, batched}`, where `batched` is how
many commits went out in the write that carried it. `knotless commit`
names the committer as the author when none is given: their handle and
`<their did>@knotless.invalid`. A request that names no author is
authored with the name `knotless` and the owner's
`<did>@knotless.invalid`. The client
reports `attempts` as 1 for a commit through a gateway, as it pushed once.
This way many agents can commit to one branch without each losing races:
the gateway turns many writers into one.
## Protected branches
A `rules` entry on the repository record protects a branch. The gateway
refuses a push to it that:
- deletes the branch: `protected branch: it is not deleted`;
- moves it anywhere but forward: `protected branch: it only moves
forward`;
- when the rule is linear (its `history` is anything but `merges`, or
absent), brings it a merge commit:
`protected branch: no merge commits; squash or rebase`. That is any
commit with two parents between the branch's tip and the new one (or in
all of its history, for a branch that does not exist yet), even if the
commit is already on other branches;
- when the rule says `queue`, moves it at all, from anyone but the owner
and the runner that lands the [merge queue](merge-queue.md#queue-only-branches):
`protected branch: pulls land on it only through the merge queue`;
- when the rule asks for [reviews](collaboration.md#required-reviews),
moves it, from anyone but the owner and the queue's runner, other than
by merging an open pull into it: `protected branch: changes land on it
only through reviewed pulls; push to another branch and open a pull`;
- by merging a pull that may not merge yet, its reviews or its checks
short, as `knotless merge --check` would say: `protected branch: <why>`.
A merge of a pull is recognized as `knotless merge` builds one: the new
tip is the pull's head, a fast-forward, or a commit on the branch's tip
(and on the head, for a merge commit) of the tree the two merge to. The
gateway then reads the pull's reviews and checks as `merge --check` does,
as the repository stands when the push arrives. In a repository that names
runners, the same holds a collaborator's push to any branch, protected or
not, whose commit's message ends with a pull's AT-URI, as every merge's
does (`merge gate: <why>` on an unprotected branch); a commit that says no
such thing passes, as the collaborator could push it anyway.
Commits through `createCommit` build on the tip with one parent, so they
pass the rest; into a `queue` branch they are refused from a collaborator
before they are batched, 403 `<branch> takes pulls only through the merge
queue: commit to another branch and open a pull`, and into a branch that
asks for reviews, 403 `<branch> takes changes only through reviewed pulls:
commit to another branch and open a pull`. Pulls merge into a linear branch
[squashed](collaboration.md#merging).
A `history` value this version does not know reads as `linear`, so a rule
that a newer client wrote is held at least as strictly. A client rewriting
the record keeps the fields it does not know. A field of a rule's `reviews` that
this version does not know blocks merges into the branch, here as in the
client.
The rules hold whoever pushes through the gateway, the owner too, but for
`queue`, which the owner passes. The owner writes their own records
without any gateway, so the rules bind the owner only as far as their own
client keeps to them.
## Organizations
Any account can be an organization: its members push to each of its
repositories, as collaborators named on each would. Who they are is
looked up where a repository is read, never copied into its record. The
organization's public `dev.knotless.org` record says it is one. A
`dev.knotless.orgMember` record for each account it invited, keyed by
that account's DID, gives their `roles`, when they were invited, when the
invitation lapses (`expiresAt`, 7 days on), and once they are in,
`joinedAt`. A member's `dev.knotless.orgAcceptance` record, keyed by the
organization's DID, says they joined. One record per member, as Bluesky
keeps one list item per account, means a change touches only that
member's record, and each has a URI of its own to point at. Membership
has two sides in as many accounts, so it is looked up in the appview's
index of both (`dev.knotless.getOrg`), not member by member. The gateway
the organization delegated to (named in its `dev.knotless.org` record's
`gateway`) makes every change, as the organization, even one the
organization asks for itself:
| Method | Caller | What it does |
| --- | --- | --- |
| `dev.knotless.putOrgMember` | the organization, or an admin who joined | invites `member`, or changes their `role` |
| `dev.knotless.removeOrgMember` | the organization, or an admin who joined | takes `member` out, or withdraws an invitation |
| `dev.knotless.joinOrg` | the invitee | takes them in, once they wrote their acceptance record and while the invitation stands (7 days); the `visibility` they ask for says which record counts while they move it |
| `dev.knotless.leaveOrg` | the member, or the invitee | takes them out of its spaces, after they deleted their acceptance record, or withdraws their invitation |
| `dev.knotless.createOrgRepo` | an admin who joined | makes a repository as the organization, as `knotless create` does |
Each takes a service token for its method. The gateway makes one change
to an organization at a time, and writes what it changed into its index
before it answers, so a member pushes the moment they are taken in. A
change goes by the organization's own records, read again for it: a
space's records take no compare and swap, so changes are kept from
crossing by making them one at a time, and each record is written over
only the version read, so a change made elsewhere meanwhile is refused
rather than lost. Taking someone in stamps
`joinedAt` on their record: a acceptance record counts only after that, so
an invitation that lapsed cannot be joined by writing the record alone.
Leaving ends the membership as the member deletes their record, before
the gateway is asked: from when the index hears of it, they push to
nothing and no reader inside the organization counts them. If the gateway
cannot be reached, `knotless org leave` warns that they may still read
its private repositories, until it takes them out.
The pages make the same changes, as the account logged in to them, through
the same code as `knotless org`: the appview asks the organization's
gateway with the session it keeps for that account
([Web data](../reference/api.md#web-data)). An account that delegated to
the appview's gateway from the command line begins an organization from
the pages through it.
Each `orgMember` record is in one place. Invitations and hidden members
are kept in the organization's space (type `dev.knotless.orgSpace`, key
`self`), which only it and its members read; an invitee may write their
acceptance record there but read nothing else. Those who joined with a
public acceptance record have theirs in its public records, where anyone
may see them. Moving between the two writes the new record before taking
out the old, and while both are there the public one counts. A PDS
without spaces keeps everything in public, and has no hidden members.
A push from a member, hidden or not, passes: the gateway keeps an index
of the organization's space, synced with the organization's own session.
Everyone else reads the roster as they can: public members on a public
repository, so a hidden member's merge, close or label there would count
for no reader, and knotless refuses them upfront. In each private
repository's space the organization keeps an `orgMember` record for
each member who pushes to it, written as it adds them to that space, which whoever reads the repository reads, so its collaborators and
runners count hidden members too. A reader inside the organization counts
only those of them still members; anyone else, every one it names. A member who was
shared a private repository on their own (`shared` on that record, or a
collaborator) stays in its space when they leave.
## Trust
The gateway holds an OAuth session for the owner, so it is trusted with
writing knotless records to the owner's account: run your own, or use one
you trust. Every reader still checks what it writes to a repository with
`index-pack`. See [Integrity and trust](security.md).