johnandersen777.bsky.social/atlibs-knotless

Switch repository

Loading…

# atlibs-knotless

An [atlibs](https://knotless.dev/johnandersen777.bsky.social/atlibs) wasm module
(knotless://johnandersen777.bsky.social/atlibs) for [knotless](https://knotless.dev) repositories:
git kept as `dev.knotless.*` records in your own atproto account. It runs as the signed-in user, held
to `records/knotless.json`'s scope, in a page or a server.

| action | input | does |
| --- | --- | --- |
| `create_repo` | `name`, `description?`, `defaultBranch?` (main) | writes the `dev.knotless.repo` record under its name, as `knotless repo create` does; refuses a name that is taken |
| `commit` | `name`, `branch`, `message`, `changes`, `parent?`, `author?`, `retries?` | a commit without a checkout, as `knotless repo commit` does |

`changes` is a list (or that list as JSON text) of `{path, content}`, `{path, patch}` (a unified diff of
that one file: `--- /dev/null` makes it, `+++ /dev/null` deletes it) or `{path, delete: true}`, each
optionally with `executable`. A new `branch` starts from `parent`, else the default branch; an empty
repository gets a root commit. `author` is `{name, email}` or `"Name <email>"`; without one, the DID at
`<did>@knotless.invalid`. It answers `{sha, parent, changed, branch, pack, url}`; `changed: false` when
the tree is as it was, and nothing is written.

## How a commit is made

knotless needs nothing but XRPC to the user's own PDS: no git server, no knot, no other host.

1. `getRecord` the repo record (its `key`), then the branch's `dev.knotless.ref`
   (`<key>.<base32 sha256 of refs/heads/<branch>>`), or the parent branch's for a new one.
2. Read the parent's commit, the trees down to each changed path and each patched file's blob from the
   repository's `dev.knotless.pack` records: the pack the ref names first, then `listRecords` pages from
   `<key>/`, newest first. Packs are parsed in the module, deltas included. A pack kept over blobs
   (`parts`) is read with `com.atproto.sync.getBlob`, its parts whole.
3. Build the blobs, trees and commit, and a pack of them (tips: the commit, bases: its parent). A pack
   past 192 KiB goes up first as 4 MB blobs (`com.atproto.repo.uploadBlob`,
   `application/x-git-packed-objects`), named in the record's `parts`, as knotless push does.
4. One `applyWrites`: create the pack, create the claim `<ref rkey>.<generation>`, update the ref to the
   next generation (a new branch: create its ref, no claim). The claim is the ref's compare-and-swap: a
   write that loses rereads the ref and builds again on the new tip, up to `retries` (5) times.

Session scope: `repo:dev.knotless.repo?action=create repo:dev.knotless.pack?action=create
repo:dev.knotless.claim?action=create repo:dev.knotless.ref?action=create&action=update
blob:application/x-git-packed-objects`.
`dev.knotless.ref` is in `anyRecords`: the module moves refs other clients made.

## Not yet

- **Big packs and the per-event cap.** A pack over blobs goes up from one event, and a blob-backed pack
  is read whole into one, so the run's `ioBytes` budget bounds both: past atlibs' 1 MiB default (a
  commit of a few hundred KiB) the host must raise it, at most to 16 MiB (`ATLIBS_BUDGET='{"ioBytes":16777216}'`
  for `atlibs run`; the page sets it). That holds a commit of about 11 MB, and a parent pack of about
  as much; a bigger compacted pack cannot be read yet (it needs reading a pack in pieces).
- **Private repositories** live in a PDS space (`com.atproto.space.*`), which this module does not write.
- `create_repo` names no runners (knotless's CLI asks the appview for its default ones) and does not
  replace a redirect a rename left under the name.
- Deleting a repository: use `knotless repo delete <url> --yes`.

## Layout

- `modules/knotless`: the module (Rust, against atlibs' `abi` SDK): `git.rs` (packs, trees),
  `patch.rs` (unified diffs), `lib.rs` (the actions)
- `web/knotless`: a page that signs in and runs both actions
- `records/knotless.json`: the session record

## Set up

atlibs is a sibling checkout: Cargo fetches the `abi` crate from git, and the tests and the page import
its JavaScript from `../atlibs`.

```bash
git clone https://knotless.dev/johnandersen777.bsky.social/atlibs ../atlibs && git -C ../atlibs checkout -q 88b2ba8
```

## Build and test

```bash
cargo wasm
cargo test -p knotless
node --test modules/*/test.mjs web/*/*.test.mjs
```

The module tests run it against an in-memory PDS holding packs real `git pack-objects` made, and
index each pack it writes with `git index-pack --strict` (git must be installed).

`live.sh` is the live round: as `LIVE_HANDLE` (an app password in `LIVE_PASSWORD`) it creates a
repository, makes three commits (puts, an executable, a patch, a delete, a new branch), clones it with
`git clone knotless://...`, checks it with `git fsck --strict`, and deletes it with `knotless repo
delete`. It needs `atlibs`, `knotless` and `jq`.

## Run

```bash
python3 -m http.server 8000 --bind 127.0.0.1 --directory web/knotless
```

Open http://127.0.0.1:8000/ (or `?module=bobvmbuilder.bsky.social/knotless` for the published module record). From
the CLI:

```bash
atlibs run modules/knotless create_repo '{"name":"demo"}' --as you.bsky.social
atlibs run modules/knotless commit '{"name":"demo","branch":"main","message":"first","changes":[{"path":"README.md","content":"# demo\n"}]}' --as you.bsky.social
git clone knotless://you.bsky.social/demo
```