chadtmiller.com/knotless

Switch repository

Loading…

# Releases

A release is one version of a package that a repository publishes: a set
of files people download and install, such as the archives of a
command-line tool. In knotless a release is a record,
`dev.knotless.release`, in the repository owner's
[atproto](https://atproto.com) account, beside the repository's refs and
packs (see [storage.md](storage.md)). Its files are blobs, the binary data
a PDS keeps next to an account's records. There is no registry server: the
owner's PDS holds the release, and anyone can read it from there.

This page explains why releases are stored this way. The commands that
publish and read them, `knotless publish`, `knotless releases` and
`knotless download`, are in [../reference/cli.md](../reference/cli.md),
and the MCP tools that do the same, `publish`, `releases` and
`read_release`, in [../reference/mcp.md](../reference/mcp.md).

## The problem

Publishing a release means putting files where people can install them
from. Here that place is an account on a stock PDS, the server that holds
the account's records and blobs and signs them. Four things about it shape
the design.

The PDS caps each blob. The reference PDS takes blobs up to 5 MB by
default, and knotless's own CLI archives are 7.6 to 8.4 MB each.

Nothing can be installed on the PDS. knotless runs on any PDS as it ships,
so it cannot add a registry endpoint, a hook, or a check the server
enforces. Every rule has to hold through the records the stock server
already accepts.

A published version must never change. Someone who installs `1.2.0` today
and again next year must get the same bytes, and two publishes racing to
the same version must not leave a mix of both.

Readers fetch through caches. A blob may come from the PDS, a CDN or
knotless-serve. A reader needs to know a file is the one published without
trusting whoever handed it over.

A reader also wants to know what a release was built from: which commit,
and which CI run.

## One record per version

Each version of each package is its own record. The record is created once
and never updated, so the version cannot change after it is published.
Deleting the record unpublishes the version.

A record is addressed by its record key (rkey), and keys sort as strings.
A release's key is built so the keys of one repository, and of one package
within it, sit together:

```
<repo key>.<package hash>.<version hash>
```

The repo key is the repository's own key, a TID (a time-ordered key). Each
hash is the same short hash refs use: the first 26 characters of the
lowercase, unpadded base32 SHA-256 of the name. Hashing keeps any package
name or version within the characters an rkey allows, at a fixed length.

Because of this layout, one `listRecords` call that starts at a prefix and
stops at the first key outside it reads a whole list. Starting at the repo
key reads every release of the repository. Starting at the repo key and
the package hash reads one package's versions. Refs and packs are read the
same way ([storage.md](storage.md#keys-and-one-repository-as-one-range)).

A record holds:

- `repo`: the repository's key.
- `package` and `version`: the names, in full.
- `files`: 1 to 64 files, each with its `name`, `size`, `sha256` and the
  blobs that hold it.
- `commit`: the commit it was built from, optional.
- `check`: the AT-URI of the check record of the run that built it,
  optional.
- `notes`: release notes in Markdown, optional.
- `createdAt`.

Package names follow npm's rules, so that the npm protocol can later be
served from the same records. Versions are Semantic Versioning 2.0.

Details:

- A package name is lowercase letters, digits, `-`, `.`, `_` and `~`, not
  starting with `.` or `_`, after an optional `@scope/`, at most 214
  bytes.
- A version is at most 256 bytes.
- A file name is letters, digits, `.`, `_`, `+` and `-`, not starting with
  `.`, at most 255 bytes, and distinct within its release. A file cannot
  be empty.
- `notes` is at most 20,000 bytes and 5,000 graphemes.
- The schema is `lexicons/dev/knotless/release.json`.

## Files larger than a blob

A file that does not fit in one blob is split into parts, each its own
blob, listed in order under the file's `parts`. A reader joins them back
together. The splitting is the code that splits large git packs on push,
so releases and packs meet the PDS's limit the same way.

Unlike a pack, a release file has no inline form: every file is blobs,
even a small one.

Details:

- Parts are `KNOTLESS_CHUNK_SIZE` bytes, 4,000,000 by default, capped at
  the `blobUploadLimit` the PDS gives in `describeServer`.
- Four parts upload at a time. When the PDS says a part is too large, the
  size halves and the upload tries again.
- Each blob is stored as `application/octet-stream`.
- Each part costs one blob upload, and bsky.social allows 1,000 a day. A
  release of four 8 MB archives costs eight.

## Publishing

Only the repository's owner publishes, logged in as that account. The
writes go to the owner's PDS, as a push's do.

A publish runs in this order:

1. Check everything locally: the package name, the version, and each
   file's name, size and uniqueness.
2. Open the repository, and refuse a private one.
3. Read the record at the release's key, and stop if it exists, before
   uploading anything.
4. Upload each file's parts.
5. Create the record with `applyWrites`.

The last step is what keeps a version fixed. `applyWrites` writes its
records together or not at all, and a create fails if the record already
exists. Two publishes racing to the same version both upload their blobs,
but only one creates the record; the other is told the version is
published already. The early read in step 3 only saves a publish from
uploading files it cannot use.

Inside a runner's step, `commit` defaults to the commit the run is
checking (`KNOTLESS_RUN_SHA`). A flag sets it otherwise, and another names
the check record.

## The latest version

The latest version of a package is the highest version with no pre-release
part. `1.1.0-rc.1` is never the latest, even when it is the highest.
`knotless download` takes the latest when no version is named.

npm keeps a `latest` tag as mutable state that each publish moves.
knotless has no such record. The latest is computed from the versions that
exist, so publishes have nothing shared to contend on, and no tag can
point at a version that is not there.

## Trusting a file

The record lives in the owner's account repository, which the PDS signs.
Each file's `size` and `sha256` are in that signed record. A reader
downloads the parts, joins them, and checks the length and the SHA-256
against the record. If either differs, the download fails with an error
that what was read is not the file its release names.

This is what makes any cache safe to read through. The blobs can come from
anywhere; the signed record says what they must be.

Anyone can read a public repository's releases, with no login:
`listRecords` for the records and `getBlob` for the parts.

## Installing from releases

knotless-serve hands out the client at `/install.sh` and `/releases/`.
Given a repository, `--releases knotless://<owner>/<name>`, it answers
them from that repository's releases of the `knotless` package:
`/releases/latest` is the latest version, `<version>/<file>` a file of
it, and `<version>/<file>.sha256` the hash its record holds. The script
is the same one that installs from a directory, so the URLs are the same.

Serve reads each file from the owner's PDS the first time it is asked
for, checks it against its record, and keeps it by its hash for good,
since a release's files never change. It lists the releases again once
a minute, so a new version is the latest within one.

The script checks each archive against the `.sha256` beside it, which
serve takes from the record. It trusts the server it came from, as
`curl | sh` always does. Serve does not: what it hands out is what the
owner's account signed.

## Images

A container image is a release too. An image is content-addressed
already: a manifest names its config and layers by their SHA-256, and an
OCI image layout keeps each blob under its digest, with an `index.json`
naming the manifest. `knotless publish --oci <dir>` publishes such a
layout as a release's files: the index, and each blob named
`sha256-<hex>`. The `sha256` each file has in the record is then the
blob's digest, so the record that makes a download safe to read through
any cache does the same for a pull. The `index.json` must name one
manifest (else `index.json names more or less than one manifest: publish
one image, of one or many platforms, at a time`), and the image is held
to a release's 64 files: the index and at most 63 blobs.

knotless-serve answers pulls in the OCI distribution protocol from those
records: `docker pull knotless.dev/<owner>/<repo>/<package>:<version>`,
or `knotless.dev/<owner>/<repo>:<version>` for the package named as the
repository.
A tag is a version, or `latest`, computed as for any release, so the
registry keeps no tags of its own either. Its manifest is the one the
release's `index.json` names, which may be an index of several
platforms.

An image can be pushed there too, as to any registry: `docker login` with
the owner's handle and an app password, then `docker push`. The password
gets a token from serve by logging in to the owner's PDS, and is kept
nowhere. Each blob is checked against its digest and uploaded to the
owner's PDS as it lands; the manifest pushed with a version as its tag
makes the release, with every blob it reaches and an `index.json` naming
it, as `knotless publish --oci` would, though with no `commit`, `check`
or `notes`. The record is the owner's, made
with their session, so serve holds nothing a rebuild from the PDS would
lose but the pushes still under way. A version is pushed once, and
`latest` never: it is computed.

knotless publishes its own image so. Once main's deploy is up, its
`image` step saves the image it runs, which `docker save` writes as an
OCI image layout, and publishes it as the release of the
`knotless-server` package at main's version, if that version has none
yet: `docker pull knotless.dev/chadtmiller.com/knotless/knotless-server`.
It is one image for every service a host runs, each started by its own
command, as Loki and Concourse ship theirs: one build to make and
publish, rather than an image a service, which would share all but their
entry point. deploy/README.md lists the commands. It is one platform,
linux/amd64, as the host builds it, and its layers are as docker keeps
them, uncompressed.

## On the pages

knotless-serve indexes release records as it indexes the rest, so a
repository's pages show what it publishes. Under its name, one line says
the release to show and links to it: the latest of the package named as
the repository, its own, marked Latest; else the newest that is its
package's latest; or, when every release is a pre-release, the newest of
all, marked Pre-release. Beside it, how many packages the
repository publishes and how many of them are images, linked to its
releases.

A Releases tab, with how many releases there are, opens the releases
page: every version of every package, newest first, each with its
notes, the commit it was built from and its files, each with its size
and SHA-256. An image shows the `docker pull` command for it rather than
its blobs.

Each file's name links to download it, from
`/<owner>/<repo>/releases/download/<package>/<version>/<file>`.
knotless-serve answers that as it answers `/releases/` for the
repository it installs from, and through the same code: it reads the
file's parts from the owner's PDS the first time it is asked for,
joins them, checks the whole against the record, and keeps it by its
hash. It hands out only what the index has, so a release taken down,
or a repository or account taken down, is not found, and it reads as
no one, so a private repository's is not found either. The file goes
out to be saved, never shown: `application/octet-stream`, `attachment`
under its own name, `nosniff` and a sandbox. Its hash is its `ETag`,
and it is `no-cache`, so a cache asks again each time, at the cost of
a `304`, and a takedown holds at once. With more than one package, a chip for each, with its latest
version, shows one package's releases alone. A record that is no release,
by its package's name or its version, is passed by. A private
repository's releases are refused, so its pages show none.

## What a release was built from

Two optional fields link a release to its source. `commit` names the
commit it was built from. `check` names the check record of the run that
built and published it, which a runner writes in its own account (see
[runner.md](runner.md)).

A check is evidence only from a runner the repository trusts. The schema
says a reader trusts it only when its runner is one of the accounts the
repository record lists in `runners`, and only once that check passed at
`commit`. No reader checks this yet.

On GitHub, Releases attach files to a tag, and only lately can a release
be made immutable. npm's provenance and GitHub's artifact attestations are
Sigstore signatures kept apart from the package. Here the record is
immutable from its creation, and the link to the commit and the check sits
in the same signed account repository as the files' hashes.

## Why not a registry on the PDS

pds.js serves npm and OCI from the PDS itself, as extensions
(`@pdsjs/npm`, `@pdsjs/oci`), with one mutable record per package that
holds every version and tag. Clients that speak npm or OCI then work
against the PDS directly.

knotless installs nothing on a PDS, so it cannot take that path. One
record per package would also bring back the contention knotless removed
elsewhere: every publish rewrites the same record. It is the lesson that
split git refs into one record each ([prior-art.md](prior-art.md)). With
one record per version, two publishes of different versions never touch
the same record.

npm-compatible package names leave room to serve the npm protocol later,
from knotless-serve rather than the PDS.

## How knotless versions itself

knotless's own CLI is published as releases of the `knotless` package
from its repository, on one track. Its version is set with knope and
changeset files rather than conventional commits.

A pull that changes what users of the CLI see adds a change file under
`.changeset/` with `knope document-change`, naming the bump. `knope
release` gathers them: it bumps the workspace version in `Cargo.toml` and
`Cargo.lock`, writes `CHANGELOG.md`, and deletes the change files. That
goes to main as a pull like any other.

A change file is easy to forget, so the `check` check asks for one: a
pull that changes the client's code, `crates/knotless/src` or
`crates/knotless-mcp/src`, fails without a new file under `.changeset/`,
unless its body says `Changeset: none` on a line of its own, for a change
no one using knotless sees.

Publishing follows from the version. The `release` workflow runs on each
push to main that changes `Cargo.toml`. If that version has no release
yet, it waits for the commit's seven checks to pass, builds the client for
macOS and Linux, each on aarch64 and x86_64, and publishes the four
archives as the owner, with the version's section of the changelog as the
notes and the commit it was built from. It then tags that commit
`v<version>`, so a git checkout finds each version by name. A version
already published makes it pass at once, so a push that changes `Cargo.toml` for another reason
publishes nothing. A version is never published from a commit whose checks
did not pass, since it could never be taken back.

All four targets build on the Linux runner, with cargo-zigbuild: the
client links no macOS framework, so zig's linker can link it for macOS.

A canary build, if there is one, would be a pre-release version, which is
never the latest.

## Keeping clients current

A client bug can harm repositories: a compaction that dropped packs
others still needed once left a repository no fresh clone could read,
and any client still running that version could do it again after the
fix was out. So the appview tells clients both what is newest and what
is too old.

Once a day the client asks its appview for `/releases/latest` and for
`dev.knotless.describeServer`, whose `client.minimum` is the oldest
version it asks for (`--client-minimum`, set only when a release is
found to do harm), with a `reason`. A newer release is said on stderr
after the command, as gh does, so output that scripts read stays the
same. A client older than the minimum refuses every command but those
that get it current (`upgrade`, `login`, `logout`, `help`), saying why.
The check fails open: an appview that does not answer, or answers
nothing, leaves what was last known, and runs nothing it would not
have. `KNOTLESS_NO_UPDATE_CHECK` turns it off, and
`KNOTLESS_IGNORE_MINIMUM` lets an old client run anyway, for someone who
knows the harm does not reach them.

`knotless upgrade` runs the appview's `/install.sh`, the same script
that installed it, into the directory the binary is in. A copy put
there by cargo or a package manager is that tool's to upgrade, so
`upgrade` says so instead of writing over it. The client never upgrades
itself unasked: a binary that changes under a script is worse than one
that says it is old.

## Not yet

- The npm protocol is not served from releases yet.
- A layer of hundreds of megabytes is many blob uploads, which a PDS
  counts against its rate limits.
- Serve hands out the app password's session for an hour to whoever has
  the token, and trusts the TLS in front of it to keep the password from
  anyone else.
- A private repository's releases, kept in a space, are refused.
- A runner does not yet give its steps its check's URI, so `check` is not
  filled in automatically in a run.
- No reader yet checks a release's `check` against the repository's
  `runners`.