~ knotless Search/
↑↓ move↵ open⇧↵ all results

chadtmiller.com/knotless

git that lives in your Atmosphere account

knotless.dev
1 collaborator

knotless 0.17.0 Latest 2 packages, 1 image

1
10

knotless-serve answers Maven's repository layout for a repository's artifacts

Merged
Merged chadtmiller.com asked to merge into main from chadtmiller.com/knotless:serve/maven-registry

knotless-serve now runs a Maven repository for each repository. Gradle's maven-publish publishes an Android or Java library to https://knotless.dev/maven/<did>/<repo key>/ with a handle and an app password, and any build that names that URL can resolve it. HUD's hudkit is the first user. This pull is stacked on knotless-serve answers the Swift Package Registry for a repository's packages (Swift), so until that lands its diff includes that pull's commit. Two things need a look: the maven object in dev.knotless.release, and how a publish is held between requests (Staging).

Claude Opus 5.5 · high

What changes

knotless-serve runs a Maven repository for each repository at /maven/<did>/<repo key>/, in Maven's layout:

  • GET/HEAD of <group path>/<artifact>/maven-metadata.xml: built from the releases. It lists the versions that aren't yanked, the highest as latest, the latest without a pre-release as release, and lastUpdated. It is sent no-cache with an ETag.
  • GET/HEAD of each file, <group path>/<artifact>/<version>/<file>.
  • .sha1, .sha256 and .sha512 beside each file and beside the metadata. MD5 is answered 404.
  • PUT of each of those, as Gradle and Maven publish.

Errors are plain text. A write without credentials gets 401 with a Basic challenge, which Gradle needs before it sends its credentials.

A publish is many requests, and a release is made once. Each file PUT is uploaded to the owner's PDS as it lands and kept in memory (Staging in App). The checksum PUT that follows each file is checked against it; MD5 is not computed, so it goes unchecked. The artifact's maven-metadata.xml, which Gradle and Maven send last, makes the release of each version kept, with every file.

Credentials log in to the owner's PDS once per publish. The session is kept by the credentials' SHA-256 for ten minutes, so a publish of some twenty requests isn't twenty createSession calls.

A version is published once: a file PUT to a published version gets 409. Snapshots get 400.

A release gains an optional maven object: its group, its artifact, and each file's SHA-1 and SHA-512. The release's package is <group>.<artifact>.

docs/design/packages.md has a new Maven artifacts section. docs/reference/api.md lists the routes. There is a change file.

Why

HUD's Android half, hudkit, has to be a versioned Gradle dependency that Grain can use, instead of a copied vendor/hud. It follows the other registries: one repository's own registry, an address holding a DID and a key, reads served from the index, and a yank keeping a version out of what is offered (here, out of maven-metadata.xml) while its files stay.

  • Files are held in memory, not in records. Maven's publish is a file at a time, with nothing that says "begin". OCI's push has the same problem, and oci.rs stages its blobs in the same way. The cost is that a restart mid-publish loses the files. The metadata PUT then names a version neither published nor kept, and is refused with "publish it again", rather than answering 201 with nothing released. The blobs already uploaded are never referenced and the PDS drops them.
  • The metadata is built, not kept. maven-metadata.xml is mutable shared state; npm's packument and Cargo's index are built from releases for the same reason.
  • <group>.<artifact> as the package name keeps it within a release's name rules. A <group>:<artifact> name would have needed the lexicon's rule widened. The price is that com.example + lib and com + example.lib collide; whichever comes second is refused as published already.
  • SemVer versions only, as releases are. Maven takes any version, but Android libraries' 1.2.0 and 1.0.0-alpha01 are SemVer.

How it was checked

a_maven_repository in tests/web.rs publishes as Gradle does, against a served repository and the PDS:

  • for each of .aar, .pom, .module and -sources.jar: the file, then its four hashes;
  • then the metadata and its hashes, for two versions.

It then reads everything back: the metadata, its SHA-1, each file and its hashes, and HEAD. It also covers:

  • the 401 challenge;
  • a yank, which leaves the metadata and is still readable;
  • every refusal: 409 republish, wrong password, another account's credentials, snapshot, an uppercase group, a non-SemVer version, a file not of the artifact, an empty file, the hash of a file never uploaded, a wrong hash, metadata naming a version lost;
  • a repository that isn't there;
  • a record no publish would make (bad bytes give 502, no hashes give 404).

Unit tests cover the layout parser, the metadata XML, the staging bounds (files per version, versions kept, sessions kept), the hashes and the credentials. Run locally: those tests, clippy and fmt for knotless-serve and knotless, and harness/lexicons.mjs.

End to end, against a local knotless-serve over the harness PDS, with Gradle 9.7.1, AGP 9.3.1 and JDK 17:

  • an Android library place.hud:tinylib:0.1.0 (with maven-publish, singleVariant("release") { withSourcesJar() }, credentials of a handle and an app password) published with ./gradlew publish;
  • the release held .aar, .pom, -sources.jar and .module;
  • an Android app depending on place.hud:tinylib:0.1.0, with only the repository's URL added, built with ./gradlew assembleDebug --refresh-dependencies;
  • publishing 0.1.0 again got 409, 0.2.0-SNAPSHOT got 400, and a wrong app password got 403, each from Gradle.

Over plain http (isAllowInsecureProtocol).

Shortcomings

  • A publish in flight lives in one serve process's memory. A restart, or a second serve behind a balancer, loses it, and the publish has to be run again. Images already have this limit.
  • MD5 is neither kept nor checked.
  • Versions that aren't SemVer, and snapshots, are refused.
  • As with the other registries: public repositories only, no commit or check on a release, and the app password sits in the build's credentials.
  • The releases page doesn't show how to use a Maven artifact yet.

Review

The lexicon first, then crates/knotless-serve/src/web/maven.rs: Writing::file, Writing::publish and Staging.

Approved

This adds a per-repository Maven repository to knotless-serve, with an optional maven object on dev.knotless.release. File PUTs are uploaded to the PDS and staged in memory. The artifact's maven-metadata.xml PUT then makes the releases.

I read the lexicon, maven.rs, the route and App wiring, the record and release changes, the docs and the change file. I did not read tests/web.rs, swift.rs or docs/design/packages.md. swift.rs belongs to the stacked pull, and I judged the layout and publish logic from maven.rs itself. I found nothing that must change.

The lexicon and Record types agree. The auth path checks session.did == did, so a cached session can't be used on another account's repository. A republish gets 409, because versions() counts yanked releases as published.

Two notes, neither blocking:

  • Writing::publish takes every staged version of the artifact out of Staging (versions(&of, true)) before it makes any release. A failure partway through, such as a PDS error on the second version, loses the rest of the staged files, and the client has to re-upload them. Taking each version only after its release is made would avoid that.
  • publish also releases staged versions that the uploaded XML doesn't name. It only checks the other direction, that every named version is known. This is harmless for Gradle and Maven, which always send the version they just uploaded, but it differs from what docs/reference/api.md says ("each version … kept").
crates/knotless-serve/src/web/maven.rsline 574Outdated
return Err(bad(says));
}
let package = asked.package();
for (version, files) in app.maven.versions(&of, true) {
review.knotless.dev

versions(&of, true) removes every staged version up front. If Target::open or create fails for one, the versions after it are already gone from Staging, and the client must re-upload them. Remove each version only once its release is made.

Approved

I read what changed since my approval at 004f018. The Maven file limit drops from 256 MiB to 64 MiB. The docs and a new test for the oversize refusal agree with it. refused_publish becomes refused_as(status), a curried closure, in both maven.rs and swift.rs. It behaves the same and is used the same way at each call site. The rest is small swift.rs cleanups (Link, Digest and Content-Disposition headers set through extend) plus a few extra unit assertions. None of it breaks what I approved. My two earlier notes (taking staged versions out before making their releases, and releasing staged versions the XML doesn't name) were non-blocking, and this round doesn't touch them.

Its head ebc01370501b on main at 6a6130767443 conflicts with main in docs/design/packages.md, where both sides changed the same lines, so it is left as it is. Merge main into it, then queue it again.

Approved

The head is the same tree as the commit I approved (ebc0137), so the code hasn't changed. Only the title or body changed. I read them again, and they match what I approved. My two earlier non-blocking notes still stand: staged versions are taken out before their releases are made, and staged versions the XML doesn't name are released too. Approving.

Merged as 450f83b