~ knotless Search/
↑↓ move↵ open⇧↵ all results

johnandersen777.bsky.social/vuln-notif

0
0
main
196 lines · 11.9 KB Raw History
# Release sources

Per-adapter reference for the release lane. The source contract and registry
live in `src/release/source.rs`; implementations live in
`src/release/sources/`. Module and data-flow detail is in
[`release-architecture.md`](release-architecture.md).

## Mode support

| Mode | Source path |
|---|---|
| `latest` | deps.dev default/latest for `npm`, `pypi`, `maven`, `nuget`, `gem`, `cargo`, `golang`; every registered type also has a full-history adapter |
| `affected` | OSV advisories matched by purl; version catalogs come from local release rows plus deps.dev when the purl type is one of the seven; artifacts only through the seven-ecosystem artifact fetchers |
| `history` | registered history adapters: `pypi`, `cargo`, `gem`, `github`, `gitlab`, `bitbucket`, `rpm`/`deb`/`apk` (distro), `composer`, `cpan`, `apache`, `gitee`, `gnu` |
| `extras` | every adapter in `SOURCE_REGISTRY`, including npm, Maven, NuGet, Go, Docker/OCI, and SourceForge |

`SOURCE_REGISTRY` has 21 purl types mapping to 18 adapter names:

| Purl type | Adapter | deps.dev system |
|---|---|---|
| `npm` | `npm` | `NPM` |
| `pypi` | `pypi` | `PYPI` |
| `maven` | `maven` | `MAVEN` |
| `nuget` | `nuget` | `NUGET` |
| `gem` | `gem` | `RUBYGEMS` |
| `cargo` | `cargo` | `CARGO` |
| `golang` | `golang` | `GO` |
| `rpm` | `distro` | — |
| `deb` | `distro` | — |
| `apk` | `distro` | — |
| `github` | `github` | — |
| `gitlab` | `gitlab` | — |
| `bitbucket` | `bitbucket` | — |
| `composer` | `composer` | — |
| `cpan` | `cpan` | — |
| `apache` | `apache` | — |
| `gitee` | `gitee` | — |
| `gnu` | `gnu` | — |
| `docker` | `docker` | — |
| `oci` | `docker` | — |
| `sourceforge` | `sourceforge` | — |

Every registered purl type now has a full-history/extras adapter. `latest`
still uses deps.dev for default selection; `history`/`extras` use the registry
adapters directly.

## Capability matrix: auth and versions

| Purl type(s) | Adapter | Auth / token | Version source |
|---|---|---|---|
| `npm` | deps.dev | none (public registry tarball fetch) | deps.dev `/v3/systems/NPM/packages/{name}`; default/deprecated ordering |
| `pypi` | PyPI | none | `https://pypi.org/pypi/{pep503-name}/json` release keys; latest uses deps.dev |
| `maven` | Maven Central `maven-metadata.xml` | `maven` | full versions; `.jar`/`-sources.jar`/`.pom` artifacts; `.sha1` never relabeled |
| `nuget` | NuGet registration | `nuget` | full versions; `.nupkg` via `packageContent`; unlisted/deprecated statuses |
| `gem` | RubyGems | none | `https://rubygems.org/api/v1/versions/{name}.json`; latest uses deps.dev |
| `cargo` | crates.io | none | `https://crates.io/api/v1/crates/{name}` versions array; latest uses deps.dev |
| `golang` | `proxy.golang.org` `@v/list` | `golang` | full versions with Go path escaping; pseudo-versions opt-in; `.zip`/`.mod`; retractions |
| `rpm`, `deb`, `apk` | distro | none for public lifecycle/UBI data; authenticated registries are typed `Unavailable` | curated RHEL table + Red Hat Product Life Cycle API + endoflife.date v1 |
| `github` | GitHub | optional `GITHUB_TOKEN` (`Authorization: Bearer`) | `api.github.com/repos/{owner}/{repo}/releases` then `/tags` fallback |
| `gitlab` | GitLab | optional `GITLAB_TOKEN` (`PRIVATE-TOKEN`) | `gitlab.com/api/v4/projects/{id}/releases` then `/repository/tags` |
| `bitbucket` | Bitbucket | none | `api.bitbucket.org/2.0/repositories/{workspace}/{repo}/refs/tags` |
| `composer` | Packagist | none | `repo.packagist.org/p2/{vendor}/{package}.json` |
| `cpan` | MetaCPAN | none | MetaCPAN `_search` release documents for the distribution |
| `apache` | Apache | none | `downloads.apache.org/{project}/` and `archive.apache.org/dist/{project}/` directory indexes |
| `gitee` | Gitee | optional `GITEE_TOKEN` (`Authorization: Bearer`) | `gitee.com/api/v5/repos/{owner}/{repo}/releases` |
| `gnu` | GNU FTP | none | `ftp.gnu.org/gnu/{project}/` autoindex |
| `docker`, `oci` | Docker Hub | anonymous pull token per operation from `auth.docker.io` | `registry-1.docker.io/v2/{repo}/tags/list` (RFC 8288 pagination) |
| `sourceforge` | SourceForge | none | project RSS feed `/projects/{project}/rss?path=/` plus `/projects/{project}/files/` listing |

Tokens are sent only in request headers, never in URLs, logs, cycle stats, or
events. `GITHUB_TOKEN`, `GITLAB_TOKEN`, and `GITEE_TOKEN` are the only token
names; every other adapter is anonymous.

## Capability matrix: artifacts, digests, statuses

| Purl type(s) | Artifact and digest path | Status signals |
|---|---|---|
| `npm` | `registry.npmjs.org` full version map; tarball URL; SHA-1/SHA-512 never relabeled; SHA-256 computed on download | `deprecated` (`kind: registry`) with the npm reason; dist-tags exposed as channels |
| `pypi` | release files; wheel preferred; declared `digests.sha256` verified against downloaded bytes | `yanked` when every file is yanked; reason from `yanked_reason` |
| `maven` | Maven Central `maven-metadata.xml` full versions; `.jar`/`-sources.jar`/`.pom`; SHA-256 computed | none (`NoStatuses`) |
| `nuget` | NuGet registration full versions/pages; `.nupkg` via `packageContent`; SHA-256 computed | `deprecated` for unlisted/deprecated entries |
| `gem` | `gem_uri`; declared `sha` and optional size, HEAD size fallback | `yanked` with `kind: registry` and `yanked_reason` |
| `cargo` | `static.crates.io` `.crate`; declared `checksum` and `crate_size` | `yanked` with the registry reason kind |
| `golang` | `proxy.golang.org` `@v/list` full versions; `.zip`/`.mod`; pseudo-versions opt-in; SHA-256 computed | `retracted` from per-version `.info` |
| `rpm`, `deb`, `apk` | RHEL/UBI only: public OCI index manifest digest/URL; other distro builds are typed `NoArtifacts` | `deprecated` (`kind: eol`) after lifecycle EOL; `archived` (`kind: eol`) for retired products |
| `github` | release assets; asset `digest` when present, otherwise download and SHA-256; `tarball_url`/`zipball_url` fallback | `deprecated` for draft/prerelease; `archived` for archived repositories |
| `gitlab` | release asset links; SHA-256 computed; tag archive fallback | `deprecated` for `upcoming_release` (`kind: draft`); `archived` for archived projects |
| `bitbucket` | repository downloads matched by file name; no declared digest, SHA-256 computed | none (`NoStatuses`) |
| `composer` | p2 `dist.url` zip; `dist.shasum` used as SHA-256 only when 64 hex | `deprecated` for `abandoned` packages |
| `cpan` | release tarball URL with declared SHA-256 | none (`NoStatuses`); backpan entries are marked deprecated in the catalog but emit no status record |
| `apache` | archive files from the project index; `.sha256` sidecars are disabled by default in this build, so SHA-256 is computed from the bytes | none (`NoStatuses`) |
| `gitee` | release assets; no declared digest, SHA-256 computed | `prerelease` for prerelease releases |
| `gnu` | archive files from the FTP index; signatures ignored; SHA-256 computed | none (`NoStatuses`) |
| `docker`, `oci` | immutable manifest URL; `Docker-Content-Digest` is authoritative, size from `Content-Length`/manifest entries | `deprecated` with `kind: mutable_tag` when a tag resolves to a new digest |
| `sourceforge` | direct download host `downloads.sourceforge.net`; size from listing; digest via downloader | none (`NoStatuses`) |

Digest semantics are uniform: a declared digest is verified against the
downloaded bytes; a missing digest is computed; the artifact CID is the raw
CIDv1 over the SHA-256. A digest mismatch or an over-cap download is a typed
artifact error and is never retried as a transient source failure.

## Rate limits and retry

| Source | Behavior |
|---|---|
| Shared registry HTTP | 4 attempts, 1s/2s/4s backoff, retry on 429 and 500/502/503/504; 60s timeout; redirects limited to 10 |
| `cargo` | every JSON request spaced at least 1s per source instance |
| `github` | surfaces `x-ratelimit-remaining`, `x-ratelimit-reset`, and `retry-after`; fails typed when exhausted |
| `gitlab` | follows response pagination links; uses `PRIVATE-TOKEN` |
| `docker`, `oci` | anonymous bearer token fetched per operation; 401/403 is typed `Unavailable` (private vs missing is indistinguishable) |
| `sourceforge` | HTML error pages and missing projects are typed `Unavailable`, never parsed as records |
| OSV | online `POST /v1/query` and `/v1/querybatch` (split at 1,000 queries); offline bulk directory reads every `*.json` |
| PDS | separate governor: 5,000 points/hour, 35,000/day, 3,000 requests/5 min; create/put 3, update 2, delete 1 |

There is no adaptive per-host bucket in this build. `source_host_state` exists
is written by `GovernedHttp` on every rate-limit block and loaded at startup.

## Unsupported reasons

`UnsupportedReason` is typed and logged; a project is never silently dropped:

| Reason | Meaning |
|---|---|
| `UnknownPurlType(String)` | purl type absent from `SOURCE_REGISTRY` |
| `NoVersions` | source has no version catalog for this project |
| `NoArtifacts` | source has versions but no fetchable artifact (for example a distro build not published as UBI) |
| `NoStatuses` | source has no typed lifecycle signal |
| `Unavailable(&'static str)` | source cannot answer, for example missing transport, authenticated Red Hat registry, SourceForge error page, or an unknown/unsupported registry type |

Extras records the rendered reason in
`release_project_state.unsupported_reason` and emits
`release_extras_skipped`. History records `unsupported` counters and keeps the
project row; latest marks `unsupported_no_purl` when the project has no
release-ecosystem purl.

## RHEL / UBI artifact mapping

The distro adapter treats every Red Hat Enterprise Linux variant as the
canonical project `red-hat-enterprise-linux` (slug and rkey). Versions and
channels come from:

1. the curated `RHEL_CURATED_CHANNELS` table (`6`, `6-els`, `7`, `7-els`,
   `8`, `8-eus`, `9`, `9-eus`, `10`);
2. the public Red Hat Product Life Cycle API
   (`access.redhat.com/product-life-cycles/api/v1/products?name=Red%20Hat%20Enterprise%20Linux`);
3. endoflife.date v1 (`endoflife.date/api/v1`) when the product is published
   there.

Point releases are only emitted when lifecycle data supplies them; they are
never invented. Channels replace dots with hyphens because channel rkeys
cannot contain dots; versions keep dots.

For RHEL/UBI, the artifact is the public OCI index at:

```text
https://registry.access.redhat.com/v2/ubi{major}/ubi/manifests/{stream}
```

- `{major}` is the leading numeric component of the version (`9` from `9.4`).
- `{stream}` is the version itself when it names a point release (`9.4`),
  otherwise the major (`9`).
- Architecture tokens (`aarch64`, `ppc64le`, `ppc64`, `s390x`) are stripped
  from the stream and select a manifest from the index; the default is
  `amd64`/`x86_64`.
- The response `Docker-Content-Digest` (or a selected index entry digest) is
  the SHA-256; the immutable manifest URL is the artifact URL; the file name is
  `ubi{major}-ubi-{stream}.oci.json`.
- A 401/403 means the registry requires authentication and produces
  `Unavailable("redhat registry requires authentication")`, so callers emit
  version labels only. A non-UBI distro build produces `NoArtifacts`. No
  subscription artifact is fabricated.

The canonical channel/version mapping is documented in
[`identity-model.md`](identity-model.md) and
[`identity-migration.md`](identity-migration.md).

## Source URL overrides

`--release-source-url` (or `RELEASE_SOURCE_URL`) is a local e2e/test seam for
adapters with configurable base URLs:

| Mode | Adapters |
|---|---|
| `history` | `composer`, `cpan`, `apache`, `gitee`, `gnu` |
| `extras` | `composer`, `cpan`, `apache`, `gitee`, `gnu`, `sourceforge` |

It does not affect deps.dev, registry artifact URLs, GitHub/GitLab/Bitbucket,
Docker Hub, distro lifecycle URLs, or OSV.

## Related

- [`release-architecture.md`](release-architecture.md) — module map, DB tables,
  event/cache contracts, retries.
- [`identity-model.md`](identity-model.md) — canonical project, channel, and
  version rules.
- `src/release/source.rs` — `SOURCE_REGISTRY`, capabilities, unsupported enum.