~ knotless Search/
↑↓ move↵ open⇧↵ all results

johnandersen777.bsky.social/vuln-notif

0
0
main
191 lines · 10.1 KB Raw History
# Canonical software identity

The Rust service stores one `software.release.project` per product, not one
project per raw CPE observation. This document describes the identity model
that `src/identity/` implements. The migration runbook is
[`identity-migration.md`](identity-migration.md).

## Formula

```text
identity     = (class, canonical_vendor, canonical_family)
identity_key = "class:canonical_vendor:canonical_family"
slug         = truncate64(join(canonical_vendor, canonical_family, "-"))
```

The identity key never contains the raw CPE `part` or a version. Every
observation of the same product, however NVD spells it, therefore maps to one
key and one project slug. Canonicalization is deterministic: there is no fuzzy
matching, stemming, or batch-order dependence.

`class` is `os`, `app`, `hardware`, or `firmware`. A `class_overrides` entry
wins (curated OS families are `os`, router firmware is `firmware`); otherwise
CPE part maps `o` → `os`, `h` → `hardware`, and everything else → `app`. The
raw part is deliberately excluded from the identity key, so the `a`/`o` RHEL
duplicates collapse.

## Alias table

Curated aliases live in `data/identity/aliases.json`. The table is embedded
with `include_str!`, validated at startup, and unit-tested. Editing this file
and rebuilding is the only supported way to merge identities.

```json
{
  "vendors": {"redhat": "red-hat", "red-hat": "red-hat", "fedoraproject": "fedora"},
  "products": {"enterprise-linux": "red-hat-enterprise-linux",
               "rhel-els": "red-hat-enterprise-linux"},
  "families": {"red-hat-enterprise-linux": "red-hat"},
  "versioned_families": ["red-hat-enterprise-linux", "ubuntu", "windows"],
  "class_overrides": {"red-hat-enterprise-linux": "os"}
}
```

| Section | Meaning |
|---|---|
| `vendors` | Normalized vendor token → canonical vendor. |
| `products` | Normalized product alias → canonical family. The longest matching prefix wins. |
| `families` | Canonical family → canonical vendor. Also makes a curated family a matchable product alias and marks it as the owner of its base slug. |
| `versioned_families` | Families whose trailing stream/version text may be split into a channel and version. |
| `class_overrides` | Family → identity class. |

## Rules

1. **Vendor normalization.** CPE input is unescaped; the token is lowercased
   and every run of non-alphanumerics becomes one `-`. The normalized token is
   then looked up in `vendors` (`Red Hat`, `redhat`, `red_hat` → `red-hat`).
2. **Product-family aliases.** The normalized product is matched against
   `products` and `families` by longest alias prefix. `enterprise_linux*`,
   `rhel_els`, `enterprise_linux_for_*`, and `codeready_linux_builder*` all map
   to `red-hat-enterprise-linux`. OpenShift Container Platform, Keycloak,
   Single Sign-On, JBoss EAP, and Ansible Automation Platform have their own
   canonical families.
3. **Vendor-prefix stripping.** The full product is tried against the alias
   tables first: `Red Hat` + `Red Hat Enterprise Linux 9` matches the RHEL
   aliases and leaves family `enterprise-linux`. Trying the full product first
   keeps a longer curated alias such as `centos-stream`. If no alias matches, a
   redundant vendor prefix is still stripped so different vendor spellings
   converge. `Canonical` + `Ubuntu Linux` resolves to family `ubuntu`, not
   `linux`.
4. **Versioned families only.** Trailing `-`-separated numeric/stream text is
   split only for entries in `versioned_families`. Unlisted applications and
   hardware never split: `apple-iphone-15` stays one project.
5. **Channel/version split.** `X` and single-digit `X.0` are NVD major
   sentinels: they produce channel `X` and no version. `X.Y` produces channel
   `X` and version `X.Y` (`9.4`, `8.10`). Multi-digit `X.0` values such as RHEL
   `10.0` and Keycloak `26.0` are real versions, not sentinels. A non-numeric
   version produces a version reference and no channel.
6. **Channels never contain dots.** Ubuntu `20.04` becomes channel `20-04`
   with version `20.04`; versions keep dots.
7. **Edition and architecture suffixes.** Edition/stream tokens `eus`, `aus`,
   `tus`, `sap`, `els`, `rt`, `nfv`, `av`, `crb`, `desktop`, `workstation`, and
   `hpc` become channel suffixes. Architecture tokens `aarch64`, `ppc64le`,
   `ppc64`, and `s390x` append to the channel; the default `x86_64` is omitted.
   Multiple suffixes are sorted and joined with `-` (`9-eus-ppc64le`).
8. **Collision suffix.** A curated family owns its base slug. Otherwise the
   lexicographically smaller identity key owns the base slug and every other
   identity gets `-<sha256(identity_key)[:6]>`. The prefixed slug is truncated
   at a hyphen boundary before the suffix is appended. The result never
   depends on database order.
9. **64-character limit.** The project slug is truncated at the last hyphen
   boundary at or before 64 characters. Project `name` must equal the rkey.
10. **Placeholders are dropped.** An absent token normalizes to empty; `n/a`,
    `any`, `unknown`, `*`, and `-` produce no identity. The legacy `n-a-n-a`
    project is never created again.

## RHEL worked example

One product, `red-hat-enterprise-linux`, replaces every legacy RHEL split.
The canonical identity is `(os, red-hat, red-hat-enterprise-linux)`; its slug
is `red-hat-enterprise-linux`.

| Observation | Legacy raw slug | Canonical project | Channel | Version |
|---|---|---|---|---|
| `cpe:2.3:o:redhat:enterprise_linux:6.0` | `redhat-enterprise-linux` | `red-hat-enterprise-linux` | `6` | none |
| `cpe:2.3:o:redhat:enterprise_linux:7.0` | `redhat-enterprise-linux` | `red-hat-enterprise-linux` | `7` | none |
| NVD `Red Hat` + `Red Hat Enterprise Linux 8` | `red-hat-red-hat-enterprise-linux-8` | `red-hat-enterprise-linux` | `8` | none |
| `cpe:2.3:o:redhat:enterprise_linux:9.4` | `redhat-enterprise-linux` | `red-hat-enterprise-linux` | `9` | `9.4` |
| `cpe:2.3:o:redhat:enterprise_linux:10.0` | `redhat-enterprise-linux` | `red-hat-enterprise-linux` | `10` | `10.0` |
| `cpe:2.3:o:redhat:enterprise_linux_eus:8` | `redhat-enterprise-linux-eus` | `red-hat-enterprise-linux` | `8-eus` | none |
| `cpe:2.3:o:redhat:rhel_els:6` | `redhat-rhel-els` | `red-hat-enterprise-linux` | `6-els` | none |
| `cpe:2.3:o:redhat:enterprise_linux:8.10` | `redhat-enterprise-linux` | `red-hat-enterprise-linux` | `8` | `8.10` |

The project therefore has channels `6`, `7`, `8`, `9`, `10`, `8-eus`, and
`6-els`, with versions such as `9.4`, `8.10`, and `10.0` mapped to them. An
`a`/`o` part collision, a `-<hash>` suffix from the old slugifier, and the
`redhat-enterprise-linux`/`redhat-rhel-els` spellings all converge here. Other
edition and architecture tokens produce channels such as `9-aus`, `9-sap`,
`9-rt`, and `9-aarch64`.

## Raw aliases vs canonical projects

Raw `part:vendor:product` observations are kept **locally only**, in the
`software` / `software_identity` / `software_purls` SQLite tables. They are not
published as standalone AT records.

Each `com.nahdig.cve.atprotoV1` `software[]` entry carries the raw fields and
the canonical project ref:

```json
{
  "project": {"uri": "at://.../software.release.project/red-hat-enterprise-linux", "cid": "bafy..."},
  "part": "o",
  "vendor": "Red Hat",
  "product": "Red Hat Enterprise Linux 8",
  "cpes": ["cpe:2.3:o:redhat:enterprise_linux:8.0:*:*:*:*:*:*:*"],
  "purls": ["pkg:rpm/redhat/enterprise-linux"]
}
```

- `software_identity(repo_did, raw_slug, canonical_slug, identity_key, source,
  confidence, updated_at)` persists the raw → canonical map.
- PURLs stay in local `software_purls` and in CVE `software[].purls[]`;
  `software_purls.software_slug` stores the canonical project slug.
- `com.nahdig.cve.atprotoV1` dedupes software refs by canonical project and
  merges the raw CPE/PURL sets from all observations.
- No `com.nahdig.software` records are written.

| `software.release.channel` | `{project}:{channel}` | Mutable named pointer | `project` and `version` are same-DID; suffix must equal `name` |
| `software.release.version` | `{project}:{version}` | Immutable | `project` and `manifest` are same-DID; suffix must equal `version` |
| `software.release.manifest` | TID | Immutable; artifacts fixed at creation | `project` is same-DID |
| `software.release.status` | TID-shaped digest | Append-only transition events | `subject` is same-DID |
| `com.nahdig.cve.atprotoV1` | CVE ID | Updated when refs change | `nvd`, `software[].project/channels/versions/manifests` are same-DID |

Record-key grammar is enforced by the vendored lexicons:

- project: 1–64 lowercase ASCII alphanumerics and interior hyphens, no dots
  or colons, no leading/trailing hyphen.
- channel: lowercase alphanumerics and interior hyphens, no dots; rkey has
  exactly one colon.
- version: same grammar as a project except dots are allowed; rkey has exactly
  one colon.
- Changing a channel means rewriting its `version` pointer. A version or
  manifest is never edited: publish a new immutable version, and mark the old
  one with a `software.release.status` record.

## Migration flow

The one-time migration is an out-of-band admin binary; the service never runs
it. The sequence is:

```text
snapshot → plan → review diff → apply --resume → verify
```

1. `snapshot` writes every affected record to per-collection JSONL.
2. `plan` inventories the local cache (`cves`, `software`, `software_purls`,
   purl2cpe) plus the remote `listRecords` inventory and writes a reviewable
   map.
3. Review every `create_project`, `repoint_software`, `repoint_purl`,
   `repoint_cve`, `normalize_purl_slug`, and `delete_release_record` entry.
   Project rkeys are permanent.
4. `apply` executes in dependency order (projects, software, purl, CVE,
   purl-slug normalization, then deletes) with a persisted cursor. Re-running
   with `--resume` skips `done` entries.
5. `verify` re-reads the repository, asserts canonical projects exist, legacy
   projects have no unmanaged backlinks, refs are deduped and canonical, and
   strongRef CIDs are current. It exits `1` on any error.

Exact commands, the write budget, rollback by snapshot, and failure recovery
are in [`identity-migration.md`](identity-migration.md). The local harness is
`scripts/identity-migrate-e2e.sh`.