# Canonical software identity The Rust service stores one `software.release.project` per product, not one project per raw CPE observation. This document describes the identity model that `src/identity/` implements. The migration runbook is [`identity-migration.md`](identity-migration.md). ## Formula ```text identity = (class, canonical_vendor, canonical_family) identity_key = "class:canonical_vendor:canonical_family" slug = truncate64(join(canonical_vendor, canonical_family, "-")) ``` The identity key never contains the raw CPE `part` or a version. Every observation of the same product, however NVD spells it, therefore maps to one key and one project slug. Canonicalization is deterministic: there is no fuzzy matching, stemming, or batch-order dependence. `class` is `os`, `app`, `hardware`, or `firmware`. A `class_overrides` entry wins (curated OS families are `os`, router firmware is `firmware`); otherwise CPE part maps `o` → `os`, `h` → `hardware`, and everything else → `app`. The raw part is deliberately excluded from the identity key, so the `a`/`o` RHEL duplicates collapse. ## Alias table Curated aliases live in `data/identity/aliases.json`. The table is embedded with `include_str!`, validated at startup, and unit-tested. Editing this file and rebuilding is the only supported way to merge identities. ```json { "vendors": {"redhat": "red-hat", "red-hat": "red-hat", "fedoraproject": "fedora"}, "products": {"enterprise-linux": "red-hat-enterprise-linux", "rhel-els": "red-hat-enterprise-linux"}, "families": {"red-hat-enterprise-linux": "red-hat"}, "versioned_families": ["red-hat-enterprise-linux", "ubuntu", "windows"], "class_overrides": {"red-hat-enterprise-linux": "os"} } ``` | Section | Meaning | |---|---| | `vendors` | Normalized vendor token → canonical vendor. | | `products` | Normalized product alias → canonical family. The longest matching prefix wins. | | `families` | Canonical family → canonical vendor. Also makes a curated family a matchable product alias and marks it as the owner of its base slug. | | `versioned_families` | Families whose trailing stream/version text may be split into a channel and version. | | `class_overrides` | Family → identity class. | ## Rules 1. **Vendor normalization.** CPE input is unescaped; the token is lowercased and every run of non-alphanumerics becomes one `-`. The normalized token is then looked up in `vendors` (`Red Hat`, `redhat`, `red_hat` → `red-hat`). 2. **Product-family aliases.** The normalized product is matched against `products` and `families` by longest alias prefix. `enterprise_linux*`, `rhel_els`, `enterprise_linux_for_*`, and `codeready_linux_builder*` all map to `red-hat-enterprise-linux`. OpenShift Container Platform, Keycloak, Single Sign-On, JBoss EAP, and Ansible Automation Platform have their own canonical families. 3. **Vendor-prefix stripping.** The full product is tried against the alias tables first: `Red Hat` + `Red Hat Enterprise Linux 9` matches the RHEL aliases and leaves family `enterprise-linux`. Trying the full product first keeps a longer curated alias such as `centos-stream`. If no alias matches, a redundant vendor prefix is still stripped so different vendor spellings converge. `Canonical` + `Ubuntu Linux` resolves to family `ubuntu`, not `linux`. 4. **Versioned families only.** Trailing `-`-separated numeric/stream text is split only for entries in `versioned_families`. Unlisted applications and hardware never split: `apple-iphone-15` stays one project. 5. **Channel/version split.** `X` and single-digit `X.0` are NVD major sentinels: they produce channel `X` and no version. `X.Y` produces channel `X` and version `X.Y` (`9.4`, `8.10`). Multi-digit `X.0` values such as RHEL `10.0` and Keycloak `26.0` are real versions, not sentinels. A non-numeric version produces a version reference and no channel. 6. **Channels never contain dots.** Ubuntu `20.04` becomes channel `20-04` with version `20.04`; versions keep dots. 7. **Edition and architecture suffixes.** Edition/stream tokens `eus`, `aus`, `tus`, `sap`, `els`, `rt`, `nfv`, `av`, `crb`, `desktop`, `workstation`, and `hpc` become channel suffixes. Architecture tokens `aarch64`, `ppc64le`, `ppc64`, and `s390x` append to the channel; the default `x86_64` is omitted. Multiple suffixes are sorted and joined with `-` (`9-eus-ppc64le`). 8. **Collision suffix.** A curated family owns its base slug. Otherwise the lexicographically smaller identity key owns the base slug and every other identity gets `-`. The prefixed slug is truncated at a hyphen boundary before the suffix is appended. The result never depends on database order. 9. **64-character limit.** The project slug is truncated at the last hyphen boundary at or before 64 characters. Project `name` must equal the rkey. 10. **Placeholders are dropped.** An absent token normalizes to empty; `n/a`, `any`, `unknown`, `*`, and `-` produce no identity. The legacy `n-a-n-a` project is never created again. ## RHEL worked example One product, `red-hat-enterprise-linux`, replaces every legacy RHEL split. The canonical identity is `(os, red-hat, red-hat-enterprise-linux)`; its slug is `red-hat-enterprise-linux`. | Observation | Legacy raw slug | Canonical project | Channel | Version | |---|---|---|---|---| | `cpe:2.3:o:redhat:enterprise_linux:6.0` | `redhat-enterprise-linux` | `red-hat-enterprise-linux` | `6` | none | | `cpe:2.3:o:redhat:enterprise_linux:7.0` | `redhat-enterprise-linux` | `red-hat-enterprise-linux` | `7` | none | | NVD `Red Hat` + `Red Hat Enterprise Linux 8` | `red-hat-red-hat-enterprise-linux-8` | `red-hat-enterprise-linux` | `8` | none | | `cpe:2.3:o:redhat:enterprise_linux:9.4` | `redhat-enterprise-linux` | `red-hat-enterprise-linux` | `9` | `9.4` | | `cpe:2.3:o:redhat:enterprise_linux:10.0` | `redhat-enterprise-linux` | `red-hat-enterprise-linux` | `10` | `10.0` | | `cpe:2.3:o:redhat:enterprise_linux_eus:8` | `redhat-enterprise-linux-eus` | `red-hat-enterprise-linux` | `8-eus` | none | | `cpe:2.3:o:redhat:rhel_els:6` | `redhat-rhel-els` | `red-hat-enterprise-linux` | `6-els` | none | | `cpe:2.3:o:redhat:enterprise_linux:8.10` | `redhat-enterprise-linux` | `red-hat-enterprise-linux` | `8` | `8.10` | The project therefore has channels `6`, `7`, `8`, `9`, `10`, `8-eus`, and `6-els`, with versions such as `9.4`, `8.10`, and `10.0` mapped to them. An `a`/`o` part collision, a `-` suffix from the old slugifier, and the `redhat-enterprise-linux`/`redhat-rhel-els` spellings all converge here. Other edition and architecture tokens produce channels such as `9-aus`, `9-sap`, `9-rt`, and `9-aarch64`. ## Raw aliases vs canonical projects Raw `part:vendor:product` observations are kept **locally only**, in the `software` / `software_identity` / `software_purls` SQLite tables. They are not published as standalone AT records. Each `com.nahdig.cve.atprotoV1` `software[]` entry carries the raw fields and the canonical project ref: ```json { "project": {"uri": "at://.../software.release.project/red-hat-enterprise-linux", "cid": "bafy..."}, "part": "o", "vendor": "Red Hat", "product": "Red Hat Enterprise Linux 8", "cpes": ["cpe:2.3:o:redhat:enterprise_linux:8.0:*:*:*:*:*:*:*"], "purls": ["pkg:rpm/redhat/enterprise-linux"] } ``` - `software_identity(repo_did, raw_slug, canonical_slug, identity_key, source, confidence, updated_at)` persists the raw → canonical map. - PURLs stay in local `software_purls` and in CVE `software[].purls[]`; `software_purls.software_slug` stores the canonical project slug. - `com.nahdig.cve.atprotoV1` dedupes software refs by canonical project and merges the raw CPE/PURL sets from all observations. - No `com.nahdig.software` records are written. | `software.release.channel` | `{project}:{channel}` | Mutable named pointer | `project` and `version` are same-DID; suffix must equal `name` | | `software.release.version` | `{project}:{version}` | Immutable | `project` and `manifest` are same-DID; suffix must equal `version` | | `software.release.manifest` | TID | Immutable; artifacts fixed at creation | `project` is same-DID | | `software.release.status` | TID-shaped digest | Append-only transition events | `subject` is same-DID | | `com.nahdig.cve.atprotoV1` | CVE ID | Updated when refs change | `nvd`, `software[].project/channels/versions/manifests` are same-DID | Record-key grammar is enforced by the vendored lexicons: - project: 1–64 lowercase ASCII alphanumerics and interior hyphens, no dots or colons, no leading/trailing hyphen. - channel: lowercase alphanumerics and interior hyphens, no dots; rkey has exactly one colon. - version: same grammar as a project except dots are allowed; rkey has exactly one colon. - Changing a channel means rewriting its `version` pointer. A version or manifest is never edited: publish a new immutable version, and mark the old one with a `software.release.status` record. ## Migration flow The one-time migration is an out-of-band admin binary; the service never runs it. The sequence is: ```text snapshot → plan → review diff → apply --resume → verify ``` 1. `snapshot` writes every affected record to per-collection JSONL. 2. `plan` inventories the local cache (`cves`, `software`, `software_purls`, purl2cpe) plus the remote `listRecords` inventory and writes a reviewable map. 3. Review every `create_project`, `repoint_software`, `repoint_purl`, `repoint_cve`, `normalize_purl_slug`, and `delete_release_record` entry. Project rkeys are permanent. 4. `apply` executes in dependency order (projects, software, purl, CVE, purl-slug normalization, then deletes) with a persisted cursor. Re-running with `--resume` skips `done` entries. 5. `verify` re-reads the repository, asserts canonical projects exist, legacy projects have no unmanaged backlinks, refs are deduped and canonical, and strongRef CIDs are current. It exits `1` on any error. Exact commands, the write budget, rollback by snapshot, and failure recovery are in [`identity-migration.md`](identity-migration.md). The local harness is `scripts/identity-migrate-e2e.sh`.