Canonical software identity
The Rust service stores one software.release.project per product, not one
project per raw CPE observation. This document describes the identity model
that src/identity/ implements. The migration runbook is
identity-migration.md.
Formula
identity = (class, canonical_vendor, canonical_family)
identity_key = "class:canonical_vendor:canonical_family"
slug = truncate64(join(canonical_vendor, canonical_family, "-"))
The identity key never contains the raw CPE part or a version. Every
observation of the same product, however NVD spells it, therefore maps to one
key and one project slug. Canonicalization is deterministic: there is no fuzzy
matching, stemming, or batch-order dependence.
class is os, app, hardware, or firmware. A class_overrides entry
wins (curated OS families are os, router firmware is firmware); otherwise
CPE part maps o → os, h → hardware, and everything else → app. The
raw part is deliberately excluded from the identity key, so the a/o RHEL
duplicates collapse.
Alias table
Curated aliases live in data/identity/aliases.json. The table is embedded
with include_str!, validated at startup, and unit-tested. Editing this file
and rebuilding is the only supported way to merge identities.
{
"vendors": {"redhat": "red-hat", "red-hat": "red-hat", "fedoraproject": "fedora"},
"products": {"enterprise-linux": "red-hat-enterprise-linux",
"rhel-els": "red-hat-enterprise-linux"},
"families": {"red-hat-enterprise-linux": "red-hat"},
"versioned_families": ["red-hat-enterprise-linux", "ubuntu", "windows"],
"class_overrides": {"red-hat-enterprise-linux": "os"}
}
| Section | Meaning |
|---|---|
vendors | Normalized vendor token → canonical vendor. |
products | Normalized product alias → canonical family. The longest matching prefix wins. |
families | Canonical family → canonical vendor. Also makes a curated family a matchable product alias and marks it as the owner of its base slug. |
versioned_families | Families whose trailing stream/version text may be split into a channel and version. |
class_overrides | Family → identity class. |
Rules
- Vendor normalization. CPE input is unescaped; the token is lowercased
and every run of non-alphanumerics becomes one
-. The normalized token is then looked up invendors(Red Hat,redhat,red_hat→red-hat). - Product-family aliases. The normalized product is matched against
productsandfamiliesby longest alias prefix.enterprise_linux*,rhel_els,enterprise_linux_for_*, andcodeready_linux_builder*all map tored-hat-enterprise-linux. OpenShift Container Platform, Keycloak, Single Sign-On, JBoss EAP, and Ansible Automation Platform have their own canonical families. - Vendor-prefix stripping. The full product is tried against the alias
tables first:
Red Hat+Red Hat Enterprise Linux 9matches the RHEL aliases and leaves familyenterprise-linux. Trying the full product first keeps a longer curated alias such ascentos-stream. If no alias matches, a redundant vendor prefix is still stripped so different vendor spellings converge.Canonical+Ubuntu Linuxresolves to familyubuntu, notlinux. - Versioned families only. Trailing
--separated numeric/stream text is split only for entries inversioned_families. Unlisted applications and hardware never split:apple-iphone-15stays one project. - Channel/version split.
Xand single-digitX.0are NVD major sentinels: they produce channelXand no version.X.Yproduces channelXand versionX.Y(9.4,8.10). Multi-digitX.0values such as RHEL10.0and Keycloak26.0are real versions, not sentinels. A non-numeric version produces a version reference and no channel. - Channels never contain dots. Ubuntu
20.04becomes channel20-04with version20.04; versions keep dots. - Edition and architecture suffixes. Edition/stream tokens
eus,aus,tus,sap,els,rt,nfv,av,crb,desktop,workstation, andhpcbecome channel suffixes. Architecture tokensaarch64,ppc64le,ppc64, ands390xappend to the channel; the defaultx86_64is omitted. Multiple suffixes are sorted and joined with-(9-eus-ppc64le). - Collision suffix. A curated family owns its base slug. Otherwise the
lexicographically smaller identity key owns the base slug and every other
identity gets
-<sha256(identity_key)[:6]>. The prefixed slug is truncated at a hyphen boundary before the suffix is appended. The result never depends on database order. - 64-character limit. The project slug is truncated at the last hyphen
boundary at or before 64 characters. Project
namemust equal the rkey. - Placeholders are dropped. An absent token normalizes to empty;
n/a,any,unknown,*, and-produce no identity. The legacyn-a-n-aproject is never created again.
RHEL worked example
One product, red-hat-enterprise-linux, replaces every legacy RHEL split.
The canonical identity is (os, red-hat, red-hat-enterprise-linux); its slug
is red-hat-enterprise-linux.
| Observation | Legacy raw slug | Canonical project | Channel | Version |
|---|---|---|---|---|
cpe:2.3:o:redhat:enterprise_linux:6.0 | redhat-enterprise-linux | red-hat-enterprise-linux | 6 | none |
cpe:2.3:o:redhat:enterprise_linux:7.0 | redhat-enterprise-linux | red-hat-enterprise-linux | 7 | none |
NVD Red Hat + Red Hat Enterprise Linux 8 | red-hat-red-hat-enterprise-linux-8 | red-hat-enterprise-linux | 8 | none |
cpe:2.3:o:redhat:enterprise_linux:9.4 | redhat-enterprise-linux | red-hat-enterprise-linux | 9 | 9.4 |
cpe:2.3:o:redhat:enterprise_linux:10.0 | redhat-enterprise-linux | red-hat-enterprise-linux | 10 | 10.0 |
cpe:2.3:o:redhat:enterprise_linux_eus:8 | redhat-enterprise-linux-eus | red-hat-enterprise-linux | 8-eus | none |
cpe:2.3:o:redhat:rhel_els:6 | redhat-rhel-els | red-hat-enterprise-linux | 6-els | none |
cpe:2.3:o:redhat:enterprise_linux:8.10 | redhat-enterprise-linux | red-hat-enterprise-linux | 8 | 8.10 |
The project therefore has channels 6, 7, 8, 9, 10, 8-eus, and
6-els, with versions such as 9.4, 8.10, and 10.0 mapped to them. An
a/o part collision, a -<hash> suffix from the old slugifier, and the
redhat-enterprise-linux/redhat-rhel-els spellings all converge here. Other
edition and architecture tokens produce channels such as 9-aus, 9-sap,
9-rt, and 9-aarch64.
Raw aliases vs canonical projects
Raw part:vendor:product observations are kept locally only, in the
software / software_identity / software_purls SQLite tables. They are not
published as standalone AT records.
Each com.nahdig.cve.atprotoV1 software[] entry carries the raw fields and
the canonical project ref:
{
"project": {"uri": "at://.../software.release.project/red-hat-enterprise-linux", "cid": "bafy..."},
"part": "o",
"vendor": "Red Hat",
"product": "Red Hat Enterprise Linux 8",
"cpes": ["cpe:2.3:o:redhat:enterprise_linux:8.0:*:*:*:*:*:*:*"],
"purls": ["pkg:rpm/redhat/enterprise-linux"]
}
software_identity(repo_did, raw_slug, canonical_slug, identity_key, source, confidence, updated_at)persists the raw → canonical map.- PURLs stay in local
software_purlsand in CVEsoftware[].purls[];software_purls.software_slugstores the canonical project slug. com.nahdig.cve.atprotoV1dedupes software refs by canonical project and merges the raw CPE/PURL sets from all observations.- No
com.nahdig.softwarerecords are written.
| software.release.channel | {project}:{channel} | Mutable named pointer | project and version are same-DID; suffix must equal name |
| software.release.version | {project}:{version} | Immutable | project and manifest are same-DID; suffix must equal version |
| software.release.manifest | TID | Immutable; artifacts fixed at creation | project is same-DID |
| software.release.status | TID-shaped digest | Append-only transition events | subject is same-DID |
| com.nahdig.cve.atprotoV1 | CVE ID | Updated when refs change | nvd, software[].project/channels/versions/manifests are same-DID |
Record-key grammar is enforced by the vendored lexicons:
- project: 1–64 lowercase ASCII alphanumerics and interior hyphens, no dots or colons, no leading/trailing hyphen.
- channel: lowercase alphanumerics and interior hyphens, no dots; rkey has exactly one colon.
- version: same grammar as a project except dots are allowed; rkey has exactly one colon.
- Changing a channel means rewriting its
versionpointer. A version or manifest is never edited: publish a new immutable version, and mark the old one with asoftware.release.statusrecord.
Migration flow
The one-time migration is an out-of-band admin binary; the service never runs it. The sequence is:
snapshot → plan → review diff → apply --resume → verify
snapshotwrites every affected record to per-collection JSONL.planinventories the local cache (cves,software,software_purls, purl2cpe) plus the remotelistRecordsinventory and writes a reviewable map.- Review every
create_project,repoint_software,repoint_purl,repoint_cve,normalize_purl_slug, anddelete_release_recordentry. Project rkeys are permanent. applyexecutes in dependency order (projects, software, purl, CVE, purl-slug normalization, then deletes) with a persisted cursor. Re-running with--resumeskipsdoneentries.verifyre-reads the repository, asserts canonical projects exist, legacy projects have no unmanaged backlinks, refs are deduped and canonical, and strongRef CIDs are current. It exits1on any error.
Exact commands, the write budget, rollback by snapshot, and failure recovery
are in identity-migration.md. The local harness is
scripts/identity-migrate-e2e.sh.