1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
# Canonical software identity
The Rust service stores one `software.release.project` per product, not one
project per raw CPE observation. This document describes the identity model
that `src/identity/` implements. The migration runbook is
[`identity-migration.md`](identity-migration.md).
## Formula
```text
identity = (class, canonical_vendor, canonical_family)
identity_key = "class:canonical_vendor:canonical_family"
slug = truncate64(join(canonical_vendor, canonical_family, "-"))
```
The identity key never contains the raw CPE `part` or a version. Every
observation of the same product, however NVD spells it, therefore maps to one
key and one project slug. Canonicalization is deterministic: there is no fuzzy
matching, stemming, or batch-order dependence.
`class` is `os`, `app`, `hardware`, or `firmware`. A `class_overrides` entry
wins (curated OS families are `os`, router firmware is `firmware`); otherwise
CPE part maps `o` → `os`, `h` → `hardware`, and everything else → `app`. The
raw part is deliberately excluded from the identity key, so the `a`/`o` RHEL
duplicates collapse.
## Alias table
Curated aliases live in `data/identity/aliases.json`. The table is embedded
with `include_str!`, validated at startup, and unit-tested. Editing this file
and rebuilding is the only supported way to merge identities.
```json
{
"vendors": {"redhat": "red-hat", "red-hat": "red-hat", "fedoraproject": "fedora"},
"products": {"enterprise-linux": "red-hat-enterprise-linux",
"rhel-els": "red-hat-enterprise-linux"},
"families": {"red-hat-enterprise-linux": "red-hat"},
"versioned_families": ["red-hat-enterprise-linux", "ubuntu", "windows"],
"class_overrides": {"red-hat-enterprise-linux": "os"}
}
```
| Section | Meaning |
|---|---|
| `vendors` | Normalized vendor token → canonical vendor. |
| `products` | Normalized product alias → canonical family. The longest matching prefix wins. |
| `families` | Canonical family → canonical vendor. Also makes a curated family a matchable product alias and marks it as the owner of its base slug. |
| `versioned_families` | Families whose trailing stream/version text may be split into a channel and version. |
| `class_overrides` | Family → identity class. |
## Rules
1. **Vendor normalization.** CPE input is unescaped; the token is lowercased
and every run of non-alphanumerics becomes one `-`. The normalized token is
then looked up in `vendors` (`Red Hat`, `redhat`, `red_hat` → `red-hat`).
2. **Product-family aliases.** The normalized product is matched against
`products` and `families` by longest alias prefix. `enterprise_linux*`,
`rhel_els`, `enterprise_linux_for_*`, and `codeready_linux_builder*` all map
to `red-hat-enterprise-linux`. OpenShift Container Platform, Keycloak,
Single Sign-On, JBoss EAP, and Ansible Automation Platform have their own
canonical families.
3. **Vendor-prefix stripping.** The full product is tried against the alias
tables first: `Red Hat` + `Red Hat Enterprise Linux 9` matches the RHEL
aliases and leaves family `enterprise-linux`. Trying the full product first
keeps a longer curated alias such as `centos-stream`. If no alias matches, a
redundant vendor prefix is still stripped so different vendor spellings
converge. `Canonical` + `Ubuntu Linux` resolves to family `ubuntu`, not
`linux`.
4. **Versioned families only.** Trailing `-`-separated numeric/stream text is
split only for entries in `versioned_families`. Unlisted applications and
hardware never split: `apple-iphone-15` stays one project.
5. **Channel/version split.** `X` and single-digit `X.0` are NVD major
sentinels: they produce channel `X` and no version. `X.Y` produces channel
`X` and version `X.Y` (`9.4`, `8.10`). Multi-digit `X.0` values such as RHEL
`10.0` and Keycloak `26.0` are real versions, not sentinels. A non-numeric
version produces a version reference and no channel.
6. **Channels never contain dots.** Ubuntu `20.04` becomes channel `20-04`
with version `20.04`; versions keep dots.
7. **Edition and architecture suffixes.** Edition/stream tokens `eus`, `aus`,
`tus`, `sap`, `els`, `rt`, `nfv`, `av`, `crb`, `desktop`, `workstation`, and
`hpc` become channel suffixes. Architecture tokens `aarch64`, `ppc64le`,
`ppc64`, and `s390x` append to the channel; the default `x86_64` is omitted.
Multiple suffixes are sorted and joined with `-` (`9-eus-ppc64le`).
8. **Collision suffix.** A curated family owns its base slug. Otherwise the
lexicographically smaller identity key owns the base slug and every other
identity gets `-<sha256(identity_key)[:6]>`. The prefixed slug is truncated
at a hyphen boundary before the suffix is appended. The result never
depends on database order.
9. **64-character limit.** The project slug is truncated at the last hyphen
boundary at or before 64 characters. Project `name` must equal the rkey.
10. **Placeholders are dropped.** An absent token normalizes to empty; `n/a`,
`any`, `unknown`, `*`, and `-` produce no identity. The legacy `n-a-n-a`
project is never created again.
## RHEL worked example
One product, `red-hat-enterprise-linux`, replaces every legacy RHEL split.
The canonical identity is `(os, red-hat, red-hat-enterprise-linux)`; its slug
is `red-hat-enterprise-linux`.
| Observation | Legacy raw slug | Canonical project | Channel | Version |
|---|---|---|---|---|
| `cpe:2.3:o:redhat:enterprise_linux:6.0` | `redhat-enterprise-linux` | `red-hat-enterprise-linux` | `6` | none |
| `cpe:2.3:o:redhat:enterprise_linux:7.0` | `redhat-enterprise-linux` | `red-hat-enterprise-linux` | `7` | none |
| NVD `Red Hat` + `Red Hat Enterprise Linux 8` | `red-hat-red-hat-enterprise-linux-8` | `red-hat-enterprise-linux` | `8` | none |
| `cpe:2.3:o:redhat:enterprise_linux:9.4` | `redhat-enterprise-linux` | `red-hat-enterprise-linux` | `9` | `9.4` |
| `cpe:2.3:o:redhat:enterprise_linux:10.0` | `redhat-enterprise-linux` | `red-hat-enterprise-linux` | `10` | `10.0` |
| `cpe:2.3:o:redhat:enterprise_linux_eus:8` | `redhat-enterprise-linux-eus` | `red-hat-enterprise-linux` | `8-eus` | none |
| `cpe:2.3:o:redhat:rhel_els:6` | `redhat-rhel-els` | `red-hat-enterprise-linux` | `6-els` | none |
| `cpe:2.3:o:redhat:enterprise_linux:8.10` | `redhat-enterprise-linux` | `red-hat-enterprise-linux` | `8` | `8.10` |
The project therefore has channels `6`, `7`, `8`, `9`, `10`, `8-eus`, and
`6-els`, with versions such as `9.4`, `8.10`, and `10.0` mapped to them. An
`a`/`o` part collision, a `-<hash>` suffix from the old slugifier, and the
`redhat-enterprise-linux`/`redhat-rhel-els` spellings all converge here. Other
edition and architecture tokens produce channels such as `9-aus`, `9-sap`,
`9-rt`, and `9-aarch64`.
## Raw aliases vs canonical projects
Raw `part:vendor:product` observations are kept **locally only**, in the
`software` / `software_identity` / `software_purls` SQLite tables. They are not
published as standalone AT records.
Each `com.nahdig.cve.atprotoV1` `software[]` entry carries the raw fields and
the canonical project ref:
```json
{
"project": {"uri": "at://.../software.release.project/red-hat-enterprise-linux", "cid": "bafy..."},
"part": "o",
"vendor": "Red Hat",
"product": "Red Hat Enterprise Linux 8",
"cpes": ["cpe:2.3:o:redhat:enterprise_linux:8.0:*:*:*:*:*:*:*"],
"purls": ["pkg:rpm/redhat/enterprise-linux"]
}
```
- `software_identity(repo_did, raw_slug, canonical_slug, identity_key, source,
confidence, updated_at)` persists the raw → canonical map.
- PURLs stay in local `software_purls` and in CVE `software[].purls[]`;
`software_purls.software_slug` stores the canonical project slug.
- `com.nahdig.cve.atprotoV1` dedupes software refs by canonical project and
merges the raw CPE/PURL sets from all observations.
- No `com.nahdig.software` records are written.
| `software.release.channel` | `{project}:{channel}` | Mutable named pointer | `project` and `version` are same-DID; suffix must equal `name` |
| `software.release.version` | `{project}:{version}` | Immutable | `project` and `manifest` are same-DID; suffix must equal `version` |
| `software.release.manifest` | TID | Immutable; artifacts fixed at creation | `project` is same-DID |
| `software.release.status` | TID-shaped digest | Append-only transition events | `subject` is same-DID |
| `com.nahdig.cve.atprotoV1` | CVE ID | Updated when refs change | `nvd`, `software[].project/channels/versions/manifests` are same-DID |
Record-key grammar is enforced by the vendored lexicons:
- project: 1–64 lowercase ASCII alphanumerics and interior hyphens, no dots
or colons, no leading/trailing hyphen.
- channel: lowercase alphanumerics and interior hyphens, no dots; rkey has
exactly one colon.
- version: same grammar as a project except dots are allowed; rkey has exactly
one colon.
- Changing a channel means rewriting its `version` pointer. A version or
manifest is never edited: publish a new immutable version, and mark the old
one with a `software.release.status` record.
## Migration flow
The one-time migration is an out-of-band admin binary; the service never runs
it. The sequence is:
```text
snapshot → plan → review diff → apply --resume → verify
```
1. `snapshot` writes every affected record to per-collection JSONL.
2. `plan` inventories the local cache (`cves`, `software`, `software_purls`,
purl2cpe) plus the remote `listRecords` inventory and writes a reviewable
map.
3. Review every `create_project`, `repoint_software`, `repoint_purl`,
`repoint_cve`, `normalize_purl_slug`, and `delete_release_record` entry.
Project rkeys are permanent.
4. `apply` executes in dependency order (projects, software, purl, CVE,
purl-slug normalization, then deletes) with a persisted cursor. Re-running
with `--resume` skips `done` entries.
5. `verify` re-reads the repository, asserts canonical projects exist, legacy
projects have no unmanaged backlinks, refs are deduped and canonical, and
strongRef CIDs are current. It exits `1` on any error.
Exact commands, the write budget, rollback by snapshot, and failure recovery
are in [`identity-migration.md`](identity-migration.md). The local harness is
`scripts/identity-migrate-e2e.sh`.