~ knotless Search/
↑↓ move↵ open⇧↵ all results

johnandersen777.bsky.social/vuln-notif

0
0
main
191 lines · 10.1 KB Raw History

Canonical software identity

The Rust service stores one software.release.project per product, not one project per raw CPE observation. This document describes the identity model that src/identity/ implements. The migration runbook is identity-migration.md.

Formula

identity     = (class, canonical_vendor, canonical_family)
identity_key = "class:canonical_vendor:canonical_family"
slug         = truncate64(join(canonical_vendor, canonical_family, "-"))

The identity key never contains the raw CPE part or a version. Every observation of the same product, however NVD spells it, therefore maps to one key and one project slug. Canonicalization is deterministic: there is no fuzzy matching, stemming, or batch-order dependence.

class is os, app, hardware, or firmware. A class_overrides entry wins (curated OS families are os, router firmware is firmware); otherwise CPE part maps o → os, h → hardware, and everything else → app. The raw part is deliberately excluded from the identity key, so the a/o RHEL duplicates collapse.

Alias table

Curated aliases live in data/identity/aliases.json. The table is embedded with include_str!, validated at startup, and unit-tested. Editing this file and rebuilding is the only supported way to merge identities.

{
  "vendors": {"redhat": "red-hat", "red-hat": "red-hat", "fedoraproject": "fedora"},
  "products": {"enterprise-linux": "red-hat-enterprise-linux",
               "rhel-els": "red-hat-enterprise-linux"},
  "families": {"red-hat-enterprise-linux": "red-hat"},
  "versioned_families": ["red-hat-enterprise-linux", "ubuntu", "windows"],
  "class_overrides": {"red-hat-enterprise-linux": "os"}
}
SectionMeaning
vendorsNormalized vendor token → canonical vendor.
productsNormalized product alias → canonical family. The longest matching prefix wins.
familiesCanonical family → canonical vendor. Also makes a curated family a matchable product alias and marks it as the owner of its base slug.
versioned_familiesFamilies whose trailing stream/version text may be split into a channel and version.
class_overridesFamily → identity class.

Rules

  1. Vendor normalization. CPE input is unescaped; the token is lowercased and every run of non-alphanumerics becomes one -. The normalized token is then looked up in vendors (Red Hat, redhat, red_hat → red-hat).
  2. Product-family aliases. The normalized product is matched against products and families by longest alias prefix. enterprise_linux*, rhel_els, enterprise_linux_for_*, and codeready_linux_builder* all map to red-hat-enterprise-linux. OpenShift Container Platform, Keycloak, Single Sign-On, JBoss EAP, and Ansible Automation Platform have their own canonical families.
  3. Vendor-prefix stripping. The full product is tried against the alias tables first: Red Hat + Red Hat Enterprise Linux 9 matches the RHEL aliases and leaves family enterprise-linux. Trying the full product first keeps a longer curated alias such as centos-stream. If no alias matches, a redundant vendor prefix is still stripped so different vendor spellings converge. Canonical + Ubuntu Linux resolves to family ubuntu, not linux.
  4. Versioned families only. Trailing --separated numeric/stream text is split only for entries in versioned_families. Unlisted applications and hardware never split: apple-iphone-15 stays one project.
  5. Channel/version split. X and single-digit X.0 are NVD major sentinels: they produce channel X and no version. X.Y produces channel X and version X.Y (9.4, 8.10). Multi-digit X.0 values such as RHEL 10.0 and Keycloak 26.0 are real versions, not sentinels. A non-numeric version produces a version reference and no channel.
  6. Channels never contain dots. Ubuntu 20.04 becomes channel 20-04 with version 20.04; versions keep dots.
  7. Edition and architecture suffixes. Edition/stream tokens eus, aus, tus, sap, els, rt, nfv, av, crb, desktop, workstation, and hpc become channel suffixes. Architecture tokens aarch64, ppc64le, ppc64, and s390x append to the channel; the default x86_64 is omitted. Multiple suffixes are sorted and joined with - (9-eus-ppc64le).
  8. Collision suffix. A curated family owns its base slug. Otherwise the lexicographically smaller identity key owns the base slug and every other identity gets -<sha256(identity_key)[:6]>. The prefixed slug is truncated at a hyphen boundary before the suffix is appended. The result never depends on database order.
  9. 64-character limit. The project slug is truncated at the last hyphen boundary at or before 64 characters. Project name must equal the rkey.
  10. Placeholders are dropped. An absent token normalizes to empty; n/a, any, unknown, *, and - produce no identity. The legacy n-a-n-a project is never created again.

RHEL worked example

One product, red-hat-enterprise-linux, replaces every legacy RHEL split. The canonical identity is (os, red-hat, red-hat-enterprise-linux); its slug is red-hat-enterprise-linux.

ObservationLegacy raw slugCanonical projectChannelVersion
cpe:2.3:o:redhat:enterprise_linux:6.0redhat-enterprise-linuxred-hat-enterprise-linux6none
cpe:2.3:o:redhat:enterprise_linux:7.0redhat-enterprise-linuxred-hat-enterprise-linux7none
NVD Red Hat + Red Hat Enterprise Linux 8red-hat-red-hat-enterprise-linux-8red-hat-enterprise-linux8none
cpe:2.3:o:redhat:enterprise_linux:9.4redhat-enterprise-linuxred-hat-enterprise-linux99.4
cpe:2.3:o:redhat:enterprise_linux:10.0redhat-enterprise-linuxred-hat-enterprise-linux1010.0
cpe:2.3:o:redhat:enterprise_linux_eus:8redhat-enterprise-linux-eusred-hat-enterprise-linux8-eusnone
cpe:2.3:o:redhat:rhel_els:6redhat-rhel-elsred-hat-enterprise-linux6-elsnone
cpe:2.3:o:redhat:enterprise_linux:8.10redhat-enterprise-linuxred-hat-enterprise-linux88.10

The project therefore has channels 6, 7, 8, 9, 10, 8-eus, and 6-els, with versions such as 9.4, 8.10, and 10.0 mapped to them. An a/o part collision, a -<hash> suffix from the old slugifier, and the redhat-enterprise-linux/redhat-rhel-els spellings all converge here. Other edition and architecture tokens produce channels such as 9-aus, 9-sap, 9-rt, and 9-aarch64.

Raw aliases vs canonical projects

Raw part:vendor:product observations are kept locally only, in the software / software_identity / software_purls SQLite tables. They are not published as standalone AT records.

Each com.nahdig.cve.atprotoV1 software[] entry carries the raw fields and the canonical project ref:

{
  "project": {"uri": "at://.../software.release.project/red-hat-enterprise-linux", "cid": "bafy..."},
  "part": "o",
  "vendor": "Red Hat",
  "product": "Red Hat Enterprise Linux 8",
  "cpes": ["cpe:2.3:o:redhat:enterprise_linux:8.0:*:*:*:*:*:*:*"],
  "purls": ["pkg:rpm/redhat/enterprise-linux"]
}
  • software_identity(repo_did, raw_slug, canonical_slug, identity_key, source, confidence, updated_at) persists the raw → canonical map.
  • PURLs stay in local software_purls and in CVE software[].purls[]; software_purls.software_slug stores the canonical project slug.
  • com.nahdig.cve.atprotoV1 dedupes software refs by canonical project and merges the raw CPE/PURL sets from all observations.
  • No com.nahdig.software records are written.

| software.release.channel | {project}:{channel} | Mutable named pointer | project and version are same-DID; suffix must equal name | | software.release.version | {project}:{version} | Immutable | project and manifest are same-DID; suffix must equal version | | software.release.manifest | TID | Immutable; artifacts fixed at creation | project is same-DID | | software.release.status | TID-shaped digest | Append-only transition events | subject is same-DID | | com.nahdig.cve.atprotoV1 | CVE ID | Updated when refs change | nvd, software[].project/channels/versions/manifests are same-DID |

Record-key grammar is enforced by the vendored lexicons:

  • project: 1–64 lowercase ASCII alphanumerics and interior hyphens, no dots or colons, no leading/trailing hyphen.
  • channel: lowercase alphanumerics and interior hyphens, no dots; rkey has exactly one colon.
  • version: same grammar as a project except dots are allowed; rkey has exactly one colon.
  • Changing a channel means rewriting its version pointer. A version or manifest is never edited: publish a new immutable version, and mark the old one with a software.release.status record.

Migration flow

The one-time migration is an out-of-band admin binary; the service never runs it. The sequence is:

snapshot → plan → review diff → apply --resume → verify
  1. snapshot writes every affected record to per-collection JSONL.
  2. plan inventories the local cache (cves, software, software_purls, purl2cpe) plus the remote listRecords inventory and writes a reviewable map.
  3. Review every create_project, repoint_software, repoint_purl, repoint_cve, normalize_purl_slug, and delete_release_record entry. Project rkeys are permanent.
  4. apply executes in dependency order (projects, software, purl, CVE, purl-slug normalization, then deletes) with a persisted cursor. Re-running with --resume skips done entries.
  5. verify re-reads the repository, asserts canonical projects exist, legacy projects have no unmanaged backlinks, refs are deduped and canonical, and strongRef CIDs are current. It exits 1 on any error.

Exact commands, the write budget, rollback by snapshot, and failure recovery are in identity-migration.md. The local harness is scripts/identity-migrate-e2e.sh.