# Identity migration runbook The Rust service stores one canonical `software.release.project` per product. Legacy caches and repositories instead hold one project per raw `{part}:{vendor}:{product}` observation, so Red Hat Enterprise Linux is split across `red-hat-red-hat-enterprise-linux-8`, `redhat-enterprise-linux`, `redhat-rhel-els`, and other variants. `identity-migrate` is the one-time, out-of-band admin tool that rewrites those references and deletes the legacy release records. There is no runtime migration phase, no startup `ALTER TABLE`, and no implicit write from `vuln-notif`: **the service never runs this migration itself**. A human runs `plan`, reviews the map, then runs `apply` and `verify`. Locations: - Binary: `target/release/identity-migrate` (`src/bin/identity-migrate.rs`). - Plan implementation: `src/migrate/`. - Curated aliases: `data/identity/aliases.json`. - Local e2e: `scripts/identity-migrate-e2e.sh`. - Fixtures: `tests/fixtures/identity-migration/`. ## Identity rules ```text identity = (class, canonical_vendor, canonical_family) slug = truncate64(join(canonical_vendor, canonical_family, '-')) ``` - `class` is `os`, `app`, `hardware`, or `firmware`. Curated OS families (RHEL, Fedora, CentOS, Debian, Ubuntu, SUSE, Oracle Linux, Amazon Linux, Windows, macOS) are always `os`. Otherwise CPE part maps `o` → `os`, `a` → `app`, `h` → `hardware`. The raw `part` is not part of the identity, so the `a`/`o` RHEL duplicates collapse. - `canonical_vendor` is CPE-unescaped, lowercased, non-alphanumerics collapsed to `-`, then looked up in the curated vendor aliases (`redhat`, `red_hat`, `Red Hat` → `red-hat`). - `canonical_family` strips a redundant vendor prefix, then applies the curated product aliases. `enterprise_linux*`, `rhel_els`, `enterprise_linux_for_*`, and `codeready_linux_builder*` all become `red-hat-enterprise-linux`. - Trailing stream/version text is split only for families listed in `versioned_families`. `X` and single-digit `X.0` are NVD major sentinels: they produce a channel and no version. `X.Y` produces channel `X` and version `X.Y` (`9.4`, `8.10`, multi-digit `10.0` preserved). Edition/arch tokens (`eus`, `els`, `aus`, `sap`, `rt`, `aarch64`, ...) become channel suffixes; default `x86_64` is omitted. - Channels never contain dots (`Ubuntu 20.04` → channel `20-04`); versions keep dots. Version and manifest records are immutable; only channels move. - Placeholder vendors/products (`n/a`, `any`, `unknown`, `*`, `-`) produce no identity. The old `n-a-n-a` project is never created again. - Slug collisions between different identities are resolved deterministically: a curated family owns the base slug, otherwise the lexicographically smaller identity key owns it and the other gets `-`. - Every write is same-DID with the repository. Cross-DID strongRefs are rejected by the tool and by `verify`. RHEL example: | Observation | Legacy raw slug | Canonical project | Channel | |---|---|---|---| | NVD `Red Hat` + `Red Hat Enterprise Linux 8` | `red-hat-red-hat-enterprise-linux-8` | `red-hat-enterprise-linux` | `8` | | CPE `cpe:2.3:o:redhat:enterprise_linux:9.4` | `redhat-enterprise-linux` | `red-hat-enterprise-linux` | `9` (version `9.4`) | | NVD `redhat` + `rhel_els` | `redhat-rhel-els` | `red-hat-enterprise-linux` | `els` | Legacy `com.nahdig.software` records are only encountered when migrating old snapshots; the current service does not publish them. Raw `vendor`, `product`, and `cpes` live in the local cache and in CVE `software[]` entries. ## Alias table `data/identity/aliases.json` is embedded with `include_str!`, validated at startup and by tests. Editing this file and rebuilding is the only supported way to merge identities; there is no fuzzy matching, stemming, or batch-order dependence. ```json { "vendors": {"redhat": "red-hat", "red-hat": "red-hat"}, "products": {"enterprise-linux": "red-hat-enterprise-linux", "rhel-els": "red-hat-enterprise-linux"}, "families": {"red-hat-enterprise-linux": "red-hat"}, "versioned_families": ["red-hat-enterprise-linux"], "class_overrides": {"red-hat-enterprise-linux": "os"} } ``` - `vendors`: normalized vendor → canonical vendor. - `products`: normalized product alias → canonical family. - `families`: canonical family → canonical vendor (used for slug ownership). - `versioned_families`: families that may split trailing version/edition text. - `class_overrides`: family → identity class. Every legacy → canonical pair appears in the plan `reason` and in the plan entries before apply, so the diff is reviewable per record. ## Migration phases `snapshot → plan → review diff → apply → verify` 1. **Snapshot.** Write every record in the migration collections to JSONL. Read-only on the cache and PDS. 2. **Plan.** Inventory the local cache (`cves`, `software`, `software_purls`, and purl2cpe when available) plus the remote `listRecords` inventory, then write a reviewable JSON map. `plan` performs no writes. 3. **Review diff.** Inspect every `delete_release_record`, `repoint_software`, `repoint_purl`, `repoint_cve`, `create_project`, and `normalize_purl_slug` entry. Do not continue if a pair is wrong; project rkeys are permanent. 4. **Apply.** Execute the map in dependency order with a persisted cursor. Already-`done` entries are skipped, so re-running is safe. 5. **Verify.** Re-read the repository, assert canonical projects exist, legacy projects have no unmanaged backlinks, CVE refs are deduped and canonical, and strongRef CIDs are current. `verify` exits `1` if any error is reported. ### Local ```bash scripts/local_plc.sh start scripts/local_atproto_pds.sh start eval "$(scripts/create_local_account.sh 2>/dev/null)" # exports ATP_PDS_URL/HANDLE/APP_PASSWORD target/release/identity-migrate snapshot \ --cache-db .state/local.db --snapshot-dir migration/snapshot-local target/release/identity-migrate plan \ --cache-db .state/local.db --purldb .state/purl2cpe.db \ --out migration/plan-local.json target/release/identity-migrate apply \ --cache-db .state/local.db --map migration/plan-local.json \ --batch 200 target/release/identity-migrate verify \ --cache-db .state/local.db --map migration/plan-local.json ``` `scripts/identity-migrate-e2e.sh` runs this whole flow against a temporary PLC and PDS, seeds the legacy RHEL split, and asserts the plan, apply report, zero-write second apply, and verify report. ### Production Stop the service first (freeze writers) and keep it stopped until `verify` passes. Never run `apply` concurrently with a service cycle. ```bash export ATP_HANDLE=did:plc:... # repository DID/handle, not a secret export ATP_APP_PASSWORD=... # privileged app password unset ATP_PDS_URL # resolve the PDS from PLC, or set it explicitly DB=.state/prod-week.db MAP=migration/identity-$(date +%Y%m%dT%H%M%S) cp "$DB" "$DB.pre-migration" # cache/rate-budget/cursor backup target/release/identity-migrate snapshot \ --cache-db "$DB" --snapshot-dir "$MAP.snapshot" target/release/identity-migrate plan \ --cache-db "$DB" --purldb .state/purl2cpe.db \ --out "$MAP.plan.json" # Review before applying. jq -r '.entries[] | .action' "$MAP.plan.json" | sort | uniq -c jq -r '.entries[] | select(.action == "create_project") | .canonical_slug' "$MAP.plan.json" jq -r '.entries[] | select(.action == "delete_release_record") | [.legacy_slug, .canonical_slug, .legacy_uri] | @tsv' "$MAP.plan.json" | sort -u jq -r '.entries[] | select(.action == "repoint_cve") | .legacy_slug' "$MAP.plan.json" | wc -l target/release/identity-migrate apply \ --cache-db "$DB" --map "$MAP.plan.json" \ --batch 200 --snapshot-dir "$MAP.snapshot" target/release/identity-migrate verify \ --cache-db "$DB" --map "$MAP.plan.json" ``` `apply` prints one `identity_apply` event with a report: `applied`, `skipped_done`, `created_projects`, `software_repoints`, `purl_repoints`, `cve_repoints`, `deleted`, and `normalized`. `verify` prints `identity_verify` with `canonical_projects`, `software_repoints`, `purl_repoints`, `cve_repoints`, `deletions`, `stale_refs`, and `errors`. If the cache holds no repository DID (or several), pass `--handle did:plc:...` explicitly to `normalize-purl-slugs`. ## Dependency order `apply` sorts and executes in this order; never delete a release record before its referrers move, because strongRef CIDs cascade. | Order | Action | Effect | |---:|---|---| | 1 | `create_project` | `software.release.project/` if missing | | 2 | `repoint_software` | `com.nahdig.software.project` strongRef | | 3 | `repoint_purl` | `com.nahdig.purl.software[].cid` | | 4 | `repoint_cve` | `com.nahdig.cve.atprotoV1.software[].project`; dedupe to one canonical ref; drop refs to records being deleted | | 5 | `normalize_purl_slug` | cache-only: `software_purls.software_slug` AT-URI → slug | | 6 | `delete_release_record` | `applyWrites` batches of legacy project/channel/version/manifest/status records | Local link rows are not rebuilt from `listRecords`. The service indexes links as it writes, repairs failed indexes from `link_repairs`, and re-indexes migrated URIs on its next pass; `verify` checks the local `links` table only for unmanaged backlinks to legacy projects. ## Rate budget PDS write costs: | Operation | Points | |---|---:| | create / put (new record) | 3 | | update (`putRecord` on existing record) | 2 | | delete | 1 | | `applyWrites` | each record in the batch at its own operation cost (deletes cost 1) | The migration's referrer moves are `putRecord` updates (2 points each); legacy record deletions are `applyWrites` deletes (1 point per record). Budget: 5,000 points/hour and 35,000 points/day per repository DID, with a 0.9 safety margin. The governor reserves points before each write and persists state in the cache DB's `rate_budget` table. | Work | Records | Approx. points | |---|---:|---:| | CVE refs | ~14.3k updates | ~28.6k | | software/purl refs + legacy deletes | ~100 | <1k | The CVE pass is close to one daily budget, so plan it as a day-long run. Use `--limit` to cap entries per invocation and `--batch` (default 200) for deletes. Stopping and resuming does not lose budget: it is persisted with the cursor. ## Rollback via snapshot `snapshot` writes one `.jsonl` file per migration collection; each line is `{"uri", "cid", "value"}` for a live record. Snapshots are full collection supersets, so they are also a cache-independent backup of the release/software/purl/CVE records. A snapshot taken before `apply`, plus the cache DB copy, is the rollback material. Restoring is a manual `putRecord` replay (the tool intentionally has no destructive rollback command): ```bash SNAP=migration/identity-YYYYmmddTHHMMSS.snapshot DID=$ATP_HANDLE ACCESS_JWT=$(curl -fsS -X POST "$ATP_PDS_URL/xrpc/com.atproto.server.createSession" \ -H 'Content-Type: application/json' \ -d "$(jq -nc --arg identifier "$ATP_HANDLE" --arg password "$ATP_APP_PASSWORD" \ '{identifier:$identifier,password:$password}')" | jq -r .accessJwt) for file in "$SNAP"/*.jsonl; do collection=$(basename "$file" .jsonl) while IFS= read -r line; do rkey=$(jq -r '.uri | split("/") | last' <<<"$line") value=$(jq -c '.value' <<<"$line") jq -nc --arg repo "$DID" --arg collection "$collection" \ --arg rkey "$rkey" --argjson record "$value" \ '{repo:$repo,collection:$collection,rkey:$rkey,record:$record}' | curl -fsS -X POST \ -H "Authorization: Bearer $ACCESS_JWT" \ -H 'Content-Type: application/json' \ -d @- "$ATP_PDS_URL/xrpc/com.atproto.repo.putRecord" done <"$file" done ``` Restore all collections, then re-run `verify` with the original plan: it should now report the pre-migration errors (legacy refs), which confirms the restore is complete. Restoring the cache DB copy rolls back the cursor and rate-budget rows as well. Do not restore only the project records: software, purl, and CVE strongRefs would still point at deleted records. ## Failure recovery and resume `apply` persists one cursor row per `(repo_did, phase, raw_slug)` in `identity_migration` with `status` `pending`, `done`, or `error` and an `attempts` counter. On a failed write it marks the entry `error`, commits, and exits nonzero. Completed entries are never repeated. ```bash # Retry pending/error entries; done entries are skipped. target/release/identity-migrate apply \ --cache-db "$DB" --map "$MAP.plan.json" --resume # Inspect progress. sqlite3 "$DB" "SELECT action, status, count(*) FROM identity_migration GROUP BY 1,2;" sqlite3 "$DB" "SELECT phase, raw_slug, attempts, error FROM identity_migration WHERE status='error';" ``` - Without `--resume`, an `error`/`pending` cursor row aborts with "pass --resume to retry". This prevents accidental repeated writes. - Partial deletes are safe: the batch's cursor rows stay `error`, and `--resume` re-sends only entries that are not `done`. - If `verify` reports `stale cid for `, the referenced record changed after its referrer was written. Re-run `plan` to pick up current CIDs, or re-run `apply --resume` if the entry is still pending. - If `verify` reports `legacy project ... still has a backlink`, an unmanaged record (not part of the plan) references a deleted legacy project. Inspect it with the local `links` table or `listRecords`; add the missing alias or repoint it, then regenerate the plan. - After a successful `apply`, run `verify`. Only when it exits `0` restart the service. ## Local e2e harness ```bash scripts/identity-migrate-e2e.sh ``` The script builds `vuln-notif` and `identity-migrate`, starts a temporary hono-plc and atproto-pds through the existing scripts, creates an account, initializes the cache schema, seeds `tests/fixtures/identity-migration/` (`cache_seed.sql` plus `legacy_records.json`), and then asserts: - the plan maps all three RHEL legacy projects and software records to `red-hat-enterprise-linux`, creates that project, repoints the purl and CVE, normalizes the legacy purl slug, and plans 7 legacy deletions; - `apply` reports the expected counters and writes a 12-record snapshot; - the canonical project exists, the CVE has one canonical ref plus its preserved `nvd` strongRef, software/purl refs carry current CIDs, and every legacy release record returns `RecordNotFound`; - a second `apply` reports `applied == 0`, `deleted == 0`, and skips every entry; - `verify` reports no errors and no stale refs. Requirements: `cargo`, `curl`, `jq`, `sqlite3`, `deno`, a `hono-plc` checkout (`HONO_PLC_DIR`), and the pinned `atproto-crates` checkout (`ATPROTO_CRATES_DIR`). Network access is not used; only localhost ports `4800`/`2587` (override with `PDS_PORT`/`PLC_PORT`). `BUILD_BINS=0` skips the cargo build, and `KEEP_TMP=1` keeps the temp directory on failure.