~ knotless Search/
↑↓ move↵ open⇧↵ all results

johnandersen777.bsky.social/vuln-notif

0
0
main
330 lines · 14.7 KB Raw History
# Identity migration runbook

The Rust service stores one canonical `software.release.project` per product.
Legacy caches and repositories instead hold one project per raw
`{part}:{vendor}:{product}` observation, so Red Hat Enterprise Linux is split
across `red-hat-red-hat-enterprise-linux-8`, `redhat-enterprise-linux`,
`redhat-rhel-els`, and other variants.

`identity-migrate` is the one-time, out-of-band admin tool that rewrites those
references and deletes the legacy release records. There is no runtime
migration phase, no startup `ALTER TABLE`, and no implicit write from
`vuln-notif`: **the service never runs this migration itself**. A human runs
`plan`, reviews the map, then runs `apply` and `verify`.

Locations:

- Binary: `target/release/identity-migrate` (`src/bin/identity-migrate.rs`).
- Plan implementation: `src/migrate/`.
- Curated aliases: `data/identity/aliases.json`.
- Local e2e: `scripts/identity-migrate-e2e.sh`.
- Fixtures: `tests/fixtures/identity-migration/`.

## Identity rules

```text
identity = (class, canonical_vendor, canonical_family)
slug     = truncate64(join(canonical_vendor, canonical_family, '-'))
```

- `class` is `os`, `app`, `hardware`, or `firmware`. Curated OS families
  (RHEL, Fedora, CentOS, Debian, Ubuntu, SUSE, Oracle Linux, Amazon Linux,
  Windows, macOS) are always `os`. Otherwise CPE part maps `o` → `os`,
  `a` → `app`, `h` → `hardware`. The raw `part` is not part of the identity,
  so the `a`/`o` RHEL duplicates collapse.
- `canonical_vendor` is CPE-unescaped, lowercased, non-alphanumerics collapsed
  to `-`, then looked up in the curated vendor aliases (`redhat`, `red_hat`,
  `Red Hat` → `red-hat`).
- `canonical_family` strips a redundant vendor prefix, then applies the curated
  product aliases. `enterprise_linux*`, `rhel_els`, `enterprise_linux_for_*`,
  and `codeready_linux_builder*` all become `red-hat-enterprise-linux`.
- Trailing stream/version text is split only for families listed in
  `versioned_families`. `X` and single-digit `X.0` are NVD major sentinels:
  they produce a channel and no version. `X.Y` produces channel `X` and version
  `X.Y` (`9.4`, `8.10`, multi-digit `10.0` preserved). Edition/arch tokens
  (`eus`, `els`, `aus`, `sap`, `rt`, `aarch64`, ...) become channel suffixes;
  default `x86_64` is omitted.
- Channels never contain dots (`Ubuntu 20.04` → channel `20-04`); versions keep
  dots. Version and manifest records are immutable; only channels move.
- Placeholder vendors/products (`n/a`, `any`, `unknown`, `*`, `-`) produce no
  identity. The old `n-a-n-a` project is never created again.
- Slug collisions between different identities are resolved deterministically:
  a curated family owns the base slug, otherwise the lexicographically smaller
  identity key owns it and the other gets `-<sha256(identity_key)[:6]>`.
- Every write is same-DID with the repository. Cross-DID strongRefs are
  rejected by the tool and by `verify`.

RHEL example:

| Observation | Legacy raw slug | Canonical project | Channel |
|---|---|---|---|
| NVD `Red Hat` + `Red Hat Enterprise Linux 8` | `red-hat-red-hat-enterprise-linux-8` | `red-hat-enterprise-linux` | `8` |
| CPE `cpe:2.3:o:redhat:enterprise_linux:9.4` | `redhat-enterprise-linux` | `red-hat-enterprise-linux` | `9` (version `9.4`) |
| NVD `redhat` + `rhel_els` | `redhat-rhel-els` | `red-hat-enterprise-linux` | `els` |

Legacy `com.nahdig.software` records are only encountered when migrating old
snapshots; the current service does not publish them. Raw `vendor`, `product`,
and `cpes` live in the local cache and in CVE `software[]` entries.

## Alias table

`data/identity/aliases.json` is embedded with `include_str!`, validated at
startup and by tests. Editing this file and rebuilding is the only supported
way to merge identities; there is no fuzzy matching, stemming, or
batch-order dependence.

```json
{
  "vendors": {"redhat": "red-hat", "red-hat": "red-hat"},
  "products": {"enterprise-linux": "red-hat-enterprise-linux",
               "rhel-els": "red-hat-enterprise-linux"},
  "families": {"red-hat-enterprise-linux": "red-hat"},
  "versioned_families": ["red-hat-enterprise-linux"],
  "class_overrides": {"red-hat-enterprise-linux": "os"}
}
```

- `vendors`: normalized vendor → canonical vendor.
- `products`: normalized product alias → canonical family.
- `families`: canonical family → canonical vendor (used for slug ownership).
- `versioned_families`: families that may split trailing version/edition text.
- `class_overrides`: family → identity class.

Every legacy → canonical pair appears in the plan `reason` and in the plan
entries before apply, so the diff is reviewable per record.

## Migration phases

`snapshot → plan → review diff → apply → verify`

1. **Snapshot.** Write every record in the migration collections to JSONL.
   Read-only on the cache and PDS.
2. **Plan.** Inventory the local cache (`cves`, `software`, `software_purls`,
   and purl2cpe when available) plus the remote `listRecords` inventory, then
   write a reviewable JSON map. `plan` performs no writes.
3. **Review diff.** Inspect every `delete_release_record`,
   `repoint_software`, `repoint_purl`, `repoint_cve`, `create_project`, and
   `normalize_purl_slug` entry. Do not continue if a pair is wrong; project
   rkeys are permanent.
4. **Apply.** Execute the map in dependency order with a persisted cursor.
   Already-`done` entries are skipped, so re-running is safe.
5. **Verify.** Re-read the repository, assert canonical projects exist, legacy
   projects have no unmanaged backlinks, CVE refs are deduped and canonical,
   and strongRef CIDs are current. `verify` exits `1` if any error is reported.

### Local

```bash
scripts/local_plc.sh start
scripts/local_atproto_pds.sh start
eval "$(scripts/create_local_account.sh 2>/dev/null)"   # exports ATP_PDS_URL/HANDLE/APP_PASSWORD

target/release/identity-migrate snapshot \
  --cache-db .state/local.db --snapshot-dir migration/snapshot-local

target/release/identity-migrate plan \
  --cache-db .state/local.db --purldb .state/purl2cpe.db \
  --out migration/plan-local.json

target/release/identity-migrate apply \
  --cache-db .state/local.db --map migration/plan-local.json \
  --batch 200

target/release/identity-migrate verify \
  --cache-db .state/local.db --map migration/plan-local.json
```

`scripts/identity-migrate-e2e.sh` runs this whole flow against a temporary PLC
and PDS, seeds the legacy RHEL split, and asserts the plan, apply report,
zero-write second apply, and verify report.

### Production

Stop the service first (freeze writers) and keep it stopped until `verify`
passes. Never run `apply` concurrently with a service cycle.

```bash
export ATP_HANDLE=did:plc:...          # repository DID/handle, not a secret
export ATP_APP_PASSWORD=...            # privileged app password
unset ATP_PDS_URL                      # resolve the PDS from PLC, or set it explicitly

DB=.state/prod-week.db
MAP=migration/identity-$(date +%Y%m%dT%H%M%S)

cp "$DB" "$DB.pre-migration"           # cache/rate-budget/cursor backup

target/release/identity-migrate snapshot \
  --cache-db "$DB" --snapshot-dir "$MAP.snapshot"

target/release/identity-migrate plan \
  --cache-db "$DB" --purldb .state/purl2cpe.db \
  --out "$MAP.plan.json"

# Review before applying.
jq -r '.entries[] | .action' "$MAP.plan.json" | sort | uniq -c
jq -r '.entries[] | select(.action == "create_project") | .canonical_slug' "$MAP.plan.json"
jq -r '.entries[] | select(.action == "delete_release_record") |
       [.legacy_slug, .canonical_slug, .legacy_uri] | @tsv' "$MAP.plan.json" | sort -u
jq -r '.entries[] | select(.action == "repoint_cve") | .legacy_slug' "$MAP.plan.json" | wc -l

target/release/identity-migrate apply \
  --cache-db "$DB" --map "$MAP.plan.json" \
  --batch 200 --snapshot-dir "$MAP.snapshot"

target/release/identity-migrate verify \
  --cache-db "$DB" --map "$MAP.plan.json"
```

`apply` prints one `identity_apply` event with a report:
`applied`, `skipped_done`, `created_projects`, `software_repoints`,
`purl_repoints`, `cve_repoints`, `deleted`, and `normalized`. `verify` prints
`identity_verify` with `canonical_projects`, `software_repoints`, `purl_repoints`,
`cve_repoints`, `deletions`, `stale_refs`, and `errors`.

If the cache holds no repository DID (or several), pass
`--handle did:plc:...` explicitly to `normalize-purl-slugs`.

## Dependency order

`apply` sorts and executes in this order; never delete a release record before
its referrers move, because strongRef CIDs cascade.

| Order | Action | Effect |
|---:|---|---|
| 1 | `create_project` | `software.release.project/<canonical>` if missing |
| 2 | `repoint_software` | `com.nahdig.software.project` strongRef |
| 3 | `repoint_purl` | `com.nahdig.purl.software[].cid` |
| 4 | `repoint_cve` | `com.nahdig.cve.atprotoV1.software[].project`; dedupe to one canonical ref; drop refs to records being deleted |
| 5 | `normalize_purl_slug` | cache-only: `software_purls.software_slug` AT-URI → slug |
| 6 | `delete_release_record` | `applyWrites` batches of legacy project/channel/version/manifest/status records |

Local link rows are not rebuilt from `listRecords`. The service indexes links
as it writes, repairs failed indexes from `link_repairs`, and re-indexes
migrated URIs on its next pass; `verify` checks the local `links` table only
for unmanaged backlinks to legacy projects.

## Rate budget

PDS write costs:

| Operation | Points |
|---|---:|
| create / put (new record) | 3 |
| update (`putRecord` on existing record) | 2 |
| delete | 1 |
| `applyWrites` | each record in the batch at its own operation cost (deletes cost 1) |

The migration's referrer moves are `putRecord` updates (2 points each); legacy
record deletions are `applyWrites` deletes (1 point per record). Budget:
5,000 points/hour and 35,000 points/day per repository DID, with a 0.9 safety
margin. The governor reserves points before each write and persists state in
the cache DB's `rate_budget` table.

| Work | Records | Approx. points |
|---|---:|---:|
| CVE refs | ~14.3k updates | ~28.6k |
| software/purl refs + legacy deletes | ~100 | <1k |

The CVE pass is close to one daily budget, so plan it as a day-long run. Use
`--limit` to cap entries per invocation and `--batch` (default 200) for deletes.
Stopping and resuming does not lose budget: it is persisted with the cursor.

## Rollback via snapshot

`snapshot` writes one `<collection>.jsonl` file per migration collection; each
line is `{"uri", "cid", "value"}` for a live record. Snapshots are full
collection supersets, so they are also a cache-independent backup of the
release/software/purl/CVE records.

A snapshot taken before `apply`, plus the cache DB copy, is the rollback
material. Restoring is a manual `putRecord` replay (the tool intentionally has
no destructive rollback command):

```bash
SNAP=migration/identity-YYYYmmddTHHMMSS.snapshot
DID=$ATP_HANDLE
ACCESS_JWT=$(curl -fsS -X POST "$ATP_PDS_URL/xrpc/com.atproto.server.createSession" \
  -H 'Content-Type: application/json' \
  -d "$(jq -nc --arg identifier "$ATP_HANDLE" --arg password "$ATP_APP_PASSWORD" \
        '{identifier:$identifier,password:$password}')" | jq -r .accessJwt)

for file in "$SNAP"/*.jsonl; do
  collection=$(basename "$file" .jsonl)
  while IFS= read -r line; do
    rkey=$(jq -r '.uri | split("/") | last' <<<"$line")
    value=$(jq -c '.value' <<<"$line")
    jq -nc --arg repo "$DID" --arg collection "$collection" \
      --arg rkey "$rkey" --argjson record "$value" \
      '{repo:$repo,collection:$collection,rkey:$rkey,record:$record}' |
      curl -fsS -X POST \
        -H "Authorization: Bearer $ACCESS_JWT" \
        -H 'Content-Type: application/json' \
        -d @- "$ATP_PDS_URL/xrpc/com.atproto.repo.putRecord"
  done <"$file"
done
```

Restore all collections, then re-run `verify` with the original plan: it
should now report the pre-migration errors (legacy refs), which confirms the
restore is complete. Restoring the cache DB copy rolls back the cursor and
rate-budget rows as well. Do not restore only the project records: software,
purl, and CVE strongRefs would still point at deleted records.

## Failure recovery and resume

`apply` persists one cursor row per `(repo_did, phase, raw_slug)` in
`identity_migration` with `status` `pending`, `done`, or `error` and an
`attempts` counter. On a failed write it marks the entry `error`, commits, and
exits nonzero. Completed entries are never repeated.

```bash
# Retry pending/error entries; done entries are skipped.
target/release/identity-migrate apply \
  --cache-db "$DB" --map "$MAP.plan.json" --resume

# Inspect progress.
sqlite3 "$DB" "SELECT action, status, count(*) FROM identity_migration GROUP BY 1,2;"
sqlite3 "$DB" "SELECT phase, raw_slug, attempts, error FROM identity_migration WHERE status='error';"
```

- Without `--resume`, an `error`/`pending` cursor row aborts with
  "pass --resume to retry". This prevents accidental repeated writes.
- Partial deletes are safe: the batch's cursor rows stay `error`, and
  `--resume` re-sends only entries that are not `done`.
- If `verify` reports `stale cid for <uri>`, the referenced record changed
  after its referrer was written. Re-run `plan` to pick up current CIDs, or
  re-run `apply --resume` if the entry is still pending.
- If `verify` reports `legacy project ... still has a backlink`, an unmanaged
  record (not part of the plan) references a deleted legacy project. Inspect
  it with the local `links` table or `listRecords`; add the missing alias or
  repoint it, then regenerate the plan.
- After a successful `apply`, run `verify`. Only when it exits `0` restart the
  service.

## Local e2e harness

```bash
scripts/identity-migrate-e2e.sh
```

The script builds `vuln-notif` and `identity-migrate`, starts a temporary
hono-plc and atproto-pds through the existing scripts, creates an account,
initializes the cache schema, seeds `tests/fixtures/identity-migration/`
(`cache_seed.sql` plus `legacy_records.json`), and then asserts:

- the plan maps all three RHEL legacy projects and software records to
  `red-hat-enterprise-linux`, creates that project, repoints the purl and CVE,
  normalizes the legacy purl slug, and plans 7 legacy deletions;
- `apply` reports the expected counters and writes a 12-record snapshot;
- the canonical project exists, the CVE has one canonical ref plus its
  preserved `nvd` strongRef, software/purl refs carry current CIDs, and every
  legacy release record returns `RecordNotFound`;
- a second `apply` reports `applied == 0`, `deleted == 0`, and skips every
  entry;
- `verify` reports no errors and no stale refs.

Requirements: `cargo`, `curl`, `jq`, `sqlite3`, `deno`, a `hono-plc` checkout
(`HONO_PLC_DIR`), and the pinned `atproto-crates` checkout
(`ATPROTO_CRATES_DIR`). Network access is not used; only localhost ports
`4800`/`2587` (override with `PDS_PORT`/`PLC_PORT`). `BUILD_BINS=0` skips the
cargo build, and `KEEP_TMP=1` keeps the temp directory on failure.