~ knotless Search/
↑↓ move↵ open⇧↵ all results

johnandersen777.bsky.social/vuln-notif

0
0
main
993 lines · 45.1 KB Raw History
# Maintainers

Prod and ops notes for the `vuln-notif` repository. Flags and events below are
verified against `src/cli.rs`, `src/bin/identity-migrate.rs`, and the PDS
governor. Release source capabilities live in
[`docs/release-sources.md`](docs/release-sources.md); module/data-flow/event
contracts live in
[`docs/release-architecture.md`](docs/release-architecture.md).

## DNS records

```bash
set -a; source ~/.nahdig_com_source; set +a

# com.nahdig.cve authority
dig +short TXT _lexicon.nahdig.com       # must match logged-in DID

# com.nahdig.cve.* authority (replace did:plc:example with the logged-in DID)
doctl compute domain records create \
  --record-name _lexicon.cve --record-ttl 3600 --record-type TXT \
  --record-data did=did:plc:example nahdig.com
dig +short TXT _lexicon.cve.nahdig.com   # must match logged-in DID
```

Negative DNS caches can keep `goat lex resolve` failing after the record is
live; wait for the negative TTL or flush the resolver.

## Publish lexicons

Publishing is an external ops step; the service binary never writes lexicons.
Use `goat` (below) or the equivalent `atpxrpc` call against the account's PDS.
Validate locally first:

```bash
scripts/check-lexicons.sh
```

The script walks all of `lexicons/` recursively and `jq`-validates every
`*.json`; it has no hardcoded list. The publish list is the ten vendored
documents: four `com.nahdig.*` schemas (two record types plus the `gitMeta`
and `projectMeta` definitions) and six `software.release.*` schemas (`defs`
plus the five record types).

```bash
set -a; source ~/.nahdig_com_source; set +a
cd vuln-notif
GOAT_USERNAME="$NAHDIG_COM_ATPROTO_HANDLE" \
GOAT_PASSWORD="$NAHDIG_COM_ATPROTO_PASSWORD" \
  goat account login
goat account check-auth

LEXICONS=(
  lexicons/com/nahdig/cve.nvdV2.json
  lexicons/com/nahdig/cve.atprotoV1.json
  lexicons/com/nahdig/release.projectMeta.json
  lexicons/com/nahdig/release.gitMeta.json
  lexicons/software.release.defs.json
  lexicons/software.release.project.json
  lexicons/software.release.channel.json
  lexicons/software.release.version.json
  lexicons/software.release.manifest.json
  lexicons/software.release.status.json
)
for f in "${LEXICONS[@]}"; do
  id=$(jq -r .id "$f")
  jq '. + {"$type":"com.atproto.lexicon.schema"}' "$f" > "/tmp/schema-$id.json"
  goat record create --rkey "$id" "/tmp/schema-$id.json"
done
```

If `goat lex publish` prints green but `goat lex resolve` says `RecordNotFound`,
use the `goat record create` loop above. Publish `software.release.defs` before
the record lexicons that reference it, and publish the `com.nahdig.*` record
schemas before the `software.release.*` records that reference their fields.
`com.nahdig.release.gitMeta` is a definition document embedded in manifest
artifact metadata; it is still published as a lexicon schema record so clients
can resolve the type.

## NVD API key

1. Request a free key: https://nvd.nist.gov/developers/request-an-api-key
2. Receive it by email.
3. Use it:

```bash
export NVD_API_KEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
# or:
vuln-notif --nvd-api-key "$NVD_API_KEY" ...
```

The key is sent as the `apiKey` request header. Without it: 5 requests/30s
and ~6s between calls. With it: 50 requests/30s.

## Rate limits

| Service | Limit |
|---|---|
| NVD API, no key | 5 requests / 30s |
| NVD API, key | 50 requests / 30s |
| PDS writes (per DID) | 5,000 points/hour, 35,000/day |
| PDS CREATE / UPDATE / DELETE | 3 / 2 / 1 points |
| Effective PDS creates | 1,666/hour, 11,666/day |
| Hosted API requests (per IP) | 3,000 / 5 min |
| `createSession` | 30 / 5 min, 300/day per account |
| `createAccount` | 100 / 5 min per IP |
| Blob upload size | 50 MiB |

`applyWrites` counts every record in the batch toward the write budget.
Last-week run: 14,349 CVEs = 43,047 points for `nvdV2` alone (> daily budget);
adding `atprotoV1` doubles it. Minimum ~2 days for `nvdV2`, ~3 days with both
formats. `uploadBlob` costs 0 write points.

The binary handles both limits itself: the native PDS governor paces every
PDS call (default 10 requests/s, burst 100, below 3,000 requests/5 min) and
reserves write points before create/put/delete/applyWrites. On 429 it honors
`Retry-After` (exact sleep) or `RateLimit-Reset`, otherwise backs off
60s → 1800s with jitter, and retries instead of failing the row.
`createSession` 429s retry with `max(300, Retry-After)`. The NVD client does
its own token-bucket pacing and outer retry (30/60/120/300s + jitter).

Per-source limits and retry behavior:

| Source | Limit / policy |
|---|---|
| NVD API | 5 requests/30s without a key, 50/30s with `NVD_API_KEY`; 403/5xx/timeout retried at 30/60/120/300s + jitter |
| Shared registry HTTP | 4 attempts, 1s/2s/4s backoff on 429 and 500/502/503/504; 60s timeout |
| crates.io (`cargo`) | adapter throttles every JSON request to 1 req/s per source instance |
| GitHub (`github`) | optional `GITHUB_TOKEN`; honors `x-ratelimit-remaining`/`x-ratelimit-reset`/`Retry-After`, fails typed on 429 or exhausted 403 |
| GitLab (`gitlab`) | optional `GITLAB_TOKEN` (`PRIVATE-TOKEN` header); response pagination links followed |
| Docker Hub (`docker`, `oci`) | anonymous pull token fetched per operation from `auth.docker.io`; sent only in `Authorization` |
| OSV (`osv`) | online `query`/`querybatch`; `querybatch` is split at 1,000 queries; offline bulk directory reads `*.json` without network |
| PDS | 5,000 points/hour, 35,000/day, 3,000 requests/5 min; create/put 3, update 2, delete 1 |

Every release-mode registry transport is wrapped in `GovernedHttp`: a per-host
token bucket (5 req/s, burst) plus a `Retry-After`/GitHub-rate-limit backoff
gate. Host state persists in `source_host_state` and is reloaded at startup.
Crate throttling and the GitHub/GitLab header checks remain source-level
controls inside those adapters.

## Write budget overrides

Budget state persists in the `rate_budget` table of the cache DB, so it
survives restarts and rides along in the S3 cache upload. Defaults are
5,000 points/hour and 35,000/day per repo DID with a 0.9 safety margin.

```bash
export PDS_WRITE_HOURLY=5000
export PDS_WRITE_DAILY=35000
export PDS_REQUESTS_PER_5MIN=3000

# equivalent CLI flags
vuln-notif --once \
  --pds-hourly-points 5000 --pds-daily-points 35000 \
  --pds-requests-per-5min 3000 ...

# refuse to start a backfill without an NVD key (also REQUIRE_NVD_KEY=1)
vuln-notif --backfill --require-nvd-key ...
```

Costs: create/put 3 points, update 2, delete 1, and `applyWrites` charges
every record in the batch at its own operation cost (deletes cost 1).
`uploadBlob` costs 0. When a window is exhausted the process sleeps until it
resets (plus jitter) and logs `rate_limit_wait`.

## Reading rate-limit events

All logs are NDJSON on stderr. Relevant events:

- `rate_limit_wait` — local budget exhausted before a write. Fields: `op`,
  `points`, `remaining_hour`, `remaining_day`, `reset`, `wait_seconds`.
- `pds_rate_limited` — PDS returned 429. Fields: `op`, `status`,
  `retry_after`, `remaining`, `reset`, `limit`, `sleep_seconds`, `cve`/`slug`.
- `nvd_retry` — outer NVD retry after 403/5xx/timeout. Fields: `attempt`,
  `error`, `sleep_seconds`.
- `nvd_api_key_configured` — startup boolean (`configured`), never the key.

`cycle_stats` and the final `run_stats` include
`pds_budget_hour_remaining`, `pds_budget_day_remaining`,
`pds_rate_limit_waits`, `pds_rate_limit_seconds`, `pds_requests_5min`,
`nvd_key_configured`, and `nvd_retries`. Budget remaining is measured against
the 0.9-margin cap, so a fresh repo starts at 4,500/31,500.

## Runtime events

stdout is NDJSON, one `{"event": ..., ...payload}` object per line. Logs are
separate NDJSON on stderr. `cycle_stats` and `run_stats` are orchestrator
events, not `RuntimeEvent`s; every other name below comes from
`src/runtime/events.rs`.

| Event | Payload fields |
|---|---|
| `prepare_started` | none |
| `s3_downloaded` | `bucket`, `path`, `bytes`, `seconds` |
| `pds_resolved` | `handle`, `pds`, `seconds` |
| `pds_logged_in` | `repo`, `seconds` |
| `cycle_started` | `cycle`, `cursor` |
| `nvd_phase_started` | `cursor` |
| `nvd_phase_completed` | `cves`, `requests`, `retries`, `seconds`, `cursor`, `error` |
| `cves_cached` | `new`, `updated_pending`, `already_done` |
| `publish_phase_started` | `limit` |
| `publish_phase_completed` | full `PublishStats` (`calls`, `present`, `created`, `failures`, `skipped`, `retries`, `get_ops`, `create_ops`, `apply_ops`, `blob_ops`, `blob_bytes`, `oversized`, `oversized_skipped`, `record_bytes_max`, `retried`, `retry_failed`, `durations`) + `error` |
| `publish_batch_started` | `offset`, `size` |
| `publish_batch_completed` | `offset`, `size`, `missing`, `created`, `present`, `failed` |
| `software_phase_started` | `limit` |
| `software_phase_completed` | `cves`, `software_created`, `purl_created`, `atproto_created`, `failures`, `error` |
| `release_phase_started` | `mode`, `limit`, `recheck_seconds` |
| `release_phase_completed` | `ReleaseCounters` (`projects_checked`, `projects_created`, `versions_created`, `channels_created`, `statuses_created`, `unchanged`, `skipped`, `unsupported_no_purl`, `failures`) + `error` |
| `release_affected_completed` | `AffectedCounters` (`projects_checked`, `projects_linked`, `versions_planned`, `versions_created`, `versions_label_only`, `cves_linked`, `unchanged`, `skipped`, `failures`, `unsupported`) + `error` |
| `release_history_started` | `limit`, `concurrency` |
| `release_history_project` | `project` plus flattened `HistoryJobCounters` (`enqueued`, `processed`, `versions_created`, `manifests_created`, `skipped`, `failed`, `reused`) |
| `release_history_skipped` | `project`, `reason` |
| `release_history_completed` | `HistoryModeCounters` (`projects_checked`, `projects_skipped`, `unsupported`, `enqueued`, `processed`, `versions_created`, `manifests_created`, `skipped`, `failed`, `reused`) + `error` |
| `release_extras_started` | `limit` |
| `release_extras_project` | `project`, `version` |
| `release_extras_skipped` | `project`, `reason` |
| `release_extras_completed` | `ExtrasCounters` (`projects_checked`, `projects_created`, `versions_created`, `manifests_created`, `statuses_created`, `unchanged`, `skipped`, `unsupported`, `failures`) + `error` |
| `link_repair_completed` | `phase` (`startup`, `publish`, `software`, `release`), `attempted`, `repaired`, `failed`, `deferred`, `error` |
| `s3_uploaded` | `bucket`, `path`, `bytes`, `seconds` |
| `s3_upload_failed` | `bucket`, `path`, `bytes`, `error` |
| `cycle_completed` | `cycle`, `duration`, full `CycleStats` |
| `cycle_failed` | `cycle`, `error` |
| `shutdown_requested` | none |
| `fatal` | `error` |
| `done` | none |

`cycle_stats` carries the complete `CycleStats` object (NVD, publish, cache,
S3, software, release, affected, history, extras, artifact, and link-repair
counters). `run_stats` sums the `SUM_KEYS`, keeps maxima for
`MAX_KEYS`, and uses the latest values for `LAST_KEYS`, then adds `cycles`,
`started_at`, `pds_seconds_avg`, and `records_per_second`.

## Our prod run — last week, newest → oldest

```bash
set -a; source ~/.nahdig_com_source; set +a; unset ATP_PDS_URL
export ATP_HANDLE="$NAHDIG_COM_ATPROTO_HANDLE"
export ATP_APP_PASSWORD="$NAHDIG_COM_ATPROTO_PASSWORD"
export S3_ENDPOINT_URL="$CVE_NAHDIG_COM_S3_ENDPOINT_URL"
export S3_BUCKET="$CVE_NAHDIG_COM_S3_BUCKET"
export S3_ACCESS_KEY_ID="$CVE_NAHDIG_COM_AWS_ACCESS_KEY"
export S3_SECRET_ACCESS_KEY="$CVE_NAHDIG_COM_AWS_SECRET_ACCESS_KEY"
export S3_REGION="$CVE_NAHDIG_COM_S3_REGION"
export S3_PATH="prod-cve-nahdig-com-week"

# first cycle fetches 7 days; repeat while pending > 0
vuln-notif --once \
  --initial-lookback-days 7 --limit 500 --order newest \
  --publish-atproto --software-limit 100 \
  --purldb .state/purl2cpe.db \
  --cache-db .state/prod-week.db
```

Loop until pending reaches 0:

```bash
while :; do
  vuln-notif --once \
    --initial-lookback-days 7 --limit 500 --order newest \
    --publish-atproto --software-limit 100 \
    --purldb .state/purl2cpe.db \
    --cache-db .state/prod-week.db
  REPO=$(sqlite3 .state/prod-week.db \
    "select repo_did from publications limit 1;")
  PENDING=$(sqlite3 .state/prod-week.db \
    "select count(*) from cves c
       left join publications p
         on p.repo_did='$REPO' and p.cve_id=c.id
      where p.status is null or p.status='pending';")
  echo "pending=$PENDING"
  [ "$PENDING" -eq 0 ] && break
  sleep 1800  # PDS repo-write-hour cap
done
```

`--limit` advances the NVD cursor; remaining rows stay pending. The cache is
resumable; a 429 only delays the loop. `--backfill` deliberately ignores the
stored cursor and starts from 1999; each fetched page replaces the stored
cursor, so stop and resume works the same way as a normal incremental run.

## Release records

`--publish-releases` publishes the default/latest version for software already
in the cache. Candidate rows resolve through `software_identity`, so one
canonical project receives all raw aliases; legacy raw-slug projects such as
`red-hat-red-hat-enterprise-linux-8` are no longer created. Projects use the
canonical identity slug (`red-hat-enterprise-linux`), and raw
Raw aliases stay in the local cache and inside CVE records; no
`com.nahdig.software` records are published.

```bash
vuln-notif --once \
  --publish-atproto --publish-releases --release-mode latest \
  --release-limit 100 --release-recheck-seconds 86400 \
  --cache-db .state/prod-week.db
```

`--release-mode latest` is the default value but is a no-op on its own:
standalone latest requires `--publish-releases`. `--release-mode all` runs the
latest pass unconditionally without that switch.

- Writes `software.release.project`, an immutable `manifest`, an immutable
  `version`, a mutable `channel` (`latest` for this mode), and optional `status`
  records. A version or manifest is never overwritten; a channel update only
  moves its `version` pointer.
- When the purl type has a source adapter, also writes
  `com.nahdig.release.projectMeta` (same-DID strongRef, only when the content
  changed) and records `publishedAt`/`rawVersion`/artifact cid+size in
  `release_version_meta`. Manifest artifacts gain `metadata` git fragments and
  a `target` object when the source declares them.
- Statuses are append-only `software.release.status` records plus
  `release_status_events` rows. A deps.dev deprecation without a registry
  signal goes through the same status module; an identical re-observation is a
  no-op and a reason change appends a new event. CVE presence never changes a
  status.
- Canonical observations supply the project, channel, and version identity.
  `com.nahdig.cve.atprotoV1` emits `channels`/`versions`/`manifests` strongRefs
  only for records that exist; otherwise it uses `versionLabels` and
  `affectedRanges`.
- Sources: deps.dev for versions/defaults; PyPI, crates.io, and RubyGems give
  SHA-256 directly; npm, Maven, NuGet, and Go are downloaded once per artifact
  to compute SHA-256.
- `--release-recheck-seconds` skips projects checked recently. Use `0` to force
  a fresh check.
- PDS writes still pass through the rate governor; every release record write
  uses the same 5k/hour and 35k/day budget.

### Affected mode

`--release-mode affected` maps OSV advisories onto the release catalog and
updates the CVEs already linked to each canonical project.

```bash
# OSV online (default), one resumable page per cycle.
while :; do
  vuln-notif --once \
    --release-mode affected --release-limit 100 --release-recheck-seconds 86400 \
    --cache-db .state/prod-week.db
  sleep 1800
done

# Network-free / reproducible: load advisory JSON from disk instead of OSV.
vuln-notif --once \
  --release-mode affected --release-limit 100 --release-recheck-seconds 0 \
  --osv-bulk-dir /srv/osv-bulk --osv-offline \
  --cache-db .state/affected.db
```

- OSV is queried online by default (`--osv-online` defaults to true). Use
  `--osv-offline` with `--osv-bulk-dir`, or `--osv-online=false` with
  `--osv-bulk-dir`; a bulk directory implies offline. Offline without a bulk
  directory is a usage error (exit 2). Bulk loading reads every `*.json`
  advisory in the directory and filters by purl per project.
- `--release-ecosystems` filters purl types (`pypi`, `npm`, `rpm`, ...) and
  `--release-sources` filters sources (`osv` matches all affected work).
- `--release-concurrency` is accepted but affected work runs in project order
  in this build; the flag affects history enumeration only.
- The catalog is existing `release_versions` rows plus deps.dev versions when
  the purl type has a deps.dev system. Affected versions without a local row
  get an immutable `manifest`+`version` only when an artifact is fetchable;
  otherwise they remain `versionLabels`. No version is invented.
- Channels point at the highest known version row in the stream and are only
  updated when that pointer changes. CVE records are `putRecord` only when the
  `$type`, `cve`, `nvd`, or `software` content changes.
- `--release-mode extras` and `all` are documented in
  [Extras mode](#extras-mode) and [All mode](#all-mode).
- Write budget: create/put costs 3 points, update 2, delete 1; `applyWrites`
  charges every record. A first affected pass can add two immutable records
  (manifest + version) per planned version, plus one channel point and one CVE
  update per linked CVE. Budget with `--release-limit` and the governor's
  `pds_budget_*_remaining` fields.
- Per-project source, database, and PDS failures are isolated: the cycle
  continues and `cycle_stats.release_affected_failures` increases. Every 10
  projects commit their local rows.

Verification after a run:

```bash
# Cycle counters (all release_affected_* fields are in cycle_stats).
grep '"event":"cycle_stats"' run.ndjson | tail -1 | jq '{
  mode: .release_mode,
  checked: .release_affected_projects_checked,
  linked: .release_affected_cves_linked,
  planned: .release_affected_versions_planned,
  created: .release_affected_versions_created,
  labels: .release_affected_versions_label_only,
  failures: .release_affected_failures
}'

# CVE refs for one project.
curl -sS -G "$ATP_PDS_URL/xrpc/com.atproto.repo.getRecord" \
  --data-urlencode "repo=$DID" \
  --data-urlencode 'collection=com.nahdig.cve.atprotoV1' \
  --data-urlencode 'rkey=CVE-2024-3094' |
  jq '.value.software[0] | {project, channels, versions, versionLabels}'

# Local link index rows written by the run.
sqlite3 .state/prod-week.db \
  "SELECT path, count(*) FROM links
   WHERE path LIKE 'software[].%'
   GROUP BY path;"
```

### Metadata and status verification (all modes)

```bash
# Project metadata (same-DID strongRef; name equals the rkey).
curl -sS -G "$ATP_PDS_URL/xrpc/com.atproto.repo.getRecord" \
  --data-urlencode "repo=$DID" \
  --data-urlencode 'collection=com.nahdig.release.projectMeta' \
  --data-urlencode 'rkey=acme-widget' |
  jq '.value | {project, name, versioningScheme, repository}'

# Manifest artifact metadata: git facts and the runtime target.
curl -sS -G "$ATP_PDS_URL/xrpc/com.atproto.repo.getRecord" \
  --data-urlencode "repo=$DID" \
  --data-urlencode 'collection=software.release.manifest' \
  --data-urlencode 'rkey=<manifest-rkey>' |
  jq '.value.artifacts[] | {metadata, target}'

# Version facts plus the full append-only status history for one version.
sqlite3 .state/prod-week.db \
  "SELECT project_slug, version, source, published_at, raw_version
     FROM release_version_meta ORDER BY project_slug, version LIMIT 20;
   SELECT status, kind, reason, created_at FROM release_status_events
     WHERE subject_uri='at://$DID/software.release.version/acme-widget:1.2.3'
     ORDER BY created_at;"
```

Append-only semantics: a status event is never updated or deleted. Current
state is the latest live same-DID `software.release.status` for the subject; a
registry reason change appends a new record under a new digest rkey, and the
old record stays addressable. `projectMeta`, statuses, and version metadata
are compared before writing, so an unchanged second run makes zero PDS writes.
If the PDS does not support `listRecords`, projectMeta writes are skipped
because "put only when changed" cannot be verified.

## History mode

`--release-mode history` enumerates each candidate's full source catalog and
publishes immutable `manifest` + `version` records. Jobs are durable and
resumable, so run it as a long-lived loop with small pages:

```bash
# One project page per cycle; 100 versions per project; 4 projects at a time.
while :; do
  vuln-notif --once \
    --release-mode history --release-limit 25 \
    --release-history-limit 100 --release-concurrency 4 \
    --release-ecosystems pypi,cargo,gem \
    --release-recheck-seconds 604800 \
    --artifact-max-bytes 104857600 \
    --cache-db .state/prod-history.db
  sleep 900
done

# Network-free source override for the local e2e (composer/cpan/apache/gnu/gitee).
vuln-notif --once \
  --release-mode history --release-recheck-seconds 0 \
  --release-source-url http://127.0.0.1:4850 \
  --cache-db /tmp/history-e2e.db
```

- Queue order is `(updated_at, project, raw version)`. A job becomes `retry`
  after a transient artifact failure and terminal `error` after three failed
  attempts. Re-running the mode never resets a terminal job; inspect
  `release_history_jobs.error` before requeueing by hand.
- `--release-history-limit` is per project per cycle. `--release-limit` bounds
  candidate projects, and `--release-concurrency` bounds how many projects are
  enumerated at once; writes still go through the single PDS governor.
- Immutable records are reused, never rewritten. A version that already exists
  is skipped at enqueue time, and a manifest whose URI/CID are recorded is
  reused.
- History adapters: PyPI, crates.io, RubyGems, GitHub, GitLab, Bitbucket,
  distro (rpm/deb/apk), Packagist, MetaCPAN, Apache, Gitee, GNU. npm, Maven,
  NuGet, Go, Docker/OCI, and SourceForge are typed `unsupported` in history:
  the first four are covered by `latest` via deps.dev, and Docker/OCI plus
  SourceForge are covered by `extras`.
- `--release-source-url` applies to the history adapters with configurable
  base URLs: `composer`, `cpan`, `apache`, `gitee`, and `gnu`. It is a local
  e2e/test seam, not a production mirror setting.
- Token environment variables: `GITHUB_TOKEN`, `GITLAB_TOKEN`, `GITEE_TOKEN`
  (or the matching flags). Tokens are sent only as request headers (Gitee uses
  `Authorization: Bearer`) and are redacted from config debug output and logs.
  They never appear in URLs, cycle stats, or events.
- Write budget: every new version costs two records (manifest + version = 6
  points); reused rows and skipped jobs cost nothing. Statuses are re-planned
  for versions materialized in the cycle and for deprecated catalog entries,
  so a registry reason change appends an event without rewriting history.
  A full-history pass over
  256k–884k versions is roughly 797k–2.68M points (23–77 days at the default
  35k/day budget). Throttle with `--release-history-limit` and
  `--release-recheck-seconds`; the governor reports
  `pds_budget_*_remaining` in every cycle.

## Extras mode

`--release-mode extras` covers every registered extras adapter, including the
deps.dev ecosystems pypi, cargo, and gem. One project per cycle page: newest
non-prerelease/non-draft version, one source-declared artifact, then project /
manifest / version / `latest` channel / status / `release.projectMeta`
records.

```bash
# One resumable page per cycle; rotate through the candidate set.
while :; do
  vuln-notif --once \
    --release-mode extras --release-limit 100 --release-recheck-seconds 86400 \
    --release-ecosystems pypi,github,rpm,docker,composer \
    --release-sources github,docker \
    --artifact-max-bytes 104857600 \
    --cache-db .state/prod-extras.db
  sleep 1800
done

# Network-free local e2e (composer stub on 127.0.0.1).
scripts/release-extras-e2e.sh
```

- Candidate selection uses the same least-recently-checked rotation as the
  other modes; `--release-limit` bounds the page and every 10 projects commit
  local rows. Filters are applied before the source is built: a project whose
  purls are all excluded is counted as `skipped` and keeps its old check time
  so a later run with different filters still visits it.
- Per-project isolation: source, artifact, database, and PDS failures increment
  `release_extras_failures` and the cycle continues. Unknown purl types and
  not-yet-adapted ecosystems are typed `unsupported`: the reason is written to
  `release_project_state.unsupported_reason` and emitted as
  `release_extras_skipped`.
- Freshness is durable: `release_project_state` stores the canonical purl and
  adapter, and the `release_projects` row caches the published URIs. A second
  run with the same newest version is `unchanged` and performs no PDS writes;
  registry status changes are evaluated when a version is (re)materialized,
  not on the unchanged fast path.
- `--dry-run` skips every release mode: only NVD fetch/cache runs, with no
  credentials and no PDS writes. Library-level extras dry-run planning exists
  but is not reachable from the CLI in this build.
- `--release-source-url` applies to extras adapters with configurable base
  URLs: `composer`, `cpan`, `apache`, `gitee`, `gnu`, and `sourceforge`.
- `--release-sources` matches purl types and adapter names (`docker` matches
  `docker`/`oci`, `distro` matches `rpm`/`deb`/`apk`); `osv` only affects
  affected work.
- Source statuses are append-only: an identical signal is a no-op and a reason
  change appends a new `software.release.status` event. Manifests carry the
  adapter's git/target facts; `release_version_meta` records the version's
  `publishedAt`/`rawVersion` when the source declares them.

### Source matrix

The full per-adapter matrix (version source, artifact/digest path, status
signals, unsupported reasons, and RHEL/UBI mapping) is
[`docs/release-sources.md`](docs/release-sources.md). Operational summary:

| Purl types | Adapter | Auth | Versions / artifacts / statuses / meta | Notes |
|---|---|---|---|---|
| `pypi` | PyPI JSON API | none | yes / yes / yanked / yes | PEP 503 names, declared SHA-256 |
| `cargo` | crates.io | none | yes / yes / yanked / yes | adapter throttles to 1 req/s |
| `gem` | RubyGems | none | yes / yes / yanked / yes | declared SHA-256, HEAD size fallback |
| `github` | GitHub releases + tags | `GITHUB_TOKEN` | yes / yes / draft+prerelease+archived / yes | asset digest, archive fallback |
| `gitlab` | GitLab releases + tags | `GITLAB_TOKEN` | yes / yes / upcoming+archived / yes | `PRIVATE-TOKEN`, pagination links |
| `bitbucket` | Bitbucket tags + downloads | none | yes / yes / no / yes | no declared digest |
| `rpm`, `deb`, `apk` | distro lifecycle + public UBI OCI | none | yes / RHEL OCI only / EOL / yes | endoflife.date + Red Hat PLC |
| `composer` | Packagist p2 | none | yes / dist zip / abandoned / yes | shasum used as SHA-256 only when 64 hex |
| `cpan` | MetaCPAN | none | yes / tarball / no status record / yes | backpan entries are marked deprecated but statuses are typed `NoStatuses` |
| `apache` | Apache download indexes | none | yes / archive files / no / yes | sidecar digests disabled; SHA-256 computed |
| `gitee` | Gitee releases | `GITEE_TOKEN` bearer | yes / assets / prerelease / yes | token never in the URL |
| `gnu` | GNU FTP index | none | yes / tarballs / no / yes | index discovery |
| `docker`, `oci` | Docker Hub registry | anonymous pull token | tags / manifest digest / tag-mutation / yes | immutable digest artifact |
| `sourceforge` | SourceForge RSS/files | none | best effort / files / no / yes | typed unavailable reasons |
| `npm`, `maven`, `nuget`, `golang` | deps.dev in `latest`; registry adapters in `history`/`extras` | none | yes / yes / registry signals / yes | full history supported |

- Tokens: `GITHUB_TOKEN`, `GITLAB_TOKEN`, `GITEE_TOKEN`; all other adapters are
  anonymous. Header values are never logged, and Gitee uses
  `Authorization: Bearer <token>` rather than `access_token`.
- Rate limits: the shared registry transport retries 429/5xx with 1s/2s/4s
  backoff and honours response pagination caps; crates.io requests are spaced
  1s apart; GitHub/GitLab surface rate-limit headers as typed failures. PDS
  writes are governed by `PDS_WRITE_HOURLY`/`PDS_WRITE_DAILY` exactly like
  every other mode.
- Write budget: a fresh project costs about 18 points (project 3, projectMeta
  3, manifest 3, version 3, channel 3, status 3 per registry signal). Re-runs
  are unchanged and cost zero. Plan blocks with `--release-limit` and inspect
  `release_extras_*` before scaling.
- Artifact downloads honour `--artifact-max-bytes`; declared digests are
  passed to the resolver, which verifies them against the downloaded bytes and
  caches the result in `release_artifacts`.

Verification after an extras run:

```bash
grep '"event":"cycle_stats"' run.ndjson | tail -1 | jq '{
  mode: .release_mode,
  checked: .release_extras_projects_checked,
  created: .release_extras_projects_created,
  versions: .release_extras_versions_created,
  manifests: .release_extras_manifests_created,
  statuses: .release_extras_statuses_created,
  unchanged: .release_extras_unchanged,
  skipped: .release_extras_skipped,
  unsupported: .release_extras_unsupported,
  failures: .release_extras_failures
}'

# Typed unsupported reasons, never silent skips.
sqlite3 .state/prod-extras.db \
  "SELECT slug, unsupported_reason FROM release_project_state
   WHERE unsupported_reason IS NOT NULL;"

# One project's newest version and its manifest strongRef.
curl -sS -G "$ATP_PDS_URL/xrpc/com.atproto.repo.getRecord" \
  --data-urlencode "repo=$DID" \
  --data-urlencode 'collection=software.release.version' \
  --data-urlencode 'rkey=acme-widget:1.2.3' | jq '.value'
```

## All mode

`--release-mode all` runs latest → affected → history (only when
`--release-history-limit > 0`) → extras in one cycle. All modes share the
candidate cursor and one artifact resolver; each keeps its own counters:

- `release_*` for latest, `release_affected_*` for affected,
  `release_history_*` for history, `release_extras_*` for extras.
- `artifact_*` aggregates every resolve in the cycle.
- `--release-sources`/`--release-ecosystems` filter affected, history, and
  extras connector work.
- Use it for a small daily sweep only. A full `all` cycle over a production
  cache can combine the OSV, history, and extras write budgets; keep
  `--release-limit`, `--release-history-limit`, and
  `--release-recheck-seconds` conservative.

```bash
vuln-notif --once \
  --release-mode all --release-limit 100 --release-recheck-seconds 86400 \
  --release-history-limit 20 --release-concurrency 4 \
  --release-ecosystems pypi,github,composer \
  --artifact-max-bytes 104857600 \
  --cache-db .state/all.db
```

## Write-budget schedules by mode

Planning numbers below assume one canonical project per product and the default
35,000-point/day budget (31,500 after the 0.9 safety margin). They are estimates
for capacity planning, not measured prod totals.

| Work | Records / updates | Approx. points | Elapsed |
|---|---:|---:|---:|
| Identity migration: CVE refs | ~14.3k updates | ~28.7k | ~1 day |
| Identity migration: software/purl refs + legacy deletes | ~100 | <1k | hours |
| `latest` / default (canonical) | ~19.2k records | ~57.6k | ~1.6 days |
| `affected` (OSV) | 100k–300k records | 330k–930k | 9–27 days |
| `history` (full catalogs) | 256k–884k records | 797k–2.68M | 23–77 days |
| `extras` | ~35.7k PURLs | source-dependent | source-dependent |

Per-mode record costs:

- `latest`: project 3 + manifest 3 + version 3 + channel 3 (+ status 3 per
  registry signal, + projectMeta 3 once). Re-observing an unchanged version
  costs zero. A channel update is a 2-point update only when the pointer moves.
- `affected`: immutable manifest + version (6 points) per affected version with
  a fetchable artifact, one channel update when the pointer moves, and one CVE
  `putRecord` update (2 points) per linked CVE only when refs change.
  `versionLabels`/`affectedRanges`-only versions cost zero PDS points.
- `history`: manifest + version (6 points) per new version; reused records and
  skipped jobs cost zero. Jobs persist in `release_history_jobs`; status
  planning may append a status record on a later pass.
- `extras`: project 3 + projectMeta 3 + manifest 3 + version 3 + channel 3
  (+ status 3 per registry signal). Unchanged projects cost zero.
- `all`: sums the mode costs in one cycle; keep `--release-limit`,
  `--release-history-limit`, and `--release-recheck-seconds` conservative.

Recommended multi-day order: migration (`snapshot → plan → review → apply →
verify`) first, then `latest --release-limit 100`, then `affected`, then
`history` for weeks, then `extras` with ecosystem filters.

Long runs are resumable:

```bash
# Foreground loop: one bounded page per cycle.
while :; do
  vuln-notif --once --release-mode history \
    --release-limit 25 --release-history-limit 100 --release-concurrency 4 \
    --cache-db .state/prod-history.db
  sleep 900
done

# Stop cleanly: SIGINT/SIGTERM drains the in-flight phase, commits, and exits.
# Resume by running the same command again; cursors, retry queues, and
# rate_budget live in the cache DB.
```

### Artifact cache and S3 mirror

- Verified artifact metadata is stored in `release_artifacts`, keyed by
  `(repo, source, package, raw version, url)`. Confirm hits with
  `cycle_stats.artifact_cache_hits`; the second resolve must not increment
  `artifact_downloads`.
- `--artifact-max-bytes N` rejects oversized streams before/while reading.
  `0` (default) means no cap; set an explicit cap in production.
- `--artifact-s3-bucket` plus `--artifact-s3-path` (default `artifacts`)
  mirrors raw bytes. The mirror uses the same S3 endpoint and credentials as
  the cache snapshot (`S3_ENDPOINT_URL`, `S3_ACCESS_KEY_ID`,
  `S3_SECRET_ACCESS_KEY`, `S3_REGION`); verify with
  `cycle_stats.artifact_mirror_puts`. Mirror failures fall back to the
  registry and increment `artifact_errors` only when the resolve itself fails.
- Without `--artifact-s3-bucket` the mirror is a no-op: `artifact_mirror_puts`
  stays `0` and only the local digest cache is consulted.
- Cache snapshot uploads (`S3_BUCKET`/`S3_PATH`) are fetched before NVD work,
  then `VACUUM INTO` + upload after each cycle. Uploads get 4 attempts with
  1s/2s/4s backoff plus 0.25s jitter and a post-upload size check; the final
  failure is logged as `s3_upload_failed` and returned, never swallowed. A
  missing remote cache is a normal first run.

### Publish retries and link repair

- CVE publish failures are recorded in `publish_errors` with an attempt count
  and `next_retry_at = now + 30 * 2^(attempts-1)` seconds, capped at one hour,
  plus up to 20% jitter. `drain_retries` runs before and after the publish
  phase, up to 100 due rows per pass. Success marks the publication
  present/created and clears the retry row; a CVE that left the cache has its
  retry row deleted.
- Local link indexing is part of the write contract: PDS write → index links
  (or queue the record) → DB commit → advance cursor. An indexing failure
  serializes the written record into `link_repairs` in the same SQLite
  transaction, so a crash cannot lose either the link rows or the repair row.
- `link_repairs` is drained at startup and after the publish, software, and
  release phases, 200 rows per drain. Retries use the same
  `30 * 2^(attempts-1)` formula capped at one hour plus jitter. A drain makes
  no PDS calls: it reindexes from the stored record JSON and is idempotent
  because indexing replaces the source's link rows.
- `link_repair_*` counters are in `cycle_stats`; each drain emits
  `link_repair_completed` with `phase`, `attempted`, `repaired`, `failed`,
  `deferred`, and `error`.

Verification after a history run:

```bash
grep '"event":"cycle_stats"' run.ndjson | tail -1 | jq '{
  mode: .release_mode,
  projects: .release_history_projects_checked,
  skipped: .release_history_skipped,
  unsupported: .release_history_unsupported,
  enqueued: .release_history_enqueued,
  processed: .release_history_processed,
  created: .release_history_created,
  failed: .release_history_failed,
  reused: .release_history_reused,
  downloads: .artifact_downloads,
  bytes: .artifact_bytes,
  cache_hits: .artifact_cache_hits,
  mirror_puts: .artifact_mirror_puts
}'

# Queue state: pending/retry/error must be inspected before a long run.
sqlite3 .state/prod-history.db \
  "SELECT status, count(*) FROM release_history_jobs GROUP BY status;"

# One immutable version and its manifest strongRef.
curl -sS -G "$ATP_PDS_URL/xrpc/com.atproto.repo.getRecord" \
  --data-urlencode "repo=$DID" \
  --data-urlencode 'collection=software.release.version' \
  --data-urlencode 'rkey=acme-widget:1.2.3' | jq '.value'

# Local e2e: fake composer source, limit/resume, artifact counters.
scripts/release-history-e2e.sh
```

## Software identity migration

`identity-migrate` is a one-time, out-of-band admin binary. The service never
runs it. Stop `vuln-notif` before `apply` and keep it stopped until `verify`
exits `0`; never run `apply` concurrently with a service cycle. The full
runbook, including the failure-recovery matrix and snapshot restore script, is
[`docs/identity-migration.md`](docs/identity-migration.md). The rules are in
[`docs/identity-model.md`](docs/identity-model.md).

Sequence: **snapshot → plan → review diff → apply --resume → verify**.

```bash
set -a; source ~/.nahdig_com_source; set +a; unset ATP_PDS_URL
export ATP_HANDLE="$NAHDIG_COM_ATPROTO_HANDLE"   # repo DID/handle, not a secret
export ATP_APP_PASSWORD="$NAHDIG_COM_ATPROTO_PASSWORD"

DB=.state/prod-week.db
MAP=migration/identity-$(date +%Y%m%dT%H%M%S)
cp "$DB" "$DB.pre-migration"                    # cache, cursor, rate-budget backup

# 1. Snapshot every migration collection to JSONL (read-only).
target/release/identity-migrate snapshot \
  --cache-db "$DB" --snapshot-dir "$MAP.snapshot"

# 2. Inventory cache + remote listRecords and write a reviewable plan.
target/release/identity-migrate plan \
  --cache-db "$DB" --purldb .state/purl2cpe.db \
  --out "$MAP.plan.json"

# 3. Review every legacy -> canonical pair before writing.
jq -r '.entries[] | .action' "$MAP.plan.json" | sort | uniq -c
jq -r '.entries[] | select(.action == "create_project") | .canonical_slug' "$MAP.plan.json"
jq -r '.entries[] | select(.action == "delete_release_record") |
       [.legacy_slug, .canonical_slug, .legacy_uri] | @tsv' "$MAP.plan.json" | sort -u

# 4. Apply in dependency order (projects, software, purl, CVE, status,
#    normalize, deletes) with a persisted cursor. 200-record applyWrites
#    delete batches.
target/release/identity-migrate apply \
  --cache-db "$DB" --map "$MAP.plan.json" \
  --batch 200 --snapshot-dir "$MAP.snapshot"

# If step 4 exits nonzero, rerun the same command with --resume:
target/release/identity-migrate apply \
  --cache-db "$DB" --map "$MAP.plan.json" \
  --batch 200 --snapshot-dir "$MAP.snapshot" --resume

# 5. Re-read the repository and fail on stale refs/backlinks.
target/release/identity-migrate verify \
  --cache-db "$DB" --map "$MAP.plan.json"
```

Useful flags (all global; `identity-migrate --help`):

| Flag | Use |
|---|---|
| `--cache-db` | SQLite cache, migration cursor, and rate-budget state |
| `--purldb` | purl2cpe SQLite for planning; absent is tolerated |
| `--out` / `--map` | Plan output / plan input for apply and verify |
| `--batch` | Records per `applyWrites` delete batch; default `200` |
| `--limit` | Max plan entries per invocation (`0` = all), for chunked runs |
| `--snapshot-dir` | JSONL snapshot; accepted by `plan` and `apply` |
| `--resume` | Retry `pending`/`error` entries, skip `done` |
| `--dry-run` | Plan only; valid only for `plan` |
| `--apply` / `--verify` | Flag forms of the subcommands; can combine |

The subcommand forms are `plan`, `apply`, `verify`, `snapshot`, and
`normalize-purl-slugs`; the last one rewrites legacy full-AT-URI
`software_purls.software_slug` values in the cache and needs no credentials.

NDJSON events on stdout (one line per run):

| Event | Fields |
|---|---|
| `identity_snapshot` | `records`, `dir` |
| `identity_plan` | `entries`, `created`, `deleted`, `out` |
| `identity_apply` | `report`: `applied`, `skipped_done`, `created_projects`, `software_repoints`, `purl_repoints`, `cve_repoints`, `status_repoints`, `deleted`, `normalized` |
| `identity_verify` | `report`: `canonical_projects`, `legacy_projects`, `software_repoints`, `purl_repoints`, `cve_repoints`, `status_repoints`, `deletions`, `stale_refs`, `errors` |
| `identity_normalize_purl_slugs` | `repo_did`, `updated` |

`verify` exits `1` when `errors` is non-empty. `apply` exits nonzero and leaves
the failing entry `error`; `--resume` retries it.

### Write budget and batching

| Operation | Points |
|---|---:|
| create / put (new record) | 3 |
| update (`putRecord` on existing record) | 2 |
| delete | 1 |
| `applyWrites` | each record at its own operation cost |

The migration is dominated by CVE ref updates (~14.3k updates × 2 ≈ 28.6k
points, close to one 35k/day budget) plus software/purl ref updates and ~100
legacy deletes (<1k). Run it as a day-long job. `apply` persists one cursor row
per `(repo_did, phase, raw_slug)` in `identity_migration`, so `--resume`
skips completed entries and stopping does not lose budget. The governor still
enforces 5,000 points/hour and 35,000/day with a 0.9 safety margin.

### Rollback

The pre-apply snapshot plus the cache DB copy is the rollback material. There
is intentionally no destructive rollback command; restore is a manual
`putRecord` replay from the snapshot, documented in
`docs/identity-migration.md#rollback-via-snapshot`. Restore all collections
(project, software, purl, CVE), not only projects, or strongRefs stay broken.
Restore the cache DB copy to roll back the cursor and rate-budget rows too.

### Verification recipe

Canonical project backlinks (expect a non-zero total and canonical project
URI):

```bash
PDS=${ATP_PDS_URL:-https://your-pds.example}
DID=$(goat account check-auth | sed -n 's/^DID: //p')
PROJ=red-hat-enterprise-linux
PROJECT="at://$DID/software.release.project/$PROJ"
curl -sS -G 'https://constellation.microcosm.blue/xrpc/blue.microcosm.links.getBacklinks' \
  --data-urlencode "subject=$PROJECT" \
  --data-urlencode 'source=com.nahdig.cve.atprotoV1:software[].project.uri' | jq .
```

Zero legacy backlinks — query each legacy RHEL slug and expect `total: 0`:

```bash
for LEGACY in red-hat-red-hat-enterprise-linux-8 \
              red-hat-red-hat-enterprise-linux-9 \
              redhat-enterprise-linux \
              redhat-enterprise-linux-eus \
              redhat-rhel-els; do
  SUBJECT="at://$DID/software.release.project/$LEGACY"
  TOTAL=$(curl -sS -G 'https://constellation.microcosm.blue/xrpc/blue.microcosm.links.getBacklinks' \
    --data-urlencode "subject=$SUBJECT" \
    --data-urlencode 'source=com.nahdig.cve.atprotoV1:software[].project.uri' | jq -r .total)
  printf '%s: %s\n' "$LEGACY" "$TOTAL"
done
```

The local equivalent checks the `links` table for unmanaged backlinks:

```bash
sqlite3 "$DB" "
  SELECT subject_uri, count(*)
  FROM links
  WHERE subject_uri LIKE 'at://$DID/software.release.project/%'
    AND subject_uri NOT LIKE '%/red-hat-enterprise-linux'
  GROUP BY subject_uri;"
```

`verify` performs the authoritative check across the repository; the queries
above are the quick prod spot-check. Also confirm the canonical project exists
(`getRecord ... software.release.project/red-hat-enterprise-linux`), the CVE
has one canonical ref plus its preserved `nvd` strongRef, and a second `apply`
reports `applied == 0` and `deleted == 0`.

## Local e2e order

Services: hono-plc `:2587`, atproto-pds `:4800`, RustFS `:9000` (S3 test only).
Start **PLC → PDS → account**; the PDS must run with
`PDS_LEXICON_DIR=$PWD/lexicons` and `PDS_DID_PLC_URL=http://127.0.0.1:2587`.

```bash
scripts/local_plc.sh start
scripts/local_atproto_pds.sh start
scripts/create_local_account.sh
scripts/local-e2e.sh
scripts/blob-e2e.sh
scripts/local_rustfs.sh start
scripts/s3-cache-e2e.sh
scripts/identity-migrate-e2e.sh
scripts/release-affected-e2e.sh
scripts/release-history-e2e.sh
scripts/release-extras-e2e.sh
scripts/check-lexicons.sh
```

`local-e2e.sh` and `identity-migrate-e2e.sh` are self-contained and start
their own temporary PLC/PDS; the explicit start commands above are for a
persistent dev setup.

## Delete old collection

```bash
OLD=com.nahdig.cve
# paginated listRecords + applyWrites delete (200/batch), then:
goat record delete --rkey "$OLD" com.atproto.lexicon.schema
```

## Verify blob record

```bash
PDS=${ATP_PDS_URL:-https://your-pds.example}
DID=$(goat account check-auth | sed -n 's/^DID: //p')
REC=$(curl -sS "$PDS/xrpc/com.atproto.repo.getRecord?repo=$DID&collection=com.nahdig.cve.nvdV2&rkey=CVE-2021-3942")
CID=$(jq -r '.value.dataBlob.ref["$link"]' <<<"$REC")
curl -sS "$PDS/xrpc/com.atproto.sync.getBlob?did=$DID&cid=$CID" | gunzip | jq .id
```

## Verify backlinks

Check the canonical project and its channel/version refs:

```bash
PROJ=red-hat-enterprise-linux
PROJECT="at://$DID/software.release.project/$PROJ"
CHANNEL="at://$DID/software.release.channel/$PROJ:9"
VERSION="at://$DID/software.release.version/$PROJ:9.4"

for PAIR in "project:$PROJECT:software[].project.uri" \
            "channel:$CHANNEL:software[].channels[].uri" \
            "version:$VERSION:software[].versions[].uri"; do
  LABEL=${PAIR%%:*}
  REST=${PAIR#*:}
  SUBJECT=${REST%:*}
  SOURCE=${REST##*:}
  echo "== $LABEL $SUBJECT"
  curl -sS -G 'https://constellation.microcosm.blue/xrpc/blue.microcosm.links.getBacklinks' \
    --data-urlencode "subject=$SUBJECT" \
    --data-urlencode "source=com.nahdig.cve.atprotoV1:$SOURCE" | jq .
done
```

For zero legacy backlinks, query the legacy RHEL slugs listed in the migration
section and assert `total: 0`; `identity-migrate verify` is the authoritative
repository-wide check.