# Maintainers Prod and ops notes for the `vuln-notif` repository. Flags and events below are verified against `src/cli.rs`, `src/bin/identity-migrate.rs`, and the PDS governor. Release source capabilities live in [`docs/release-sources.md`](docs/release-sources.md); module/data-flow/event contracts live in [`docs/release-architecture.md`](docs/release-architecture.md). ## DNS records ```bash set -a; source ~/.nahdig_com_source; set +a # com.nahdig.cve authority dig +short TXT _lexicon.nahdig.com # must match logged-in DID # com.nahdig.cve.* authority (replace did:plc:example with the logged-in DID) doctl compute domain records create \ --record-name _lexicon.cve --record-ttl 3600 --record-type TXT \ --record-data did=did:plc:example nahdig.com dig +short TXT _lexicon.cve.nahdig.com # must match logged-in DID ``` Negative DNS caches can keep `goat lex resolve` failing after the record is live; wait for the negative TTL or flush the resolver. ## Publish lexicons Publishing is an external ops step; the service binary never writes lexicons. Use `goat` (below) or the equivalent `atpxrpc` call against the account's PDS. Validate locally first: ```bash scripts/check-lexicons.sh ``` The script walks all of `lexicons/` recursively and `jq`-validates every `*.json`; it has no hardcoded list. The publish list is the ten vendored documents: four `com.nahdig.*` schemas (two record types plus the `gitMeta` and `projectMeta` definitions) and six `software.release.*` schemas (`defs` plus the five record types). ```bash set -a; source ~/.nahdig_com_source; set +a cd vuln-notif GOAT_USERNAME="$NAHDIG_COM_ATPROTO_HANDLE" \ GOAT_PASSWORD="$NAHDIG_COM_ATPROTO_PASSWORD" \ goat account login goat account check-auth LEXICONS=( lexicons/com/nahdig/cve.nvdV2.json lexicons/com/nahdig/cve.atprotoV1.json lexicons/com/nahdig/release.projectMeta.json lexicons/com/nahdig/release.gitMeta.json lexicons/software.release.defs.json lexicons/software.release.project.json lexicons/software.release.channel.json lexicons/software.release.version.json lexicons/software.release.manifest.json lexicons/software.release.status.json ) for f in "${LEXICONS[@]}"; do id=$(jq -r .id "$f") jq '. + {"$type":"com.atproto.lexicon.schema"}' "$f" > "/tmp/schema-$id.json" goat record create --rkey "$id" "/tmp/schema-$id.json" done ``` If `goat lex publish` prints green but `goat lex resolve` says `RecordNotFound`, use the `goat record create` loop above. Publish `software.release.defs` before the record lexicons that reference it, and publish the `com.nahdig.*` record schemas before the `software.release.*` records that reference their fields. `com.nahdig.release.gitMeta` is a definition document embedded in manifest artifact metadata; it is still published as a lexicon schema record so clients can resolve the type. ## NVD API key 1. Request a free key: https://nvd.nist.gov/developers/request-an-api-key 2. Receive it by email. 3. Use it: ```bash export NVD_API_KEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx # or: vuln-notif --nvd-api-key "$NVD_API_KEY" ... ``` The key is sent as the `apiKey` request header. Without it: 5 requests/30s and ~6s between calls. With it: 50 requests/30s. ## Rate limits | Service | Limit | |---|---| | NVD API, no key | 5 requests / 30s | | NVD API, key | 50 requests / 30s | | PDS writes (per DID) | 5,000 points/hour, 35,000/day | | PDS CREATE / UPDATE / DELETE | 3 / 2 / 1 points | | Effective PDS creates | 1,666/hour, 11,666/day | | Hosted API requests (per IP) | 3,000 / 5 min | | `createSession` | 30 / 5 min, 300/day per account | | `createAccount` | 100 / 5 min per IP | | Blob upload size | 50 MiB | `applyWrites` counts every record in the batch toward the write budget. Last-week run: 14,349 CVEs = 43,047 points for `nvdV2` alone (> daily budget); adding `atprotoV1` doubles it. Minimum ~2 days for `nvdV2`, ~3 days with both formats. `uploadBlob` costs 0 write points. The binary handles both limits itself: the native PDS governor paces every PDS call (default 10 requests/s, burst 100, below 3,000 requests/5 min) and reserves write points before create/put/delete/applyWrites. On 429 it honors `Retry-After` (exact sleep) or `RateLimit-Reset`, otherwise backs off 60s → 1800s with jitter, and retries instead of failing the row. `createSession` 429s retry with `max(300, Retry-After)`. The NVD client does its own token-bucket pacing and outer retry (30/60/120/300s + jitter). Per-source limits and retry behavior: | Source | Limit / policy | |---|---| | NVD API | 5 requests/30s without a key, 50/30s with `NVD_API_KEY`; 403/5xx/timeout retried at 30/60/120/300s + jitter | | Shared registry HTTP | 4 attempts, 1s/2s/4s backoff on 429 and 500/502/503/504; 60s timeout | | crates.io (`cargo`) | adapter throttles every JSON request to 1 req/s per source instance | | GitHub (`github`) | optional `GITHUB_TOKEN`; honors `x-ratelimit-remaining`/`x-ratelimit-reset`/`Retry-After`, fails typed on 429 or exhausted 403 | | GitLab (`gitlab`) | optional `GITLAB_TOKEN` (`PRIVATE-TOKEN` header); response pagination links followed | | Docker Hub (`docker`, `oci`) | anonymous pull token fetched per operation from `auth.docker.io`; sent only in `Authorization` | | OSV (`osv`) | online `query`/`querybatch`; `querybatch` is split at 1,000 queries; offline bulk directory reads `*.json` without network | | PDS | 5,000 points/hour, 35,000/day, 3,000 requests/5 min; create/put 3, update 2, delete 1 | Every release-mode registry transport is wrapped in `GovernedHttp`: a per-host token bucket (5 req/s, burst) plus a `Retry-After`/GitHub-rate-limit backoff gate. Host state persists in `source_host_state` and is reloaded at startup. Crate throttling and the GitHub/GitLab header checks remain source-level controls inside those adapters. ## Write budget overrides Budget state persists in the `rate_budget` table of the cache DB, so it survives restarts and rides along in the S3 cache upload. Defaults are 5,000 points/hour and 35,000/day per repo DID with a 0.9 safety margin. ```bash export PDS_WRITE_HOURLY=5000 export PDS_WRITE_DAILY=35000 export PDS_REQUESTS_PER_5MIN=3000 # equivalent CLI flags vuln-notif --once \ --pds-hourly-points 5000 --pds-daily-points 35000 \ --pds-requests-per-5min 3000 ... # refuse to start a backfill without an NVD key (also REQUIRE_NVD_KEY=1) vuln-notif --backfill --require-nvd-key ... ``` Costs: create/put 3 points, update 2, delete 1, and `applyWrites` charges every record in the batch at its own operation cost (deletes cost 1). `uploadBlob` costs 0. When a window is exhausted the process sleeps until it resets (plus jitter) and logs `rate_limit_wait`. ## Reading rate-limit events All logs are NDJSON on stderr. Relevant events: - `rate_limit_wait` — local budget exhausted before a write. Fields: `op`, `points`, `remaining_hour`, `remaining_day`, `reset`, `wait_seconds`. - `pds_rate_limited` — PDS returned 429. Fields: `op`, `status`, `retry_after`, `remaining`, `reset`, `limit`, `sleep_seconds`, `cve`/`slug`. - `nvd_retry` — outer NVD retry after 403/5xx/timeout. Fields: `attempt`, `error`, `sleep_seconds`. - `nvd_api_key_configured` — startup boolean (`configured`), never the key. `cycle_stats` and the final `run_stats` include `pds_budget_hour_remaining`, `pds_budget_day_remaining`, `pds_rate_limit_waits`, `pds_rate_limit_seconds`, `pds_requests_5min`, `nvd_key_configured`, and `nvd_retries`. Budget remaining is measured against the 0.9-margin cap, so a fresh repo starts at 4,500/31,500. ## Runtime events stdout is NDJSON, one `{"event": ..., ...payload}` object per line. Logs are separate NDJSON on stderr. `cycle_stats` and `run_stats` are orchestrator events, not `RuntimeEvent`s; every other name below comes from `src/runtime/events.rs`. | Event | Payload fields | |---|---| | `prepare_started` | none | | `s3_downloaded` | `bucket`, `path`, `bytes`, `seconds` | | `pds_resolved` | `handle`, `pds`, `seconds` | | `pds_logged_in` | `repo`, `seconds` | | `cycle_started` | `cycle`, `cursor` | | `nvd_phase_started` | `cursor` | | `nvd_phase_completed` | `cves`, `requests`, `retries`, `seconds`, `cursor`, `error` | | `cves_cached` | `new`, `updated_pending`, `already_done` | | `publish_phase_started` | `limit` | | `publish_phase_completed` | full `PublishStats` (`calls`, `present`, `created`, `failures`, `skipped`, `retries`, `get_ops`, `create_ops`, `apply_ops`, `blob_ops`, `blob_bytes`, `oversized`, `oversized_skipped`, `record_bytes_max`, `retried`, `retry_failed`, `durations`) + `error` | | `publish_batch_started` | `offset`, `size` | | `publish_batch_completed` | `offset`, `size`, `missing`, `created`, `present`, `failed` | | `software_phase_started` | `limit` | | `software_phase_completed` | `cves`, `software_created`, `purl_created`, `atproto_created`, `failures`, `error` | | `release_phase_started` | `mode`, `limit`, `recheck_seconds` | | `release_phase_completed` | `ReleaseCounters` (`projects_checked`, `projects_created`, `versions_created`, `channels_created`, `statuses_created`, `unchanged`, `skipped`, `unsupported_no_purl`, `failures`) + `error` | | `release_affected_completed` | `AffectedCounters` (`projects_checked`, `projects_linked`, `versions_planned`, `versions_created`, `versions_label_only`, `cves_linked`, `unchanged`, `skipped`, `failures`, `unsupported`) + `error` | | `release_history_started` | `limit`, `concurrency` | | `release_history_project` | `project` plus flattened `HistoryJobCounters` (`enqueued`, `processed`, `versions_created`, `manifests_created`, `skipped`, `failed`, `reused`) | | `release_history_skipped` | `project`, `reason` | | `release_history_completed` | `HistoryModeCounters` (`projects_checked`, `projects_skipped`, `unsupported`, `enqueued`, `processed`, `versions_created`, `manifests_created`, `skipped`, `failed`, `reused`) + `error` | | `release_extras_started` | `limit` | | `release_extras_project` | `project`, `version` | | `release_extras_skipped` | `project`, `reason` | | `release_extras_completed` | `ExtrasCounters` (`projects_checked`, `projects_created`, `versions_created`, `manifests_created`, `statuses_created`, `unchanged`, `skipped`, `unsupported`, `failures`) + `error` | | `link_repair_completed` | `phase` (`startup`, `publish`, `software`, `release`), `attempted`, `repaired`, `failed`, `deferred`, `error` | | `s3_uploaded` | `bucket`, `path`, `bytes`, `seconds` | | `s3_upload_failed` | `bucket`, `path`, `bytes`, `error` | | `cycle_completed` | `cycle`, `duration`, full `CycleStats` | | `cycle_failed` | `cycle`, `error` | | `shutdown_requested` | none | | `fatal` | `error` | | `done` | none | `cycle_stats` carries the complete `CycleStats` object (NVD, publish, cache, S3, software, release, affected, history, extras, artifact, and link-repair counters). `run_stats` sums the `SUM_KEYS`, keeps maxima for `MAX_KEYS`, and uses the latest values for `LAST_KEYS`, then adds `cycles`, `started_at`, `pds_seconds_avg`, and `records_per_second`. ## Our prod run — last week, newest → oldest ```bash set -a; source ~/.nahdig_com_source; set +a; unset ATP_PDS_URL export ATP_HANDLE="$NAHDIG_COM_ATPROTO_HANDLE" export ATP_APP_PASSWORD="$NAHDIG_COM_ATPROTO_PASSWORD" export S3_ENDPOINT_URL="$CVE_NAHDIG_COM_S3_ENDPOINT_URL" export S3_BUCKET="$CVE_NAHDIG_COM_S3_BUCKET" export S3_ACCESS_KEY_ID="$CVE_NAHDIG_COM_AWS_ACCESS_KEY" export S3_SECRET_ACCESS_KEY="$CVE_NAHDIG_COM_AWS_SECRET_ACCESS_KEY" export S3_REGION="$CVE_NAHDIG_COM_S3_REGION" export S3_PATH="prod-cve-nahdig-com-week" # first cycle fetches 7 days; repeat while pending > 0 vuln-notif --once \ --initial-lookback-days 7 --limit 500 --order newest \ --publish-atproto --software-limit 100 \ --purldb .state/purl2cpe.db \ --cache-db .state/prod-week.db ``` Loop until pending reaches 0: ```bash while :; do vuln-notif --once \ --initial-lookback-days 7 --limit 500 --order newest \ --publish-atproto --software-limit 100 \ --purldb .state/purl2cpe.db \ --cache-db .state/prod-week.db REPO=$(sqlite3 .state/prod-week.db \ "select repo_did from publications limit 1;") PENDING=$(sqlite3 .state/prod-week.db \ "select count(*) from cves c left join publications p on p.repo_did='$REPO' and p.cve_id=c.id where p.status is null or p.status='pending';") echo "pending=$PENDING" [ "$PENDING" -eq 0 ] && break sleep 1800 # PDS repo-write-hour cap done ``` `--limit` advances the NVD cursor; remaining rows stay pending. The cache is resumable; a 429 only delays the loop. `--backfill` deliberately ignores the stored cursor and starts from 1999; each fetched page replaces the stored cursor, so stop and resume works the same way as a normal incremental run. ## Release records `--publish-releases` publishes the default/latest version for software already in the cache. Candidate rows resolve through `software_identity`, so one canonical project receives all raw aliases; legacy raw-slug projects such as `red-hat-red-hat-enterprise-linux-8` are no longer created. Projects use the canonical identity slug (`red-hat-enterprise-linux`), and raw Raw aliases stay in the local cache and inside CVE records; no `com.nahdig.software` records are published. ```bash vuln-notif --once \ --publish-atproto --publish-releases --release-mode latest \ --release-limit 100 --release-recheck-seconds 86400 \ --cache-db .state/prod-week.db ``` `--release-mode latest` is the default value but is a no-op on its own: standalone latest requires `--publish-releases`. `--release-mode all` runs the latest pass unconditionally without that switch. - Writes `software.release.project`, an immutable `manifest`, an immutable `version`, a mutable `channel` (`latest` for this mode), and optional `status` records. A version or manifest is never overwritten; a channel update only moves its `version` pointer. - When the purl type has a source adapter, also writes `com.nahdig.release.projectMeta` (same-DID strongRef, only when the content changed) and records `publishedAt`/`rawVersion`/artifact cid+size in `release_version_meta`. Manifest artifacts gain `metadata` git fragments and a `target` object when the source declares them. - Statuses are append-only `software.release.status` records plus `release_status_events` rows. A deps.dev deprecation without a registry signal goes through the same status module; an identical re-observation is a no-op and a reason change appends a new event. CVE presence never changes a status. - Canonical observations supply the project, channel, and version identity. `com.nahdig.cve.atprotoV1` emits `channels`/`versions`/`manifests` strongRefs only for records that exist; otherwise it uses `versionLabels` and `affectedRanges`. - Sources: deps.dev for versions/defaults; PyPI, crates.io, and RubyGems give SHA-256 directly; npm, Maven, NuGet, and Go are downloaded once per artifact to compute SHA-256. - `--release-recheck-seconds` skips projects checked recently. Use `0` to force a fresh check. - PDS writes still pass through the rate governor; every release record write uses the same 5k/hour and 35k/day budget. ### Affected mode `--release-mode affected` maps OSV advisories onto the release catalog and updates the CVEs already linked to each canonical project. ```bash # OSV online (default), one resumable page per cycle. while :; do vuln-notif --once \ --release-mode affected --release-limit 100 --release-recheck-seconds 86400 \ --cache-db .state/prod-week.db sleep 1800 done # Network-free / reproducible: load advisory JSON from disk instead of OSV. vuln-notif --once \ --release-mode affected --release-limit 100 --release-recheck-seconds 0 \ --osv-bulk-dir /srv/osv-bulk --osv-offline \ --cache-db .state/affected.db ``` - OSV is queried online by default (`--osv-online` defaults to true). Use `--osv-offline` with `--osv-bulk-dir`, or `--osv-online=false` with `--osv-bulk-dir`; a bulk directory implies offline. Offline without a bulk directory is a usage error (exit 2). Bulk loading reads every `*.json` advisory in the directory and filters by purl per project. - `--release-ecosystems` filters purl types (`pypi`, `npm`, `rpm`, ...) and `--release-sources` filters sources (`osv` matches all affected work). - `--release-concurrency` is accepted but affected work runs in project order in this build; the flag affects history enumeration only. - The catalog is existing `release_versions` rows plus deps.dev versions when the purl type has a deps.dev system. Affected versions without a local row get an immutable `manifest`+`version` only when an artifact is fetchable; otherwise they remain `versionLabels`. No version is invented. - Channels point at the highest known version row in the stream and are only updated when that pointer changes. CVE records are `putRecord` only when the `$type`, `cve`, `nvd`, or `software` content changes. - `--release-mode extras` and `all` are documented in [Extras mode](#extras-mode) and [All mode](#all-mode). - Write budget: create/put costs 3 points, update 2, delete 1; `applyWrites` charges every record. A first affected pass can add two immutable records (manifest + version) per planned version, plus one channel point and one CVE update per linked CVE. Budget with `--release-limit` and the governor's `pds_budget_*_remaining` fields. - Per-project source, database, and PDS failures are isolated: the cycle continues and `cycle_stats.release_affected_failures` increases. Every 10 projects commit their local rows. Verification after a run: ```bash # Cycle counters (all release_affected_* fields are in cycle_stats). grep '"event":"cycle_stats"' run.ndjson | tail -1 | jq '{ mode: .release_mode, checked: .release_affected_projects_checked, linked: .release_affected_cves_linked, planned: .release_affected_versions_planned, created: .release_affected_versions_created, labels: .release_affected_versions_label_only, failures: .release_affected_failures }' # CVE refs for one project. curl -sS -G "$ATP_PDS_URL/xrpc/com.atproto.repo.getRecord" \ --data-urlencode "repo=$DID" \ --data-urlencode 'collection=com.nahdig.cve.atprotoV1' \ --data-urlencode 'rkey=CVE-2024-3094' | jq '.value.software[0] | {project, channels, versions, versionLabels}' # Local link index rows written by the run. sqlite3 .state/prod-week.db \ "SELECT path, count(*) FROM links WHERE path LIKE 'software[].%' GROUP BY path;" ``` ### Metadata and status verification (all modes) ```bash # Project metadata (same-DID strongRef; name equals the rkey). curl -sS -G "$ATP_PDS_URL/xrpc/com.atproto.repo.getRecord" \ --data-urlencode "repo=$DID" \ --data-urlencode 'collection=com.nahdig.release.projectMeta' \ --data-urlencode 'rkey=acme-widget' | jq '.value | {project, name, versioningScheme, repository}' # Manifest artifact metadata: git facts and the runtime target. curl -sS -G "$ATP_PDS_URL/xrpc/com.atproto.repo.getRecord" \ --data-urlencode "repo=$DID" \ --data-urlencode 'collection=software.release.manifest' \ --data-urlencode 'rkey=' | jq '.value.artifacts[] | {metadata, target}' # Version facts plus the full append-only status history for one version. sqlite3 .state/prod-week.db \ "SELECT project_slug, version, source, published_at, raw_version FROM release_version_meta ORDER BY project_slug, version LIMIT 20; SELECT status, kind, reason, created_at FROM release_status_events WHERE subject_uri='at://$DID/software.release.version/acme-widget:1.2.3' ORDER BY created_at;" ``` Append-only semantics: a status event is never updated or deleted. Current state is the latest live same-DID `software.release.status` for the subject; a registry reason change appends a new record under a new digest rkey, and the old record stays addressable. `projectMeta`, statuses, and version metadata are compared before writing, so an unchanged second run makes zero PDS writes. If the PDS does not support `listRecords`, projectMeta writes are skipped because "put only when changed" cannot be verified. ## History mode `--release-mode history` enumerates each candidate's full source catalog and publishes immutable `manifest` + `version` records. Jobs are durable and resumable, so run it as a long-lived loop with small pages: ```bash # One project page per cycle; 100 versions per project; 4 projects at a time. while :; do vuln-notif --once \ --release-mode history --release-limit 25 \ --release-history-limit 100 --release-concurrency 4 \ --release-ecosystems pypi,cargo,gem \ --release-recheck-seconds 604800 \ --artifact-max-bytes 104857600 \ --cache-db .state/prod-history.db sleep 900 done # Network-free source override for the local e2e (composer/cpan/apache/gnu/gitee). vuln-notif --once \ --release-mode history --release-recheck-seconds 0 \ --release-source-url http://127.0.0.1:4850 \ --cache-db /tmp/history-e2e.db ``` - Queue order is `(updated_at, project, raw version)`. A job becomes `retry` after a transient artifact failure and terminal `error` after three failed attempts. Re-running the mode never resets a terminal job; inspect `release_history_jobs.error` before requeueing by hand. - `--release-history-limit` is per project per cycle. `--release-limit` bounds candidate projects, and `--release-concurrency` bounds how many projects are enumerated at once; writes still go through the single PDS governor. - Immutable records are reused, never rewritten. A version that already exists is skipped at enqueue time, and a manifest whose URI/CID are recorded is reused. - History adapters: PyPI, crates.io, RubyGems, GitHub, GitLab, Bitbucket, distro (rpm/deb/apk), Packagist, MetaCPAN, Apache, Gitee, GNU. npm, Maven, NuGet, Go, Docker/OCI, and SourceForge are typed `unsupported` in history: the first four are covered by `latest` via deps.dev, and Docker/OCI plus SourceForge are covered by `extras`. - `--release-source-url` applies to the history adapters with configurable base URLs: `composer`, `cpan`, `apache`, `gitee`, and `gnu`. It is a local e2e/test seam, not a production mirror setting. - Token environment variables: `GITHUB_TOKEN`, `GITLAB_TOKEN`, `GITEE_TOKEN` (or the matching flags). Tokens are sent only as request headers (Gitee uses `Authorization: Bearer`) and are redacted from config debug output and logs. They never appear in URLs, cycle stats, or events. - Write budget: every new version costs two records (manifest + version = 6 points); reused rows and skipped jobs cost nothing. Statuses are re-planned for versions materialized in the cycle and for deprecated catalog entries, so a registry reason change appends an event without rewriting history. A full-history pass over 256k–884k versions is roughly 797k–2.68M points (23–77 days at the default 35k/day budget). Throttle with `--release-history-limit` and `--release-recheck-seconds`; the governor reports `pds_budget_*_remaining` in every cycle. ## Extras mode `--release-mode extras` covers every registered extras adapter, including the deps.dev ecosystems pypi, cargo, and gem. One project per cycle page: newest non-prerelease/non-draft version, one source-declared artifact, then project / manifest / version / `latest` channel / status / `release.projectMeta` records. ```bash # One resumable page per cycle; rotate through the candidate set. while :; do vuln-notif --once \ --release-mode extras --release-limit 100 --release-recheck-seconds 86400 \ --release-ecosystems pypi,github,rpm,docker,composer \ --release-sources github,docker \ --artifact-max-bytes 104857600 \ --cache-db .state/prod-extras.db sleep 1800 done # Network-free local e2e (composer stub on 127.0.0.1). scripts/release-extras-e2e.sh ``` - Candidate selection uses the same least-recently-checked rotation as the other modes; `--release-limit` bounds the page and every 10 projects commit local rows. Filters are applied before the source is built: a project whose purls are all excluded is counted as `skipped` and keeps its old check time so a later run with different filters still visits it. - Per-project isolation: source, artifact, database, and PDS failures increment `release_extras_failures` and the cycle continues. Unknown purl types and not-yet-adapted ecosystems are typed `unsupported`: the reason is written to `release_project_state.unsupported_reason` and emitted as `release_extras_skipped`. - Freshness is durable: `release_project_state` stores the canonical purl and adapter, and the `release_projects` row caches the published URIs. A second run with the same newest version is `unchanged` and performs no PDS writes; registry status changes are evaluated when a version is (re)materialized, not on the unchanged fast path. - `--dry-run` skips every release mode: only NVD fetch/cache runs, with no credentials and no PDS writes. Library-level extras dry-run planning exists but is not reachable from the CLI in this build. - `--release-source-url` applies to extras adapters with configurable base URLs: `composer`, `cpan`, `apache`, `gitee`, `gnu`, and `sourceforge`. - `--release-sources` matches purl types and adapter names (`docker` matches `docker`/`oci`, `distro` matches `rpm`/`deb`/`apk`); `osv` only affects affected work. - Source statuses are append-only: an identical signal is a no-op and a reason change appends a new `software.release.status` event. Manifests carry the adapter's git/target facts; `release_version_meta` records the version's `publishedAt`/`rawVersion` when the source declares them. ### Source matrix The full per-adapter matrix (version source, artifact/digest path, status signals, unsupported reasons, and RHEL/UBI mapping) is [`docs/release-sources.md`](docs/release-sources.md). Operational summary: | Purl types | Adapter | Auth | Versions / artifacts / statuses / meta | Notes | |---|---|---|---|---| | `pypi` | PyPI JSON API | none | yes / yes / yanked / yes | PEP 503 names, declared SHA-256 | | `cargo` | crates.io | none | yes / yes / yanked / yes | adapter throttles to 1 req/s | | `gem` | RubyGems | none | yes / yes / yanked / yes | declared SHA-256, HEAD size fallback | | `github` | GitHub releases + tags | `GITHUB_TOKEN` | yes / yes / draft+prerelease+archived / yes | asset digest, archive fallback | | `gitlab` | GitLab releases + tags | `GITLAB_TOKEN` | yes / yes / upcoming+archived / yes | `PRIVATE-TOKEN`, pagination links | | `bitbucket` | Bitbucket tags + downloads | none | yes / yes / no / yes | no declared digest | | `rpm`, `deb`, `apk` | distro lifecycle + public UBI OCI | none | yes / RHEL OCI only / EOL / yes | endoflife.date + Red Hat PLC | | `composer` | Packagist p2 | none | yes / dist zip / abandoned / yes | shasum used as SHA-256 only when 64 hex | | `cpan` | MetaCPAN | none | yes / tarball / no status record / yes | backpan entries are marked deprecated but statuses are typed `NoStatuses` | | `apache` | Apache download indexes | none | yes / archive files / no / yes | sidecar digests disabled; SHA-256 computed | | `gitee` | Gitee releases | `GITEE_TOKEN` bearer | yes / assets / prerelease / yes | token never in the URL | | `gnu` | GNU FTP index | none | yes / tarballs / no / yes | index discovery | | `docker`, `oci` | Docker Hub registry | anonymous pull token | tags / manifest digest / tag-mutation / yes | immutable digest artifact | | `sourceforge` | SourceForge RSS/files | none | best effort / files / no / yes | typed unavailable reasons | | `npm`, `maven`, `nuget`, `golang` | deps.dev in `latest`; registry adapters in `history`/`extras` | none | yes / yes / registry signals / yes | full history supported | - Tokens: `GITHUB_TOKEN`, `GITLAB_TOKEN`, `GITEE_TOKEN`; all other adapters are anonymous. Header values are never logged, and Gitee uses `Authorization: Bearer ` rather than `access_token`. - Rate limits: the shared registry transport retries 429/5xx with 1s/2s/4s backoff and honours response pagination caps; crates.io requests are spaced 1s apart; GitHub/GitLab surface rate-limit headers as typed failures. PDS writes are governed by `PDS_WRITE_HOURLY`/`PDS_WRITE_DAILY` exactly like every other mode. - Write budget: a fresh project costs about 18 points (project 3, projectMeta 3, manifest 3, version 3, channel 3, status 3 per registry signal). Re-runs are unchanged and cost zero. Plan blocks with `--release-limit` and inspect `release_extras_*` before scaling. - Artifact downloads honour `--artifact-max-bytes`; declared digests are passed to the resolver, which verifies them against the downloaded bytes and caches the result in `release_artifacts`. Verification after an extras run: ```bash grep '"event":"cycle_stats"' run.ndjson | tail -1 | jq '{ mode: .release_mode, checked: .release_extras_projects_checked, created: .release_extras_projects_created, versions: .release_extras_versions_created, manifests: .release_extras_manifests_created, statuses: .release_extras_statuses_created, unchanged: .release_extras_unchanged, skipped: .release_extras_skipped, unsupported: .release_extras_unsupported, failures: .release_extras_failures }' # Typed unsupported reasons, never silent skips. sqlite3 .state/prod-extras.db \ "SELECT slug, unsupported_reason FROM release_project_state WHERE unsupported_reason IS NOT NULL;" # One project's newest version and its manifest strongRef. curl -sS -G "$ATP_PDS_URL/xrpc/com.atproto.repo.getRecord" \ --data-urlencode "repo=$DID" \ --data-urlencode 'collection=software.release.version' \ --data-urlencode 'rkey=acme-widget:1.2.3' | jq '.value' ``` ## All mode `--release-mode all` runs latest → affected → history (only when `--release-history-limit > 0`) → extras in one cycle. All modes share the candidate cursor and one artifact resolver; each keeps its own counters: - `release_*` for latest, `release_affected_*` for affected, `release_history_*` for history, `release_extras_*` for extras. - `artifact_*` aggregates every resolve in the cycle. - `--release-sources`/`--release-ecosystems` filter affected, history, and extras connector work. - Use it for a small daily sweep only. A full `all` cycle over a production cache can combine the OSV, history, and extras write budgets; keep `--release-limit`, `--release-history-limit`, and `--release-recheck-seconds` conservative. ```bash vuln-notif --once \ --release-mode all --release-limit 100 --release-recheck-seconds 86400 \ --release-history-limit 20 --release-concurrency 4 \ --release-ecosystems pypi,github,composer \ --artifact-max-bytes 104857600 \ --cache-db .state/all.db ``` ## Write-budget schedules by mode Planning numbers below assume one canonical project per product and the default 35,000-point/day budget (31,500 after the 0.9 safety margin). They are estimates for capacity planning, not measured prod totals. | Work | Records / updates | Approx. points | Elapsed | |---|---:|---:|---:| | Identity migration: CVE refs | ~14.3k updates | ~28.7k | ~1 day | | Identity migration: software/purl refs + legacy deletes | ~100 | <1k | hours | | `latest` / default (canonical) | ~19.2k records | ~57.6k | ~1.6 days | | `affected` (OSV) | 100k–300k records | 330k–930k | 9–27 days | | `history` (full catalogs) | 256k–884k records | 797k–2.68M | 23–77 days | | `extras` | ~35.7k PURLs | source-dependent | source-dependent | Per-mode record costs: - `latest`: project 3 + manifest 3 + version 3 + channel 3 (+ status 3 per registry signal, + projectMeta 3 once). Re-observing an unchanged version costs zero. A channel update is a 2-point update only when the pointer moves. - `affected`: immutable manifest + version (6 points) per affected version with a fetchable artifact, one channel update when the pointer moves, and one CVE `putRecord` update (2 points) per linked CVE only when refs change. `versionLabels`/`affectedRanges`-only versions cost zero PDS points. - `history`: manifest + version (6 points) per new version; reused records and skipped jobs cost zero. Jobs persist in `release_history_jobs`; status planning may append a status record on a later pass. - `extras`: project 3 + projectMeta 3 + manifest 3 + version 3 + channel 3 (+ status 3 per registry signal). Unchanged projects cost zero. - `all`: sums the mode costs in one cycle; keep `--release-limit`, `--release-history-limit`, and `--release-recheck-seconds` conservative. Recommended multi-day order: migration (`snapshot → plan → review → apply → verify`) first, then `latest --release-limit 100`, then `affected`, then `history` for weeks, then `extras` with ecosystem filters. Long runs are resumable: ```bash # Foreground loop: one bounded page per cycle. while :; do vuln-notif --once --release-mode history \ --release-limit 25 --release-history-limit 100 --release-concurrency 4 \ --cache-db .state/prod-history.db sleep 900 done # Stop cleanly: SIGINT/SIGTERM drains the in-flight phase, commits, and exits. # Resume by running the same command again; cursors, retry queues, and # rate_budget live in the cache DB. ``` ### Artifact cache and S3 mirror - Verified artifact metadata is stored in `release_artifacts`, keyed by `(repo, source, package, raw version, url)`. Confirm hits with `cycle_stats.artifact_cache_hits`; the second resolve must not increment `artifact_downloads`. - `--artifact-max-bytes N` rejects oversized streams before/while reading. `0` (default) means no cap; set an explicit cap in production. - `--artifact-s3-bucket` plus `--artifact-s3-path` (default `artifacts`) mirrors raw bytes. The mirror uses the same S3 endpoint and credentials as the cache snapshot (`S3_ENDPOINT_URL`, `S3_ACCESS_KEY_ID`, `S3_SECRET_ACCESS_KEY`, `S3_REGION`); verify with `cycle_stats.artifact_mirror_puts`. Mirror failures fall back to the registry and increment `artifact_errors` only when the resolve itself fails. - Without `--artifact-s3-bucket` the mirror is a no-op: `artifact_mirror_puts` stays `0` and only the local digest cache is consulted. - Cache snapshot uploads (`S3_BUCKET`/`S3_PATH`) are fetched before NVD work, then `VACUUM INTO` + upload after each cycle. Uploads get 4 attempts with 1s/2s/4s backoff plus 0.25s jitter and a post-upload size check; the final failure is logged as `s3_upload_failed` and returned, never swallowed. A missing remote cache is a normal first run. ### Publish retries and link repair - CVE publish failures are recorded in `publish_errors` with an attempt count and `next_retry_at = now + 30 * 2^(attempts-1)` seconds, capped at one hour, plus up to 20% jitter. `drain_retries` runs before and after the publish phase, up to 100 due rows per pass. Success marks the publication present/created and clears the retry row; a CVE that left the cache has its retry row deleted. - Local link indexing is part of the write contract: PDS write → index links (or queue the record) → DB commit → advance cursor. An indexing failure serializes the written record into `link_repairs` in the same SQLite transaction, so a crash cannot lose either the link rows or the repair row. - `link_repairs` is drained at startup and after the publish, software, and release phases, 200 rows per drain. Retries use the same `30 * 2^(attempts-1)` formula capped at one hour plus jitter. A drain makes no PDS calls: it reindexes from the stored record JSON and is idempotent because indexing replaces the source's link rows. - `link_repair_*` counters are in `cycle_stats`; each drain emits `link_repair_completed` with `phase`, `attempted`, `repaired`, `failed`, `deferred`, and `error`. Verification after a history run: ```bash grep '"event":"cycle_stats"' run.ndjson | tail -1 | jq '{ mode: .release_mode, projects: .release_history_projects_checked, skipped: .release_history_skipped, unsupported: .release_history_unsupported, enqueued: .release_history_enqueued, processed: .release_history_processed, created: .release_history_created, failed: .release_history_failed, reused: .release_history_reused, downloads: .artifact_downloads, bytes: .artifact_bytes, cache_hits: .artifact_cache_hits, mirror_puts: .artifact_mirror_puts }' # Queue state: pending/retry/error must be inspected before a long run. sqlite3 .state/prod-history.db \ "SELECT status, count(*) FROM release_history_jobs GROUP BY status;" # One immutable version and its manifest strongRef. curl -sS -G "$ATP_PDS_URL/xrpc/com.atproto.repo.getRecord" \ --data-urlencode "repo=$DID" \ --data-urlencode 'collection=software.release.version' \ --data-urlencode 'rkey=acme-widget:1.2.3' | jq '.value' # Local e2e: fake composer source, limit/resume, artifact counters. scripts/release-history-e2e.sh ``` ## Software identity migration `identity-migrate` is a one-time, out-of-band admin binary. The service never runs it. Stop `vuln-notif` before `apply` and keep it stopped until `verify` exits `0`; never run `apply` concurrently with a service cycle. The full runbook, including the failure-recovery matrix and snapshot restore script, is [`docs/identity-migration.md`](docs/identity-migration.md). The rules are in [`docs/identity-model.md`](docs/identity-model.md). Sequence: **snapshot → plan → review diff → apply --resume → verify**. ```bash set -a; source ~/.nahdig_com_source; set +a; unset ATP_PDS_URL export ATP_HANDLE="$NAHDIG_COM_ATPROTO_HANDLE" # repo DID/handle, not a secret export ATP_APP_PASSWORD="$NAHDIG_COM_ATPROTO_PASSWORD" DB=.state/prod-week.db MAP=migration/identity-$(date +%Y%m%dT%H%M%S) cp "$DB" "$DB.pre-migration" # cache, cursor, rate-budget backup # 1. Snapshot every migration collection to JSONL (read-only). target/release/identity-migrate snapshot \ --cache-db "$DB" --snapshot-dir "$MAP.snapshot" # 2. Inventory cache + remote listRecords and write a reviewable plan. target/release/identity-migrate plan \ --cache-db "$DB" --purldb .state/purl2cpe.db \ --out "$MAP.plan.json" # 3. Review every legacy -> canonical pair before writing. jq -r '.entries[] | .action' "$MAP.plan.json" | sort | uniq -c jq -r '.entries[] | select(.action == "create_project") | .canonical_slug' "$MAP.plan.json" jq -r '.entries[] | select(.action == "delete_release_record") | [.legacy_slug, .canonical_slug, .legacy_uri] | @tsv' "$MAP.plan.json" | sort -u # 4. Apply in dependency order (projects, software, purl, CVE, status, # normalize, deletes) with a persisted cursor. 200-record applyWrites # delete batches. target/release/identity-migrate apply \ --cache-db "$DB" --map "$MAP.plan.json" \ --batch 200 --snapshot-dir "$MAP.snapshot" # If step 4 exits nonzero, rerun the same command with --resume: target/release/identity-migrate apply \ --cache-db "$DB" --map "$MAP.plan.json" \ --batch 200 --snapshot-dir "$MAP.snapshot" --resume # 5. Re-read the repository and fail on stale refs/backlinks. target/release/identity-migrate verify \ --cache-db "$DB" --map "$MAP.plan.json" ``` Useful flags (all global; `identity-migrate --help`): | Flag | Use | |---|---| | `--cache-db` | SQLite cache, migration cursor, and rate-budget state | | `--purldb` | purl2cpe SQLite for planning; absent is tolerated | | `--out` / `--map` | Plan output / plan input for apply and verify | | `--batch` | Records per `applyWrites` delete batch; default `200` | | `--limit` | Max plan entries per invocation (`0` = all), for chunked runs | | `--snapshot-dir` | JSONL snapshot; accepted by `plan` and `apply` | | `--resume` | Retry `pending`/`error` entries, skip `done` | | `--dry-run` | Plan only; valid only for `plan` | | `--apply` / `--verify` | Flag forms of the subcommands; can combine | The subcommand forms are `plan`, `apply`, `verify`, `snapshot`, and `normalize-purl-slugs`; the last one rewrites legacy full-AT-URI `software_purls.software_slug` values in the cache and needs no credentials. NDJSON events on stdout (one line per run): | Event | Fields | |---|---| | `identity_snapshot` | `records`, `dir` | | `identity_plan` | `entries`, `created`, `deleted`, `out` | | `identity_apply` | `report`: `applied`, `skipped_done`, `created_projects`, `software_repoints`, `purl_repoints`, `cve_repoints`, `status_repoints`, `deleted`, `normalized` | | `identity_verify` | `report`: `canonical_projects`, `legacy_projects`, `software_repoints`, `purl_repoints`, `cve_repoints`, `status_repoints`, `deletions`, `stale_refs`, `errors` | | `identity_normalize_purl_slugs` | `repo_did`, `updated` | `verify` exits `1` when `errors` is non-empty. `apply` exits nonzero and leaves the failing entry `error`; `--resume` retries it. ### Write budget and batching | Operation | Points | |---|---:| | create / put (new record) | 3 | | update (`putRecord` on existing record) | 2 | | delete | 1 | | `applyWrites` | each record at its own operation cost | The migration is dominated by CVE ref updates (~14.3k updates × 2 ≈ 28.6k points, close to one 35k/day budget) plus software/purl ref updates and ~100 legacy deletes (<1k). Run it as a day-long job. `apply` persists one cursor row per `(repo_did, phase, raw_slug)` in `identity_migration`, so `--resume` skips completed entries and stopping does not lose budget. The governor still enforces 5,000 points/hour and 35,000/day with a 0.9 safety margin. ### Rollback The pre-apply snapshot plus the cache DB copy is the rollback material. There is intentionally no destructive rollback command; restore is a manual `putRecord` replay from the snapshot, documented in `docs/identity-migration.md#rollback-via-snapshot`. Restore all collections (project, software, purl, CVE), not only projects, or strongRefs stay broken. Restore the cache DB copy to roll back the cursor and rate-budget rows too. ### Verification recipe Canonical project backlinks (expect a non-zero total and canonical project URI): ```bash PDS=${ATP_PDS_URL:-https://your-pds.example} DID=$(goat account check-auth | sed -n 's/^DID: //p') PROJ=red-hat-enterprise-linux PROJECT="at://$DID/software.release.project/$PROJ" curl -sS -G 'https://constellation.microcosm.blue/xrpc/blue.microcosm.links.getBacklinks' \ --data-urlencode "subject=$PROJECT" \ --data-urlencode 'source=com.nahdig.cve.atprotoV1:software[].project.uri' | jq . ``` Zero legacy backlinks — query each legacy RHEL slug and expect `total: 0`: ```bash for LEGACY in red-hat-red-hat-enterprise-linux-8 \ red-hat-red-hat-enterprise-linux-9 \ redhat-enterprise-linux \ redhat-enterprise-linux-eus \ redhat-rhel-els; do SUBJECT="at://$DID/software.release.project/$LEGACY" TOTAL=$(curl -sS -G 'https://constellation.microcosm.blue/xrpc/blue.microcosm.links.getBacklinks' \ --data-urlencode "subject=$SUBJECT" \ --data-urlencode 'source=com.nahdig.cve.atprotoV1:software[].project.uri' | jq -r .total) printf '%s: %s\n' "$LEGACY" "$TOTAL" done ``` The local equivalent checks the `links` table for unmanaged backlinks: ```bash sqlite3 "$DB" " SELECT subject_uri, count(*) FROM links WHERE subject_uri LIKE 'at://$DID/software.release.project/%' AND subject_uri NOT LIKE '%/red-hat-enterprise-linux' GROUP BY subject_uri;" ``` `verify` performs the authoritative check across the repository; the queries above are the quick prod spot-check. Also confirm the canonical project exists (`getRecord ... software.release.project/red-hat-enterprise-linux`), the CVE has one canonical ref plus its preserved `nvd` strongRef, and a second `apply` reports `applied == 0` and `deleted == 0`. ## Local e2e order Services: hono-plc `:2587`, atproto-pds `:4800`, RustFS `:9000` (S3 test only). Start **PLC → PDS → account**; the PDS must run with `PDS_LEXICON_DIR=$PWD/lexicons` and `PDS_DID_PLC_URL=http://127.0.0.1:2587`. ```bash scripts/local_plc.sh start scripts/local_atproto_pds.sh start scripts/create_local_account.sh scripts/local-e2e.sh scripts/blob-e2e.sh scripts/local_rustfs.sh start scripts/s3-cache-e2e.sh scripts/identity-migrate-e2e.sh scripts/release-affected-e2e.sh scripts/release-history-e2e.sh scripts/release-extras-e2e.sh scripts/check-lexicons.sh ``` `local-e2e.sh` and `identity-migrate-e2e.sh` are self-contained and start their own temporary PLC/PDS; the explicit start commands above are for a persistent dev setup. ## Delete old collection ```bash OLD=com.nahdig.cve # paginated listRecords + applyWrites delete (200/batch), then: goat record delete --rkey "$OLD" com.atproto.lexicon.schema ``` ## Verify blob record ```bash PDS=${ATP_PDS_URL:-https://your-pds.example} DID=$(goat account check-auth | sed -n 's/^DID: //p') REC=$(curl -sS "$PDS/xrpc/com.atproto.repo.getRecord?repo=$DID&collection=com.nahdig.cve.nvdV2&rkey=CVE-2021-3942") CID=$(jq -r '.value.dataBlob.ref["$link"]' <<<"$REC") curl -sS "$PDS/xrpc/com.atproto.sync.getBlob?did=$DID&cid=$CID" | gunzip | jq .id ``` ## Verify backlinks Check the canonical project and its channel/version refs: ```bash PROJ=red-hat-enterprise-linux PROJECT="at://$DID/software.release.project/$PROJ" CHANNEL="at://$DID/software.release.channel/$PROJ:9" VERSION="at://$DID/software.release.version/$PROJ:9.4" for PAIR in "project:$PROJECT:software[].project.uri" \ "channel:$CHANNEL:software[].channels[].uri" \ "version:$VERSION:software[].versions[].uri"; do LABEL=${PAIR%%:*} REST=${PAIR#*:} SUBJECT=${REST%:*} SOURCE=${REST##*:} echo "== $LABEL $SUBJECT" curl -sS -G 'https://constellation.microcosm.blue/xrpc/blue.microcosm.links.getBacklinks' \ --data-urlencode "subject=$SUBJECT" \ --data-urlencode "source=com.nahdig.cve.atprotoV1:$SOURCE" | jq . done ``` For zero legacy backlinks, query the legacy RHEL slugs listed in the migration section and assert `total: 0`; `identity-migrate verify` is the authoritative repository-wide check.