~ knotless Search/
↑↓ move↵ open⇧↵ all results

johnandersen777.bsky.social/vuln-notif

0
0
main
993 lines · 45.1 KB Raw History

Maintainers

Prod and ops notes for the vuln-notif repository. Flags and events below are verified against src/cli.rs, src/bin/identity-migrate.rs, and the PDS governor. Release source capabilities live in docs/release-sources.md; module/data-flow/event contracts live in docs/release-architecture.md.

DNS records

set -a; source ~/.nahdig_com_source; set +a

# com.nahdig.cve authority
dig +short TXT _lexicon.nahdig.com       # must match logged-in DID

# com.nahdig.cve.* authority (replace did:plc:example with the logged-in DID)
doctl compute domain records create \
  --record-name _lexicon.cve --record-ttl 3600 --record-type TXT \
  --record-data did=did:plc:example nahdig.com
dig +short TXT _lexicon.cve.nahdig.com   # must match logged-in DID

Negative DNS caches can keep goat lex resolve failing after the record is live; wait for the negative TTL or flush the resolver.

Publish lexicons

Publishing is an external ops step; the service binary never writes lexicons. Use goat (below) or the equivalent atpxrpc call against the account's PDS. Validate locally first:

scripts/check-lexicons.sh

The script walks all of lexicons/ recursively and jq-validates every *.json; it has no hardcoded list. The publish list is the ten vendored documents: four com.nahdig.* schemas (two record types plus the gitMeta and projectMeta definitions) and six software.release.* schemas (defs plus the five record types).

set -a; source ~/.nahdig_com_source; set +a
cd vuln-notif
GOAT_USERNAME="$NAHDIG_COM_ATPROTO_HANDLE" \
GOAT_PASSWORD="$NAHDIG_COM_ATPROTO_PASSWORD" \
  goat account login
goat account check-auth

LEXICONS=(
  lexicons/com/nahdig/cve.nvdV2.json
  lexicons/com/nahdig/cve.atprotoV1.json
  lexicons/com/nahdig/release.projectMeta.json
  lexicons/com/nahdig/release.gitMeta.json
  lexicons/software.release.defs.json
  lexicons/software.release.project.json
  lexicons/software.release.channel.json
  lexicons/software.release.version.json
  lexicons/software.release.manifest.json
  lexicons/software.release.status.json
)
for f in "${LEXICONS[@]}"; do
  id=$(jq -r .id "$f")
  jq '. + {"$type":"com.atproto.lexicon.schema"}' "$f" > "/tmp/schema-$id.json"
  goat record create --rkey "$id" "/tmp/schema-$id.json"
done

If goat lex publish prints green but goat lex resolve says RecordNotFound, use the goat record create loop above. Publish software.release.defs before the record lexicons that reference it, and publish the com.nahdig.* record schemas before the software.release.* records that reference their fields. com.nahdig.release.gitMeta is a definition document embedded in manifest artifact metadata; it is still published as a lexicon schema record so clients can resolve the type.

NVD API key

  1. Request a free key: https://nvd.nist.gov/developers/request-an-api-key
  2. Receive it by email.
  3. Use it:
export NVD_API_KEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
# or:
vuln-notif --nvd-api-key "$NVD_API_KEY" ...

The key is sent as the apiKey request header. Without it: 5 requests/30s and ~6s between calls. With it: 50 requests/30s.

Rate limits

ServiceLimit
NVD API, no key5 requests / 30s
NVD API, key50 requests / 30s
PDS writes (per DID)5,000 points/hour, 35,000/day
PDS CREATE / UPDATE / DELETE3 / 2 / 1 points
Effective PDS creates1,666/hour, 11,666/day
Hosted API requests (per IP)3,000 / 5 min
createSession30 / 5 min, 300/day per account
createAccount100 / 5 min per IP
Blob upload size50 MiB

applyWrites counts every record in the batch toward the write budget. Last-week run: 14,349 CVEs = 43,047 points for nvdV2 alone (> daily budget); adding atprotoV1 doubles it. Minimum ~2 days for nvdV2, ~3 days with both formats. uploadBlob costs 0 write points.

The binary handles both limits itself: the native PDS governor paces every PDS call (default 10 requests/s, burst 100, below 3,000 requests/5 min) and reserves write points before create/put/delete/applyWrites. On 429 it honors Retry-After (exact sleep) or RateLimit-Reset, otherwise backs off 60s → 1800s with jitter, and retries instead of failing the row. createSession 429s retry with max(300, Retry-After). The NVD client does its own token-bucket pacing and outer retry (30/60/120/300s + jitter).

Per-source limits and retry behavior:

SourceLimit / policy
NVD API5 requests/30s without a key, 50/30s with NVD_API_KEY; 403/5xx/timeout retried at 30/60/120/300s + jitter
Shared registry HTTP4 attempts, 1s/2s/4s backoff on 429 and 500/502/503/504; 60s timeout
crates.io (cargo)adapter throttles every JSON request to 1 req/s per source instance
GitHub (github)optional GITHUB_TOKEN; honors x-ratelimit-remaining/x-ratelimit-reset/Retry-After, fails typed on 429 or exhausted 403
GitLab (gitlab)optional GITLAB_TOKEN (PRIVATE-TOKEN header); response pagination links followed
Docker Hub (docker, oci)anonymous pull token fetched per operation from auth.docker.io; sent only in Authorization
OSV (osv)online query/querybatch; querybatch is split at 1,000 queries; offline bulk directory reads *.json without network
PDS5,000 points/hour, 35,000/day, 3,000 requests/5 min; create/put 3, update 2, delete 1

Every release-mode registry transport is wrapped in GovernedHttp: a per-host token bucket (5 req/s, burst) plus a Retry-After/GitHub-rate-limit backoff gate. Host state persists in source_host_state and is reloaded at startup. Crate throttling and the GitHub/GitLab header checks remain source-level controls inside those adapters.

Write budget overrides

Budget state persists in the rate_budget table of the cache DB, so it survives restarts and rides along in the S3 cache upload. Defaults are 5,000 points/hour and 35,000/day per repo DID with a 0.9 safety margin.

export PDS_WRITE_HOURLY=5000
export PDS_WRITE_DAILY=35000
export PDS_REQUESTS_PER_5MIN=3000

# equivalent CLI flags
vuln-notif --once \
  --pds-hourly-points 5000 --pds-daily-points 35000 \
  --pds-requests-per-5min 3000 ...

# refuse to start a backfill without an NVD key (also REQUIRE_NVD_KEY=1)
vuln-notif --backfill --require-nvd-key ...

Costs: create/put 3 points, update 2, delete 1, and applyWrites charges every record in the batch at its own operation cost (deletes cost 1). uploadBlob costs 0. When a window is exhausted the process sleeps until it resets (plus jitter) and logs rate_limit_wait.

Reading rate-limit events

All logs are NDJSON on stderr. Relevant events:

  • rate_limit_wait — local budget exhausted before a write. Fields: op, points, remaining_hour, remaining_day, reset, wait_seconds.
  • pds_rate_limited — PDS returned 429. Fields: op, status, retry_after, remaining, reset, limit, sleep_seconds, cve/slug.
  • nvd_retry — outer NVD retry after 403/5xx/timeout. Fields: attempt, error, sleep_seconds.
  • nvd_api_key_configured — startup boolean (configured), never the key.

cycle_stats and the final run_stats include pds_budget_hour_remaining, pds_budget_day_remaining, pds_rate_limit_waits, pds_rate_limit_seconds, pds_requests_5min, nvd_key_configured, and nvd_retries. Budget remaining is measured against the 0.9-margin cap, so a fresh repo starts at 4,500/31,500.

Runtime events

stdout is NDJSON, one {"event": ..., ...payload} object per line. Logs are separate NDJSON on stderr. cycle_stats and run_stats are orchestrator events, not RuntimeEvents; every other name below comes from src/runtime/events.rs.

EventPayload fields
prepare_startednone
s3_downloadedbucket, path, bytes, seconds
pds_resolvedhandle, pds, seconds
pds_logged_inrepo, seconds
cycle_startedcycle, cursor
nvd_phase_startedcursor
nvd_phase_completedcves, requests, retries, seconds, cursor, error
cves_cachednew, updated_pending, already_done
publish_phase_startedlimit
publish_phase_completedfull PublishStats (calls, present, created, failures, skipped, retries, get_ops, create_ops, apply_ops, blob_ops, blob_bytes, oversized, oversized_skipped, record_bytes_max, retried, retry_failed, durations) + error
publish_batch_startedoffset, size
publish_batch_completedoffset, size, missing, created, present, failed
software_phase_startedlimit
software_phase_completedcves, software_created, purl_created, atproto_created, failures, error
release_phase_startedmode, limit, recheck_seconds
release_phase_completedReleaseCounters (projects_checked, projects_created, versions_created, channels_created, statuses_created, unchanged, skipped, unsupported_no_purl, failures) + error
release_affected_completedAffectedCounters (projects_checked, projects_linked, versions_planned, versions_created, versions_label_only, cves_linked, unchanged, skipped, failures, unsupported) + error
release_history_startedlimit, concurrency
release_history_projectproject plus flattened HistoryJobCounters (enqueued, processed, versions_created, manifests_created, skipped, failed, reused)
release_history_skippedproject, reason
release_history_completedHistoryModeCounters (projects_checked, projects_skipped, unsupported, enqueued, processed, versions_created, manifests_created, skipped, failed, reused) + error
release_extras_startedlimit
release_extras_projectproject, version
release_extras_skippedproject, reason
release_extras_completedExtrasCounters (projects_checked, projects_created, versions_created, manifests_created, statuses_created, unchanged, skipped, unsupported, failures) + error
link_repair_completedphase (startup, publish, software, release), attempted, repaired, failed, deferred, error
s3_uploadedbucket, path, bytes, seconds
s3_upload_failedbucket, path, bytes, error
cycle_completedcycle, duration, full CycleStats
cycle_failedcycle, error
shutdown_requestednone
fatalerror
donenone

cycle_stats carries the complete CycleStats object (NVD, publish, cache, S3, software, release, affected, history, extras, artifact, and link-repair counters). run_stats sums the SUM_KEYS, keeps maxima for MAX_KEYS, and uses the latest values for LAST_KEYS, then adds cycles, started_at, pds_seconds_avg, and records_per_second.

Our prod run — last week, newest → oldest

set -a; source ~/.nahdig_com_source; set +a; unset ATP_PDS_URL
export ATP_HANDLE="$NAHDIG_COM_ATPROTO_HANDLE"
export ATP_APP_PASSWORD="$NAHDIG_COM_ATPROTO_PASSWORD"
export S3_ENDPOINT_URL="$CVE_NAHDIG_COM_S3_ENDPOINT_URL"
export S3_BUCKET="$CVE_NAHDIG_COM_S3_BUCKET"
export S3_ACCESS_KEY_ID="$CVE_NAHDIG_COM_AWS_ACCESS_KEY"
export S3_SECRET_ACCESS_KEY="$CVE_NAHDIG_COM_AWS_SECRET_ACCESS_KEY"
export S3_REGION="$CVE_NAHDIG_COM_S3_REGION"
export S3_PATH="prod-cve-nahdig-com-week"

# first cycle fetches 7 days; repeat while pending > 0
vuln-notif --once \
  --initial-lookback-days 7 --limit 500 --order newest \
  --publish-atproto --software-limit 100 \
  --purldb .state/purl2cpe.db \
  --cache-db .state/prod-week.db

Loop until pending reaches 0:

while :; do
  vuln-notif --once \
    --initial-lookback-days 7 --limit 500 --order newest \
    --publish-atproto --software-limit 100 \
    --purldb .state/purl2cpe.db \
    --cache-db .state/prod-week.db
  REPO=$(sqlite3 .state/prod-week.db \
    "select repo_did from publications limit 1;")
  PENDING=$(sqlite3 .state/prod-week.db \
    "select count(*) from cves c
       left join publications p
         on p.repo_did='$REPO' and p.cve_id=c.id
      where p.status is null or p.status='pending';")
  echo "pending=$PENDING"
  [ "$PENDING" -eq 0 ] && break
  sleep 1800  # PDS repo-write-hour cap
done

--limit advances the NVD cursor; remaining rows stay pending. The cache is resumable; a 429 only delays the loop. --backfill deliberately ignores the stored cursor and starts from 1999; each fetched page replaces the stored cursor, so stop and resume works the same way as a normal incremental run.

Release records

--publish-releases publishes the default/latest version for software already in the cache. Candidate rows resolve through software_identity, so one canonical project receives all raw aliases; legacy raw-slug projects such as red-hat-red-hat-enterprise-linux-8 are no longer created. Projects use the canonical identity slug (red-hat-enterprise-linux), and raw Raw aliases stay in the local cache and inside CVE records; no com.nahdig.software records are published.

vuln-notif --once \
  --publish-atproto --publish-releases --release-mode latest \
  --release-limit 100 --release-recheck-seconds 86400 \
  --cache-db .state/prod-week.db

--release-mode latest is the default value but is a no-op on its own: standalone latest requires --publish-releases. --release-mode all runs the latest pass unconditionally without that switch.

  • Writes software.release.project, an immutable manifest, an immutable version, a mutable channel (latest for this mode), and optional status records. A version or manifest is never overwritten; a channel update only moves its version pointer.
  • When the purl type has a source adapter, also writes com.nahdig.release.projectMeta (same-DID strongRef, only when the content changed) and records publishedAt/rawVersion/artifact cid+size in release_version_meta. Manifest artifacts gain metadata git fragments and a target object when the source declares them.
  • Statuses are append-only software.release.status records plus release_status_events rows. A deps.dev deprecation without a registry signal goes through the same status module; an identical re-observation is a no-op and a reason change appends a new event. CVE presence never changes a status.
  • Canonical observations supply the project, channel, and version identity. com.nahdig.cve.atprotoV1 emits channels/versions/manifests strongRefs only for records that exist; otherwise it uses versionLabels and affectedRanges.
  • Sources: deps.dev for versions/defaults; PyPI, crates.io, and RubyGems give SHA-256 directly; npm, Maven, NuGet, and Go are downloaded once per artifact to compute SHA-256.
  • --release-recheck-seconds skips projects checked recently. Use 0 to force a fresh check.
  • PDS writes still pass through the rate governor; every release record write uses the same 5k/hour and 35k/day budget.

Affected mode

--release-mode affected maps OSV advisories onto the release catalog and updates the CVEs already linked to each canonical project.

# OSV online (default), one resumable page per cycle.
while :; do
  vuln-notif --once \
    --release-mode affected --release-limit 100 --release-recheck-seconds 86400 \
    --cache-db .state/prod-week.db
  sleep 1800
done

# Network-free / reproducible: load advisory JSON from disk instead of OSV.
vuln-notif --once \
  --release-mode affected --release-limit 100 --release-recheck-seconds 0 \
  --osv-bulk-dir /srv/osv-bulk --osv-offline \
  --cache-db .state/affected.db
  • OSV is queried online by default (--osv-online defaults to true). Use --osv-offline with --osv-bulk-dir, or --osv-online=false with --osv-bulk-dir; a bulk directory implies offline. Offline without a bulk directory is a usage error (exit 2). Bulk loading reads every *.json advisory in the directory and filters by purl per project.
  • --release-ecosystems filters purl types (pypi, npm, rpm, ...) and --release-sources filters sources (osv matches all affected work).
  • --release-concurrency is accepted but affected work runs in project order in this build; the flag affects history enumeration only.
  • The catalog is existing release_versions rows plus deps.dev versions when the purl type has a deps.dev system. Affected versions without a local row get an immutable manifest+version only when an artifact is fetchable; otherwise they remain versionLabels. No version is invented.
  • Channels point at the highest known version row in the stream and are only updated when that pointer changes. CVE records are putRecord only when the $type, cve, nvd, or software content changes.
  • --release-mode extras and all are documented in Extras mode and All mode.
  • Write budget: create/put costs 3 points, update 2, delete 1; applyWrites charges every record. A first affected pass can add two immutable records (manifest + version) per planned version, plus one channel point and one CVE update per linked CVE. Budget with --release-limit and the governor's pds_budget_*_remaining fields.
  • Per-project source, database, and PDS failures are isolated: the cycle continues and cycle_stats.release_affected_failures increases. Every 10 projects commit their local rows.

Verification after a run:

# Cycle counters (all release_affected_* fields are in cycle_stats).
grep '"event":"cycle_stats"' run.ndjson | tail -1 | jq '{
  mode: .release_mode,
  checked: .release_affected_projects_checked,
  linked: .release_affected_cves_linked,
  planned: .release_affected_versions_planned,
  created: .release_affected_versions_created,
  labels: .release_affected_versions_label_only,
  failures: .release_affected_failures
}'

# CVE refs for one project.
curl -sS -G "$ATP_PDS_URL/xrpc/com.atproto.repo.getRecord" \
  --data-urlencode "repo=$DID" \
  --data-urlencode 'collection=com.nahdig.cve.atprotoV1' \
  --data-urlencode 'rkey=CVE-2024-3094' |
  jq '.value.software[0] | {project, channels, versions, versionLabels}'

# Local link index rows written by the run.
sqlite3 .state/prod-week.db \
  "SELECT path, count(*) FROM links
   WHERE path LIKE 'software[].%'
   GROUP BY path;"

Metadata and status verification (all modes)

# Project metadata (same-DID strongRef; name equals the rkey).
curl -sS -G "$ATP_PDS_URL/xrpc/com.atproto.repo.getRecord" \
  --data-urlencode "repo=$DID" \
  --data-urlencode 'collection=com.nahdig.release.projectMeta' \
  --data-urlencode 'rkey=acme-widget' |
  jq '.value | {project, name, versioningScheme, repository}'

# Manifest artifact metadata: git facts and the runtime target.
curl -sS -G "$ATP_PDS_URL/xrpc/com.atproto.repo.getRecord" \
  --data-urlencode "repo=$DID" \
  --data-urlencode 'collection=software.release.manifest' \
  --data-urlencode 'rkey=<manifest-rkey>' |
  jq '.value.artifacts[] | {metadata, target}'

# Version facts plus the full append-only status history for one version.
sqlite3 .state/prod-week.db \
  "SELECT project_slug, version, source, published_at, raw_version
     FROM release_version_meta ORDER BY project_slug, version LIMIT 20;
   SELECT status, kind, reason, created_at FROM release_status_events
     WHERE subject_uri='at://$DID/software.release.version/acme-widget:1.2.3'
     ORDER BY created_at;"

Append-only semantics: a status event is never updated or deleted. Current state is the latest live same-DID software.release.status for the subject; a registry reason change appends a new record under a new digest rkey, and the old record stays addressable. projectMeta, statuses, and version metadata are compared before writing, so an unchanged second run makes zero PDS writes. If the PDS does not support listRecords, projectMeta writes are skipped because "put only when changed" cannot be verified.

History mode

--release-mode history enumerates each candidate's full source catalog and publishes immutable manifest + version records. Jobs are durable and resumable, so run it as a long-lived loop with small pages:

# One project page per cycle; 100 versions per project; 4 projects at a time.
while :; do
  vuln-notif --once \
    --release-mode history --release-limit 25 \
    --release-history-limit 100 --release-concurrency 4 \
    --release-ecosystems pypi,cargo,gem \
    --release-recheck-seconds 604800 \
    --artifact-max-bytes 104857600 \
    --cache-db .state/prod-history.db
  sleep 900
done

# Network-free source override for the local e2e (composer/cpan/apache/gnu/gitee).
vuln-notif --once \
  --release-mode history --release-recheck-seconds 0 \
  --release-source-url http://127.0.0.1:4850 \
  --cache-db /tmp/history-e2e.db
  • Queue order is (updated_at, project, raw version). A job becomes retry after a transient artifact failure and terminal error after three failed attempts. Re-running the mode never resets a terminal job; inspect release_history_jobs.error before requeueing by hand.
  • --release-history-limit is per project per cycle. --release-limit bounds candidate projects, and --release-concurrency bounds how many projects are enumerated at once; writes still go through the single PDS governor.
  • Immutable records are reused, never rewritten. A version that already exists is skipped at enqueue time, and a manifest whose URI/CID are recorded is reused.
  • History adapters: PyPI, crates.io, RubyGems, GitHub, GitLab, Bitbucket, distro (rpm/deb/apk), Packagist, MetaCPAN, Apache, Gitee, GNU. npm, Maven, NuGet, Go, Docker/OCI, and SourceForge are typed unsupported in history: the first four are covered by latest via deps.dev, and Docker/OCI plus SourceForge are covered by extras.
  • --release-source-url applies to the history adapters with configurable base URLs: composer, cpan, apache, gitee, and gnu. It is a local e2e/test seam, not a production mirror setting.
  • Token environment variables: GITHUB_TOKEN, GITLAB_TOKEN, GITEE_TOKEN (or the matching flags). Tokens are sent only as request headers (Gitee uses Authorization: Bearer) and are redacted from config debug output and logs. They never appear in URLs, cycle stats, or events.
  • Write budget: every new version costs two records (manifest + version = 6 points); reused rows and skipped jobs cost nothing. Statuses are re-planned for versions materialized in the cycle and for deprecated catalog entries, so a registry reason change appends an event without rewriting history. A full-history pass over 256k–884k versions is roughly 797k–2.68M points (23–77 days at the default 35k/day budget). Throttle with --release-history-limit and --release-recheck-seconds; the governor reports pds_budget_*_remaining in every cycle.

Extras mode

--release-mode extras covers every registered extras adapter, including the deps.dev ecosystems pypi, cargo, and gem. One project per cycle page: newest non-prerelease/non-draft version, one source-declared artifact, then project / manifest / version / latest channel / status / release.projectMeta records.

# One resumable page per cycle; rotate through the candidate set.
while :; do
  vuln-notif --once \
    --release-mode extras --release-limit 100 --release-recheck-seconds 86400 \
    --release-ecosystems pypi,github,rpm,docker,composer \
    --release-sources github,docker \
    --artifact-max-bytes 104857600 \
    --cache-db .state/prod-extras.db
  sleep 1800
done

# Network-free local e2e (composer stub on 127.0.0.1).
scripts/release-extras-e2e.sh
  • Candidate selection uses the same least-recently-checked rotation as the other modes; --release-limit bounds the page and every 10 projects commit local rows. Filters are applied before the source is built: a project whose purls are all excluded is counted as skipped and keeps its old check time so a later run with different filters still visits it.
  • Per-project isolation: source, artifact, database, and PDS failures increment release_extras_failures and the cycle continues. Unknown purl types and not-yet-adapted ecosystems are typed unsupported: the reason is written to release_project_state.unsupported_reason and emitted as release_extras_skipped.
  • Freshness is durable: release_project_state stores the canonical purl and adapter, and the release_projects row caches the published URIs. A second run with the same newest version is unchanged and performs no PDS writes; registry status changes are evaluated when a version is (re)materialized, not on the unchanged fast path.
  • --dry-run skips every release mode: only NVD fetch/cache runs, with no credentials and no PDS writes. Library-level extras dry-run planning exists but is not reachable from the CLI in this build.
  • --release-source-url applies to extras adapters with configurable base URLs: composer, cpan, apache, gitee, gnu, and sourceforge.
  • --release-sources matches purl types and adapter names (docker matches docker/oci, distro matches rpm/deb/apk); osv only affects affected work.
  • Source statuses are append-only: an identical signal is a no-op and a reason change appends a new software.release.status event. Manifests carry the adapter's git/target facts; release_version_meta records the version's publishedAt/rawVersion when the source declares them.

Source matrix

The full per-adapter matrix (version source, artifact/digest path, status signals, unsupported reasons, and RHEL/UBI mapping) is docs/release-sources.md. Operational summary:

Purl typesAdapterAuthVersions / artifacts / statuses / metaNotes
pypiPyPI JSON APInoneyes / yes / yanked / yesPEP 503 names, declared SHA-256
cargocrates.iononeyes / yes / yanked / yesadapter throttles to 1 req/s
gemRubyGemsnoneyes / yes / yanked / yesdeclared SHA-256, HEAD size fallback
githubGitHub releases + tagsGITHUB_TOKENyes / yes / draft+prerelease+archived / yesasset digest, archive fallback
gitlabGitLab releases + tagsGITLAB_TOKENyes / yes / upcoming+archived / yesPRIVATE-TOKEN, pagination links
bitbucketBitbucket tags + downloadsnoneyes / yes / no / yesno declared digest
rpm, deb, apkdistro lifecycle + public UBI OCInoneyes / RHEL OCI only / EOL / yesendoflife.date + Red Hat PLC
composerPackagist p2noneyes / dist zip / abandoned / yesshasum used as SHA-256 only when 64 hex
cpanMetaCPANnoneyes / tarball / no status record / yesbackpan entries are marked deprecated but statuses are typed NoStatuses
apacheApache download indexesnoneyes / archive files / no / yessidecar digests disabled; SHA-256 computed
giteeGitee releasesGITEE_TOKEN beareryes / assets / prerelease / yestoken never in the URL
gnuGNU FTP indexnoneyes / tarballs / no / yesindex discovery
docker, ociDocker Hub registryanonymous pull tokentags / manifest digest / tag-mutation / yesimmutable digest artifact
sourceforgeSourceForge RSS/filesnonebest effort / files / no / yestyped unavailable reasons
npm, maven, nuget, golangdeps.dev in latest; registry adapters in history/extrasnoneyes / yes / registry signals / yesfull history supported
  • Tokens: GITHUB_TOKEN, GITLAB_TOKEN, GITEE_TOKEN; all other adapters are anonymous. Header values are never logged, and Gitee uses Authorization: Bearer <token> rather than access_token.
  • Rate limits: the shared registry transport retries 429/5xx with 1s/2s/4s backoff and honours response pagination caps; crates.io requests are spaced 1s apart; GitHub/GitLab surface rate-limit headers as typed failures. PDS writes are governed by PDS_WRITE_HOURLY/PDS_WRITE_DAILY exactly like every other mode.
  • Write budget: a fresh project costs about 18 points (project 3, projectMeta 3, manifest 3, version 3, channel 3, status 3 per registry signal). Re-runs are unchanged and cost zero. Plan blocks with --release-limit and inspect release_extras_* before scaling.
  • Artifact downloads honour --artifact-max-bytes; declared digests are passed to the resolver, which verifies them against the downloaded bytes and caches the result in release_artifacts.

Verification after an extras run:

grep '"event":"cycle_stats"' run.ndjson | tail -1 | jq '{
  mode: .release_mode,
  checked: .release_extras_projects_checked,
  created: .release_extras_projects_created,
  versions: .release_extras_versions_created,
  manifests: .release_extras_manifests_created,
  statuses: .release_extras_statuses_created,
  unchanged: .release_extras_unchanged,
  skipped: .release_extras_skipped,
  unsupported: .release_extras_unsupported,
  failures: .release_extras_failures
}'

# Typed unsupported reasons, never silent skips.
sqlite3 .state/prod-extras.db \
  "SELECT slug, unsupported_reason FROM release_project_state
   WHERE unsupported_reason IS NOT NULL;"

# One project's newest version and its manifest strongRef.
curl -sS -G "$ATP_PDS_URL/xrpc/com.atproto.repo.getRecord" \
  --data-urlencode "repo=$DID" \
  --data-urlencode 'collection=software.release.version' \
  --data-urlencode 'rkey=acme-widget:1.2.3' | jq '.value'

All mode

--release-mode all runs latest → affected → history (only when --release-history-limit > 0) → extras in one cycle. All modes share the candidate cursor and one artifact resolver; each keeps its own counters:

  • release_* for latest, release_affected_* for affected, release_history_* for history, release_extras_* for extras.
  • artifact_* aggregates every resolve in the cycle.
  • --release-sources/--release-ecosystems filter affected, history, and extras connector work.
  • Use it for a small daily sweep only. A full all cycle over a production cache can combine the OSV, history, and extras write budgets; keep --release-limit, --release-history-limit, and --release-recheck-seconds conservative.
vuln-notif --once \
  --release-mode all --release-limit 100 --release-recheck-seconds 86400 \
  --release-history-limit 20 --release-concurrency 4 \
  --release-ecosystems pypi,github,composer \
  --artifact-max-bytes 104857600 \
  --cache-db .state/all.db

Write-budget schedules by mode

Planning numbers below assume one canonical project per product and the default 35,000-point/day budget (31,500 after the 0.9 safety margin). They are estimates for capacity planning, not measured prod totals.

WorkRecords / updatesApprox. pointsElapsed
Identity migration: CVE refs~14.3k updates~28.7k~1 day
Identity migration: software/purl refs + legacy deletes~100<1khours
latest / default (canonical)~19.2k records~57.6k~1.6 days
affected (OSV)100k–300k records330k–930k9–27 days
history (full catalogs)256k–884k records797k–2.68M23–77 days
extras~35.7k PURLssource-dependentsource-dependent

Per-mode record costs:

  • latest: project 3 + manifest 3 + version 3 + channel 3 (+ status 3 per registry signal, + projectMeta 3 once). Re-observing an unchanged version costs zero. A channel update is a 2-point update only when the pointer moves.
  • affected: immutable manifest + version (6 points) per affected version with a fetchable artifact, one channel update when the pointer moves, and one CVE putRecord update (2 points) per linked CVE only when refs change. versionLabels/affectedRanges-only versions cost zero PDS points.
  • history: manifest + version (6 points) per new version; reused records and skipped jobs cost zero. Jobs persist in release_history_jobs; status planning may append a status record on a later pass.
  • extras: project 3 + projectMeta 3 + manifest 3 + version 3 + channel 3 (+ status 3 per registry signal). Unchanged projects cost zero.
  • all: sums the mode costs in one cycle; keep --release-limit, --release-history-limit, and --release-recheck-seconds conservative.

Recommended multi-day order: migration (snapshot → plan → review → apply → verify) first, then latest --release-limit 100, then affected, then history for weeks, then extras with ecosystem filters.

Long runs are resumable:

# Foreground loop: one bounded page per cycle.
while :; do
  vuln-notif --once --release-mode history \
    --release-limit 25 --release-history-limit 100 --release-concurrency 4 \
    --cache-db .state/prod-history.db
  sleep 900
done

# Stop cleanly: SIGINT/SIGTERM drains the in-flight phase, commits, and exits.
# Resume by running the same command again; cursors, retry queues, and
# rate_budget live in the cache DB.

Artifact cache and S3 mirror

  • Verified artifact metadata is stored in release_artifacts, keyed by (repo, source, package, raw version, url). Confirm hits with cycle_stats.artifact_cache_hits; the second resolve must not increment artifact_downloads.
  • --artifact-max-bytes N rejects oversized streams before/while reading. 0 (default) means no cap; set an explicit cap in production.
  • --artifact-s3-bucket plus --artifact-s3-path (default artifacts) mirrors raw bytes. The mirror uses the same S3 endpoint and credentials as the cache snapshot (S3_ENDPOINT_URL, S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY, S3_REGION); verify with cycle_stats.artifact_mirror_puts. Mirror failures fall back to the registry and increment artifact_errors only when the resolve itself fails.
  • Without --artifact-s3-bucket the mirror is a no-op: artifact_mirror_puts stays 0 and only the local digest cache is consulted.
  • Cache snapshot uploads (S3_BUCKET/S3_PATH) are fetched before NVD work, then VACUUM INTO + upload after each cycle. Uploads get 4 attempts with 1s/2s/4s backoff plus 0.25s jitter and a post-upload size check; the final failure is logged as s3_upload_failed and returned, never swallowed. A missing remote cache is a normal first run.

Publish retries and link repair

  • CVE publish failures are recorded in publish_errors with an attempt count and next_retry_at = now + 30 * 2^(attempts-1) seconds, capped at one hour, plus up to 20% jitter. drain_retries runs before and after the publish phase, up to 100 due rows per pass. Success marks the publication present/created and clears the retry row; a CVE that left the cache has its retry row deleted.
  • Local link indexing is part of the write contract: PDS write → index links (or queue the record) → DB commit → advance cursor. An indexing failure serializes the written record into link_repairs in the same SQLite transaction, so a crash cannot lose either the link rows or the repair row.
  • link_repairs is drained at startup and after the publish, software, and release phases, 200 rows per drain. Retries use the same 30 * 2^(attempts-1) formula capped at one hour plus jitter. A drain makes no PDS calls: it reindexes from the stored record JSON and is idempotent because indexing replaces the source's link rows.
  • link_repair_* counters are in cycle_stats; each drain emits link_repair_completed with phase, attempted, repaired, failed, deferred, and error.

Verification after a history run:

grep '"event":"cycle_stats"' run.ndjson | tail -1 | jq '{
  mode: .release_mode,
  projects: .release_history_projects_checked,
  skipped: .release_history_skipped,
  unsupported: .release_history_unsupported,
  enqueued: .release_history_enqueued,
  processed: .release_history_processed,
  created: .release_history_created,
  failed: .release_history_failed,
  reused: .release_history_reused,
  downloads: .artifact_downloads,
  bytes: .artifact_bytes,
  cache_hits: .artifact_cache_hits,
  mirror_puts: .artifact_mirror_puts
}'

# Queue state: pending/retry/error must be inspected before a long run.
sqlite3 .state/prod-history.db \
  "SELECT status, count(*) FROM release_history_jobs GROUP BY status;"

# One immutable version and its manifest strongRef.
curl -sS -G "$ATP_PDS_URL/xrpc/com.atproto.repo.getRecord" \
  --data-urlencode "repo=$DID" \
  --data-urlencode 'collection=software.release.version' \
  --data-urlencode 'rkey=acme-widget:1.2.3' | jq '.value'

# Local e2e: fake composer source, limit/resume, artifact counters.
scripts/release-history-e2e.sh

Software identity migration

identity-migrate is a one-time, out-of-band admin binary. The service never runs it. Stop vuln-notif before apply and keep it stopped until verify exits 0; never run apply concurrently with a service cycle. The full runbook, including the failure-recovery matrix and snapshot restore script, is docs/identity-migration.md. The rules are in docs/identity-model.md.

Sequence: snapshot → plan → review diff → apply --resume → verify.

set -a; source ~/.nahdig_com_source; set +a; unset ATP_PDS_URL
export ATP_HANDLE="$NAHDIG_COM_ATPROTO_HANDLE"   # repo DID/handle, not a secret
export ATP_APP_PASSWORD="$NAHDIG_COM_ATPROTO_PASSWORD"

DB=.state/prod-week.db
MAP=migration/identity-$(date +%Y%m%dT%H%M%S)
cp "$DB" "$DB.pre-migration"                    # cache, cursor, rate-budget backup

# 1. Snapshot every migration collection to JSONL (read-only).
target/release/identity-migrate snapshot \
  --cache-db "$DB" --snapshot-dir "$MAP.snapshot"

# 2. Inventory cache + remote listRecords and write a reviewable plan.
target/release/identity-migrate plan \
  --cache-db "$DB" --purldb .state/purl2cpe.db \
  --out "$MAP.plan.json"

# 3. Review every legacy -> canonical pair before writing.
jq -r '.entries[] | .action' "$MAP.plan.json" | sort | uniq -c
jq -r '.entries[] | select(.action == "create_project") | .canonical_slug' "$MAP.plan.json"
jq -r '.entries[] | select(.action == "delete_release_record") |
       [.legacy_slug, .canonical_slug, .legacy_uri] | @tsv' "$MAP.plan.json" | sort -u

# 4. Apply in dependency order (projects, software, purl, CVE, status,
#    normalize, deletes) with a persisted cursor. 200-record applyWrites
#    delete batches.
target/release/identity-migrate apply \
  --cache-db "$DB" --map "$MAP.plan.json" \
  --batch 200 --snapshot-dir "$MAP.snapshot"

# If step 4 exits nonzero, rerun the same command with --resume:
target/release/identity-migrate apply \
  --cache-db "$DB" --map "$MAP.plan.json" \
  --batch 200 --snapshot-dir "$MAP.snapshot" --resume

# 5. Re-read the repository and fail on stale refs/backlinks.
target/release/identity-migrate verify \
  --cache-db "$DB" --map "$MAP.plan.json"

Useful flags (all global; identity-migrate --help):

FlagUse
--cache-dbSQLite cache, migration cursor, and rate-budget state
--purldbpurl2cpe SQLite for planning; absent is tolerated
--out / --mapPlan output / plan input for apply and verify
--batchRecords per applyWrites delete batch; default 200
--limitMax plan entries per invocation (0 = all), for chunked runs
--snapshot-dirJSONL snapshot; accepted by plan and apply
--resumeRetry pending/error entries, skip done
--dry-runPlan only; valid only for plan
--apply / --verifyFlag forms of the subcommands; can combine

The subcommand forms are plan, apply, verify, snapshot, and normalize-purl-slugs; the last one rewrites legacy full-AT-URI software_purls.software_slug values in the cache and needs no credentials.

NDJSON events on stdout (one line per run):

EventFields
identity_snapshotrecords, dir
identity_planentries, created, deleted, out
identity_applyreport: applied, skipped_done, created_projects, software_repoints, purl_repoints, cve_repoints, status_repoints, deleted, normalized
identity_verifyreport: canonical_projects, legacy_projects, software_repoints, purl_repoints, cve_repoints, status_repoints, deletions, stale_refs, errors
identity_normalize_purl_slugsrepo_did, updated

verify exits 1 when errors is non-empty. apply exits nonzero and leaves the failing entry error; --resume retries it.

Write budget and batching

OperationPoints
create / put (new record)3
update (putRecord on existing record)2
delete1
applyWriteseach record at its own operation cost

The migration is dominated by CVE ref updates (~14.3k updates × 2 ≈ 28.6k points, close to one 35k/day budget) plus software/purl ref updates and ~100 legacy deletes (<1k). Run it as a day-long job. apply persists one cursor row per (repo_did, phase, raw_slug) in identity_migration, so --resume skips completed entries and stopping does not lose budget. The governor still enforces 5,000 points/hour and 35,000/day with a 0.9 safety margin.

Rollback

The pre-apply snapshot plus the cache DB copy is the rollback material. There is intentionally no destructive rollback command; restore is a manual putRecord replay from the snapshot, documented in docs/identity-migration.md#rollback-via-snapshot. Restore all collections (project, software, purl, CVE), not only projects, or strongRefs stay broken. Restore the cache DB copy to roll back the cursor and rate-budget rows too.

Verification recipe

Canonical project backlinks (expect a non-zero total and canonical project URI):

PDS=${ATP_PDS_URL:-https://your-pds.example}
DID=$(goat account check-auth | sed -n 's/^DID: //p')
PROJ=red-hat-enterprise-linux
PROJECT="at://$DID/software.release.project/$PROJ"
curl -sS -G 'https://constellation.microcosm.blue/xrpc/blue.microcosm.links.getBacklinks' \
  --data-urlencode "subject=$PROJECT" \
  --data-urlencode 'source=com.nahdig.cve.atprotoV1:software[].project.uri' | jq .

Zero legacy backlinks — query each legacy RHEL slug and expect total: 0:

for LEGACY in red-hat-red-hat-enterprise-linux-8 \
              red-hat-red-hat-enterprise-linux-9 \
              redhat-enterprise-linux \
              redhat-enterprise-linux-eus \
              redhat-rhel-els; do
  SUBJECT="at://$DID/software.release.project/$LEGACY"
  TOTAL=$(curl -sS -G 'https://constellation.microcosm.blue/xrpc/blue.microcosm.links.getBacklinks' \
    --data-urlencode "subject=$SUBJECT" \
    --data-urlencode 'source=com.nahdig.cve.atprotoV1:software[].project.uri' | jq -r .total)
  printf '%s: %s\n' "$LEGACY" "$TOTAL"
done

The local equivalent checks the links table for unmanaged backlinks:

sqlite3 "$DB" "
  SELECT subject_uri, count(*)
  FROM links
  WHERE subject_uri LIKE 'at://$DID/software.release.project/%'
    AND subject_uri NOT LIKE '%/red-hat-enterprise-linux'
  GROUP BY subject_uri;"

verify performs the authoritative check across the repository; the queries above are the quick prod spot-check. Also confirm the canonical project exists (getRecord ... software.release.project/red-hat-enterprise-linux), the CVE has one canonical ref plus its preserved nvd strongRef, and a second apply reports applied == 0 and deleted == 0.

Local e2e order

Services: hono-plc :2587, atproto-pds :4800, RustFS :9000 (S3 test only). Start PLC → PDS → account; the PDS must run with PDS_LEXICON_DIR=$PWD/lexicons and PDS_DID_PLC_URL=http://127.0.0.1:2587.

scripts/local_plc.sh start
scripts/local_atproto_pds.sh start
scripts/create_local_account.sh
scripts/local-e2e.sh
scripts/blob-e2e.sh
scripts/local_rustfs.sh start
scripts/s3-cache-e2e.sh
scripts/identity-migrate-e2e.sh
scripts/release-affected-e2e.sh
scripts/release-history-e2e.sh
scripts/release-extras-e2e.sh
scripts/check-lexicons.sh

local-e2e.sh and identity-migrate-e2e.sh are self-contained and start their own temporary PLC/PDS; the explicit start commands above are for a persistent dev setup.

Delete old collection

OLD=com.nahdig.cve
# paginated listRecords + applyWrites delete (200/batch), then:
goat record delete --rkey "$OLD" com.atproto.lexicon.schema

Verify blob record

PDS=${ATP_PDS_URL:-https://your-pds.example}
DID=$(goat account check-auth | sed -n 's/^DID: //p')
REC=$(curl -sS "$PDS/xrpc/com.atproto.repo.getRecord?repo=$DID&collection=com.nahdig.cve.nvdV2&rkey=CVE-2021-3942")
CID=$(jq -r '.value.dataBlob.ref["$link"]' <<<"$REC")
curl -sS "$PDS/xrpc/com.atproto.sync.getBlob?did=$DID&cid=$CID" | gunzip | jq .id

Verify backlinks

Check the canonical project and its channel/version refs:

PROJ=red-hat-enterprise-linux
PROJECT="at://$DID/software.release.project/$PROJ"
CHANNEL="at://$DID/software.release.channel/$PROJ:9"
VERSION="at://$DID/software.release.version/$PROJ:9.4"

for PAIR in "project:$PROJECT:software[].project.uri" \
            "channel:$CHANNEL:software[].channels[].uri" \
            "version:$VERSION:software[].versions[].uri"; do
  LABEL=${PAIR%%:*}
  REST=${PAIR#*:}
  SUBJECT=${REST%:*}
  SOURCE=${REST##*:}
  echo "== $LABEL $SUBJECT"
  curl -sS -G 'https://constellation.microcosm.blue/xrpc/blue.microcosm.links.getBacklinks' \
    --data-urlencode "subject=$SUBJECT" \
    --data-urlencode "source=com.nahdig.cve.atprotoV1:$SOURCE" | jq .
done

For zero legacy backlinks, query the legacy RHEL slugs listed in the migration section and assert total: 0; identity-migrate verify is the authoritative repository-wide check.