Maintainers
Prod and ops notes for the vuln-notif repository. Flags and events below are
verified against src/cli.rs, src/bin/identity-migrate.rs, and the PDS
governor. Release source capabilities live in
docs/release-sources.md; module/data-flow/event
contracts live in
docs/release-architecture.md.
DNS records
set -a; source ~/.nahdig_com_source; set +a
# com.nahdig.cve authority
dig +short TXT _lexicon.nahdig.com # must match logged-in DID
# com.nahdig.cve.* authority (replace did:plc:example with the logged-in DID)
doctl compute domain records create \
--record-name _lexicon.cve --record-ttl 3600 --record-type TXT \
--record-data did=did:plc:example nahdig.com
dig +short TXT _lexicon.cve.nahdig.com # must match logged-in DID
Negative DNS caches can keep goat lex resolve failing after the record is
live; wait for the negative TTL or flush the resolver.
Publish lexicons
Publishing is an external ops step; the service binary never writes lexicons.
Use goat (below) or the equivalent atpxrpc call against the account's PDS.
Validate locally first:
scripts/check-lexicons.sh
The script walks all of lexicons/ recursively and jq-validates every
*.json; it has no hardcoded list. The publish list is the ten vendored
documents: four com.nahdig.* schemas (two record types plus the gitMeta
and projectMeta definitions) and six software.release.* schemas (defs
plus the five record types).
set -a; source ~/.nahdig_com_source; set +a
cd vuln-notif
GOAT_USERNAME="$NAHDIG_COM_ATPROTO_HANDLE" \
GOAT_PASSWORD="$NAHDIG_COM_ATPROTO_PASSWORD" \
goat account login
goat account check-auth
LEXICONS=(
lexicons/com/nahdig/cve.nvdV2.json
lexicons/com/nahdig/cve.atprotoV1.json
lexicons/com/nahdig/release.projectMeta.json
lexicons/com/nahdig/release.gitMeta.json
lexicons/software.release.defs.json
lexicons/software.release.project.json
lexicons/software.release.channel.json
lexicons/software.release.version.json
lexicons/software.release.manifest.json
lexicons/software.release.status.json
)
for f in "${LEXICONS[@]}"; do
id=$(jq -r .id "$f")
jq '. + {"$type":"com.atproto.lexicon.schema"}' "$f" > "/tmp/schema-$id.json"
goat record create --rkey "$id" "/tmp/schema-$id.json"
done
If goat lex publish prints green but goat lex resolve says RecordNotFound,
use the goat record create loop above. Publish software.release.defs before
the record lexicons that reference it, and publish the com.nahdig.* record
schemas before the software.release.* records that reference their fields.
com.nahdig.release.gitMeta is a definition document embedded in manifest
artifact metadata; it is still published as a lexicon schema record so clients
can resolve the type.
NVD API key
- Request a free key: https://nvd.nist.gov/developers/request-an-api-key
- Receive it by email.
- Use it:
export NVD_API_KEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
# or:
vuln-notif --nvd-api-key "$NVD_API_KEY" ...
The key is sent as the apiKey request header. Without it: 5 requests/30s
and ~6s between calls. With it: 50 requests/30s.
Rate limits
| Service | Limit |
|---|---|
| NVD API, no key | 5 requests / 30s |
| NVD API, key | 50 requests / 30s |
| PDS writes (per DID) | 5,000 points/hour, 35,000/day |
| PDS CREATE / UPDATE / DELETE | 3 / 2 / 1 points |
| Effective PDS creates | 1,666/hour, 11,666/day |
| Hosted API requests (per IP) | 3,000 / 5 min |
createSession | 30 / 5 min, 300/day per account |
createAccount | 100 / 5 min per IP |
| Blob upload size | 50 MiB |
applyWrites counts every record in the batch toward the write budget.
Last-week run: 14,349 CVEs = 43,047 points for nvdV2 alone (> daily budget);
adding atprotoV1 doubles it. Minimum ~2 days for nvdV2, ~3 days with both
formats. uploadBlob costs 0 write points.
The binary handles both limits itself: the native PDS governor paces every
PDS call (default 10 requests/s, burst 100, below 3,000 requests/5 min) and
reserves write points before create/put/delete/applyWrites. On 429 it honors
Retry-After (exact sleep) or RateLimit-Reset, otherwise backs off
60s → 1800s with jitter, and retries instead of failing the row.
createSession 429s retry with max(300, Retry-After). The NVD client does
its own token-bucket pacing and outer retry (30/60/120/300s + jitter).
Per-source limits and retry behavior:
| Source | Limit / policy |
|---|---|
| NVD API | 5 requests/30s without a key, 50/30s with NVD_API_KEY; 403/5xx/timeout retried at 30/60/120/300s + jitter |
| Shared registry HTTP | 4 attempts, 1s/2s/4s backoff on 429 and 500/502/503/504; 60s timeout |
crates.io (cargo) | adapter throttles every JSON request to 1 req/s per source instance |
GitHub (github) | optional GITHUB_TOKEN; honors x-ratelimit-remaining/x-ratelimit-reset/Retry-After, fails typed on 429 or exhausted 403 |
GitLab (gitlab) | optional GITLAB_TOKEN (PRIVATE-TOKEN header); response pagination links followed |
Docker Hub (docker, oci) | anonymous pull token fetched per operation from auth.docker.io; sent only in Authorization |
OSV (osv) | online query/querybatch; querybatch is split at 1,000 queries; offline bulk directory reads *.json without network |
| PDS | 5,000 points/hour, 35,000/day, 3,000 requests/5 min; create/put 3, update 2, delete 1 |
Every release-mode registry transport is wrapped in GovernedHttp: a per-host
token bucket (5 req/s, burst) plus a Retry-After/GitHub-rate-limit backoff
gate. Host state persists in source_host_state and is reloaded at startup.
Crate throttling and the GitHub/GitLab header checks remain source-level
controls inside those adapters.
Write budget overrides
Budget state persists in the rate_budget table of the cache DB, so it
survives restarts and rides along in the S3 cache upload. Defaults are
5,000 points/hour and 35,000/day per repo DID with a 0.9 safety margin.
export PDS_WRITE_HOURLY=5000
export PDS_WRITE_DAILY=35000
export PDS_REQUESTS_PER_5MIN=3000
# equivalent CLI flags
vuln-notif --once \
--pds-hourly-points 5000 --pds-daily-points 35000 \
--pds-requests-per-5min 3000 ...
# refuse to start a backfill without an NVD key (also REQUIRE_NVD_KEY=1)
vuln-notif --backfill --require-nvd-key ...
Costs: create/put 3 points, update 2, delete 1, and applyWrites charges
every record in the batch at its own operation cost (deletes cost 1).
uploadBlob costs 0. When a window is exhausted the process sleeps until it
resets (plus jitter) and logs rate_limit_wait.
Reading rate-limit events
All logs are NDJSON on stderr. Relevant events:
rate_limit_wait— local budget exhausted before a write. Fields:op,points,remaining_hour,remaining_day,reset,wait_seconds.pds_rate_limited— PDS returned 429. Fields:op,status,retry_after,remaining,reset,limit,sleep_seconds,cve/slug.nvd_retry— outer NVD retry after 403/5xx/timeout. Fields:attempt,error,sleep_seconds.nvd_api_key_configured— startup boolean (configured), never the key.
cycle_stats and the final run_stats include
pds_budget_hour_remaining, pds_budget_day_remaining,
pds_rate_limit_waits, pds_rate_limit_seconds, pds_requests_5min,
nvd_key_configured, and nvd_retries. Budget remaining is measured against
the 0.9-margin cap, so a fresh repo starts at 4,500/31,500.
Runtime events
stdout is NDJSON, one {"event": ..., ...payload} object per line. Logs are
separate NDJSON on stderr. cycle_stats and run_stats are orchestrator
events, not RuntimeEvents; every other name below comes from
src/runtime/events.rs.
| Event | Payload fields |
|---|---|
prepare_started | none |
s3_downloaded | bucket, path, bytes, seconds |
pds_resolved | handle, pds, seconds |
pds_logged_in | repo, seconds |
cycle_started | cycle, cursor |
nvd_phase_started | cursor |
nvd_phase_completed | cves, requests, retries, seconds, cursor, error |
cves_cached | new, updated_pending, already_done |
publish_phase_started | limit |
publish_phase_completed | full PublishStats (calls, present, created, failures, skipped, retries, get_ops, create_ops, apply_ops, blob_ops, blob_bytes, oversized, oversized_skipped, record_bytes_max, retried, retry_failed, durations) + error |
publish_batch_started | offset, size |
publish_batch_completed | offset, size, missing, created, present, failed |
software_phase_started | limit |
software_phase_completed | cves, software_created, purl_created, atproto_created, failures, error |
release_phase_started | mode, limit, recheck_seconds |
release_phase_completed | ReleaseCounters (projects_checked, projects_created, versions_created, channels_created, statuses_created, unchanged, skipped, unsupported_no_purl, failures) + error |
release_affected_completed | AffectedCounters (projects_checked, projects_linked, versions_planned, versions_created, versions_label_only, cves_linked, unchanged, skipped, failures, unsupported) + error |
release_history_started | limit, concurrency |
release_history_project | project plus flattened HistoryJobCounters (enqueued, processed, versions_created, manifests_created, skipped, failed, reused) |
release_history_skipped | project, reason |
release_history_completed | HistoryModeCounters (projects_checked, projects_skipped, unsupported, enqueued, processed, versions_created, manifests_created, skipped, failed, reused) + error |
release_extras_started | limit |
release_extras_project | project, version |
release_extras_skipped | project, reason |
release_extras_completed | ExtrasCounters (projects_checked, projects_created, versions_created, manifests_created, statuses_created, unchanged, skipped, unsupported, failures) + error |
link_repair_completed | phase (startup, publish, software, release), attempted, repaired, failed, deferred, error |
s3_uploaded | bucket, path, bytes, seconds |
s3_upload_failed | bucket, path, bytes, error |
cycle_completed | cycle, duration, full CycleStats |
cycle_failed | cycle, error |
shutdown_requested | none |
fatal | error |
done | none |
cycle_stats carries the complete CycleStats object (NVD, publish, cache,
S3, software, release, affected, history, extras, artifact, and link-repair
counters). run_stats sums the SUM_KEYS, keeps maxima for
MAX_KEYS, and uses the latest values for LAST_KEYS, then adds cycles,
started_at, pds_seconds_avg, and records_per_second.
Our prod run — last week, newest → oldest
set -a; source ~/.nahdig_com_source; set +a; unset ATP_PDS_URL
export ATP_HANDLE="$NAHDIG_COM_ATPROTO_HANDLE"
export ATP_APP_PASSWORD="$NAHDIG_COM_ATPROTO_PASSWORD"
export S3_ENDPOINT_URL="$CVE_NAHDIG_COM_S3_ENDPOINT_URL"
export S3_BUCKET="$CVE_NAHDIG_COM_S3_BUCKET"
export S3_ACCESS_KEY_ID="$CVE_NAHDIG_COM_AWS_ACCESS_KEY"
export S3_SECRET_ACCESS_KEY="$CVE_NAHDIG_COM_AWS_SECRET_ACCESS_KEY"
export S3_REGION="$CVE_NAHDIG_COM_S3_REGION"
export S3_PATH="prod-cve-nahdig-com-week"
# first cycle fetches 7 days; repeat while pending > 0
vuln-notif --once \
--initial-lookback-days 7 --limit 500 --order newest \
--publish-atproto --software-limit 100 \
--purldb .state/purl2cpe.db \
--cache-db .state/prod-week.db
Loop until pending reaches 0:
while :; do
vuln-notif --once \
--initial-lookback-days 7 --limit 500 --order newest \
--publish-atproto --software-limit 100 \
--purldb .state/purl2cpe.db \
--cache-db .state/prod-week.db
REPO=$(sqlite3 .state/prod-week.db \
"select repo_did from publications limit 1;")
PENDING=$(sqlite3 .state/prod-week.db \
"select count(*) from cves c
left join publications p
on p.repo_did='$REPO' and p.cve_id=c.id
where p.status is null or p.status='pending';")
echo "pending=$PENDING"
[ "$PENDING" -eq 0 ] && break
sleep 1800 # PDS repo-write-hour cap
done
--limit advances the NVD cursor; remaining rows stay pending. The cache is
resumable; a 429 only delays the loop. --backfill deliberately ignores the
stored cursor and starts from 1999; each fetched page replaces the stored
cursor, so stop and resume works the same way as a normal incremental run.
Release records
--publish-releases publishes the default/latest version for software already
in the cache. Candidate rows resolve through software_identity, so one
canonical project receives all raw aliases; legacy raw-slug projects such as
red-hat-red-hat-enterprise-linux-8 are no longer created. Projects use the
canonical identity slug (red-hat-enterprise-linux), and raw
Raw aliases stay in the local cache and inside CVE records; no
com.nahdig.software records are published.
vuln-notif --once \
--publish-atproto --publish-releases --release-mode latest \
--release-limit 100 --release-recheck-seconds 86400 \
--cache-db .state/prod-week.db
--release-mode latest is the default value but is a no-op on its own:
standalone latest requires --publish-releases. --release-mode all runs the
latest pass unconditionally without that switch.
- Writes
software.release.project, an immutablemanifest, an immutableversion, a mutablechannel(latestfor this mode), and optionalstatusrecords. A version or manifest is never overwritten; a channel update only moves itsversionpointer. - When the purl type has a source adapter, also writes
com.nahdig.release.projectMeta(same-DID strongRef, only when the content changed) and recordspublishedAt/rawVersion/artifact cid+size inrelease_version_meta. Manifest artifacts gainmetadatagit fragments and atargetobject when the source declares them. - Statuses are append-only
software.release.statusrecords plusrelease_status_eventsrows. A deps.dev deprecation without a registry signal goes through the same status module; an identical re-observation is a no-op and a reason change appends a new event. CVE presence never changes a status. - Canonical observations supply the project, channel, and version identity.
com.nahdig.cve.atprotoV1emitschannels/versions/manifestsstrongRefs only for records that exist; otherwise it usesversionLabelsandaffectedRanges. - Sources: deps.dev for versions/defaults; PyPI, crates.io, and RubyGems give SHA-256 directly; npm, Maven, NuGet, and Go are downloaded once per artifact to compute SHA-256.
--release-recheck-secondsskips projects checked recently. Use0to force a fresh check.- PDS writes still pass through the rate governor; every release record write uses the same 5k/hour and 35k/day budget.
Affected mode
--release-mode affected maps OSV advisories onto the release catalog and
updates the CVEs already linked to each canonical project.
# OSV online (default), one resumable page per cycle.
while :; do
vuln-notif --once \
--release-mode affected --release-limit 100 --release-recheck-seconds 86400 \
--cache-db .state/prod-week.db
sleep 1800
done
# Network-free / reproducible: load advisory JSON from disk instead of OSV.
vuln-notif --once \
--release-mode affected --release-limit 100 --release-recheck-seconds 0 \
--osv-bulk-dir /srv/osv-bulk --osv-offline \
--cache-db .state/affected.db
- OSV is queried online by default (
--osv-onlinedefaults to true). Use--osv-offlinewith--osv-bulk-dir, or--osv-online=falsewith--osv-bulk-dir; a bulk directory implies offline. Offline without a bulk directory is a usage error (exit 2). Bulk loading reads every*.jsonadvisory in the directory and filters by purl per project. --release-ecosystemsfilters purl types (pypi,npm,rpm, ...) and--release-sourcesfilters sources (osvmatches all affected work).--release-concurrencyis accepted but affected work runs in project order in this build; the flag affects history enumeration only.- The catalog is existing
release_versionsrows plus deps.dev versions when the purl type has a deps.dev system. Affected versions without a local row get an immutablemanifest+versiononly when an artifact is fetchable; otherwise they remainversionLabels. No version is invented. - Channels point at the highest known version row in the stream and are only
updated when that pointer changes. CVE records are
putRecordonly when the$type,cve,nvd, orsoftwarecontent changes. --release-mode extrasandallare documented in Extras mode and All mode.- Write budget: create/put costs 3 points, update 2, delete 1;
applyWritescharges every record. A first affected pass can add two immutable records (manifest + version) per planned version, plus one channel point and one CVE update per linked CVE. Budget with--release-limitand the governor'spds_budget_*_remainingfields. - Per-project source, database, and PDS failures are isolated: the cycle
continues and
cycle_stats.release_affected_failuresincreases. Every 10 projects commit their local rows.
Verification after a run:
# Cycle counters (all release_affected_* fields are in cycle_stats).
grep '"event":"cycle_stats"' run.ndjson | tail -1 | jq '{
mode: .release_mode,
checked: .release_affected_projects_checked,
linked: .release_affected_cves_linked,
planned: .release_affected_versions_planned,
created: .release_affected_versions_created,
labels: .release_affected_versions_label_only,
failures: .release_affected_failures
}'
# CVE refs for one project.
curl -sS -G "$ATP_PDS_URL/xrpc/com.atproto.repo.getRecord" \
--data-urlencode "repo=$DID" \
--data-urlencode 'collection=com.nahdig.cve.atprotoV1' \
--data-urlencode 'rkey=CVE-2024-3094' |
jq '.value.software[0] | {project, channels, versions, versionLabels}'
# Local link index rows written by the run.
sqlite3 .state/prod-week.db \
"SELECT path, count(*) FROM links
WHERE path LIKE 'software[].%'
GROUP BY path;"
Metadata and status verification (all modes)
# Project metadata (same-DID strongRef; name equals the rkey).
curl -sS -G "$ATP_PDS_URL/xrpc/com.atproto.repo.getRecord" \
--data-urlencode "repo=$DID" \
--data-urlencode 'collection=com.nahdig.release.projectMeta' \
--data-urlencode 'rkey=acme-widget' |
jq '.value | {project, name, versioningScheme, repository}'
# Manifest artifact metadata: git facts and the runtime target.
curl -sS -G "$ATP_PDS_URL/xrpc/com.atproto.repo.getRecord" \
--data-urlencode "repo=$DID" \
--data-urlencode 'collection=software.release.manifest' \
--data-urlencode 'rkey=<manifest-rkey>' |
jq '.value.artifacts[] | {metadata, target}'
# Version facts plus the full append-only status history for one version.
sqlite3 .state/prod-week.db \
"SELECT project_slug, version, source, published_at, raw_version
FROM release_version_meta ORDER BY project_slug, version LIMIT 20;
SELECT status, kind, reason, created_at FROM release_status_events
WHERE subject_uri='at://$DID/software.release.version/acme-widget:1.2.3'
ORDER BY created_at;"
Append-only semantics: a status event is never updated or deleted. Current
state is the latest live same-DID software.release.status for the subject; a
registry reason change appends a new record under a new digest rkey, and the
old record stays addressable. projectMeta, statuses, and version metadata
are compared before writing, so an unchanged second run makes zero PDS writes.
If the PDS does not support listRecords, projectMeta writes are skipped
because "put only when changed" cannot be verified.
History mode
--release-mode history enumerates each candidate's full source catalog and
publishes immutable manifest + version records. Jobs are durable and
resumable, so run it as a long-lived loop with small pages:
# One project page per cycle; 100 versions per project; 4 projects at a time.
while :; do
vuln-notif --once \
--release-mode history --release-limit 25 \
--release-history-limit 100 --release-concurrency 4 \
--release-ecosystems pypi,cargo,gem \
--release-recheck-seconds 604800 \
--artifact-max-bytes 104857600 \
--cache-db .state/prod-history.db
sleep 900
done
# Network-free source override for the local e2e (composer/cpan/apache/gnu/gitee).
vuln-notif --once \
--release-mode history --release-recheck-seconds 0 \
--release-source-url http://127.0.0.1:4850 \
--cache-db /tmp/history-e2e.db
- Queue order is
(updated_at, project, raw version). A job becomesretryafter a transient artifact failure and terminalerrorafter three failed attempts. Re-running the mode never resets a terminal job; inspectrelease_history_jobs.errorbefore requeueing by hand. --release-history-limitis per project per cycle.--release-limitbounds candidate projects, and--release-concurrencybounds how many projects are enumerated at once; writes still go through the single PDS governor.- Immutable records are reused, never rewritten. A version that already exists is skipped at enqueue time, and a manifest whose URI/CID are recorded is reused.
- History adapters: PyPI, crates.io, RubyGems, GitHub, GitLab, Bitbucket,
distro (rpm/deb/apk), Packagist, MetaCPAN, Apache, Gitee, GNU. npm, Maven,
NuGet, Go, Docker/OCI, and SourceForge are typed
unsupportedin history: the first four are covered bylatestvia deps.dev, and Docker/OCI plus SourceForge are covered byextras. --release-source-urlapplies to the history adapters with configurable base URLs:composer,cpan,apache,gitee, andgnu. It is a local e2e/test seam, not a production mirror setting.- Token environment variables:
GITHUB_TOKEN,GITLAB_TOKEN,GITEE_TOKEN(or the matching flags). Tokens are sent only as request headers (Gitee usesAuthorization: Bearer) and are redacted from config debug output and logs. They never appear in URLs, cycle stats, or events. - Write budget: every new version costs two records (manifest + version = 6
points); reused rows and skipped jobs cost nothing. Statuses are re-planned
for versions materialized in the cycle and for deprecated catalog entries,
so a registry reason change appends an event without rewriting history.
A full-history pass over
256k–884k versions is roughly 797k–2.68M points (23–77 days at the default
35k/day budget). Throttle with
--release-history-limitand--release-recheck-seconds; the governor reportspds_budget_*_remainingin every cycle.
Extras mode
--release-mode extras covers every registered extras adapter, including the
deps.dev ecosystems pypi, cargo, and gem. One project per cycle page: newest
non-prerelease/non-draft version, one source-declared artifact, then project /
manifest / version / latest channel / status / release.projectMeta
records.
# One resumable page per cycle; rotate through the candidate set.
while :; do
vuln-notif --once \
--release-mode extras --release-limit 100 --release-recheck-seconds 86400 \
--release-ecosystems pypi,github,rpm,docker,composer \
--release-sources github,docker \
--artifact-max-bytes 104857600 \
--cache-db .state/prod-extras.db
sleep 1800
done
# Network-free local e2e (composer stub on 127.0.0.1).
scripts/release-extras-e2e.sh
- Candidate selection uses the same least-recently-checked rotation as the
other modes;
--release-limitbounds the page and every 10 projects commit local rows. Filters are applied before the source is built: a project whose purls are all excluded is counted asskippedand keeps its old check time so a later run with different filters still visits it. - Per-project isolation: source, artifact, database, and PDS failures increment
release_extras_failuresand the cycle continues. Unknown purl types and not-yet-adapted ecosystems are typedunsupported: the reason is written torelease_project_state.unsupported_reasonand emitted asrelease_extras_skipped. - Freshness is durable:
release_project_statestores the canonical purl and adapter, and therelease_projectsrow caches the published URIs. A second run with the same newest version isunchangedand performs no PDS writes; registry status changes are evaluated when a version is (re)materialized, not on the unchanged fast path. --dry-runskips every release mode: only NVD fetch/cache runs, with no credentials and no PDS writes. Library-level extras dry-run planning exists but is not reachable from the CLI in this build.--release-source-urlapplies to extras adapters with configurable base URLs:composer,cpan,apache,gitee,gnu, andsourceforge.--release-sourcesmatches purl types and adapter names (dockermatchesdocker/oci,distromatchesrpm/deb/apk);osvonly affects affected work.- Source statuses are append-only: an identical signal is a no-op and a reason
change appends a new
software.release.statusevent. Manifests carry the adapter's git/target facts;release_version_metarecords the version'spublishedAt/rawVersionwhen the source declares them.
Source matrix
The full per-adapter matrix (version source, artifact/digest path, status
signals, unsupported reasons, and RHEL/UBI mapping) is
docs/release-sources.md. Operational summary:
| Purl types | Adapter | Auth | Versions / artifacts / statuses / meta | Notes |
|---|---|---|---|---|
pypi | PyPI JSON API | none | yes / yes / yanked / yes | PEP 503 names, declared SHA-256 |
cargo | crates.io | none | yes / yes / yanked / yes | adapter throttles to 1 req/s |
gem | RubyGems | none | yes / yes / yanked / yes | declared SHA-256, HEAD size fallback |
github | GitHub releases + tags | GITHUB_TOKEN | yes / yes / draft+prerelease+archived / yes | asset digest, archive fallback |
gitlab | GitLab releases + tags | GITLAB_TOKEN | yes / yes / upcoming+archived / yes | PRIVATE-TOKEN, pagination links |
bitbucket | Bitbucket tags + downloads | none | yes / yes / no / yes | no declared digest |
rpm, deb, apk | distro lifecycle + public UBI OCI | none | yes / RHEL OCI only / EOL / yes | endoflife.date + Red Hat PLC |
composer | Packagist p2 | none | yes / dist zip / abandoned / yes | shasum used as SHA-256 only when 64 hex |
cpan | MetaCPAN | none | yes / tarball / no status record / yes | backpan entries are marked deprecated but statuses are typed NoStatuses |
apache | Apache download indexes | none | yes / archive files / no / yes | sidecar digests disabled; SHA-256 computed |
gitee | Gitee releases | GITEE_TOKEN bearer | yes / assets / prerelease / yes | token never in the URL |
gnu | GNU FTP index | none | yes / tarballs / no / yes | index discovery |
docker, oci | Docker Hub registry | anonymous pull token | tags / manifest digest / tag-mutation / yes | immutable digest artifact |
sourceforge | SourceForge RSS/files | none | best effort / files / no / yes | typed unavailable reasons |
npm, maven, nuget, golang | deps.dev in latest; registry adapters in history/extras | none | yes / yes / registry signals / yes | full history supported |
- Tokens:
GITHUB_TOKEN,GITLAB_TOKEN,GITEE_TOKEN; all other adapters are anonymous. Header values are never logged, and Gitee usesAuthorization: Bearer <token>rather thanaccess_token. - Rate limits: the shared registry transport retries 429/5xx with 1s/2s/4s
backoff and honours response pagination caps; crates.io requests are spaced
1s apart; GitHub/GitLab surface rate-limit headers as typed failures. PDS
writes are governed by
PDS_WRITE_HOURLY/PDS_WRITE_DAILYexactly like every other mode. - Write budget: a fresh project costs about 18 points (project 3, projectMeta
3, manifest 3, version 3, channel 3, status 3 per registry signal). Re-runs
are unchanged and cost zero. Plan blocks with
--release-limitand inspectrelease_extras_*before scaling. - Artifact downloads honour
--artifact-max-bytes; declared digests are passed to the resolver, which verifies them against the downloaded bytes and caches the result inrelease_artifacts.
Verification after an extras run:
grep '"event":"cycle_stats"' run.ndjson | tail -1 | jq '{
mode: .release_mode,
checked: .release_extras_projects_checked,
created: .release_extras_projects_created,
versions: .release_extras_versions_created,
manifests: .release_extras_manifests_created,
statuses: .release_extras_statuses_created,
unchanged: .release_extras_unchanged,
skipped: .release_extras_skipped,
unsupported: .release_extras_unsupported,
failures: .release_extras_failures
}'
# Typed unsupported reasons, never silent skips.
sqlite3 .state/prod-extras.db \
"SELECT slug, unsupported_reason FROM release_project_state
WHERE unsupported_reason IS NOT NULL;"
# One project's newest version and its manifest strongRef.
curl -sS -G "$ATP_PDS_URL/xrpc/com.atproto.repo.getRecord" \
--data-urlencode "repo=$DID" \
--data-urlencode 'collection=software.release.version' \
--data-urlencode 'rkey=acme-widget:1.2.3' | jq '.value'
All mode
--release-mode all runs latest → affected → history (only when
--release-history-limit > 0) → extras in one cycle. All modes share the
candidate cursor and one artifact resolver; each keeps its own counters:
release_*for latest,release_affected_*for affected,release_history_*for history,release_extras_*for extras.artifact_*aggregates every resolve in the cycle.--release-sources/--release-ecosystemsfilter affected, history, and extras connector work.- Use it for a small daily sweep only. A full
allcycle over a production cache can combine the OSV, history, and extras write budgets; keep--release-limit,--release-history-limit, and--release-recheck-secondsconservative.
vuln-notif --once \
--release-mode all --release-limit 100 --release-recheck-seconds 86400 \
--release-history-limit 20 --release-concurrency 4 \
--release-ecosystems pypi,github,composer \
--artifact-max-bytes 104857600 \
--cache-db .state/all.db
Write-budget schedules by mode
Planning numbers below assume one canonical project per product and the default 35,000-point/day budget (31,500 after the 0.9 safety margin). They are estimates for capacity planning, not measured prod totals.
| Work | Records / updates | Approx. points | Elapsed |
|---|---|---|---|
| Identity migration: CVE refs | ~14.3k updates | ~28.7k | ~1 day |
| Identity migration: software/purl refs + legacy deletes | ~100 | <1k | hours |
latest / default (canonical) | ~19.2k records | ~57.6k | ~1.6 days |
affected (OSV) | 100k–300k records | 330k–930k | 9–27 days |
history (full catalogs) | 256k–884k records | 797k–2.68M | 23–77 days |
extras | ~35.7k PURLs | source-dependent | source-dependent |
Per-mode record costs:
latest: project 3 + manifest 3 + version 3 + channel 3 (+ status 3 per registry signal, + projectMeta 3 once). Re-observing an unchanged version costs zero. A channel update is a 2-point update only when the pointer moves.affected: immutable manifest + version (6 points) per affected version with a fetchable artifact, one channel update when the pointer moves, and one CVEputRecordupdate (2 points) per linked CVE only when refs change.versionLabels/affectedRanges-only versions cost zero PDS points.history: manifest + version (6 points) per new version; reused records and skipped jobs cost zero. Jobs persist inrelease_history_jobs; status planning may append a status record on a later pass.extras: project 3 + projectMeta 3 + manifest 3 + version 3 + channel 3 (+ status 3 per registry signal). Unchanged projects cost zero.all: sums the mode costs in one cycle; keep--release-limit,--release-history-limit, and--release-recheck-secondsconservative.
Recommended multi-day order: migration (snapshot → plan → review → apply → verify) first, then latest --release-limit 100, then affected, then
history for weeks, then extras with ecosystem filters.
Long runs are resumable:
# Foreground loop: one bounded page per cycle.
while :; do
vuln-notif --once --release-mode history \
--release-limit 25 --release-history-limit 100 --release-concurrency 4 \
--cache-db .state/prod-history.db
sleep 900
done
# Stop cleanly: SIGINT/SIGTERM drains the in-flight phase, commits, and exits.
# Resume by running the same command again; cursors, retry queues, and
# rate_budget live in the cache DB.
Artifact cache and S3 mirror
- Verified artifact metadata is stored in
release_artifacts, keyed by(repo, source, package, raw version, url). Confirm hits withcycle_stats.artifact_cache_hits; the second resolve must not incrementartifact_downloads. --artifact-max-bytes Nrejects oversized streams before/while reading.0(default) means no cap; set an explicit cap in production.--artifact-s3-bucketplus--artifact-s3-path(defaultartifacts) mirrors raw bytes. The mirror uses the same S3 endpoint and credentials as the cache snapshot (S3_ENDPOINT_URL,S3_ACCESS_KEY_ID,S3_SECRET_ACCESS_KEY,S3_REGION); verify withcycle_stats.artifact_mirror_puts. Mirror failures fall back to the registry and incrementartifact_errorsonly when the resolve itself fails.- Without
--artifact-s3-bucketthe mirror is a no-op:artifact_mirror_putsstays0and only the local digest cache is consulted. - Cache snapshot uploads (
S3_BUCKET/S3_PATH) are fetched before NVD work, thenVACUUM INTO+ upload after each cycle. Uploads get 4 attempts with 1s/2s/4s backoff plus 0.25s jitter and a post-upload size check; the final failure is logged ass3_upload_failedand returned, never swallowed. A missing remote cache is a normal first run.
Publish retries and link repair
- CVE publish failures are recorded in
publish_errorswith an attempt count andnext_retry_at = now + 30 * 2^(attempts-1)seconds, capped at one hour, plus up to 20% jitter.drain_retriesruns before and after the publish phase, up to 100 due rows per pass. Success marks the publication present/created and clears the retry row; a CVE that left the cache has its retry row deleted. - Local link indexing is part of the write contract: PDS write → index links
(or queue the record) → DB commit → advance cursor. An indexing failure
serializes the written record into
link_repairsin the same SQLite transaction, so a crash cannot lose either the link rows or the repair row. link_repairsis drained at startup and after the publish, software, and release phases, 200 rows per drain. Retries use the same30 * 2^(attempts-1)formula capped at one hour plus jitter. A drain makes no PDS calls: it reindexes from the stored record JSON and is idempotent because indexing replaces the source's link rows.link_repair_*counters are incycle_stats; each drain emitslink_repair_completedwithphase,attempted,repaired,failed,deferred, anderror.
Verification after a history run:
grep '"event":"cycle_stats"' run.ndjson | tail -1 | jq '{
mode: .release_mode,
projects: .release_history_projects_checked,
skipped: .release_history_skipped,
unsupported: .release_history_unsupported,
enqueued: .release_history_enqueued,
processed: .release_history_processed,
created: .release_history_created,
failed: .release_history_failed,
reused: .release_history_reused,
downloads: .artifact_downloads,
bytes: .artifact_bytes,
cache_hits: .artifact_cache_hits,
mirror_puts: .artifact_mirror_puts
}'
# Queue state: pending/retry/error must be inspected before a long run.
sqlite3 .state/prod-history.db \
"SELECT status, count(*) FROM release_history_jobs GROUP BY status;"
# One immutable version and its manifest strongRef.
curl -sS -G "$ATP_PDS_URL/xrpc/com.atproto.repo.getRecord" \
--data-urlencode "repo=$DID" \
--data-urlencode 'collection=software.release.version' \
--data-urlencode 'rkey=acme-widget:1.2.3' | jq '.value'
# Local e2e: fake composer source, limit/resume, artifact counters.
scripts/release-history-e2e.sh
Software identity migration
identity-migrate is a one-time, out-of-band admin binary. The service never
runs it. Stop vuln-notif before apply and keep it stopped until verify
exits 0; never run apply concurrently with a service cycle. The full
runbook, including the failure-recovery matrix and snapshot restore script, is
docs/identity-migration.md. The rules are in
docs/identity-model.md.
Sequence: snapshot → plan → review diff → apply --resume → verify.
set -a; source ~/.nahdig_com_source; set +a; unset ATP_PDS_URL
export ATP_HANDLE="$NAHDIG_COM_ATPROTO_HANDLE" # repo DID/handle, not a secret
export ATP_APP_PASSWORD="$NAHDIG_COM_ATPROTO_PASSWORD"
DB=.state/prod-week.db
MAP=migration/identity-$(date +%Y%m%dT%H%M%S)
cp "$DB" "$DB.pre-migration" # cache, cursor, rate-budget backup
# 1. Snapshot every migration collection to JSONL (read-only).
target/release/identity-migrate snapshot \
--cache-db "$DB" --snapshot-dir "$MAP.snapshot"
# 2. Inventory cache + remote listRecords and write a reviewable plan.
target/release/identity-migrate plan \
--cache-db "$DB" --purldb .state/purl2cpe.db \
--out "$MAP.plan.json"
# 3. Review every legacy -> canonical pair before writing.
jq -r '.entries[] | .action' "$MAP.plan.json" | sort | uniq -c
jq -r '.entries[] | select(.action == "create_project") | .canonical_slug' "$MAP.plan.json"
jq -r '.entries[] | select(.action == "delete_release_record") |
[.legacy_slug, .canonical_slug, .legacy_uri] | @tsv' "$MAP.plan.json" | sort -u
# 4. Apply in dependency order (projects, software, purl, CVE, status,
# normalize, deletes) with a persisted cursor. 200-record applyWrites
# delete batches.
target/release/identity-migrate apply \
--cache-db "$DB" --map "$MAP.plan.json" \
--batch 200 --snapshot-dir "$MAP.snapshot"
# If step 4 exits nonzero, rerun the same command with --resume:
target/release/identity-migrate apply \
--cache-db "$DB" --map "$MAP.plan.json" \
--batch 200 --snapshot-dir "$MAP.snapshot" --resume
# 5. Re-read the repository and fail on stale refs/backlinks.
target/release/identity-migrate verify \
--cache-db "$DB" --map "$MAP.plan.json"
Useful flags (all global; identity-migrate --help):
| Flag | Use |
|---|---|
--cache-db | SQLite cache, migration cursor, and rate-budget state |
--purldb | purl2cpe SQLite for planning; absent is tolerated |
--out / --map | Plan output / plan input for apply and verify |
--batch | Records per applyWrites delete batch; default 200 |
--limit | Max plan entries per invocation (0 = all), for chunked runs |
--snapshot-dir | JSONL snapshot; accepted by plan and apply |
--resume | Retry pending/error entries, skip done |
--dry-run | Plan only; valid only for plan |
--apply / --verify | Flag forms of the subcommands; can combine |
The subcommand forms are plan, apply, verify, snapshot, and
normalize-purl-slugs; the last one rewrites legacy full-AT-URI
software_purls.software_slug values in the cache and needs no credentials.
NDJSON events on stdout (one line per run):
| Event | Fields |
|---|---|
identity_snapshot | records, dir |
identity_plan | entries, created, deleted, out |
identity_apply | report: applied, skipped_done, created_projects, software_repoints, purl_repoints, cve_repoints, status_repoints, deleted, normalized |
identity_verify | report: canonical_projects, legacy_projects, software_repoints, purl_repoints, cve_repoints, status_repoints, deletions, stale_refs, errors |
identity_normalize_purl_slugs | repo_did, updated |
verify exits 1 when errors is non-empty. apply exits nonzero and leaves
the failing entry error; --resume retries it.
Write budget and batching
| Operation | Points |
|---|---|
| create / put (new record) | 3 |
update (putRecord on existing record) | 2 |
| delete | 1 |
applyWrites | each record at its own operation cost |
The migration is dominated by CVE ref updates (~14.3k updates × 2 ≈ 28.6k
points, close to one 35k/day budget) plus software/purl ref updates and ~100
legacy deletes (<1k). Run it as a day-long job. apply persists one cursor row
per (repo_did, phase, raw_slug) in identity_migration, so --resume
skips completed entries and stopping does not lose budget. The governor still
enforces 5,000 points/hour and 35,000/day with a 0.9 safety margin.
Rollback
The pre-apply snapshot plus the cache DB copy is the rollback material. There
is intentionally no destructive rollback command; restore is a manual
putRecord replay from the snapshot, documented in
docs/identity-migration.md#rollback-via-snapshot. Restore all collections
(project, software, purl, CVE), not only projects, or strongRefs stay broken.
Restore the cache DB copy to roll back the cursor and rate-budget rows too.
Verification recipe
Canonical project backlinks (expect a non-zero total and canonical project URI):
PDS=${ATP_PDS_URL:-https://your-pds.example}
DID=$(goat account check-auth | sed -n 's/^DID: //p')
PROJ=red-hat-enterprise-linux
PROJECT="at://$DID/software.release.project/$PROJ"
curl -sS -G 'https://constellation.microcosm.blue/xrpc/blue.microcosm.links.getBacklinks' \
--data-urlencode "subject=$PROJECT" \
--data-urlencode 'source=com.nahdig.cve.atprotoV1:software[].project.uri' | jq .
Zero legacy backlinks — query each legacy RHEL slug and expect total: 0:
for LEGACY in red-hat-red-hat-enterprise-linux-8 \
red-hat-red-hat-enterprise-linux-9 \
redhat-enterprise-linux \
redhat-enterprise-linux-eus \
redhat-rhel-els; do
SUBJECT="at://$DID/software.release.project/$LEGACY"
TOTAL=$(curl -sS -G 'https://constellation.microcosm.blue/xrpc/blue.microcosm.links.getBacklinks' \
--data-urlencode "subject=$SUBJECT" \
--data-urlencode 'source=com.nahdig.cve.atprotoV1:software[].project.uri' | jq -r .total)
printf '%s: %s\n' "$LEGACY" "$TOTAL"
done
The local equivalent checks the links table for unmanaged backlinks:
sqlite3 "$DB" "
SELECT subject_uri, count(*)
FROM links
WHERE subject_uri LIKE 'at://$DID/software.release.project/%'
AND subject_uri NOT LIKE '%/red-hat-enterprise-linux'
GROUP BY subject_uri;"
verify performs the authoritative check across the repository; the queries
above are the quick prod spot-check. Also confirm the canonical project exists
(getRecord ... software.release.project/red-hat-enterprise-linux), the CVE
has one canonical ref plus its preserved nvd strongRef, and a second apply
reports applied == 0 and deleted == 0.
Local e2e order
Services: hono-plc :2587, atproto-pds :4800, RustFS :9000 (S3 test only).
Start PLC → PDS → account; the PDS must run with
PDS_LEXICON_DIR=$PWD/lexicons and PDS_DID_PLC_URL=http://127.0.0.1:2587.
scripts/local_plc.sh start
scripts/local_atproto_pds.sh start
scripts/create_local_account.sh
scripts/local-e2e.sh
scripts/blob-e2e.sh
scripts/local_rustfs.sh start
scripts/s3-cache-e2e.sh
scripts/identity-migrate-e2e.sh
scripts/release-affected-e2e.sh
scripts/release-history-e2e.sh
scripts/release-extras-e2e.sh
scripts/check-lexicons.sh
local-e2e.sh and identity-migrate-e2e.sh are self-contained and start
their own temporary PLC/PDS; the explicit start commands above are for a
persistent dev setup.
Delete old collection
OLD=com.nahdig.cve
# paginated listRecords + applyWrites delete (200/batch), then:
goat record delete --rkey "$OLD" com.atproto.lexicon.schema
Verify blob record
PDS=${ATP_PDS_URL:-https://your-pds.example}
DID=$(goat account check-auth | sed -n 's/^DID: //p')
REC=$(curl -sS "$PDS/xrpc/com.atproto.repo.getRecord?repo=$DID&collection=com.nahdig.cve.nvdV2&rkey=CVE-2021-3942")
CID=$(jq -r '.value.dataBlob.ref["$link"]' <<<"$REC")
curl -sS "$PDS/xrpc/com.atproto.sync.getBlob?did=$DID&cid=$CID" | gunzip | jq .id
Verify backlinks
Check the canonical project and its channel/version refs:
PROJ=red-hat-enterprise-linux
PROJECT="at://$DID/software.release.project/$PROJ"
CHANNEL="at://$DID/software.release.channel/$PROJ:9"
VERSION="at://$DID/software.release.version/$PROJ:9.4"
for PAIR in "project:$PROJECT:software[].project.uri" \
"channel:$CHANNEL:software[].channels[].uri" \
"version:$VERSION:software[].versions[].uri"; do
LABEL=${PAIR%%:*}
REST=${PAIR#*:}
SUBJECT=${REST%:*}
SOURCE=${REST##*:}
echo "== $LABEL $SUBJECT"
curl -sS -G 'https://constellation.microcosm.blue/xrpc/blue.microcosm.links.getBacklinks' \
--data-urlencode "subject=$SUBJECT" \
--data-urlencode "source=com.nahdig.cve.atprotoV1:$SOURCE" | jq .
done
For zero legacy backlinks, query the legacy RHEL slugs listed in the migration
section and assert total: 0; identity-migrate verify is the authoritative
repository-wide check.