~ knotless Search/
↑↓ move↵ open⇧↵ all results

johnandersen777.bsky.social/vuln-notif

0
0
main
276 lines · 14.5 KB Raw History

Release architecture

How the Rust service ingests releases, publishes software.release.* records, and keeps local state consistent. Source capabilities are in release-sources.md; identity rules are in identity-model.md.

Process and module map

The binary is two reactors:

  • src/runtime/orchestrator.rs — signals, scheduling, event output, run totals.
  • src/runtime/mod.rs — MirrorRuntime: cache, NVD client, publisher, cursor, phase state machine.
  • src/runtime/phases.rs — the only bridge from the reactor to lane modules.
  • src/runtime/events.rs — typed ProgressEvent (lanes) and RuntimeEvent (output); progress is forwarded on a bounded channel.
  • src/db/mod.rs — every SQLite operation runs on one dedicated OS thread behind DbHandle; callers await command replies.

Release modules:

ModuleContract
src/release/mod.rsReleaseConfig, AffectedConfig, HistoryModeConfig, ExtrasConfig, ReleaseCursor, and per-mode counters
src/release/source.rsReleaseSource trait, SourceCapabilities, UnsupportedReason, SOURCE_REGISTRY, build_source*
src/release/sources/one adapter per source; OSV is a lookup client, not a ReleaseSource
src/release/service.rsmode orchestration (sync_releases*, sync_affected*, sync_history*, sync_extras*, sync_all*), record prep, project state
src/release/planner.rsdeps.dev default selection (select_release) and OSV range matching (plan_versions)
src/release/versioning.rsversion schemes, normalization, comparison
src/release/records.rswire shapes and rkeys for software.release.*
src/release/artifacts.rsartifact resolver: cap, verify/compute digest, CID, cache, S3 mirror, counters
src/release/artifact.rspurl selection and per-ecosystem artifact fetchers (npm, PyPI, cargo, RubyGems, Maven, NuGet, Go)
src/release/http.rsRegistryHttp seam and production reqwest transport with retries/caps
src/release/history.rsresumable release_history_jobs queue
src/release/cve_links.rsatprotoV1 channels/versions/manifests/versionLabels/affectedRanges refs
src/release/status.rsappend-only status transition planning
src/release/metadata.rsproject metadata, git/target facts, versioning scheme
src/identity/deterministic canonical identity and CVE ref building
src/software.rslocal raw aliases and atprotoV1 records
src/links.rs, src/links/repair.rslocal link index, durability queue, repair drain
src/pds/publisher, same-DID assertions, governs writes and blobs
src/migrate/, src/bin/identity-migrate.rsout-of-band one-time identity migration

Data flow

Common cycle

S3 snapshot download (optional)
  -> open cache DB; load cursor; --backfill resets to 1999; otherwise initialize
     --initial-lookback-days when no cursor
  -> build RateGovernor; resolve PDS URL; login (unless --dry-run)
  -> NVD fetch page -> upsert cves -> set cursor -> commit
  -> publish phase: drain due publish_errors, publish pending rows, drain again
  -> software phase when --publish-atproto: raw aliases, purls, atprotoV1
  -> release mode dispatch (below)
  -> drain link_repairs for release work
  -> VACUUM INTO + S3 snapshot upload (optional)

Per-project failures inside a mode are isolated and counted; the next candidate proceeds. Release work commits local rows every 10 projects (history commits per concurrency chunk and every 50 jobs).

latest

release_candidates_after(limit, recheck_seconds, after_slug)
  -> parse_purl + choose_purl (leaf match, then ecosystem priority)
  -> depsdev_versions(system, name)
  -> select_release (default, else newest non-deprecated representable)
  -> build_source for pypi/cargo/gem only; npm/maven/nuget/golang have no adapter
  -> resolve artifact through ArtifactResolver (cap, digest, cache, mirror)
  -> put project -> manifest -> version -> latest channel
  -> plan/apply statuses (registry signal + deps.dev deprecation)
  -> projectMeta when the source exposes it and the PDS supports listRecords
  -> mark release_* rows and release_project_state

The standalone mode runs only when --publish-releases is set. all runs the same pass unconditionally.

affected

release_candidates_after(...) -> choose_affected_purl (ecosystem/source filters)
  -> resolve existing project strongRef
  -> local backlinks(software[].project.uri) -> CVE ids; no CVEs = skip
  -> OSV: online query/querybatch by versionless purl, or bulk-dir advisories
  -> affected_catalog = release_versions rows + deps.dev versions (7 ecosystems)
  -> plan_versions per CVE: introduced/fixed/last_affected ranges
  -> affected versions with an artifact -> immutable manifest + version
     affected versions without an artifact -> versionLabels
     range-only observations -> affectedRanges
  -> ensure channel points at the highest known version row
  -> update atprotoV1 software[] refs only when content changed
  -> write release_cve_links and local links

Artifacts in affected mode use the seven-ecosystem artifact_for path; a distro or other purl whose artifact cannot be resolved stays label-only. OSV never creates or yanks a version; only registry signals drive statuses.

history

release_candidates_after(...)
  -> build_history_source (pypi, cargo, gem, github, gitlab, bitbucket,
     distro, composer, cpan, apache, gitee, gnu)
  -> source.versions() -> enqueue_versions
       (skip versions that already have a release_versions row; never reset
        an existing pending/retry/done/error job)
  -> process_jobs(limit = --release-history-limit):
       resolve one artifact through HistoryArtifacts
       -> reuse an existing manifest/version when recorded
       -> else publish immutable manifest + version
  -> plan statuses for materialized/deprecated entries

Jobs run pending -> retry -> done, or terminal error after three failed attempts. skipped jobs carry a typed reason and are not retried automatically. Projects are enumerated concurrently (--release-concurrency); writes still serialize through one PDS governor.

extras

release_candidates_after(...) -> choose_affected_purl filters
  -> build_source (every registered adapter)
  -> source.versions() -> newest stable (prerelease/draft markers excluded;
     fall back to newest prerelease when every version is prerelease)
  -> source.artifacts() -> resolve_declared through the shared resolver
  -> project -> projectMeta -> manifest -> version -> latest channel
  -> source.statuses() only when capabilities.statuses
  -> release_version_meta and release_project_state

An unchanged project (same newest version, existing version row and channel) performs no PDS writes. A source with no artifact is skipped with reason no_artifact; an unknown/unsupported purl type is unsupported with the rendered UnsupportedReason in release_project_state.unsupported_reason.

all

latest -> affected -> history (only when --release-history-limit > 0) -> extras, sharing one candidate rotation cursor and one artifact resolver. Each mode keeps its own counters; artifact_* aggregates every resolve. The runtime guards every release arm with !dry_run, so --dry-run skips all release modes; sync_all_with_http_resolver's library-level dry-run extras branch is not reachable from the CLI.

Database tables

Existing NVD/publish state:

TablePurpose
metacursor and small key/value state
cvesraw NVD JSON keyed by CVE id
publicationsper-repo CVE publish status, URI/CID, blob metadata
softwareraw CPE alias rows and canonical project refs
software_purlsPURL lookup rows; software_slug stores the canonical slug
cve_atprotoatprotoV1 publish status
linkslocal outgoing-link index (source URI/CID, path, subject URI/CID)

Release current-state indexes:

TablePurpose
release_projectscanonical project row, selected purl/system, latest version, record refs
release_versionsimmutable version row and manifest refs
release_channelsmutable channel pointer per project
release_statuscurrent/latest status rows by subject

Release caches, queues, and identity state:

TablePurpose
software_identityraw slug -> canonical slug/identity key
project_aliasesdeclared alias-key mapping; no current caller outside the DB layer
project_channelsdeclared channel/latest/EOL state; no current caller outside the DB layer
release_artifactsverified artifact cache, PK (repo, source, package, version, url)
release_manifestsmanifest key -> URI/CID/artifacts JSON
release_history_jobsdurable version jobs with status/attempts/error
release_osvcached OSV advisories by purl/id
release_cve_linksCVE -> project/version/channel/manifest refs and labels
release_source_statedeclared per-source cursor/etag; no current caller outside the DB layer
release_status_eventsappend-only status transitions
release_project_statecanonical purl, source, raw versions, checked_at, unsupported reason
release_version_metapublished_at, raw version, artifact cid/size
identity_migrationmigration cursor per (repo, phase, raw_slug)
link_repairsserialized records awaiting reindex
publish_errorsCVE publish retry state
source_host_stateper-host token-bucket window and Retry-After backoff; read/written by GovernedHttp
rate_budgetpersistent PDS hour/day points per repo DID

Schema is additive and created with CREATE TABLE IF NOT EXISTS; no ALTER TABLE or app-startup migration runs.

Event and cache contracts

  • Lanes report ProgressEvent; the runtime forwards it as a RuntimeEvent. RuntimeEvent::payload serializes the full stats struct plus error for terminal phase events; CycleCompleted includes the complete CycleStats.
  • stdout is NDJSON {"event": name, ...payload}; stderr is the log stream. The full event/field table is in ../MAINTAINERS.md#runtime-events.
  • Artifact cache key: (repo_did, source, package, raw_version, url). A cache hit returns the stored digest/CID/size/mime and increments artifact_cache_hits without downloading.
  • Manifest reuse: release_manifests is keyed by (repo, project_slug, manifest_key) and holds the published URI/CID and artifacts JSON. An existing version row is never overwritten.
  • S3 mirror key: the resolver key is artifacts/<sha256(url)>; --artifact-s3-path (default artifacts) is the object-key prefix, so the default object key is artifacts/artifacts/<sha256(url)>. The mirror is read first; a miss falls back to the registry; a failed put is best-effort and does not fail the resolve.
  • OSV cache: release_osv stores aliases/affected/modified per (repo, purl, osv_id); summary and severity are advisory-only and are not persisted.
  • Local links: links is replaced per source URI on every successful write, so reindexing is idempotent.

Idempotency and immutability

  • software.release.project rkeys are permanent. name equals the rkey; renames are expressed with renamedTo, never an in-place edit.
  • software.release.version and software.release.manifest are immutable: re-observing one reuses the row/record; it never rewrites bytes or artifacts.
  • software.release.channel is a mutable pointer; a new record is written only when the pointer changes.
  • software.release.status is append-only. The rkey is a digest over (subject, status, reason); current state is the latest live same-DID event.
  • com.nahdig.release.projectMeta is compared before putRecord; an unchanged second run makes zero PDS writes. Without PDS listRecords, metadata writes are skipped because "only when changed" cannot be verified.
  • com.nahdig.cve.atprotoV1 is putRecord only when $type, cve, nvd, or software content changes.
  • History jobs are never reset: re-enumeration skips existing versions and preserves pending/retry/error attempts.
  • Artifact resolution reuses the digest cache; declared digests must match the downloaded bytes. Manifest/version writes reuse recorded rows.
  • Cross-DID strongRefs are rejected before every write: project, channel, version, manifest, status, software, purl, and atprotoV1.

Failure and retry semantics

FailureBehavior
NVD 403/5xx/timeoutbounded outer retry at 30/60/120/300s + jitter; non-200 loops are bounded
NVD rate limitclient token bucket; 5 req/30s without a key, 50/30s with a key
PDS 429honor Retry-After/RateLimit-Reset, else 60s -> 1800s jittered backoff, bounded attempts
PDS write budgetgovernor sleeps until the hour/day window resets; state persists in rate_budget
Publish failurerecorded in publish_errors; retried before/after the next publish phase with capped exponential backoff; success clears the row
Link index failurerecord serialized to link_repairs in the same transaction; drained at startup and after phases; no PDS calls
Registry 429/5xxshared HTTP transport: 4 attempts, 1/2/4s backoff; artifact size-cap and digest-mismatch errors are not retried
Artifact over captyped TooLarge; affected falls back to versionLabels, latest/extras count a project failure, history fails the job (retried, then terminal error); never retried as a transient source error
Artifact digest mismatchtyped DigestMismatch; never accepted or cached
Source errorper-project failure counter; cycle continues; unsupported types are typed and logged
History job failureattempts incremented; pending/retry until 3 attempts, then terminal error; --resume is not needed because the next cycle resumes pending/retry jobs
S3 cache upload4 attempts, 1/2/4s + jitter, post-upload size check; final failure is logged and returned
S3 artifact mirrorbest-effort; miss re-downloads; failure does not fail the resolve
Fatal runtime eventorchestrator sends shutdown; process exits nonzero

--dry-run keeps credentials optional, fetches/caches NVD data only, skips every release mode, and writes no PDS records. identity-migrate is the only tool that rewrites canonical refs; the service never runs it.

Related

  • release-sources.md — adapter matrix, auth, digests, status signals, unsupported reasons, RHEL/UBI mapping.
  • ../MAINTAINERS.md — prod commands, rate tables, events, write-budget schedules, migration runbook.