Release architecture
How the Rust service ingests releases, publishes software.release.* records,
and keeps local state consistent. Source capabilities are in
release-sources.md; identity rules are in
identity-model.md.
Process and module map
The binary is two reactors:
src/runtime/orchestrator.rs— signals, scheduling, event output, run totals.src/runtime/mod.rs—MirrorRuntime: cache, NVD client, publisher, cursor, phase state machine.src/runtime/phases.rs— the only bridge from the reactor to lane modules.src/runtime/events.rs— typedProgressEvent(lanes) andRuntimeEvent(output); progress is forwarded on a bounded channel.src/db/mod.rs— every SQLite operation runs on one dedicated OS thread behindDbHandle; callers await command replies.
Release modules:
| Module | Contract |
|---|---|
src/release/mod.rs | ReleaseConfig, AffectedConfig, HistoryModeConfig, ExtrasConfig, ReleaseCursor, and per-mode counters |
src/release/source.rs | ReleaseSource trait, SourceCapabilities, UnsupportedReason, SOURCE_REGISTRY, build_source* |
src/release/sources/ | one adapter per source; OSV is a lookup client, not a ReleaseSource |
src/release/service.rs | mode orchestration (sync_releases*, sync_affected*, sync_history*, sync_extras*, sync_all*), record prep, project state |
src/release/planner.rs | deps.dev default selection (select_release) and OSV range matching (plan_versions) |
src/release/versioning.rs | version schemes, normalization, comparison |
src/release/records.rs | wire shapes and rkeys for software.release.* |
src/release/artifacts.rs | artifact resolver: cap, verify/compute digest, CID, cache, S3 mirror, counters |
src/release/artifact.rs | purl selection and per-ecosystem artifact fetchers (npm, PyPI, cargo, RubyGems, Maven, NuGet, Go) |
src/release/http.rs | RegistryHttp seam and production reqwest transport with retries/caps |
src/release/history.rs | resumable release_history_jobs queue |
src/release/cve_links.rs | atprotoV1 channels/versions/manifests/versionLabels/affectedRanges refs |
src/release/status.rs | append-only status transition planning |
src/release/metadata.rs | project metadata, git/target facts, versioning scheme |
src/identity/ | deterministic canonical identity and CVE ref building |
src/software.rs | local raw aliases and atprotoV1 records |
src/links.rs, src/links/repair.rs | local link index, durability queue, repair drain |
src/pds/ | publisher, same-DID assertions, governs writes and blobs |
src/migrate/, src/bin/identity-migrate.rs | out-of-band one-time identity migration |
Data flow
Common cycle
S3 snapshot download (optional)
-> open cache DB; load cursor; --backfill resets to 1999; otherwise initialize
--initial-lookback-days when no cursor
-> build RateGovernor; resolve PDS URL; login (unless --dry-run)
-> NVD fetch page -> upsert cves -> set cursor -> commit
-> publish phase: drain due publish_errors, publish pending rows, drain again
-> software phase when --publish-atproto: raw aliases, purls, atprotoV1
-> release mode dispatch (below)
-> drain link_repairs for release work
-> VACUUM INTO + S3 snapshot upload (optional)
Per-project failures inside a mode are isolated and counted; the next candidate proceeds. Release work commits local rows every 10 projects (history commits per concurrency chunk and every 50 jobs).
latest
release_candidates_after(limit, recheck_seconds, after_slug)
-> parse_purl + choose_purl (leaf match, then ecosystem priority)
-> depsdev_versions(system, name)
-> select_release (default, else newest non-deprecated representable)
-> build_source for pypi/cargo/gem only; npm/maven/nuget/golang have no adapter
-> resolve artifact through ArtifactResolver (cap, digest, cache, mirror)
-> put project -> manifest -> version -> latest channel
-> plan/apply statuses (registry signal + deps.dev deprecation)
-> projectMeta when the source exposes it and the PDS supports listRecords
-> mark release_* rows and release_project_state
The standalone mode runs only when --publish-releases is set. all runs the
same pass unconditionally.
affected
release_candidates_after(...) -> choose_affected_purl (ecosystem/source filters)
-> resolve existing project strongRef
-> local backlinks(software[].project.uri) -> CVE ids; no CVEs = skip
-> OSV: online query/querybatch by versionless purl, or bulk-dir advisories
-> affected_catalog = release_versions rows + deps.dev versions (7 ecosystems)
-> plan_versions per CVE: introduced/fixed/last_affected ranges
-> affected versions with an artifact -> immutable manifest + version
affected versions without an artifact -> versionLabels
range-only observations -> affectedRanges
-> ensure channel points at the highest known version row
-> update atprotoV1 software[] refs only when content changed
-> write release_cve_links and local links
Artifacts in affected mode use the seven-ecosystem artifact_for path; a
distro or other purl whose artifact cannot be resolved stays label-only. OSV
never creates or yanks a version; only registry signals drive statuses.
history
release_candidates_after(...)
-> build_history_source (pypi, cargo, gem, github, gitlab, bitbucket,
distro, composer, cpan, apache, gitee, gnu)
-> source.versions() -> enqueue_versions
(skip versions that already have a release_versions row; never reset
an existing pending/retry/done/error job)
-> process_jobs(limit = --release-history-limit):
resolve one artifact through HistoryArtifacts
-> reuse an existing manifest/version when recorded
-> else publish immutable manifest + version
-> plan statuses for materialized/deprecated entries
Jobs run pending -> retry -> done, or terminal error after three failed
attempts. skipped jobs carry a typed reason and are not retried
automatically. Projects are enumerated concurrently
(--release-concurrency); writes still serialize through one PDS governor.
extras
release_candidates_after(...) -> choose_affected_purl filters
-> build_source (every registered adapter)
-> source.versions() -> newest stable (prerelease/draft markers excluded;
fall back to newest prerelease when every version is prerelease)
-> source.artifacts() -> resolve_declared through the shared resolver
-> project -> projectMeta -> manifest -> version -> latest channel
-> source.statuses() only when capabilities.statuses
-> release_version_meta and release_project_state
An unchanged project (same newest version, existing version row and channel)
performs no PDS writes. A source with no artifact is skipped with reason
no_artifact; an unknown/unsupported purl type is unsupported with the
rendered UnsupportedReason in release_project_state.unsupported_reason.
all
latest -> affected -> history (only when --release-history-limit > 0) -> extras, sharing one candidate rotation cursor and one artifact resolver. Each
mode keeps its own counters; artifact_* aggregates every resolve. The runtime
guards every release arm with !dry_run, so --dry-run skips all release
modes; sync_all_with_http_resolver's library-level dry-run extras branch is
not reachable from the CLI.
Database tables
Existing NVD/publish state:
| Table | Purpose |
|---|---|
meta | cursor and small key/value state |
cves | raw NVD JSON keyed by CVE id |
publications | per-repo CVE publish status, URI/CID, blob metadata |
software | raw CPE alias rows and canonical project refs |
software_purls | PURL lookup rows; software_slug stores the canonical slug |
cve_atproto | atprotoV1 publish status |
links | local outgoing-link index (source URI/CID, path, subject URI/CID) |
Release current-state indexes:
| Table | Purpose |
|---|---|
release_projects | canonical project row, selected purl/system, latest version, record refs |
release_versions | immutable version row and manifest refs |
release_channels | mutable channel pointer per project |
release_status | current/latest status rows by subject |
Release caches, queues, and identity state:
| Table | Purpose |
|---|---|
software_identity | raw slug -> canonical slug/identity key |
project_aliases | declared alias-key mapping; no current caller outside the DB layer |
project_channels | declared channel/latest/EOL state; no current caller outside the DB layer |
release_artifacts | verified artifact cache, PK (repo, source, package, version, url) |
release_manifests | manifest key -> URI/CID/artifacts JSON |
release_history_jobs | durable version jobs with status/attempts/error |
release_osv | cached OSV advisories by purl/id |
release_cve_links | CVE -> project/version/channel/manifest refs and labels |
release_source_state | declared per-source cursor/etag; no current caller outside the DB layer |
release_status_events | append-only status transitions |
release_project_state | canonical purl, source, raw versions, checked_at, unsupported reason |
release_version_meta | published_at, raw version, artifact cid/size |
identity_migration | migration cursor per (repo, phase, raw_slug) |
link_repairs | serialized records awaiting reindex |
publish_errors | CVE publish retry state |
source_host_state | per-host token-bucket window and Retry-After backoff; read/written by GovernedHttp |
rate_budget | persistent PDS hour/day points per repo DID |
Schema is additive and created with CREATE TABLE IF NOT EXISTS; no
ALTER TABLE or app-startup migration runs.
Event and cache contracts
- Lanes report
ProgressEvent; the runtime forwards it as aRuntimeEvent.RuntimeEvent::payloadserializes the full stats struct pluserrorfor terminal phase events;CycleCompletedincludes the completeCycleStats. - stdout is NDJSON
{"event": name, ...payload}; stderr is the log stream. The full event/field table is in../MAINTAINERS.md#runtime-events. - Artifact cache key:
(repo_did, source, package, raw_version, url). A cache hit returns the stored digest/CID/size/mime and incrementsartifact_cache_hitswithout downloading. - Manifest reuse:
release_manifestsis keyed by(repo, project_slug, manifest_key)and holds the published URI/CID and artifacts JSON. An existing version row is never overwritten. - S3 mirror key: the resolver key is
artifacts/<sha256(url)>;--artifact-s3-path(defaultartifacts) is the object-key prefix, so the default object key isartifacts/artifacts/<sha256(url)>. The mirror is read first; a miss falls back to the registry; a failed put is best-effort and does not fail the resolve. - OSV cache:
release_osvstores aliases/affected/modified per(repo, purl, osv_id);summaryandseverityare advisory-only and are not persisted. - Local links:
linksis replaced per source URI on every successful write, so reindexing is idempotent.
Idempotency and immutability
software.release.projectrkeys are permanent.nameequals the rkey; renames are expressed withrenamedTo, never an in-place edit.software.release.versionandsoftware.release.manifestare immutable: re-observing one reuses the row/record; it never rewrites bytes or artifacts.software.release.channelis a mutable pointer; a new record is written only when the pointer changes.software.release.statusis append-only. The rkey is a digest over(subject, status, reason); current state is the latest live same-DID event.com.nahdig.release.projectMetais compared beforeputRecord; an unchanged second run makes zero PDS writes. Without PDSlistRecords, metadata writes are skipped because "only when changed" cannot be verified.com.nahdig.cve.atprotoV1isputRecordonly when$type,cve,nvd, orsoftwarecontent changes.- History jobs are never reset: re-enumeration skips existing versions and preserves pending/retry/error attempts.
- Artifact resolution reuses the digest cache; declared digests must match the downloaded bytes. Manifest/version writes reuse recorded rows.
- Cross-DID strongRefs are rejected before every write: project, channel, version, manifest, status, software, purl, and atprotoV1.
Failure and retry semantics
| Failure | Behavior |
|---|---|
| NVD 403/5xx/timeout | bounded outer retry at 30/60/120/300s + jitter; non-200 loops are bounded |
| NVD rate limit | client token bucket; 5 req/30s without a key, 50/30s with a key |
| PDS 429 | honor Retry-After/RateLimit-Reset, else 60s -> 1800s jittered backoff, bounded attempts |
| PDS write budget | governor sleeps until the hour/day window resets; state persists in rate_budget |
| Publish failure | recorded in publish_errors; retried before/after the next publish phase with capped exponential backoff; success clears the row |
| Link index failure | record serialized to link_repairs in the same transaction; drained at startup and after phases; no PDS calls |
| Registry 429/5xx | shared HTTP transport: 4 attempts, 1/2/4s backoff; artifact size-cap and digest-mismatch errors are not retried |
| Artifact over cap | typed TooLarge; affected falls back to versionLabels, latest/extras count a project failure, history fails the job (retried, then terminal error); never retried as a transient source error |
| Artifact digest mismatch | typed DigestMismatch; never accepted or cached |
| Source error | per-project failure counter; cycle continues; unsupported types are typed and logged |
| History job failure | attempts incremented; pending/retry until 3 attempts, then terminal error; --resume is not needed because the next cycle resumes pending/retry jobs |
| S3 cache upload | 4 attempts, 1/2/4s + jitter, post-upload size check; final failure is logged and returned |
| S3 artifact mirror | best-effort; miss re-downloads; failure does not fail the resolve |
| Fatal runtime event | orchestrator sends shutdown; process exits nonzero |
--dry-run keeps credentials optional, fetches/caches NVD data only, skips
every release mode, and writes no PDS records. identity-migrate is the only
tool that rewrites canonical refs; the service never runs it.
Related
release-sources.md— adapter matrix, auth, digests, status signals, unsupported reasons, RHEL/UBI mapping.../MAINTAINERS.md— prod commands, rate tables, events, write-budget schedules, migration runbook.