1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
# Maintainers
Prod and ops notes for the `vuln-notif` repository. Flags and events below are
verified against `src/cli.rs`, `src/bin/identity-migrate.rs`, and the PDS
governor. Release source capabilities live in
[`docs/release-sources.md`](docs/release-sources.md); module/data-flow/event
contracts live in
[`docs/release-architecture.md`](docs/release-architecture.md).
## DNS records
```bash
set -a; source ~/.nahdig_com_source; set +a
# com.nahdig.cve authority
dig +short TXT _lexicon.nahdig.com # must match logged-in DID
# com.nahdig.cve.* authority (replace did:plc:example with the logged-in DID)
doctl compute domain records create \
--record-name _lexicon.cve --record-ttl 3600 --record-type TXT \
--record-data did=did:plc:example nahdig.com
dig +short TXT _lexicon.cve.nahdig.com # must match logged-in DID
```
Negative DNS caches can keep `goat lex resolve` failing after the record is
live; wait for the negative TTL or flush the resolver.
## Publish lexicons
Publishing is an external ops step; the service binary never writes lexicons.
Use `goat` (below) or the equivalent `atpxrpc` call against the account's PDS.
Validate locally first:
```bash
scripts/check-lexicons.sh
```
The script walks all of `lexicons/` recursively and `jq`-validates every
`*.json`; it has no hardcoded list. The publish list is the ten vendored
documents: four `com.nahdig.*` schemas (two record types plus the `gitMeta`
and `projectMeta` definitions) and six `software.release.*` schemas (`defs`
plus the five record types).
```bash
set -a; source ~/.nahdig_com_source; set +a
cd vuln-notif
GOAT_USERNAME="$NAHDIG_COM_ATPROTO_HANDLE" \
GOAT_PASSWORD="$NAHDIG_COM_ATPROTO_PASSWORD" \
goat account login
goat account check-auth
LEXICONS=(
lexicons/com/nahdig/cve.nvdV2.json
lexicons/com/nahdig/cve.atprotoV1.json
lexicons/com/nahdig/release.projectMeta.json
lexicons/com/nahdig/release.gitMeta.json
lexicons/software.release.defs.json
lexicons/software.release.project.json
lexicons/software.release.channel.json
lexicons/software.release.version.json
lexicons/software.release.manifest.json
lexicons/software.release.status.json
)
for f in "${LEXICONS[@]}"; do
id=$(jq -r .id "$f")
jq '. + {"$type":"com.atproto.lexicon.schema"}' "$f" > "/tmp/schema-$id.json"
goat record create --rkey "$id" "/tmp/schema-$id.json"
done
```
If `goat lex publish` prints green but `goat lex resolve` says `RecordNotFound`,
use the `goat record create` loop above. Publish `software.release.defs` before
the record lexicons that reference it, and publish the `com.nahdig.*` record
schemas before the `software.release.*` records that reference their fields.
`com.nahdig.release.gitMeta` is a definition document embedded in manifest
artifact metadata; it is still published as a lexicon schema record so clients
can resolve the type.
## NVD API key
1. Request a free key: https://nvd.nist.gov/developers/request-an-api-key
2. Receive it by email.
3. Use it:
```bash
export NVD_API_KEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
# or:
vuln-notif --nvd-api-key "$NVD_API_KEY" ...
```
The key is sent as the `apiKey` request header. Without it: 5 requests/30s
and ~6s between calls. With it: 50 requests/30s.
## Rate limits
| Service | Limit |
|---|---|
| NVD API, no key | 5 requests / 30s |
| NVD API, key | 50 requests / 30s |
| PDS writes (per DID) | 5,000 points/hour, 35,000/day |
| PDS CREATE / UPDATE / DELETE | 3 / 2 / 1 points |
| Effective PDS creates | 1,666/hour, 11,666/day |
| Hosted API requests (per IP) | 3,000 / 5 min |
| `createSession` | 30 / 5 min, 300/day per account |
| `createAccount` | 100 / 5 min per IP |
| Blob upload size | 50 MiB |
`applyWrites` counts every record in the batch toward the write budget.
Last-week run: 14,349 CVEs = 43,047 points for `nvdV2` alone (> daily budget);
adding `atprotoV1` doubles it. Minimum ~2 days for `nvdV2`, ~3 days with both
formats. `uploadBlob` costs 0 write points.
The binary handles both limits itself: the native PDS governor paces every
PDS call (default 10 requests/s, burst 100, below 3,000 requests/5 min) and
reserves write points before create/put/delete/applyWrites. On 429 it honors
`Retry-After` (exact sleep) or `RateLimit-Reset`, otherwise backs off
60s → 1800s with jitter, and retries instead of failing the row.
`createSession` 429s retry with `max(300, Retry-After)`. The NVD client does
its own token-bucket pacing and outer retry (30/60/120/300s + jitter).
Per-source limits and retry behavior:
| Source | Limit / policy |
|---|---|
| NVD API | 5 requests/30s without a key, 50/30s with `NVD_API_KEY`; 403/5xx/timeout retried at 30/60/120/300s + jitter |
| Shared registry HTTP | 4 attempts, 1s/2s/4s backoff on 429 and 500/502/503/504; 60s timeout |
| crates.io (`cargo`) | adapter throttles every JSON request to 1 req/s per source instance |
| GitHub (`github`) | optional `GITHUB_TOKEN`; honors `x-ratelimit-remaining`/`x-ratelimit-reset`/`Retry-After`, fails typed on 429 or exhausted 403 |
| GitLab (`gitlab`) | optional `GITLAB_TOKEN` (`PRIVATE-TOKEN` header); response pagination links followed |
| Docker Hub (`docker`, `oci`) | anonymous pull token fetched per operation from `auth.docker.io`; sent only in `Authorization` |
| OSV (`osv`) | online `query`/`querybatch`; `querybatch` is split at 1,000 queries; offline bulk directory reads `*.json` without network |
| PDS | 5,000 points/hour, 35,000/day, 3,000 requests/5 min; create/put 3, update 2, delete 1 |
Every release-mode registry transport is wrapped in `GovernedHttp`: a per-host
token bucket (5 req/s, burst) plus a `Retry-After`/GitHub-rate-limit backoff
gate. Host state persists in `source_host_state` and is reloaded at startup.
Crate throttling and the GitHub/GitLab header checks remain source-level
controls inside those adapters.
## Write budget overrides
Budget state persists in the `rate_budget` table of the cache DB, so it
survives restarts and rides along in the S3 cache upload. Defaults are
5,000 points/hour and 35,000/day per repo DID with a 0.9 safety margin.
```bash
export PDS_WRITE_HOURLY=5000
export PDS_WRITE_DAILY=35000
export PDS_REQUESTS_PER_5MIN=3000
# equivalent CLI flags
vuln-notif --once \
--pds-hourly-points 5000 --pds-daily-points 35000 \
--pds-requests-per-5min 3000 ...
# refuse to start a backfill without an NVD key (also REQUIRE_NVD_KEY=1)
vuln-notif --backfill --require-nvd-key ...
```
Costs: create/put 3 points, update 2, delete 1, and `applyWrites` charges
every record in the batch at its own operation cost (deletes cost 1).
`uploadBlob` costs 0. When a window is exhausted the process sleeps until it
resets (plus jitter) and logs `rate_limit_wait`.
## Reading rate-limit events
All logs are NDJSON on stderr. Relevant events:
- `rate_limit_wait` — local budget exhausted before a write. Fields: `op`,
`points`, `remaining_hour`, `remaining_day`, `reset`, `wait_seconds`.
- `pds_rate_limited` — PDS returned 429. Fields: `op`, `status`,
`retry_after`, `remaining`, `reset`, `limit`, `sleep_seconds`, `cve`/`slug`.
- `nvd_retry` — outer NVD retry after 403/5xx/timeout. Fields: `attempt`,
`error`, `sleep_seconds`.
- `nvd_api_key_configured` — startup boolean (`configured`), never the key.
`cycle_stats` and the final `run_stats` include
`pds_budget_hour_remaining`, `pds_budget_day_remaining`,
`pds_rate_limit_waits`, `pds_rate_limit_seconds`, `pds_requests_5min`,
`nvd_key_configured`, and `nvd_retries`. Budget remaining is measured against
the 0.9-margin cap, so a fresh repo starts at 4,500/31,500.
## Runtime events
stdout is NDJSON, one `{"event": ..., ...payload}` object per line. Logs are
separate NDJSON on stderr. `cycle_stats` and `run_stats` are orchestrator
events, not `RuntimeEvent`s; every other name below comes from
`src/runtime/events.rs`.
| Event | Payload fields |
|---|---|
| `prepare_started` | none |
| `s3_downloaded` | `bucket`, `path`, `bytes`, `seconds` |
| `pds_resolved` | `handle`, `pds`, `seconds` |
| `pds_logged_in` | `repo`, `seconds` |
| `cycle_started` | `cycle`, `cursor` |
| `nvd_phase_started` | `cursor` |
| `nvd_phase_completed` | `cves`, `requests`, `retries`, `seconds`, `cursor`, `error` |
| `cves_cached` | `new`, `updated_pending`, `already_done` |
| `publish_phase_started` | `limit` |
| `publish_phase_completed` | full `PublishStats` (`calls`, `present`, `created`, `failures`, `skipped`, `retries`, `get_ops`, `create_ops`, `apply_ops`, `blob_ops`, `blob_bytes`, `oversized`, `oversized_skipped`, `record_bytes_max`, `retried`, `retry_failed`, `durations`) + `error` |
| `publish_batch_started` | `offset`, `size` |
| `publish_batch_completed` | `offset`, `size`, `missing`, `created`, `present`, `failed` |
| `software_phase_started` | `limit` |
| `software_phase_completed` | `cves`, `software_created`, `purl_created`, `atproto_created`, `failures`, `error` |
| `release_phase_started` | `mode`, `limit`, `recheck_seconds` |
| `release_phase_completed` | `ReleaseCounters` (`projects_checked`, `projects_created`, `versions_created`, `channels_created`, `statuses_created`, `unchanged`, `skipped`, `unsupported_no_purl`, `failures`) + `error` |
| `release_affected_completed` | `AffectedCounters` (`projects_checked`, `projects_linked`, `versions_planned`, `versions_created`, `versions_label_only`, `cves_linked`, `unchanged`, `skipped`, `failures`, `unsupported`) + `error` |
| `release_history_started` | `limit`, `concurrency` |
| `release_history_project` | `project` plus flattened `HistoryJobCounters` (`enqueued`, `processed`, `versions_created`, `manifests_created`, `skipped`, `failed`, `reused`) |
| `release_history_skipped` | `project`, `reason` |
| `release_history_completed` | `HistoryModeCounters` (`projects_checked`, `projects_skipped`, `unsupported`, `enqueued`, `processed`, `versions_created`, `manifests_created`, `skipped`, `failed`, `reused`) + `error` |
| `release_extras_started` | `limit` |
| `release_extras_project` | `project`, `version` |
| `release_extras_skipped` | `project`, `reason` |
| `release_extras_completed` | `ExtrasCounters` (`projects_checked`, `projects_created`, `versions_created`, `manifests_created`, `statuses_created`, `unchanged`, `skipped`, `unsupported`, `failures`) + `error` |
| `link_repair_completed` | `phase` (`startup`, `publish`, `software`, `release`), `attempted`, `repaired`, `failed`, `deferred`, `error` |
| `s3_uploaded` | `bucket`, `path`, `bytes`, `seconds` |
| `s3_upload_failed` | `bucket`, `path`, `bytes`, `error` |
| `cycle_completed` | `cycle`, `duration`, full `CycleStats` |
| `cycle_failed` | `cycle`, `error` |
| `shutdown_requested` | none |
| `fatal` | `error` |
| `done` | none |
`cycle_stats` carries the complete `CycleStats` object (NVD, publish, cache,
S3, software, release, affected, history, extras, artifact, and link-repair
counters). `run_stats` sums the `SUM_KEYS`, keeps maxima for
`MAX_KEYS`, and uses the latest values for `LAST_KEYS`, then adds `cycles`,
`started_at`, `pds_seconds_avg`, and `records_per_second`.
## Our prod run — last week, newest → oldest
```bash
set -a; source ~/.nahdig_com_source; set +a; unset ATP_PDS_URL
export ATP_HANDLE="$NAHDIG_COM_ATPROTO_HANDLE"
export ATP_APP_PASSWORD="$NAHDIG_COM_ATPROTO_PASSWORD"
export S3_ENDPOINT_URL="$CVE_NAHDIG_COM_S3_ENDPOINT_URL"
export S3_BUCKET="$CVE_NAHDIG_COM_S3_BUCKET"
export S3_ACCESS_KEY_ID="$CVE_NAHDIG_COM_AWS_ACCESS_KEY"
export S3_SECRET_ACCESS_KEY="$CVE_NAHDIG_COM_AWS_SECRET_ACCESS_KEY"
export S3_REGION="$CVE_NAHDIG_COM_S3_REGION"
export S3_PATH="prod-cve-nahdig-com-week"
# first cycle fetches 7 days; repeat while pending > 0
vuln-notif --once \
--initial-lookback-days 7 --limit 500 --order newest \
--publish-atproto --software-limit 100 \
--purldb .state/purl2cpe.db \
--cache-db .state/prod-week.db
```
Loop until pending reaches 0:
```bash
while :; do
vuln-notif --once \
--initial-lookback-days 7 --limit 500 --order newest \
--publish-atproto --software-limit 100 \
--purldb .state/purl2cpe.db \
--cache-db .state/prod-week.db
REPO=$(sqlite3 .state/prod-week.db \
"select repo_did from publications limit 1;")
PENDING=$(sqlite3 .state/prod-week.db \
"select count(*) from cves c
left join publications p
on p.repo_did='$REPO' and p.cve_id=c.id
where p.status is null or p.status='pending';")
echo "pending=$PENDING"
[ "$PENDING" -eq 0 ] && break
sleep 1800 # PDS repo-write-hour cap
done
```
`--limit` advances the NVD cursor; remaining rows stay pending. The cache is
resumable; a 429 only delays the loop. `--backfill` deliberately ignores the
stored cursor and starts from 1999; each fetched page replaces the stored
cursor, so stop and resume works the same way as a normal incremental run.
## Release records
`--publish-releases` publishes the default/latest version for software already
in the cache. Candidate rows resolve through `software_identity`, so one
canonical project receives all raw aliases; legacy raw-slug projects such as
`red-hat-red-hat-enterprise-linux-8` are no longer created. Projects use the
canonical identity slug (`red-hat-enterprise-linux`), and raw
Raw aliases stay in the local cache and inside CVE records; no
`com.nahdig.software` records are published.
```bash
vuln-notif --once \
--publish-atproto --publish-releases --release-mode latest \
--release-limit 100 --release-recheck-seconds 86400 \
--cache-db .state/prod-week.db
```
`--release-mode latest` is the default value but is a no-op on its own:
standalone latest requires `--publish-releases`. `--release-mode all` runs the
latest pass unconditionally without that switch.
- Writes `software.release.project`, an immutable `manifest`, an immutable
`version`, a mutable `channel` (`latest` for this mode), and optional `status`
records. A version or manifest is never overwritten; a channel update only
moves its `version` pointer.
- When the purl type has a source adapter, also writes
`com.nahdig.release.projectMeta` (same-DID strongRef, only when the content
changed) and records `publishedAt`/`rawVersion`/artifact cid+size in
`release_version_meta`. Manifest artifacts gain `metadata` git fragments and
a `target` object when the source declares them.
- Statuses are append-only `software.release.status` records plus
`release_status_events` rows. A deps.dev deprecation without a registry
signal goes through the same status module; an identical re-observation is a
no-op and a reason change appends a new event. CVE presence never changes a
status.
- Canonical observations supply the project, channel, and version identity.
`com.nahdig.cve.atprotoV1` emits `channels`/`versions`/`manifests` strongRefs
only for records that exist; otherwise it uses `versionLabels` and
`affectedRanges`.
- Sources: deps.dev for versions/defaults; PyPI, crates.io, and RubyGems give
SHA-256 directly; npm, Maven, NuGet, and Go are downloaded once per artifact
to compute SHA-256.
- `--release-recheck-seconds` skips projects checked recently. Use `0` to force
a fresh check.
- PDS writes still pass through the rate governor; every release record write
uses the same 5k/hour and 35k/day budget.
### Affected mode
`--release-mode affected` maps OSV advisories onto the release catalog and
updates the CVEs already linked to each canonical project.
```bash
# OSV online (default), one resumable page per cycle.
while :; do
vuln-notif --once \
--release-mode affected --release-limit 100 --release-recheck-seconds 86400 \
--cache-db .state/prod-week.db
sleep 1800
done
# Network-free / reproducible: load advisory JSON from disk instead of OSV.
vuln-notif --once \
--release-mode affected --release-limit 100 --release-recheck-seconds 0 \
--osv-bulk-dir /srv/osv-bulk --osv-offline \
--cache-db .state/affected.db
```
- OSV is queried online by default (`--osv-online` defaults to true). Use
`--osv-offline` with `--osv-bulk-dir`, or `--osv-online=false` with
`--osv-bulk-dir`; a bulk directory implies offline. Offline without a bulk
directory is a usage error (exit 2). Bulk loading reads every `*.json`
advisory in the directory and filters by purl per project.
- `--release-ecosystems` filters purl types (`pypi`, `npm`, `rpm`, ...) and
`--release-sources` filters sources (`osv` matches all affected work).
- `--release-concurrency` is accepted but affected work runs in project order
in this build; the flag affects history enumeration only.
- The catalog is existing `release_versions` rows plus deps.dev versions when
the purl type has a deps.dev system. Affected versions without a local row
get an immutable `manifest`+`version` only when an artifact is fetchable;
otherwise they remain `versionLabels`. No version is invented.
- Channels point at the highest known version row in the stream and are only
updated when that pointer changes. CVE records are `putRecord` only when the
`$type`, `cve`, `nvd`, or `software` content changes.
- `--release-mode extras` and `all` are documented in
[Extras mode](#extras-mode) and [All mode](#all-mode).
- Write budget: create/put costs 3 points, update 2, delete 1; `applyWrites`
charges every record. A first affected pass can add two immutable records
(manifest + version) per planned version, plus one channel point and one CVE
update per linked CVE. Budget with `--release-limit` and the governor's
`pds_budget_*_remaining` fields.
- Per-project source, database, and PDS failures are isolated: the cycle
continues and `cycle_stats.release_affected_failures` increases. Every 10
projects commit their local rows.
Verification after a run:
```bash
# Cycle counters (all release_affected_* fields are in cycle_stats).
grep '"event":"cycle_stats"' run.ndjson | tail -1 | jq '{
mode: .release_mode,
checked: .release_affected_projects_checked,
linked: .release_affected_cves_linked,
planned: .release_affected_versions_planned,
created: .release_affected_versions_created,
labels: .release_affected_versions_label_only,
failures: .release_affected_failures
}'
# CVE refs for one project.
curl -sS -G "$ATP_PDS_URL/xrpc/com.atproto.repo.getRecord" \
--data-urlencode "repo=$DID" \
--data-urlencode 'collection=com.nahdig.cve.atprotoV1' \
--data-urlencode 'rkey=CVE-2024-3094' |
jq '.value.software[0] | {project, channels, versions, versionLabels}'
# Local link index rows written by the run.
sqlite3 .state/prod-week.db \
"SELECT path, count(*) FROM links
WHERE path LIKE 'software[].%'
GROUP BY path;"
```
### Metadata and status verification (all modes)
```bash
# Project metadata (same-DID strongRef; name equals the rkey).
curl -sS -G "$ATP_PDS_URL/xrpc/com.atproto.repo.getRecord" \
--data-urlencode "repo=$DID" \
--data-urlencode 'collection=com.nahdig.release.projectMeta' \
--data-urlencode 'rkey=acme-widget' |
jq '.value | {project, name, versioningScheme, repository}'
# Manifest artifact metadata: git facts and the runtime target.
curl -sS -G "$ATP_PDS_URL/xrpc/com.atproto.repo.getRecord" \
--data-urlencode "repo=$DID" \
--data-urlencode 'collection=software.release.manifest' \
--data-urlencode 'rkey=<manifest-rkey>' |
jq '.value.artifacts[] | {metadata, target}'
# Version facts plus the full append-only status history for one version.
sqlite3 .state/prod-week.db \
"SELECT project_slug, version, source, published_at, raw_version
FROM release_version_meta ORDER BY project_slug, version LIMIT 20;
SELECT status, kind, reason, created_at FROM release_status_events
WHERE subject_uri='at://$DID/software.release.version/acme-widget:1.2.3'
ORDER BY created_at;"
```
Append-only semantics: a status event is never updated or deleted. Current
state is the latest live same-DID `software.release.status` for the subject; a
registry reason change appends a new record under a new digest rkey, and the
old record stays addressable. `projectMeta`, statuses, and version metadata
are compared before writing, so an unchanged second run makes zero PDS writes.
If the PDS does not support `listRecords`, projectMeta writes are skipped
because "put only when changed" cannot be verified.
## History mode
`--release-mode history` enumerates each candidate's full source catalog and
publishes immutable `manifest` + `version` records. Jobs are durable and
resumable, so run it as a long-lived loop with small pages:
```bash
# One project page per cycle; 100 versions per project; 4 projects at a time.
while :; do
vuln-notif --once \
--release-mode history --release-limit 25 \
--release-history-limit 100 --release-concurrency 4 \
--release-ecosystems pypi,cargo,gem \
--release-recheck-seconds 604800 \
--artifact-max-bytes 104857600 \
--cache-db .state/prod-history.db
sleep 900
done
# Network-free source override for the local e2e (composer/cpan/apache/gnu/gitee).
vuln-notif --once \
--release-mode history --release-recheck-seconds 0 \
--release-source-url http://127.0.0.1:4850 \
--cache-db /tmp/history-e2e.db
```
- Queue order is `(updated_at, project, raw version)`. A job becomes `retry`
after a transient artifact failure and terminal `error` after three failed
attempts. Re-running the mode never resets a terminal job; inspect
`release_history_jobs.error` before requeueing by hand.
- `--release-history-limit` is per project per cycle. `--release-limit` bounds
candidate projects, and `--release-concurrency` bounds how many projects are
enumerated at once; writes still go through the single PDS governor.
- Immutable records are reused, never rewritten. A version that already exists
is skipped at enqueue time, and a manifest whose URI/CID are recorded is
reused.
- History adapters: PyPI, crates.io, RubyGems, GitHub, GitLab, Bitbucket,
distro (rpm/deb/apk), Packagist, MetaCPAN, Apache, Gitee, GNU. npm, Maven,
NuGet, Go, Docker/OCI, and SourceForge are typed `unsupported` in history:
the first four are covered by `latest` via deps.dev, and Docker/OCI plus
SourceForge are covered by `extras`.
- `--release-source-url` applies to the history adapters with configurable
base URLs: `composer`, `cpan`, `apache`, `gitee`, and `gnu`. It is a local
e2e/test seam, not a production mirror setting.
- Token environment variables: `GITHUB_TOKEN`, `GITLAB_TOKEN`, `GITEE_TOKEN`
(or the matching flags). Tokens are sent only as request headers (Gitee uses
`Authorization: Bearer`) and are redacted from config debug output and logs.
They never appear in URLs, cycle stats, or events.
- Write budget: every new version costs two records (manifest + version = 6
points); reused rows and skipped jobs cost nothing. Statuses are re-planned
for versions materialized in the cycle and for deprecated catalog entries,
so a registry reason change appends an event without rewriting history.
A full-history pass over
256k–884k versions is roughly 797k–2.68M points (23–77 days at the default
35k/day budget). Throttle with `--release-history-limit` and
`--release-recheck-seconds`; the governor reports
`pds_budget_*_remaining` in every cycle.
## Extras mode
`--release-mode extras` covers every registered extras adapter, including the
deps.dev ecosystems pypi, cargo, and gem. One project per cycle page: newest
non-prerelease/non-draft version, one source-declared artifact, then project /
manifest / version / `latest` channel / status / `release.projectMeta`
records.
```bash
# One resumable page per cycle; rotate through the candidate set.
while :; do
vuln-notif --once \
--release-mode extras --release-limit 100 --release-recheck-seconds 86400 \
--release-ecosystems pypi,github,rpm,docker,composer \
--release-sources github,docker \
--artifact-max-bytes 104857600 \
--cache-db .state/prod-extras.db
sleep 1800
done
# Network-free local e2e (composer stub on 127.0.0.1).
scripts/release-extras-e2e.sh
```
- Candidate selection uses the same least-recently-checked rotation as the
other modes; `--release-limit` bounds the page and every 10 projects commit
local rows. Filters are applied before the source is built: a project whose
purls are all excluded is counted as `skipped` and keeps its old check time
so a later run with different filters still visits it.
- Per-project isolation: source, artifact, database, and PDS failures increment
`release_extras_failures` and the cycle continues. Unknown purl types and
not-yet-adapted ecosystems are typed `unsupported`: the reason is written to
`release_project_state.unsupported_reason` and emitted as
`release_extras_skipped`.
- Freshness is durable: `release_project_state` stores the canonical purl and
adapter, and the `release_projects` row caches the published URIs. A second
run with the same newest version is `unchanged` and performs no PDS writes;
registry status changes are evaluated when a version is (re)materialized,
not on the unchanged fast path.
- `--dry-run` skips every release mode: only NVD fetch/cache runs, with no
credentials and no PDS writes. Library-level extras dry-run planning exists
but is not reachable from the CLI in this build.
- `--release-source-url` applies to extras adapters with configurable base
URLs: `composer`, `cpan`, `apache`, `gitee`, `gnu`, and `sourceforge`.
- `--release-sources` matches purl types and adapter names (`docker` matches
`docker`/`oci`, `distro` matches `rpm`/`deb`/`apk`); `osv` only affects
affected work.
- Source statuses are append-only: an identical signal is a no-op and a reason
change appends a new `software.release.status` event. Manifests carry the
adapter's git/target facts; `release_version_meta` records the version's
`publishedAt`/`rawVersion` when the source declares them.
### Source matrix
The full per-adapter matrix (version source, artifact/digest path, status
signals, unsupported reasons, and RHEL/UBI mapping) is
[`docs/release-sources.md`](docs/release-sources.md). Operational summary:
| Purl types | Adapter | Auth | Versions / artifacts / statuses / meta | Notes |
|---|---|---|---|---|
| `pypi` | PyPI JSON API | none | yes / yes / yanked / yes | PEP 503 names, declared SHA-256 |
| `cargo` | crates.io | none | yes / yes / yanked / yes | adapter throttles to 1 req/s |
| `gem` | RubyGems | none | yes / yes / yanked / yes | declared SHA-256, HEAD size fallback |
| `github` | GitHub releases + tags | `GITHUB_TOKEN` | yes / yes / draft+prerelease+archived / yes | asset digest, archive fallback |
| `gitlab` | GitLab releases + tags | `GITLAB_TOKEN` | yes / yes / upcoming+archived / yes | `PRIVATE-TOKEN`, pagination links |
| `bitbucket` | Bitbucket tags + downloads | none | yes / yes / no / yes | no declared digest |
| `rpm`, `deb`, `apk` | distro lifecycle + public UBI OCI | none | yes / RHEL OCI only / EOL / yes | endoflife.date + Red Hat PLC |
| `composer` | Packagist p2 | none | yes / dist zip / abandoned / yes | shasum used as SHA-256 only when 64 hex |
| `cpan` | MetaCPAN | none | yes / tarball / no status record / yes | backpan entries are marked deprecated but statuses are typed `NoStatuses` |
| `apache` | Apache download indexes | none | yes / archive files / no / yes | sidecar digests disabled; SHA-256 computed |
| `gitee` | Gitee releases | `GITEE_TOKEN` bearer | yes / assets / prerelease / yes | token never in the URL |
| `gnu` | GNU FTP index | none | yes / tarballs / no / yes | index discovery |
| `docker`, `oci` | Docker Hub registry | anonymous pull token | tags / manifest digest / tag-mutation / yes | immutable digest artifact |
| `sourceforge` | SourceForge RSS/files | none | best effort / files / no / yes | typed unavailable reasons |
| `npm`, `maven`, `nuget`, `golang` | deps.dev in `latest`; registry adapters in `history`/`extras` | none | yes / yes / registry signals / yes | full history supported |
- Tokens: `GITHUB_TOKEN`, `GITLAB_TOKEN`, `GITEE_TOKEN`; all other adapters are
anonymous. Header values are never logged, and Gitee uses
`Authorization: Bearer <token>` rather than `access_token`.
- Rate limits: the shared registry transport retries 429/5xx with 1s/2s/4s
backoff and honours response pagination caps; crates.io requests are spaced
1s apart; GitHub/GitLab surface rate-limit headers as typed failures. PDS
writes are governed by `PDS_WRITE_HOURLY`/`PDS_WRITE_DAILY` exactly like
every other mode.
- Write budget: a fresh project costs about 18 points (project 3, projectMeta
3, manifest 3, version 3, channel 3, status 3 per registry signal). Re-runs
are unchanged and cost zero. Plan blocks with `--release-limit` and inspect
`release_extras_*` before scaling.
- Artifact downloads honour `--artifact-max-bytes`; declared digests are
passed to the resolver, which verifies them against the downloaded bytes and
caches the result in `release_artifacts`.
Verification after an extras run:
```bash
grep '"event":"cycle_stats"' run.ndjson | tail -1 | jq '{
mode: .release_mode,
checked: .release_extras_projects_checked,
created: .release_extras_projects_created,
versions: .release_extras_versions_created,
manifests: .release_extras_manifests_created,
statuses: .release_extras_statuses_created,
unchanged: .release_extras_unchanged,
skipped: .release_extras_skipped,
unsupported: .release_extras_unsupported,
failures: .release_extras_failures
}'
# Typed unsupported reasons, never silent skips.
sqlite3 .state/prod-extras.db \
"SELECT slug, unsupported_reason FROM release_project_state
WHERE unsupported_reason IS NOT NULL;"
# One project's newest version and its manifest strongRef.
curl -sS -G "$ATP_PDS_URL/xrpc/com.atproto.repo.getRecord" \
--data-urlencode "repo=$DID" \
--data-urlencode 'collection=software.release.version' \
--data-urlencode 'rkey=acme-widget:1.2.3' | jq '.value'
```
## All mode
`--release-mode all` runs latest → affected → history (only when
`--release-history-limit > 0`) → extras in one cycle. All modes share the
candidate cursor and one artifact resolver; each keeps its own counters:
- `release_*` for latest, `release_affected_*` for affected,
`release_history_*` for history, `release_extras_*` for extras.
- `artifact_*` aggregates every resolve in the cycle.
- `--release-sources`/`--release-ecosystems` filter affected, history, and
extras connector work.
- Use it for a small daily sweep only. A full `all` cycle over a production
cache can combine the OSV, history, and extras write budgets; keep
`--release-limit`, `--release-history-limit`, and
`--release-recheck-seconds` conservative.
```bash
vuln-notif --once \
--release-mode all --release-limit 100 --release-recheck-seconds 86400 \
--release-history-limit 20 --release-concurrency 4 \
--release-ecosystems pypi,github,composer \
--artifact-max-bytes 104857600 \
--cache-db .state/all.db
```
## Write-budget schedules by mode
Planning numbers below assume one canonical project per product and the default
35,000-point/day budget (31,500 after the 0.9 safety margin). They are estimates
for capacity planning, not measured prod totals.
| Work | Records / updates | Approx. points | Elapsed |
|---|---:|---:|---:|
| Identity migration: CVE refs | ~14.3k updates | ~28.7k | ~1 day |
| Identity migration: software/purl refs + legacy deletes | ~100 | <1k | hours |
| `latest` / default (canonical) | ~19.2k records | ~57.6k | ~1.6 days |
| `affected` (OSV) | 100k–300k records | 330k–930k | 9–27 days |
| `history` (full catalogs) | 256k–884k records | 797k–2.68M | 23–77 days |
| `extras` | ~35.7k PURLs | source-dependent | source-dependent |
Per-mode record costs:
- `latest`: project 3 + manifest 3 + version 3 + channel 3 (+ status 3 per
registry signal, + projectMeta 3 once). Re-observing an unchanged version
costs zero. A channel update is a 2-point update only when the pointer moves.
- `affected`: immutable manifest + version (6 points) per affected version with
a fetchable artifact, one channel update when the pointer moves, and one CVE
`putRecord` update (2 points) per linked CVE only when refs change.
`versionLabels`/`affectedRanges`-only versions cost zero PDS points.
- `history`: manifest + version (6 points) per new version; reused records and
skipped jobs cost zero. Jobs persist in `release_history_jobs`; status
planning may append a status record on a later pass.
- `extras`: project 3 + projectMeta 3 + manifest 3 + version 3 + channel 3
(+ status 3 per registry signal). Unchanged projects cost zero.
- `all`: sums the mode costs in one cycle; keep `--release-limit`,
`--release-history-limit`, and `--release-recheck-seconds` conservative.
Recommended multi-day order: migration (`snapshot → plan → review → apply →
verify`) first, then `latest --release-limit 100`, then `affected`, then
`history` for weeks, then `extras` with ecosystem filters.
Long runs are resumable:
```bash
# Foreground loop: one bounded page per cycle.
while :; do
vuln-notif --once --release-mode history \
--release-limit 25 --release-history-limit 100 --release-concurrency 4 \
--cache-db .state/prod-history.db
sleep 900
done
# Stop cleanly: SIGINT/SIGTERM drains the in-flight phase, commits, and exits.
# Resume by running the same command again; cursors, retry queues, and
# rate_budget live in the cache DB.
```
### Artifact cache and S3 mirror
- Verified artifact metadata is stored in `release_artifacts`, keyed by
`(repo, source, package, raw version, url)`. Confirm hits with
`cycle_stats.artifact_cache_hits`; the second resolve must not increment
`artifact_downloads`.
- `--artifact-max-bytes N` rejects oversized streams before/while reading.
`0` (default) means no cap; set an explicit cap in production.
- `--artifact-s3-bucket` plus `--artifact-s3-path` (default `artifacts`)
mirrors raw bytes. The mirror uses the same S3 endpoint and credentials as
the cache snapshot (`S3_ENDPOINT_URL`, `S3_ACCESS_KEY_ID`,
`S3_SECRET_ACCESS_KEY`, `S3_REGION`); verify with
`cycle_stats.artifact_mirror_puts`. Mirror failures fall back to the
registry and increment `artifact_errors` only when the resolve itself fails.
- Without `--artifact-s3-bucket` the mirror is a no-op: `artifact_mirror_puts`
stays `0` and only the local digest cache is consulted.
- Cache snapshot uploads (`S3_BUCKET`/`S3_PATH`) are fetched before NVD work,
then `VACUUM INTO` + upload after each cycle. Uploads get 4 attempts with
1s/2s/4s backoff plus 0.25s jitter and a post-upload size check; the final
failure is logged as `s3_upload_failed` and returned, never swallowed. A
missing remote cache is a normal first run.
### Publish retries and link repair
- CVE publish failures are recorded in `publish_errors` with an attempt count
and `next_retry_at = now + 30 * 2^(attempts-1)` seconds, capped at one hour,
plus up to 20% jitter. `drain_retries` runs before and after the publish
phase, up to 100 due rows per pass. Success marks the publication
present/created and clears the retry row; a CVE that left the cache has its
retry row deleted.
- Local link indexing is part of the write contract: PDS write → index links
(or queue the record) → DB commit → advance cursor. An indexing failure
serializes the written record into `link_repairs` in the same SQLite
transaction, so a crash cannot lose either the link rows or the repair row.
- `link_repairs` is drained at startup and after the publish, software, and
release phases, 200 rows per drain. Retries use the same
`30 * 2^(attempts-1)` formula capped at one hour plus jitter. A drain makes
no PDS calls: it reindexes from the stored record JSON and is idempotent
because indexing replaces the source's link rows.
- `link_repair_*` counters are in `cycle_stats`; each drain emits
`link_repair_completed` with `phase`, `attempted`, `repaired`, `failed`,
`deferred`, and `error`.
Verification after a history run:
```bash
grep '"event":"cycle_stats"' run.ndjson | tail -1 | jq '{
mode: .release_mode,
projects: .release_history_projects_checked,
skipped: .release_history_skipped,
unsupported: .release_history_unsupported,
enqueued: .release_history_enqueued,
processed: .release_history_processed,
created: .release_history_created,
failed: .release_history_failed,
reused: .release_history_reused,
downloads: .artifact_downloads,
bytes: .artifact_bytes,
cache_hits: .artifact_cache_hits,
mirror_puts: .artifact_mirror_puts
}'
# Queue state: pending/retry/error must be inspected before a long run.
sqlite3 .state/prod-history.db \
"SELECT status, count(*) FROM release_history_jobs GROUP BY status;"
# One immutable version and its manifest strongRef.
curl -sS -G "$ATP_PDS_URL/xrpc/com.atproto.repo.getRecord" \
--data-urlencode "repo=$DID" \
--data-urlencode 'collection=software.release.version' \
--data-urlencode 'rkey=acme-widget:1.2.3' | jq '.value'
# Local e2e: fake composer source, limit/resume, artifact counters.
scripts/release-history-e2e.sh
```
## Software identity migration
`identity-migrate` is a one-time, out-of-band admin binary. The service never
runs it. Stop `vuln-notif` before `apply` and keep it stopped until `verify`
exits `0`; never run `apply` concurrently with a service cycle. The full
runbook, including the failure-recovery matrix and snapshot restore script, is
[`docs/identity-migration.md`](docs/identity-migration.md). The rules are in
[`docs/identity-model.md`](docs/identity-model.md).
Sequence: **snapshot → plan → review diff → apply --resume → verify**.
```bash
set -a; source ~/.nahdig_com_source; set +a; unset ATP_PDS_URL
export ATP_HANDLE="$NAHDIG_COM_ATPROTO_HANDLE" # repo DID/handle, not a secret
export ATP_APP_PASSWORD="$NAHDIG_COM_ATPROTO_PASSWORD"
DB=.state/prod-week.db
MAP=migration/identity-$(date +%Y%m%dT%H%M%S)
cp "$DB" "$DB.pre-migration" # cache, cursor, rate-budget backup
# 1. Snapshot every migration collection to JSONL (read-only).
target/release/identity-migrate snapshot \
--cache-db "$DB" --snapshot-dir "$MAP.snapshot"
# 2. Inventory cache + remote listRecords and write a reviewable plan.
target/release/identity-migrate plan \
--cache-db "$DB" --purldb .state/purl2cpe.db \
--out "$MAP.plan.json"
# 3. Review every legacy -> canonical pair before writing.
jq -r '.entries[] | .action' "$MAP.plan.json" | sort | uniq -c
jq -r '.entries[] | select(.action == "create_project") | .canonical_slug' "$MAP.plan.json"
jq -r '.entries[] | select(.action == "delete_release_record") |
[.legacy_slug, .canonical_slug, .legacy_uri] | @tsv' "$MAP.plan.json" | sort -u
# 4. Apply in dependency order (projects, software, purl, CVE, status,
# normalize, deletes) with a persisted cursor. 200-record applyWrites
# delete batches.
target/release/identity-migrate apply \
--cache-db "$DB" --map "$MAP.plan.json" \
--batch 200 --snapshot-dir "$MAP.snapshot"
# If step 4 exits nonzero, rerun the same command with --resume:
target/release/identity-migrate apply \
--cache-db "$DB" --map "$MAP.plan.json" \
--batch 200 --snapshot-dir "$MAP.snapshot" --resume
# 5. Re-read the repository and fail on stale refs/backlinks.
target/release/identity-migrate verify \
--cache-db "$DB" --map "$MAP.plan.json"
```
Useful flags (all global; `identity-migrate --help`):
| Flag | Use |
|---|---|
| `--cache-db` | SQLite cache, migration cursor, and rate-budget state |
| `--purldb` | purl2cpe SQLite for planning; absent is tolerated |
| `--out` / `--map` | Plan output / plan input for apply and verify |
| `--batch` | Records per `applyWrites` delete batch; default `200` |
| `--limit` | Max plan entries per invocation (`0` = all), for chunked runs |
| `--snapshot-dir` | JSONL snapshot; accepted by `plan` and `apply` |
| `--resume` | Retry `pending`/`error` entries, skip `done` |
| `--dry-run` | Plan only; valid only for `plan` |
| `--apply` / `--verify` | Flag forms of the subcommands; can combine |
The subcommand forms are `plan`, `apply`, `verify`, `snapshot`, and
`normalize-purl-slugs`; the last one rewrites legacy full-AT-URI
`software_purls.software_slug` values in the cache and needs no credentials.
NDJSON events on stdout (one line per run):
| Event | Fields |
|---|---|
| `identity_snapshot` | `records`, `dir` |
| `identity_plan` | `entries`, `created`, `deleted`, `out` |
| `identity_apply` | `report`: `applied`, `skipped_done`, `created_projects`, `software_repoints`, `purl_repoints`, `cve_repoints`, `status_repoints`, `deleted`, `normalized` |
| `identity_verify` | `report`: `canonical_projects`, `legacy_projects`, `software_repoints`, `purl_repoints`, `cve_repoints`, `status_repoints`, `deletions`, `stale_refs`, `errors` |
| `identity_normalize_purl_slugs` | `repo_did`, `updated` |
`verify` exits `1` when `errors` is non-empty. `apply` exits nonzero and leaves
the failing entry `error`; `--resume` retries it.
### Write budget and batching
| Operation | Points |
|---|---:|
| create / put (new record) | 3 |
| update (`putRecord` on existing record) | 2 |
| delete | 1 |
| `applyWrites` | each record at its own operation cost |
The migration is dominated by CVE ref updates (~14.3k updates × 2 ≈ 28.6k
points, close to one 35k/day budget) plus software/purl ref updates and ~100
legacy deletes (<1k). Run it as a day-long job. `apply` persists one cursor row
per `(repo_did, phase, raw_slug)` in `identity_migration`, so `--resume`
skips completed entries and stopping does not lose budget. The governor still
enforces 5,000 points/hour and 35,000/day with a 0.9 safety margin.
### Rollback
The pre-apply snapshot plus the cache DB copy is the rollback material. There
is intentionally no destructive rollback command; restore is a manual
`putRecord` replay from the snapshot, documented in
`docs/identity-migration.md#rollback-via-snapshot`. Restore all collections
(project, software, purl, CVE), not only projects, or strongRefs stay broken.
Restore the cache DB copy to roll back the cursor and rate-budget rows too.
### Verification recipe
Canonical project backlinks (expect a non-zero total and canonical project
URI):
```bash
PDS=${ATP_PDS_URL:-https://your-pds.example}
DID=$(goat account check-auth | sed -n 's/^DID: //p')
PROJ=red-hat-enterprise-linux
PROJECT="at://$DID/software.release.project/$PROJ"
curl -sS -G 'https://constellation.microcosm.blue/xrpc/blue.microcosm.links.getBacklinks' \
--data-urlencode "subject=$PROJECT" \
--data-urlencode 'source=com.nahdig.cve.atprotoV1:software[].project.uri' | jq .
```
Zero legacy backlinks — query each legacy RHEL slug and expect `total: 0`:
```bash
for LEGACY in red-hat-red-hat-enterprise-linux-8 \
red-hat-red-hat-enterprise-linux-9 \
redhat-enterprise-linux \
redhat-enterprise-linux-eus \
redhat-rhel-els; do
SUBJECT="at://$DID/software.release.project/$LEGACY"
TOTAL=$(curl -sS -G 'https://constellation.microcosm.blue/xrpc/blue.microcosm.links.getBacklinks' \
--data-urlencode "subject=$SUBJECT" \
--data-urlencode 'source=com.nahdig.cve.atprotoV1:software[].project.uri' | jq -r .total)
printf '%s: %s\n' "$LEGACY" "$TOTAL"
done
```
The local equivalent checks the `links` table for unmanaged backlinks:
```bash
sqlite3 "$DB" "
SELECT subject_uri, count(*)
FROM links
WHERE subject_uri LIKE 'at://$DID/software.release.project/%'
AND subject_uri NOT LIKE '%/red-hat-enterprise-linux'
GROUP BY subject_uri;"
```
`verify` performs the authoritative check across the repository; the queries
above are the quick prod spot-check. Also confirm the canonical project exists
(`getRecord ... software.release.project/red-hat-enterprise-linux`), the CVE
has one canonical ref plus its preserved `nvd` strongRef, and a second `apply`
reports `applied == 0` and `deleted == 0`.
## Local e2e order
Services: hono-plc `:2587`, atproto-pds `:4800`, RustFS `:9000` (S3 test only).
Start **PLC → PDS → account**; the PDS must run with
`PDS_LEXICON_DIR=$PWD/lexicons` and `PDS_DID_PLC_URL=http://127.0.0.1:2587`.
```bash
scripts/local_plc.sh start
scripts/local_atproto_pds.sh start
scripts/create_local_account.sh
scripts/local-e2e.sh
scripts/blob-e2e.sh
scripts/local_rustfs.sh start
scripts/s3-cache-e2e.sh
scripts/identity-migrate-e2e.sh
scripts/release-affected-e2e.sh
scripts/release-history-e2e.sh
scripts/release-extras-e2e.sh
scripts/check-lexicons.sh
```
`local-e2e.sh` and `identity-migrate-e2e.sh` are self-contained and start
their own temporary PLC/PDS; the explicit start commands above are for a
persistent dev setup.
## Delete old collection
```bash
OLD=com.nahdig.cve
# paginated listRecords + applyWrites delete (200/batch), then:
goat record delete --rkey "$OLD" com.atproto.lexicon.schema
```
## Verify blob record
```bash
PDS=${ATP_PDS_URL:-https://your-pds.example}
DID=$(goat account check-auth | sed -n 's/^DID: //p')
REC=$(curl -sS "$PDS/xrpc/com.atproto.repo.getRecord?repo=$DID&collection=com.nahdig.cve.nvdV2&rkey=CVE-2021-3942")
CID=$(jq -r '.value.dataBlob.ref["$link"]' <<<"$REC")
curl -sS "$PDS/xrpc/com.atproto.sync.getBlob?did=$DID&cid=$CID" | gunzip | jq .id
```
## Verify backlinks
Check the canonical project and its channel/version refs:
```bash
PROJ=red-hat-enterprise-linux
PROJECT="at://$DID/software.release.project/$PROJ"
CHANNEL="at://$DID/software.release.channel/$PROJ:9"
VERSION="at://$DID/software.release.version/$PROJ:9.4"
for PAIR in "project:$PROJECT:software[].project.uri" \
"channel:$CHANNEL:software[].channels[].uri" \
"version:$VERSION:software[].versions[].uri"; do
LABEL=${PAIR%%:*}
REST=${PAIR#*:}
SUBJECT=${REST%:*}
SOURCE=${REST##*:}
echo "== $LABEL $SUBJECT"
curl -sS -G 'https://constellation.microcosm.blue/xrpc/blue.microcosm.links.getBacklinks' \
--data-urlencode "subject=$SUBJECT" \
--data-urlencode "source=com.nahdig.cve.atprotoV1:$SOURCE" | jq .
done
```
For zero legacy backlinks, query the legacy RHEL slugs listed in the migration
section and assert `total: 0`; `identity-migrate verify` is the authoritative
repository-wide check.