1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
version: 1
generated: 2026-10-05
repo: /home/johnandersen777/src/publicdomainrelay-kcp/socialweb-computer-kcp
about: >
The live backlog. docs/ARCHITECTURE.md section 14 mirrors this file: an item is written here and
in section 14 when it is opened, and moves to section 15, with its before-and-after readings,
when it is closed. Closed items are NOT kept here, because a backlog that lists its own history
reads as complete. An item is a claim about the design, so each one names the instrument that
would close it, and an item whose instrument does not exist is not yet an item.
This file was declared the live backlog by AGENTS.md, README.md, handoff.yaml and section 14 of
the architecture document, and it did not exist. Four references to a missing file is the defect
class this repo keeps finding in the other direction, so the file is written rather than the
references struck: section 14 already held the items, and a mirror that does not exist is a
backlog nobody can read.
Opened and closed on 2026-10-05 and therefore NOT listed below: CCM-1 (the cloud controller
manager is now in the process group and stamps a live Node with its provider ID) and AUTO-1 (the
scale decision carries a target total and three concurrent pods produce three nodes). Both are in
section 15 with their readings.
Opened and closed on 2026-10-06 and therefore NOT listed below: MARKET-2 (a market cluster's
tenant credential is signed from its own ClientCA on disk, chosen by the substrate, so the vault
is no longer on the path between the provider and the tenant apiserver). It is in section 15 with
its readings, including the production failure it closed and the test watched to fail without it.
Opened and closed on 2026-10-06 and therefore NOT listed below: MOUNT-1 (a cluster's kubeconfig
mount child was created before the Cluster CR reported a status.URL, and kcp copies that URL into
the workspace's spec.URL only while reconciling the mount, so the child kept an empty spec.URL and
the mount served kcp itself). It is in section 15 with its readings, including the production
reading and the four subtests watched to fail against the old unconditional create.
Opened and closed on 2026-10-07 and therefore NOT listed below: ESO-3 (the tenant add-ons applied
a ClusterSecretStore to a tenant cluster no External Secrets Operator was ever installed into, so
the pass failed every 2 s with an unknown kind until the pass installed the operator first). It is
in section 15 with its readings, including the production failure it closed and the ordering test
watched to fail with the sequence inverted.
Opened and closed on 2026-10-07 and therefore NOT listed below: ESO-4 (the tenant add-ons pass
awaited the External Secrets controller rollout, so every pass on a cluster with zero nodes - the
normal resting state - stalled 180 s waiting for a pod the node-scale pass, which runs after the
pass the wait blocks, is what would schedule). It is in section 15 with its readings, including
the production process listing and the test watched to fail without the skip.
Opened and closed on 2026-10-07 and therefore NOT listed below: MARKET-3 (a market cluster's scale
pass named the next node from the tenant cluster's joined-node list alone, so - because a market
Provision returns before its machine joins - every provision in a pass resolved to the same ordinal
and re-wrote the one claim already in flight: the count included the claim, the naming did not).
It is in section 15 with its readings, including the production log cadence, the claim and node
counts before the change, and the test watched to fail against the old ordinal.
Opened and closed on 2026-10-07 and therefore NOT listed below: MARKET-4 (a market node was counted
in flight while its claim fell short of Running and not after, so a claim that had reached Running
with its k3s agent not yet registered was in NEITHER half of the count: the pass read zero and
rented the whole target again - 15 machines for a ceiling of 10). The count is now the tenant
cluster's own node list. It is in section 15 with its readings, including the production node
count and the two tests watched to fail against the old predicate.
Opened and closed on 2026-10-07 and therefore NOT listed below: MARKET-5 (a cluster's first node
was named at construction, so a restart re-claimed node1 whatever its claim said, and
kcpclient.WriteNodeClaimTo leaves a claim whose spec matches exactly as it was -- Failed status and
the door's old words included -- so the provider read its own fresh claim as a failure and released
it 2 s later). It is in section 15 with its readings, including the production claim/release pair
and the test watched to fail without the free-ordinal rule reaching the primary host.
Opened and closed on 2026-10-07 and therefore NOT listed below: FLAKE-2 (the door's node-session
test failed about one run in four, because its fake guest let ssh2 generate the host key it then
handed back to ssh2 to parse, and ssh2's own generator occasionally emits a key ssh2's own parser
refuses). It is in section 15 with its readings, including both of the causes that were proposed and
refuted first, the rate measured with and without concurrency, and the fixed key that closed it.
Opened and closed on 2026-10-07 and therefore NOT listed below: MARKET-6 (a Node whose guest the
door's restart reaped stayed in the tenant cluster reading Ready and took pods, and the door had the
word for it all along -- it wrote Failed for a claim it reaped and for one it could not confirm
released, and Released is its own vocabulary for the first). It is in section 15 with its readings,
including pipeline 74's trace, the rate, and the reason the obvious reconcile was recorded as unsafe
before the phase distinction made it safe.
Opened and closed on 2026-10-07 and therefore NOT listed below: GL-2 (a GitLab runner off the kcp
host could not reach a tenant apiserver, or so it was recorded -- the apiserver binds loopback and
the only route is the mount, a reverse proxy carrying bearers, and whether exec survives it was
unmeasured). It closed by measurement rather than by a change: an off-host `kubectl exec` reaches
containerd through `kcp.socialweb.computer`, and the kubernetes executor that ran five green jobs at
once runs every job script with `pods/exec`. It is in section 15 with the readings, including what
the front exposes to a caller with no credential.
Opened and closed on 2026-10-07 and therefore NOT listed below: GL-1 (the minting CLI had never
been run against a real cluster's data dir -- and could not have been, because the container image
built eight other commands and not this one). It is in section 15 with the readings, including the
identity the apiserver reported back for the credential it minted, and the refusal that protects
an existing set.
Opened and closed on 2026-10-07 and therefore NOT listed below: MINT-1 (the rename that gives a mint
credential its cluster's key ran inside the clusterprovider, which the entrypoint starts AFTER the
gateway, so every mint in between named a file that was about to be renamed and was refused). The
clusterprovider gained a `migrate` subcommand and the entrypoint runs it before the gateway starts.
It is in section 15 with its readings, and the ordering is pinned by a test that reads the
entrypoint.
Opened and closed on 2026-10-07 and therefore NOT listed below: MARKET-1 (the startup sweep could
only reap records written by the substrate the gateway was configured for, because a node record did
not say which substrate wrote it -- and the identity inside one is the argv THAT substrate gave the
process). The record now carries its substrate and the sweep resolves the identity per record. It is
in section 15 with the closing reading the item asked for: a state dir carrying a firecracker record,
swept by a gateway configured for market, leaving no process from that record running.
Opened and closed on 2026-10-07 and therefore NOT listed below: ESO-3 (the tenant store's vault
server is the cluster's own subnet gateway, which a guest substrate binds and routes and the market
substrate does neither -- so the demo ExternalSecret read SecretSyncedError on every market cluster
and had never synced there). The store is no longer written where nothing can reach it, and the
operator still is. It is in section 15 with the production readings, including the loopback-only
listener and the count of binders on the gateway address.
Opened and closed on 2026-10-07 and therefore NOT listed below: ESO-2 (the tenant grant admitted the
External Secrets controller by the literal namespace `default`, so an install anywhere else was
denied and the denial read as a broken ExternalSecret). Taking the reading found a second half: the
installer ignored `ESO_NAMESPACE` altogether -- the upstream bundle names `default` in sixteen places
and defines no Namespace object -- so the grant and the install could not have agreed however the
render was fixed. Both are closed, and it is in section 15 with the readings, taken in both
directions.
Opened and closed on 2026-10-07 and therefore NOT listed below: FLAKE-3 (the OpenBao live gate lent
itself a port by listening on :0 and closing, so a second `go test` of this repo running at the same
time could take it, and the gate's plain-HTTP probe then reached somebody else's TLS listener). The
start now retries with a fresh port when either the vault cannot bind or something else answers.
It is in section 15 with the reading: ten concurrent runs, all green.
Opened and closed on 2026-10-07 and therefore NOT listed below: FLAKE-1 (a gate test was recorded as
resolving a live registry tag, so --fast could fail for reasons unrelated to a change). It closed
the other way round: the package reaches no registry at all, the failures were the IMAGE'S NAME
moving because it carries the digest of an agent binary that is built twice in one test process, and
the test now SKIPS with both digests and the mechanism when an input changes under it, where it used
to report a red about something else. It is in section 15 with the reproduction, the fix, and the
instrument watched to fire.
Opened and closed on 2026-10-07 and therefore NOT listed below: ESO-4 (the External Secrets Operator
ran as three Deployments in each tenant, so at zero nodes nothing synced, and on the market substrate
the store could not reach the cluster vault at all). It closed with plan 0005 M4-ESO-BESIDE: the
controller and its webhook are two members of each cluster's process group, the store dials the
vault on loopback, and the tenant keeps only CRDs, ClusterRoles, a binding to the beside identity and
URL webhook registrations. The instrument passed: a market-dialled cluster at ZERO nodes refused a
data-less ExternalSecret through the webhook on its loopback URL and synced users/openbao-demo from
its own vault 500 ms after the store was applied, with no ESO pod or Deployment in the tenant. It is
in section 15 with the readings and the three things watched to fail first.
Closed on 2026-10-07 by plan 0005 M6-LIMITS and therefore NOT listed under `limits`: a
process-group member that binds no port and leaves no record is not swept (every member now dies
with the provider, Pdeathsig SIGKILL from a thread held for the child's life); the instance
snapshot is load-bearing and absent reads as empty (absent is now ErrNoSnapshot, so no Node is
deleted over it); a host added to the vault's certificate is not served until the vault restarts
(the re-sign now sends SIGHUP to the verified vault process and reads the served leaf back); and
the per-pass Released-node reconcile, which listed the tenant's Nodes on every pass for ever (a
per-cluster memo now makes a pass over remembered releases cost the tenant nothing). Section 15
has each one's readings and the test watched to fail first.
open:
- id: P5
title: Plan 0005 (zero-ssh market nodes) - COMPLETE 2026-10-10; every item and acceptance check done with prod evidence.
body: >
docs/plans/0005-zero-ssh-market-nodes.yaml holds each item's evidence. Final configuration on prod:
debug key unset, SCALE_HOLD=0s, image sha256-5b823ce0, door/provider at 22d1841. Verified 2026-10-12:
all 40 items done, ACC-ZERO, ACC-MESH and ACC-NOSSH done with evidence, so the 0005 gate of plans
0006-0008 is satisfied; their items start as their own depends_on allow.
instrument: >
None open.
- id: P10
title: plan 0010, GitLab CI and the post-prod integration tests (R0-REASSESS done 2026-10-12).
body: >
docs/plans/0010-gitlab-ci-and-integration-tests.yaml. First item R0-REASSESS: re-assess every
gap and approach, ask the user the listed questions, revise the plan. Nothing else starts before it.
Gaps: no repeatable acceptance, one bidder, restart leak, kernel/rootfs contract, dead substrate
code, thin NetworkPolicy coverage, leaderElect guard, debug-ssh guard, hand-built GitLab, leftovers.
OPEN NOTE (2026-10-12, flagged by the 0006-0008 fold; 0010 not edited): 0010 G4-SCHED-SINGLE-SERVER
(GAP-7) proposes that clusterstack REFUSE to render kube-scheduler leaderElect false for a spec with
more than one k3s server, keeping false for one server; plan 0006 M3-CP-SLOTS (decided by the user
2026-10-12) turns market scheduler leader election ON for every cluster (leaseDuration 4s, renewDeadline
3s, retryPeriod 1s; reverts 1cbf945), because two k3s slots overlap during every blue/green swap. The two
edit the same render (internal/clusterstack/scheduler.go) with opposite intent; once M3-CP-SLOTS lands,
GAP-7's premise (one server, election off) no longer holds. R0-REASSESS must reconcile: drop
G4-SCHED-SINGLE-SERVER, or reduce it to "refuse leaderElect false always" and sequence it after
M3-CP-SLOTS. Unresolved until R0-REASSESS answers.
instrument: >
R0-REASSESS answered and recorded in the plan, including the leaderElect conflict with 0006 M3-CP-SLOTS.
- id: P6-M0
title: OPEN - the one binary's linux budget (cross-OS gate and kcp spike done).
principle_note: >
Organizing principle of P6-P8 (user 2026-10-12; plan 0006 `principle`, ARCHITECTURE section 0): kcp is
a core part of each APPLICATION - an application instance (bidder handle, run request, gateway, market
market network, dev instance) = its own kcp + a manifest set + a profile; state = kcp objects; components =
DenoPods/HostProcesses; modes = profiles; composition = applying sets / nesting workspaces;
applications meet only over the atproto market protocol, never by sharing a kcp; port 0 behind the
edge with addresses in kcp (check: 0006 M1-NO-FIXED-PORTS); every workstream tests in its own market
market network, bobvmbuilder + alice are a final soak only. The one traditional use is the virtual clusters
project (items tagged kcp_use: traditional-exception). Decomposing today's multi-server processes into
application instances: FOLDED 2026-10-12 (fifth fold; plan 0006 principle.decomposition) from
multi-server-to-kcp.md with the user's section-7 answers - one root object per application (0006
M2-APP-CONTROLLER); STRICT kcp per application inside a market network too (the market network is a MarketNetwork composing
children that each run their own kcp; the nested-workspace market network rejected); every hono-bidder and
request-vm-ssh option kept kcp-native, sign-in as a MarketIdentity (0007 M1-MARKET-IDENTITY); kcp only, no
standalone deno CLIs; a fresh kcp per requester invocation with a durable state dir; sessions and keys
only in each application's OpenBao; cross-app values only through the composer; iroh the only guest
transport (0008 M1-SSH-GATEWAY-IROH, then M1-STRIP-LEGACY-TRANSPORTS).
body: >
OPEN: the plan 0005 gate is satisfied (ACC-ZERO, ACC-MESH, ACC-NOSSH done); items start per their
depends_on. M0-XOS-GATE DONE 2026-10-12 (merged; check.sh gates darwin/windows build+vet). M0-KCP-SPIKE
DONE 2026-10-12 (kcp-spike.md; branch kcp-spike 5f8ef8c, main d6e1881): the user chose OUT OF PROCESS -
the unpatched release kcp v0.33.0 as an embedded asset (176.8 MB raw / 59.2 MB zstd-19), alternating
blue/green slot units on kine -> a dedicated postgres behind our own L4 front on 127.0.0.1:6443 - for
the GATEWAY; a bidder's kcp is one slot on kine-sqlite, its upgrade a short restart VMs never notice
(decided by the user 2026-10-12, correcting d6f778e). Remaining: M0-SIZE-BUDGET and M0-PG-SOURCE (new
2026-10-12: spike with fallback - zonky embedded-postgres binaries, theseus-rs/postgresql-binaries, or a
build from source with the build-host script; choice and sha256 pins recorded) (plan 0006,
docs/plans/0006-socialweb-computer-unified-binary.yaml). Both can start now.
instrument: >
The linux size budget, with the kcp and postgres asset lines, enforced by check.sh; the postgres source
and per-platform sha256 pins recorded.
- id: P6-M1
title: OPEN - CLI (with the dev tree and a required host-backend), host inventory, rustfs/t4 leftovers removed, per-OS assets, instance-keyed leases, `dev up|down|status` per worktree, the one `install`/`uninstall` root command.
body: >
OPEN as P6-M0. Plan 0006 items M1-CLI, M1-HOST-INVENTORY, M1-STORE-CLEANUP (the tenant store is kine
over the cluster's own postgres; vaultlayout central-store, socialweb-vault central-store options,
versions.json rustfs/t4 entries deleted), M1-ASSETS, M1-PORT-LEASE-INSTANCE (new 2026-10-12: leases
keyed by (instance, cluster, purpose), one per-user lease file in dev), M1-NO-FIXED-PORTS (new
2026-10-12, fourth fold: a check.sh --fast test fails on any fixed port in a manifest or default behind
the edge; edge exceptions in deploy/edge-ports.yaml, today's literals in a ratchet that only shrinks),
M1-DEV-INSTANCE (new 2026-10-12:
`socialweb-computer dev up|down|status|ls|logs|env|check|restart` per worktree, instance id from the
worktree path, XDG state keyed by instance, one devsup per instance, port 0, no root), M1-INSTALL.
M1-K8S-FORK and M1-KCP-INPROC closed 2026-10-12, route not chosen. Assets are embedded per GOOS_GOARCH
and collected only when no loaded unit names them (dev: one per-user cache, `dev gc`); the core installs
to bin/<sha256>/ with current/previous symlinks. Decided by the user 2026-10-12: `socialweb-computer
install bidder|gateway` is the one explicit, idempotent root step, with `uninstall`: account (never
created on demand elsewhere), bidder subuid/subgid range for per-VM user namespaces, kvm before linger,
delegation drop-in, AppArmor profile for the embedded pasta, node password, socialweb.slice, the core
unit files and the BOOTSTRAP-LAYER unit files (kcp store, kine, slot(s), front; M2-BOOTSTRAP-LAYER),
linger, `host-backend: systemd-user` in the account's config.json; every other unit (vaults, cluster
members, VM units) is a HostProcess placed at runtime. The dev run mode is kcp-native now (decided by
the user 2026-10-12): host-backend required with no default (install writes systemd-user, dev passes
proc), gateway dev store sqlite (quick) or postgres (market network/prodlike), k3s node password by a per-instance
userns bind-mount, AppArmor report-only in dev, one uid for every role in dev. Fifth fold 2026-10-12:
M1-CLI states the application-verb contract (apply the root object with --profile, watch, render
pending actions; no verb opens a listener or spawns deno; options.json gains `profile`; its scan test
lands with M2-APP-CONTROLLER so the binary-base slice is not widened); M1-DEV-INSTANCE's `--profile market-network`
is a MarketNetwork composing child applications with their own kcps (exits 2 naming M2-APP-CONTROLLER
until it lands).
instrument: >
Every executable the code runs has a fate in the host inventory; no rustfs or t4 name outside the
store measurement in stack.go; an extracted asset that no longer matches its pin is replaced, not run;
install run twice changes nothing the second time, uninstall --purge restores every host file it
touched, every installed unit file verifies; upgrade refuses a bad manifest and rollback retargets
current; serve without --host-backend exits 2; two instances leasing from one file never share a port;
two worktrees on hat-8 run `dev up` / `dev down` in parallel and share nothing.
- id: P6-M2
title: OPEN - user-manager units adopted never re-created, the host contract, the VM tree audit, kcp on port 0, postgres (with its upgrade paths) and OpenBao CRDs, apply from Go.
body: >
OPEN as P6-M0. M2-HOST-UNITS DONE 2026-10-12 (internal/hostsvc merged; lingering-account proof left to
M2-HOST-CONTRACT). Plan 0006 items M2-HOST-UNITS (was
M2-HOST-SCOPES), M2-HOST-CONTRACT (adds the subuid range, newuidmap and ACL checks), M2-VM-TREE-AUDIT,
M2-KCP-STORE (one kine per slot over a dedicated postgres unit on a unix socket, started before kcp,
not a CRD object), M2-KCP-FRONT (our L4 front on 127.0.0.1:6443, fd store, caddy never touched,
install_trust false), M2-KCP-SLOTS (was M2-KCP-UNIT: blue/green slot units, leader election, shard
base URL = the front, five shared root files, shard-admin consumers only), M2-KCP-PORT0 (no kcp patch:
the port read from the unit's single LISTEN socket in /proc), M2-POSTGRES (minor upgrade = unit restart, major = logical replication
with a write freeze; no container-era data dir), M2-OPENBAO-SERVER, M2-MANIFESTS, M2-TEMPLATE, M2-APPLY.
Third fold 2026-10-12 (decided by the user: dev run mode kcp-native now; bidder kcp one slot on
kine-sqlite): new M2-HOST-CONFORMANCE (one Manager contract suite, proc on every check.sh --fast,
systemd-user behind livegate), M2-HOST-BACKEND-PROC (devsup: per-instance parent, subreaper, process
groups, tree kill, fd store, ns-exec), M2-BOOTSTRAP-LAYER (kcp store, kine, slot(s), front and core:
installed units in production, devsup units in dev, before kcp; mirrored into kcp read-only),
M2-HOSTPROCESS-KIND (HostProcess objects reconciled by the in-binary supervisor onto systemd-user or
proc; Postgres, OpenBaoServer, cluster members and ComputeVMs own HostProcess children). M2-HOST-UNITS
is the systemd-user backend (done, evidence kept); M2-KCP-STORE/SLOTS: the bidder's kcp is one slot on
one sqlite kine with no postgres and no leader election; M2-VM-TREE-AUDIT is backend-aware.
Transient SERVICES in the user
manager of a lingering account over the private socket with deadlines, PropExecStart(argv, false),
CollectMode=inactive, MemorySwapMax=0, no AllowedCPUs/IOWeight, own slice; systemd 253+ with probed
features; the invariant that no socialweb code runs in a VM unit, audited from its cgroup. Fifth fold
2026-10-12: new M2-APP-CONTROLLER - each application ONE root object (MarketBidder, MarketRequester,
Gateway, MarketNetwork) in its own kcp, rendered by the core's appctl into owned children; a profile
switch deletes what it omits; option coverage (every hono-bidder and request-vm-ssh option maps to a
field or profile, removal only for a legacy transport); a MarketNetwork composes child applications,
each with its OWN kcp (bootstrap layer as HostProcess objects in the network's kcp), copying network
values onto their root objects as the only cross-app discovery.
instrument: >
A stand-in unit survives `systemctl --user restart`, kill -9, stop, daemon-reload and daemon-reexec of
its starter and is adopted; a failed unit's Result is readable until reset; the tree audit fails a core
binary in a VM unit; a Postgres object survives its controller's restart; a restarted bao is unsealed
on the next pass; a second apply of every set changes no resourceVersion; two kcp slots on one kine ->
postgres store serve the same workspaces and a front flip fails no /readyz poll; the contract suite is
green on proc and systemd-user; two dev instances each serve their own kcp on port 0; a HostProcess
reconciles identically on both backends and a deleted bootstrap mirror stops nothing; a bidder kcp
restart leaves every stand-in VM unit's InvocationID unchanged.
- id: P6-M3
title: OPEN - serve gateway kcp on the host with kcp first, each cluster a slice of member units behind a per-cluster apiserver front, blue/green k3s+kine slots, the gateway as a user unit, continuity across upgrades proved.
body: >
OPEN as P6-M0. Plan 0006 items M3-FOLD (nodeboot deleted, nodeimage to CI), M3-CLUSTER-UNITS (each
cluster a systemd slice of member units, no resident supervisor; kine and bao After=/Wants= the
cluster's postgres unit), M3-APISERVER-FRONT (a per-cluster L4 front on the existing apiserver port;
tickets, kubeconfigs, status.URL unchanged), M3-CP-SLOTS (new k3s+kine slot beside the old on the same
postgres, front swaps, old stops; idle first; scheduler leaderElect true with a 4 s lease; each slot
its own hostname so k3s's wrangler lease never has two holders), M3-LINK-HANDOVER (the link keeps its
iroh key; dumbpipe duplicate-endpoint behaviour measured), M3-BAO-SYNC, M3-SERVE-GATEWAY-KCP (order:
kcp, sets, root vault, rest; fresh state dir), M3-HOST-INSTALL (user unit, default KillMode),
M3-CP-CONTINUITY (decided by the user 2026-10-12: continuity for upgrades of the finished system only,
blue/green behind the front; the end-to-end market network proof), M3-KCP-ROLL (`units roll kcp`: pg_dump, new
slot beside the old, /readyz on its own port, flip our front, old exits, lease
kube-system/kcp-controllers deleted; storage-incompatible release pairs take a planned outage; on a
bidder it is a short restart of the one slot, VMs untouched, decided by the user 2026-10-12). Third fold
2026-10-12: cluster members are HostProcess children (either backend, StopSlice), fronts keep their fd
through UnitSpec.FDStore (devsup on proc), k3s slots get their hostname and, in dev, the node password
through `internal ns-exec`. Open note: M3-CP-SLOTS' scheduler leader election conflicts with plan 0010
G4-SCHED-SINGLE-SERVER (see P10). Fifth fold 2026-10-12: M3-SERVE-GATEWAY-KCP applies a Gateway root
object (profiles prod and market-network; the door's TEST_* seams become market-network profile values; door accounts are
MarketIdentities) and depends on M2-APP-CONTROLLER.
instrument: >
A zero-node market cluster's /readyz never fails across `systemctl --user restart` of the gateway unit
and an upgrade, with every member unit's InvocationID unchanged; a postgres restart stops no member;
two stand-in k3s slots on one postgres swap behind the front with 0 failed polls and never both hold
kube-system/k3s; on the market network (node via runComputeContract) every upgrade kind keeps the node Ready
and owned with no vm.delete; a vault seeded with stale entries is reduced to the live set in one pass;
a kcp roll fails no /readyz poll through the kcp front and leaves no controller gap over 5 s (61 s
today without the lease delete); a profile switch of a root object deletes the children it omits; two
bidder children of one MarketNetwork run two distinct kcps.
- id: P6-M4
title: OPEN - a reproducible static build per linux platform within its size budget; later, a parent kcp for nesting dev instances.
body: >
OPEN as P6-M0. Plan 0006 items M4-BUILD and M4-DEV-PARENT-KCP (new 2026-10-12, a later item the user
accepted: `dev host`, a parent application instance whose root:dev:<instance> workspaces hold each child instance's
HostProcess objects).
instrument: >
Two clean builds byte-identical; the binary within the budget in versions.json.
- id: P6-M5
title: OPEN - the finished binary deployed fresh to prod (the user runs it); the container path deleted.
body: >
OPEN as P6-M0. Plan 0006 items M5-FRESH-DEPLOY (was M5-CUTOVER; decided by the user 2026-10-12: no
backwards compatibility, migration, cutover, rollback or prod-downtime planning until the finished
solution exists; the gateway runs no containers; install + upgrade on an empty state dir, no volume
copy, no rollback script), M5-RETIRE (delete deploy/control-plane-container, container-entrypoint.sh,
run-container.sh and the docker pieces of scripts/deploy.sh; no waiting period) and the acceptance
checks ACC6-ONE-BINARY, ACC6-STATE, ACC6-UPGRADE, ACC6-CP-SWAP, ACC6-BAO-SYNC, ACC6-REPRO, ACC6-XOS,
ACC6-DEV (new 2026-10-12: two dev instances in parallel on hat-8, no root, no systemd).
instrument: >
After the fresh deploy: door /api/config 200, the bootstrap cluster Ready, None nodes 0, no container
on the host; after M5-RETIRE nothing in the repo invokes docker, kubectl or socat outside test assets.
- id: P7-M1
title: OPEN - the per-VM userns spike, compute-provider-kcp's shape, the lifecycle on record, the firecracker learnings inventoried, guest images from a build-host script, the new-NSID checklist and a working QR deploy.
body: >
OPEN: the plan 0005 gate is satisfied (ACC-ZERO, ACC-MESH, ACC-NOSSH done); items start per their
depends_on, incl. the plan 0006 and 0008 items they name. Plan 0007 (docs/plans/0007-compute-provider-kcp.yaml), revised 2026-10-11 and 2026-10-12 from
docs/research/2026-10-11-vm-lifecycle/, items M0-VM-USERNS (new 2026-10-12: spike with fallback -
firecracker opening /dev/kvm in a per-VM subuid userns with pasta; a single-threaded stage of ours vs
util-linux unshare >= 2.38; newuidmap/newgidmap host prerequisites; stand-ins plus one RFP-flow guest),
M1-DECIDE, M1-FC-INVENTORY, M1-API (OS-neutral status, no network index: every guest has the identical
address plan), M1-COMMON-EXTRACT, M1-GUEST-IMAGE (kernel + rootfs per hypervisor x arch built by
scripts/build-guest-image.sh on any Linux host - CI shells out to it later, no CI service now, decided
by the user 2026-10-12 - pinned by sha256, fetched and verified), M1-QR-DEPLOY (new 2026-10-12:
did-key-associator/scripts/deploy.sh ships hono-backend/mod.ts alone though it imports
../typescript-helpers and ./cli-args-env.json, so running it would take the QR backend down; only
static files were deployed 2026-10-12, backup /root/qr-static-backup/20261012T014155Z; fix = a
self-contained compiled backend, a --static-only mode and a --dry-run), M1-PROTOCOL-CHECKLIST (new
2026-10-12: a new NSID needs, in order, the lexicon, the typescript-helpers role-set scope, the
regenerated did-key-associator client scope deployed to qr.fedfork.com, and every account re-signed in
through the QR - refresh keeps old scopes, 403 ScopeMissingError otherwise; a check test holds every
written lexicon in its role set). compute-provider-firecracker is the gold standard; both providers stay.
Fifth fold 2026-10-12, EARLY by the user's order: M1-MARKET-IDENTITY (no dependency; the research's
M4-MARKET-IDENTITY) - kind MarketIdentity (modes oauthQr, oauthBrowser, pdsAccount, localRepo = the
ephemeral-PDS mode kept for bidder and requester, oauthDelegated for door accounts); every interactive
step (QR, association y/N, browser OAuth, session expiry) becomes status.pendingAction that the
foreground CLI renders, approvals are spec patches; no key, token or code in the object (vault refs
only); tests complete sign-in through hono-pds's session minting (createSessionInjector). M1-FC-INVENTORY
and M1-COMMON-EXTRACT record the guest JSR registry as not inherited (stripped, 0008).
instrument: >
Every exported function and logger event of the firecracker provider maps to a requirement entry;
ComputeVM fixtures round-trip through Go and TypeScript with no backend enum; enriched user_data
byte-identical across the extraction; an image whose sha256 differs is refused; the userns stage chosen
with readings; the compiled QR backend starts from an empty dir; every NSID the bidder or requester
writes is in its role set; a market network bidder runs without --skip-qr, its association approved through its
MarketIdentity.
- id: P7-M2
title: OPEN - the VM runtime with no socialweb code in the VM's tree, parity with the firecracker provider, the Verdict controller, no leaked VM or work dir.
body: >
OPEN as P7-M1. Live E8 (2026-10-12, RFP flow): SendCtrlAltDel -> firecracker exit 0.21 s on an idle
guest; today's provider never noticed a VMM that exited (nodeboot parents and the runner container
stayed up) and never removes destroyed guests' work dirs (32 dirs, 63 GB on hat-8); M2-VM-RUNTIME and
M2-CONTROLLER must make both impossible (Watch -> Lost within seconds; finalizer only after the work
dir is gone; orphan work dirs collected). Plan 0007 items M2-VM-RUNTIME (replaces the deleted M2-UNPRIVILEGED-RUNNER: one
transient service per VM, pasta -> /bin/sh trampoline -> socialweb-vmexec execve's firecracker
--config-file; graceful stop = SendCtrlAltDel, a grace window measured live, then KillSignal=SIGKILL;
arm64 straight to SIGKILL; console to a capped file in the work dir; decided by the user 2026-10-12:
each VM unit in its own subuid sub-range of the account's reserved range so a VMM escape owns nothing,
and a separate pure-Go cidata seed drive with in-guest root growth, no debugfs/e2fsck/resize2fs on
hosts), M2-BACKPORT (parity at the VMSpec, seed drive and fc.json; the host-side key scan is deleted,
not ported), M2-CONTROLLER (tri-state Verdict: identity mismatch
= Lost + StopUnit now, unreachable bus or socket = Unknown, boot id changed = Lost). Third fold
2026-10-12 (decided by the user): every VM unit is a HostProcess child of its ComputeVM placed on
systemd-user (production) or proc (dev); the userns stage is M0-VM-USERNS's choice; a bidder kcp
restart stops no VM and the controller re-adopts every VM after it.
instrument: >
Parity tests green on both providers; a stand-in VMM unit's tree is pasta + the stand-in only and its
stop ends within TimeoutStopSec; a stand-in VMM that exits on its own ends its unit and is Lost within
2 s; one offline test per Verdict row; a Lost guest is never booted again; 32 seeded orphan work dirs
are gone after one pass past the grace; two stand-in VM units map disjoint subuid sub-ranges and the
VMM uid owns no bidder file.
- id: P7-M3
title: OPEN - the TypeScript provider, the bidder flag that selects it, a NODE-LOSS clock our own outage never feeds; a market network bidder on its own PDS account, per-instance OIDC issuers, the bidder composer.
body: >
OPEN as P7-M1. Plan 0007 items M3-TS-PROVIDER (no guest JSR registry since the fifth fold),
M3-BIDDER-FLAG (adds the handle and the constant guest host address), M3-NODE-LOSS-CLOCK (NotReady
counts from max(lastTransitionTime, pathUpSince): a cluster cold start, link or front restart never
releases a node whose path returns; no depends_on, can start now). Fifth fold 2026-10-12 (startable
now unless noted): M3-BIDDER-ACCOUNT-PDS (today the market network bidder ignores bob's PDS pod, the
handle+password path builds a local repo and --atproto-pds-url is read nowhere; it becomes a
remote-PDS session), M3-OIDC-ISSUER-INSTANCE (oidc-issuer-hono's module-global issuer state moves into
the instance), M3-BIDDER-COMPOSER (after M1-MARKET-IDENTITY and 0008 M1-STRIP-LEGACY-TRANSPORTS:
hono-bidder's 825-line composition becomes lib/hono-factory-market-bidder-app driven by a component
spec; no standalone mode; the harnesses compose through it).
instrument: >
Against a fake kcp, provision resolves on Running and destroy only after 404; createBidConfig
equals the firecracker record; a node NotReady during our link outage is kept for 90 s after the link
returns, and one NotReady before a provider restart is still released on time; a bidder's offering is
listed by its own PDS account with no repo served by the bidder; two issuers in one process name their
own issuer; hono-bidder/mod.ts under 150 lines with every bidder test green.
- id: P7-M4
title: OPEN - a restarted bidder keeps its issuer, its keys and its contracts (the TypeScript store NOW); guest host keys are guest-reported and verified.
body: >
OPEN as P7-M1. Plan 0007 items M4-CONTRACT-STORE-FILE (new 2026-10-12, split from M4-CONTRACT-RESTORE
by the user and startable NOW: market-bidder ContractStore load() before subscribe and awaited save,
a file-backed store for compute-provider-firecracker, terminated-receipt tombstones persisted; bidder
restarts kept losing receipts and a guest leaked 2026-10-12T01:31Z), M4-BIDDER-VAULT, M4-BIDDER-BAO-SYNC,
M4-CONTRACT-RESTORE (now only the ComputeVM-backed store), M4-ONNETWORK, M4-HOST-KEYS (follows
M1-PROTOCOL-CHECKLIST and needs M1-QR-DEPLOY; decided by the user 2026-10-12: a bidder-signed
com.publicdomainrelay.temp.compute.events.vm.hostKeys record in the market.event envelope, reported by
a cloud-init module with proof of possession; request-vm-ssh and the door use it with
StrictHostKeyChecking=yes; the host-side ssh-keyscan is deleted).
Live on hat-8 2026-10-07 a restarted bidder lost its receipts,
vm.delete failed and the guest leaked; its issuer URL also moves, leaving one jwt mount per issuer
in cluster vaults. The keys move into an OpenBao beside the bidder, reconciled against live
ComputeVMs, with one stable issuer per bidder. Fifth fold 2026-10-12 (user: sessions and every key only
in the application's OpenBao, never kcp Secrets): M4-BIDDER-VAULT holds EVERY relay keypair by name
(1 + P relays, +1 for an in-process repo, all fresh per process today), the OAuth session and DPoP key
(MarketIdentity sessionRef) and a pdsAccount password; it depends on M3-OIDC-ISSUER-INSTANCE and
M1-MARKET-IDENTITY. M4-HOST-KEYS' known_hosts rule lands on the iroh ProxyCommand only (after the 0008
strip).
instrument: >
A bidder seeded from its ContractStore answers vm.delete 200 and destroys once; three starts keep
the same did, issuer URL and JWKS; restarts and an upgrade leave one jwt mount per bidder in the
cluster vault; request-vm-ssh verifies the guest-reported key and a wrong key fails verification.
- id: P7-M4-TERMINATION
title: DOING - the bidder tells the requester when and why a guest ended (vm.terminated).
body: >
DOING as P7-M1. Plan 0007 item M4-TERMINATION-EVENT (decided by the user 2026-10-12): new lexicon
com.publicdomainrelay.temp.compute.events.vm.terminated, bidder-authored, in the existing market.event
envelope; one of plan 0007's two protocol additions with vm.hostKeys (ownership.not_owned,
ACC7-NO-PROTOCOL-CHANGE). TS side done in atproto-market pre-iroh (eb2aa82 lexicon, 7c4a8f4 bidder
publish + vm.delete ack + tombstone, bdd423b requester watch, onTerminated hook, ssh abort, no
redundant vm.delete). Live on hat-8 2026-10-12 through the RFP flow: CtrlAltDel 01:44:39.873Z ->
requester vm_session_ended_by_bidder + vm_delete_skipped 01:44:45.429Z (5.6 s), late vm.delete
'already-terminated' on both paths at 01:33:07Z; needed typescript-helpers 5b071fe (bidder role
scope), did-key-associator 3cde452 (QR client scope) and a bidder QR re-sign-in. Remaining:
the Go controller and compute-provider-kcp emit it from Lost verdicts at re-population and from unit
failure (in practice after M2-CONTROLLER and M3-TS-PROVIDER, though its depends_on is empty); the door
acts on it in M3-NODE-LOSS-CLOCK / M6-*. Its in-memory tombstones are persisted by
M4-CONTRACT-STORE-FILE (startable now). The scope steps it needed are now M1-PROTOCOL-CHECKLIST.
instrument: >
test/vm_terminated_{lexicon,bidder,requester}_test.ts green; live on the market network a guest powered
off from inside ends request-vm-ssh within seconds (vm_session_ended_by_bidder) and no vm.delete is
sent; a late vm.delete answers 200 with 'already-terminated', never 'unknown receipt'.
- id: P7-M5
title: OPEN - run bidder as a user unit of the shared lingering bidder account, no docker, least-privilege deno.
body: >
OPEN as P7-M1. Plan 0007 item M5-RUN-BIDDER: one shared lingering account (swc-bidder) for all
handles, created only by `install bidder` with its subuid/subgid range, units namespaced by handle,
kvm added before linger, default KillMode, the QR guard in Go,
the bidder DenoPod with no -A, --allow-run or --allow-ffi and /dev/kvm and $XDG_RUNTIME_DIR denied.
Third fold 2026-10-12 (decided by the user): the handle's kcp is one slot on kine-sqlite behind a port-0
front in the bootstrap layer (corrects "kine -> postgres"); `run bidder --host-backend=proc` runs in a
dev instance with no account; --host-backend is required. Fifth fold 2026-10-12: M5-RUN-BIDDER applies a
MarketBidder root object with --profile prod-qr (no PDS, oauthQr) | market network (own PDS DenoPod, pdsAccount) |
dev-local (localRepo) and renders MarketIdentity pending actions in the foreground; hono-bidder runs only
as the bidder DenoPod's entry from then on (exits 2 outside one; the standalone launcher and
bob-bidder.sh path deleted). New M5-BIDDER-COMPONENTS (user 2026-10-12): one `guest-<provider>` DenoPod
per provider owns the provider relay, the OIDC issuer, /v1/on-network and /v1/host-keys, so a core
restart never moves what guests dial; no guest-jsr pod (stripped).
instrument: >
The bidder user unit verifies with no KillMode=mixed; deno in the bidder pod cannot open /dev/kvm or
dial the user bus or the systemd private socket; `run bidder` shows the QR from status and returns when
Ready; a bidder-core restart leaves the guest's issuer and onNetwork serving.
- id: P7-M6
title: OPEN - bidder and gateway upgrades leave VMs running, in-flight claims resume across a door restart, recovery cases, reboot = Lost, all through the RFP flow.
body: >
OPEN as P7-M1. Plan 0007 items M6-BIDDER-UPGRADE (also measures the stop grace window under a k3s
workload; E8 read 0.21 s idle), M6-CLAIM-RESUME (decided by the user 2026-10-12: per-claim
secrets-server key in the cluster OpenBao + status.secrets on the NodeClaim, write-ahead acceptPending;
no depends_on, can start now), M6-GATEWAY-REPOPULATE (record scan only for acceptPending claims and
unclaimed receipts; decided by the user 2026-10-12: an unclaimed receipt is adopted first through
claim phase Adopting - matched by node name = guest name, vault role node-<name> and the vm.hostKeys
record; joined + Ready -> adopted, not joined but its cluster exists -> adopted on join within the
grace, cluster gone or no join -> released once), M6-LIVE-RECOVERY (kill -9, systemctl restart, logout with linger, user-manager
restart = reboot semantics), M6-NOSSH-JOIN and the acceptance checks ACC7-* (incl.
ACC7-NO-RESIDENT-CODE, ACC7-NO-LEAK, ACC7-CLAIM-RESUME). The gateway rebuilds its ownership from
NodeClaims, receipts, market records and Nodes, including VMs bidders elsewhere rented to it.
instrument: >
TestABidderUpgradeLeavesItsVMRunningAndStillReleasesIt, TestADoorRestartMidProvisioningStillJoinsTheNode
and TestAGatewayUpgradeKeepsEveryMarketNodeReadyAndOwned, through the RFP flow, with zero ssh sessions,
the VM tree audit passing after every boot, restart and upgrade, and no work dir left after a release.
- id: P8-M1
title: OPEN - a pinned deno and the TypeScript closure inside the binary, run offline.
principle_note: >
P8 under the organizing principle (see P6-M0 principle_note; plan 0008 `principle`): DenoPods are an
application's TypeScript components as objects of its own kcp; every DenoPod binds port 0 and the
runner records status.address from the pod's port file (port-audit.md blocker 1, in M1-MARKET-NETWORK-PARALLEL);
the market network is the seed of the model and each workstream's own market network. How today's multi-server
deno processes split into DenoPods per application: folded 2026-10-12 (see P6-M0 principle_note; plan
0008 fifth fold).
body: >
OPEN: the plan 0005 gate is satisfied (ACC-ZERO, ACC-MESH, ACC-NOSSH done); items start per their
depends_on, incl. plan 0006 M1-ASSETS. Plan 0008 (docs/plans/0008-embedded-deno-and-typescript-denopods.yaml)
items M1-DENO, M1-TS-SOURCE, M1-TS-CLOSURE. Production runs the embedded archive; other builds
take this checkout from SOCIALWEB_COMPUTER_SOURCE_ROOT or find it by walking up. DENO_DIR is keyed by
the closure's sha256 and every run passes --no-code-cache (13349d6: a stale code cache ran old
bidder code after a restart).
instrument: >
An upgraded closure never runs the old code cache; an entry imported from jsr and npm runs with no
network; two bundles of the same commits are byte-identical.
- id: P8-M1-MARKET-NETWORK
title: OPEN, EARLY - today's market network runs N per host (no fixed ports, no one-network-per-host check).
body: >
Plan 0008 item M1-MARKET-NETWORK-PARALLEL (new 2026-10-12): live tests in parallel worktrees collide on the market network's
fixed ports 2583-2589, the deno-kcp kcp 6443 and kine 23791 and on the one-network-per-host check
(internal/marketnetwork/live_test.go:115) until this lands; ports come from the instance-keyed lease file
(0006 M1-PORT-LEASE-INSTANCE) or bind 0 and are read back. Guests still only through the market network's
requester (runComputeContract -> RFP -> bid -> accept -> cloud-init). Until it lands, market-network runs on
hat-8 are serialized. Fifth fold 2026-10-12 (decision text only; status untouched): ports are read back
from the port file, never log lines (precursor of M2-DENOPOD-LISTENERS); the market network bidder's key, repo KV
and session live in its per-run dir; accept.sh reads the bidder DID from the market network record, not stdout.
Today's market network keeps four applications in one kcp, which the user's strict rule forbids: recorded as
transitional, retired by M4-MARKET-NETWORK-PORT; this item adds no new cross-application coupling.
instrument: >
No market network port literal outside the market network record; two market networks from two worktrees each rent a guest through the
RFP flow on hat-8 at the same time.
- id: P8-M1-IROH
title: OPEN, EARLY - the ssh gateway reaches guests over iroh, then every non-iroh guest transport is stripped.
body: >
Decided by the user 2026-10-12, in this order. Plan 0008 M1-SSH-GATEWAY-IROH (no dependency, startable
now): socialweb-computer-ssh lib/socialweb-computer-requester-inproc asks for the `tunnel` transport
(mod.ts:268) and runs sessions over the websocket tunnel; it moves to iroh like request-vm-ssh. Then
M1-STRIP-LEGACY-TRANSPORTS: the cloud-init-common tunnel, fedproxy-ssh, fedproxy-web and web-terminal modules,
websocat ProxyCommand paths, relay-fqdn guest addresses and the guest JSR registry are deleted across
atproto-market, hono-compute-provider and socialweb-computer-ssh (request-vm-ssh --user-data-transport
removed as a legacy transport). The ssh gateway's Live market CI (socialweb-computer-ssh live.yml) stays
green throughout. Its broader kcp-native move stays M4-RUN-GATEWAY-SSH.
instrument: >
The scan tests find no tunnel, fedproxy-ssh, fedproxy-web or web-terminal module, no websocat, fedproxy-client,
web-terminal or JSR_URL in source and no transport other than iroh requested; the Live market workflow green
after each of the two changes; a market network guest reached over iroh.
- id: P8-M2
title: OPEN - deno-kcp and kcp-libs forked into this repo (runner build-tagged) and run in-process; one service DenoPod template with explicit permissions.
body: >
OPEN as P8-M1. Plan 0008 items M2-DENOKCP-FORK (execrunner's Setpgid moves into GOOS-tagged files so
the cross-OS gate stays green; builds on k8s.io v0.36.4 since the kcp route is out of process, so it
depends on 0006 M0-KCP-SPIKE and M0-XOS-GATE, both done, and can start now; its runner sets SWC_UNIT so
devsup attributes deno processes), M2-DENOKCP-INPROC (deno dies with the core user unit; tested with
`systemctl --user restart`, kill -9 and an upgrade, and on proc in a dev instance since 2026-10-12),
M2-SERVICE-TEMPLATE (required explicit
permissions, no -A, /dev/kvm and $XDG_RUNTIME_DIR denied; user 2026-10-11). Fifth fold 2026-10-12: new
M2-DENOPOD-LISTENERS (createServe writes a listener report - tcp, unix, relay ingress URLs - into the run
dir; the owned runner sets DenoPod.status.listeners; peers and probes resolve from status; no stdout
scraping), new M2-MARKET-NETWORK (startable now: one MarketNetwork value type and one fetch adapter in
typescript-helpers replace the four fetch patches; values reach an application only through its root
object); M2-SERVICE-TEMPLATE renders values from application profiles, the component spec, the
MarketNetwork value and the report path, never a port, HOME or stdin.
instrument: >
A DenoPod applied to the kcp runs on the embedded deno; a restart, a kill and an upgrade each restart
every recorded DenoPod on the new closure exactly once; the template refuses -A; a port-0 DenoPod is
resolved by its reported address; no fetch patch outside the adapter.
- id: P8-M3
title: OPEN - the door runs as a DenoPod; its sessions and requester key live in the root vault.
body: >
OPEN as P8-M1. Plan 0008 items M3-DOOR-DENOPOD and M3-DOOR-STATE-VAULT (fifth fold 2026-10-12, user:
secrets only in the application's OpenBao): oauth-sessions.json and requester-key.hex move into the
gateway's root vault; each door account is a MarketIdentity (oauthDelegated). The node fulfiller stays in
the door DenoPod; a split is revisited after 0007 M6-CLAIM-RESUME (decided by Claude, user may veto).
instrument: >
On a fresh state dir the door DenoPod serves /api/config and, after a core restart, lists the claims
it wrote before it (no container-era state is opened; fresh deploy, user 2026-10-12); a recreated door
with an empty state dir serves an existing session and the same requester DID.
- id: P8-M4
title: OPEN - run gateway ssh, run request, and the market network driven by socialweb-computer.
body: >
OPEN as P8-M1. Plan 0008 items M4-RUN-GATEWAY-SSH, M4-RUN-REQUEST, M4-MARKET-NETWORK-PORT (market network units
named socialweb-market-network-<runid>-..., never matched by production prefixes; since 2026-10-12 `market-network up` =
`dev up --profile market-network`, market network ports render to 0, no lock, N market networks per host) and its acceptance checks
ACC8-*. run request keeps its contracts as ComputeContract objects in its own kcp on port 0. Fifth fold
2026-10-12 (user's answers): M4-RUN-REQUEST reworked - a FRESH kcp per invocation whose state dir (kcp
root + vault under state/<invocation>) persists, `run request release|resume <name>` restart that same
kcp; a MarketRequester root object with a MarketIdentity (oauthQr with no repo, or localRepo = the
ephemeral-PDS mode); every request-vm-ssh option kept as a field or profile (--user-data-transport
removed as a legacy transport); request-vm-ssh reduced to the DenoPod entry (kcp only); ssh in the
foreground over iroh. M4-MARKET-NETWORK-PORT is the STRICT market network: the dev instance is a MarketNetwork whose
bidder, gateway and requester-invocation children each run their own kcp; values only by copy; the
shim never resolves another application's name; system tests via internal/apptest.Start reading root
status. M4-RUN-GATEWAY-SSH after M1-SSH-GATEWAY-IROH, M1-STRIP-LEGACY-TRANSPORTS and 0006
M2-APP-CONTROLLER.
instrument: >
The market network's live test passes twice through `socialweb-computer market network`, and `run request`
reaches onNetwork, lists the contract with no kcp running, and `release <name>` restarts the
invocation's kcp and gets vm.delete 200; no two applications in a market network share a kcp.
- id: P11-M0
title: QUEUED - Windows bidder spikes (plan 0011 M0).
body: >
QUEUED behind plans 0006 and 0007 (complete on Linux with ACC6-* and ACC7-* recorded) and the plan 0008
items 0011 names. Plan 0011 (docs/plans/0011-windows-bidder-hcs.yaml) items M0-REVISIT, M0-HOST-BASELINE,
M0-SCRATCH-BACKEND, M0-HCS-ACCOUNT, M0-HCS-SURVIVAL, M0-HCS-IDENTITY, M0-HNS-NETWORK, M0-COM-CONSOLE,
M0-BINARY-SWAP, M0-GATE. M0-REVISIT first: re-read what 0006/0007/0008 landed (hostsvc, vmruntime,
guestnet seams; the assets scheme; ComputeVM status) and amend 0011 before anything else.
instrument: >
M0-REVISIT's amendments recorded with the SHAs read; each spike's Windows reading recorded; M0-GATE's
go/no-go recorded.
- id: P11-M1
title: QUEUED - Windows HCS backend, HNS network, Hyper-V guest image, windows DenoPod runner, SCM host units.
body: >
QUEUED as P11-M0. Plan 0011 items M1-VMHOST-HCS, M1-NETWORK-HNS, M1-GUEST-IMAGE-HV,
M1-DENOPOD-RUNNER-WINDOWS, M1-HOSTUNIT-SCM, filling the Windows halves of the seams 0006/0007/0008
left as ErrUnsupported stubs.
instrument: >
Each item's tests green on the Windows host; the cross-OS gate still green on Linux.
- id: P11-M2
title: QUEUED - the Windows bidder as an SCM service, its installer, the versions/<sha> upgrade, a windows/amd64 release.
body: >
QUEUED as P11-M0. Plan 0011 items M2-BIDDER-SERVICE-WINDOWS, M2-INSTALLER-WINDOWS,
M2-BINARY-UPGRADE-WINDOWS, M2-RELEASE-WINDOWS.
instrument: >
The service installs and upgrades on the Windows host with a running VM untouched; a reproducible
windows/amd64 build within its budget.
- id: P11-M3
title: QUEUED - a live rent on Windows through the RFP flow; survival, release, reboot = Lost.
body: >
QUEUED as P11-M0. Plan 0011 items M3-LIVE-RENT, M3-LIVE-SURVIVE, M3-LIVE-REBOOT-LOST (then M4-DOCS):
a guest rented through requester -> RFP -> bid -> accept -> cloud-init on the Windows bidder, surviving a
bidder kill and upgrade, released by vm.delete, and Lost after a host reboot.
instrument: >
The three live readings recorded with UTC timestamps and zero ssh sessions.
- id: P12-M0
title: QUEUED - macOS bidder spikes (plan 0012 M0).
body: >
QUEUED behind plans 0006 and 0007 (complete on Linux with ACC6-* and ACC7-* recorded) and the plan 0008
items 0012 names. Plan 0012 (docs/plans/0012-macos-bidder-vz.yaml) items M0-REVISIT, M0-HOST-BASELINE,
M0-VFKIT-ASSET, M0-SCRATCH-BACKEND, M0-VZ-LIFETIME, M0-LAUNCHD-DOMAIN (headless, no login),
M0-REBOOT-LOST-POWER (reboot = Lost, power and FileVault), M0-PROC-IDENTITY (kinfo_proc),
M0-NETWORK-VZ (VZ NAT vs gvproxy), M0-ARCH-MODULES (arm64), M0-RESOURCES (admission), M0-GATE.
instrument: >
M0-REVISIT's amendments recorded with the SHAs read; each spike's macOS reading recorded; M0-GATE's
go/no-go recorded.
- id: P12-M1
title: QUEUED - vfkit launchd backend, the launchd broker, launchd host units, darwin network, arm64 guest image, darwin DenoPod runner, admission.
body: >
QUEUED as P12-M0. Plan 0012 items M1-VMHOST-VZ, M1-LAUNCHD-BROKER, M1-HOSTUNIT-LAUNCHD,
M1-NETWORK-DARWIN, M1-GUEST-IMAGE-ARM64, M1-DENOPOD-RUNNER-DARWIN, M1-ADMISSION-DARWIN.
instrument: >
Each item's tests green on the Mac; the cross-OS gate still green on Linux.
- id: P12-M2
title: QUEUED - the macOS bidder as a LaunchDaemon, its installer, the upgrade, a darwin/arm64 release.
body: >
QUEUED as P12-M0. Plan 0012 items M2-BIDDER-DAEMON-DARWIN, M2-INSTALLER-DARWIN,
M2-BINARY-UPGRADE-DARWIN, M2-RELEASE-DARWIN.
instrument: >
The daemon installs and upgrades on the Mac with a running VM untouched; a reproducible darwin/arm64
build within its budget.
- id: P12-M3
title: QUEUED - a live rent on macOS with no login through the RFP flow; survival, logout, release, reboot = Lost.
body: >
QUEUED as P12-M0. Plan 0012 items M3-LIVE-RENT, M3-LIVE-SURVIVE, M3-LIVE-REBOOT-LOST (then M4-DOCS): a
guest rented through requester -> RFP -> bid -> accept -> cloud-init on a Mac with nobody logged in,
surviving a bidder kill, an upgrade and a logout, released by vm.delete, and Lost after a reboot.
instrument: >
The live readings recorded with UTC timestamps and zero ssh sessions.
- id: PORT-1
title: OPEN - the kcp process piles up loopback connections to the fixed-port front of an old tenant cluster until the gateway host has no ephemeral port left.
body: >
Opened 2026-10-15 from a live reading on the prod gateway host. `listen tcp 127.0.0.1:0` failed
host-wide with "address already in use" because about 28000 loopback connections from the kcp
process were open to the fixed-port front of each of two old tenant clusters (canary on port
20000, goat-demo on port 20018), which exhausted the ephemeral range for every port 0 listener on
the host. The relay in internal/apiserverfront/front.go closes both ends when either end closes,
so the connections are not held by the front; they are held on the kcp side, as dials from kcp to
the front of an old cluster. Deleting the two clusters released the
ports and the host recovered. The origin is not found: which kcp component dials the front
(the mount, the kubeconfig mount child, a controller informer) and why it opens a new connection
instead of reusing or closing one is open. Every listener on the host binds port 0, so one cluster
front that leaks connections takes the whole host down, which makes this a host-wide failure
with a per-cluster cause.
instrument: >
The count of established loopback connections from the kcp process to each cluster front on the
gateway host, read with `ss -tn state established '( dport = :<front port> )'`, stays flat across
one hour with the cluster idle, for a healthy cluster and for a cluster whose apiserver is stopped;
and the count of sockets in the ephemeral range (`ss -tan | wc -l` against
`net.ipv4.ip_local_port_range`) shows headroom while a cluster is deleted and recreated.
- id: PAGE-CONTRACT-ALLOC
text: >
The browser page (kcp-wasm 637670b72 on socialweb 674abaeb) still allocates ~1.3 GB over the two
busiest minutes of a contract (sign-in end, composition, RFP, Held, release); idle after release is
16-36 MiB/min (target 100 MiB/min met). Measured on headless desktop Chrome 2026-10-25 11:19-11:28
UTC, session ca3ie1. Instrument: a heap profile pair (localStorage kcp.heapProfile, pprof
-sample_index=alloc_space -base) across one contract on the iPhone, read for the top allocators.
- id: PAGE-GOROUTINES
text: >
The browser page holds ~7,690 goroutines idle (memstats goroutines=, 2026-10-25), about the same
before and after the churn fixes. Instrument: a goroutine profile from the page (kcp.heapProfile
uploads goroutine-<session>-<t>.pb.gz) grouped by top stack; close when the count is explained per
informer/component or reduced.
- id: PAGE-PENDING-28S
text: >
One iPhone run sat ~28 s in Pending before the sign-in step "preparing" (peer handoff 2026-10-25,
item b). Not reproduced on desktop. Instrument: the page debug log timeline of a fresh-load
sign-in on the iPhone (debug log on -> preparing) over three loads.
- id: PAGE-KUBECTL-EARLY
text: >
The page runs kubectl for a new contract right after adding the EphemeralInfra entry, before the
infra controller creates the ComputeContract, so its first status read shows nothing (cosmetic;
peer handoff item c). Closed when the page waits on the ComputeContract's creation (watch) before
reading it.
- id: PAGE-REFRESH-E2E
text: >
The browser OAuth refresh worker (kcp-wasm oauth_refresh_worker.js via oauthclient TokenRefresher,
socialweb 2e6a3e08) is deployed but no refresh has been observed end to end. Instrument: a page held
past the access token's exp (or the token forced stale) showing one refresher call, the rotated set
persisted, and the session staying Ready.
- id: PAGE-STREAM-INGEST
text: >
Browser stream ingest is option A (Go wsconn js build) until the user decides A vs B2 (a plain-JS
net worker feeding Go over a MessagePort transport hook in wsconn). Decision item for the user; the
seam would be wsconn.SetTransport.
- id: RELAY-MST-INVERT
text: >
Our Go relay (relay.mini-cloud-0002.chadig.com) logged "failed to invert commit MST ... could not
find <cid>" for two hat-9 commits replayed from cursor 0 on 2026-10-25 05:06 UTC: the commit CARs of
older commits from our PDS lack MST blocks needed to verify the diff. Instrument: getRepo/
subscribeRepos of a bidder PDS replayed from 0 into the relay with the verify log; close when every
replayed commit verifies or the PDS's CAR contents are shown complete.
- id: RFP-WINDOW-RKEY
text: >
The bidder drops an RFP as closed from its rkey TID alone before reading createdAt (8dc74cef); a
requester that writes an old-looking rkey with a fresh createdAt is refused. Ours write both at the
same moment. Close by checking createdAt when the rkey says closed, or by documenting the rule in
the market lexicon.
- id: RELEASE-NO-CONTRACT-WAIT
text: >
`run request release` on a contract that failed at the RFP stage (no VM) waits the full 3 m and
exits 1 ("waiting for the ComputeContract within 3m0s") because it waits for
status.contract.release, which such a contract never gets (seen 2026-10-24/25). Close when release
of a VM-less contract returns at once with exit 0.
- id: PAGE-REQUESTER-DEFAULT-POLICY
text: >
Page requester default after the fenced sandbox (wasm option C, kcp-wasm plan 0037; owner wasm,
then linux-kcp). The CLI's `run request` defaults contract.policy to requester-tangled-vouch
(cmd/socialweb-computer/request.go, requestRun.passthrough, requester.DefaultPolicyWorkflow); the
page builds its own ComputeContracts and keeps its requester policy, because kcp.wasm's
sandbox.Runner is not Fenced (internal/sandbox/worker/worker.go) and every verdict would be Deny.
Close when the page's policy engine reports Fenced and Ready and a page request under
requester-tangled-vouch gets a bid from a vouched bidder.
execution_order:
written: "2026-10-12, third fold of plans 0006-0008 (user request (i)); waves computed from depends_on across plans 0005-0012 (no unresolved id, no cycle). Re-derived after the fifth fold (multi-server-to-kcp.md + the user's section-7 answers): still no unresolved id or cycle; critical path unchanged. Advisory: depends_on governs; re-derive after every merge."
early_fifth_fold: >
EARLY by the user's order (2026-10-12): 0008 M1-SSH-GATEWAY-IROH (wave 1) then 0008
M1-STRIP-LEGACY-TRANSPORTS (wave 2), with the ssh gateway's Live market CI green throughout; 0007
M1-MARKET-IDENTITY (wave 1, the identity CRD). Also startable now: 0007 M3-BIDDER-ACCOUNT-PDS,
M3-OIDC-ISSUER-INSTANCE, 0008 M2-MARKET-NETWORK; then 0008 M2-DENOPOD-LISTENERS (wave 2, after
M2-DENOKCP-FORK) and 0007 M3-BIDDER-COMPOSER (wave 3).
strict_market_network_rule: >
Everything the market network needs for STRICT per-application kcp (user 2026-10-12) comes before any workstream
moves its live tests onto `dev up --profile market-network`: the early items above, 0006 M2-APP-CONTROLLER (wave 8,
after M2-HOSTPROCESS-KIND and 0008 M2-SERVICE-TEMPLATE) -> 0008 M4-RUN-REQUEST (9) -> 0008
M4-MARKET-NETWORK-PORT (10). The market network's bidder runs the firecracker provider until 0007 M5-RUN-BIDDER (12), so the
strict market network does not wait for the VM chain. Until it lands, workstreams test in today's market network made parallel
by 0008 M1-MARKET-NETWORK-PARALLEL - a recorded transitional shape (four applications in one kcp, a shim that reads
other workspaces) that no item extends.
critical_path: >
15 waves to 0006 M5-RETIRE: 0006 M0-SIZE-BUDGET or M0-PG-SOURCE (1) -> M1-ASSETS (2) -> M2-KCP-STORE (3)
-> M2-KCP-SLOTS (4) -> M2-KCP-PORT0 (5) -> M2-APPLY + M2-BOOTSTRAP-LAYER (6) -> M2-HOSTPROCESS-KIND (7)
-> M2-POSTGRES (8) -> M3-CLUSTER-UNITS (9) -> M3-BAO-SYNC / M3-APISERVER-FRONT (10) ->
M3-SERVE-GATEWAY-KCP / M3-CP-SLOTS (11) -> M3-HOST-INSTALL (12) -> M3-CP-CONTINUITY (13) ->
M5-FRESH-DEPLOY (14) -> M5-RETIRE (15). The VM chain rides it: 0007 M0-VM-USERNS (1) + M1-GUEST-IMAGE
(3) -> M2-VM-RUNTIME (4); M2-HOSTPROCESS-KIND (7) -> 0007 M2-CONTROLLER (8) -> M4-BIDDER-VAULT (9) ->
M4-BIDDER-BAO-SYNC (11) -> M5-RUN-BIDDER (12) -> M6-* (13) -> M6-NOSSH-JOIN (14). Shortening it means
landing wave 1-3 platform items first: M0-PG-SOURCE, M0-SIZE-BUDGET, M1-ASSETS, M2-KCP-STORE.
wave_1_startable_now:
- "0006 M0-PG-SOURCE (spike, fallback = build from source with the build-host script)"
- "0006 M0-SIZE-BUDGET"
- "0006 M1-CLI"
- "0006 M1-HOST-INVENTORY"
- "0006 M1-PORT-LEASE-INSTANCE (early: unblocks parallel market networks)"
- "0006 M1-NO-FIXED-PORTS (early: the fixed-port ratchet; worktree WT-leases)"
- "0006 M1-STORE-CLEANUP"
- "0006 M2-HOST-CONFORMANCE (early: unblocks the proc backend)"
- "0007 M0-VM-USERNS (spike, live hat-8: one RFP-flow guest)"
- "0007 M1-DECIDE"
- "0007 M1-FC-INVENTORY"
- "0007 M1-COMMON-EXTRACT (after M4-CONTRACT-STORE-FILE in the same repo, see worktrees)"
- "0007 M1-QR-DEPLOY"
- "0007 M3-NODE-LOSS-CLOCK"
- "0007 M4-CONTRACT-STORE-FILE (urgent: receipts lost on bidder restart, guest leaked 2026-10-12T01:31Z; live hat-8)"
- "0007 M6-CLAIM-RESUME (market network test live hat-8)"
- "0008 M2-DENOKCP-FORK"
- "0007 M4-TERMINATION-EVENT is doing: its TS half is done; its Go half needs M2-CONTROLLER and M3-TS-PROVIDER in practice (not in its depends_on)"
- "0008 M1-SSH-GATEWAY-IROH (EARLY, fifth fold: the ssh gateway onto iroh; Live market CI green)"
- "0007 M1-MARKET-IDENTITY (EARLY, fifth fold: the identity CRD; hono-pds session minting completes sign-in in tests)"
- "0007 M3-BIDDER-ACCOUNT-PDS (fifth fold: a market network bidder on its own PDS account)"
- "0007 M3-OIDC-ISSUER-INSTANCE (fifth fold: no module-global issuer state)"
- "0008 M2-MARKET-NETWORK (fifth fold: one network value and fetch adapter)"
wave_2: ["0006 M1-ASSETS", "0006 M2-HOST-BACKEND-PROC", "0006 M2-MANIFESTS", "0006 M3-FOLD", "0007 M1-PROTOCOL-CHECKLIST", "0008 M1-MARKET-NETWORK-PARALLEL (early: ends the fixed-port collisions)", "0008 M1-STRIP-LEGACY-TRANSPORTS (early, after M1-SSH-GATEWAY-IROH)", "0008 M2-DENOPOD-LISTENERS"]
wave_3: ["0006 M1-DEV-INSTANCE", "0006 M1-INSTALL", "0006 M2-HOST-CONTRACT", "0006 M2-KCP-FRONT", "0006 M2-KCP-STORE", "0006 M2-TEMPLATE", "0006 M2-VM-TREE-AUDIT", "0007 M1-API", "0007 M1-GUEST-IMAGE", "0007 M3-BIDDER-COMPOSER", "0007 M4-HOST-KEYS", "0008 M1-DENO"]
wave_4_to_6: ["0006 M2-KCP-SLOTS, 0007 M2-VM-RUNTIME, 0007 M3-TS-PROVIDER, 0008 M1-TS-SOURCE (4)", "0006 M2-KCP-PORT0, 0007 M2-BACKPORT, M3-BIDDER-FLAG, M4-ONNETWORK, 0008 M1-TS-CLOSURE (5)", "0006 M2-APPLY, M2-BOOTSTRAP-LAYER, M3-KCP-ROLL, M4-BUILD, 0008 M2-DENOKCP-INPROC (6)"]
wave_7_to_10_application_model: ["0006 M2-HOSTPROCESS-KIND, 0008 M2-SERVICE-TEMPLATE (7)", "0006 M2-APP-CONTROLLER, M2-OPENBAO-SERVER (8)", "0007 M4-BIDDER-VAULT, 0008 M4-RUN-REQUEST, M4-RUN-GATEWAY-SSH (9)", "0008 M4-MARKET-NETWORK-PORT (10)", "later: 0006 M3-SERVE-GATEWAY-KCP (11), 0007 M5-RUN-BIDDER (12), 0008 M3-DOOR-DENOPOD (12), 0007 M5-BIDDER-COMPONENTS and 0008 M3-DOOR-STATE-VAULT (13)"]
vertical_slices:
- "binary-base (started 2026-10-12; user-approved dependency relaxation): 0006 M1-CLI, M2-HOST-CONFORMANCE, M2-HOST-BACKEND-PROC and a minimal `dev up --profile minimal` - kine-sqlite + one kcp slot, port 0, /proc read-back, kcp and kine from explicit paths in a dev build, two instances side by side. Possible because 0006 M2-KCP-STORE and M2-KCP-SLOTS now carry variant_dependencies: M1-ASSETS and M0-PG-SOURCE gate only their embedded-asset and postgres variants, and M2-KCP-FRONT gates only multi-slot blue/green, not a single dev slot. Item status/evidence untouched until the slice merges (the lead records them)."
parallel_worktrees_wave_1:
- "WT-hostsvc (this repo: internal/hostsvc/, internal/devsup/, deploy/check.sh --fast step): M2-HOST-CONFORMANCE, then M2-HOST-BACKEND-PROC, then M2-VM-TREE-AUDIT (internal/hostsvc/treeaudit) once M1-ASSETS lands."
- "WT-cli (this repo: cmd/socialweb-computer/main.go, internal/clicfg/): M1-CLI, then M2-MANIFESTS -> M2-TEMPLATE (deploy/embed.go, deploy/sets.yaml, internal/manifests/); M3-FOLD in its own worktree after M1-CLI merges (cmd/socialweb-* shims)."
- "WT-pins (this repo: internal/versions/versions.json, docs/host-inventory.yaml, deploy/check.sh size gate; scratch branch for the pg spike): M0-PG-SOURCE, M0-SIZE-BUDGET, M1-HOST-INVENTORY, then M1-ASSETS (internal/assets/, scripts/stage-assets.sh)."
- "WT-leases (this repo: internal/portallocator/, internal/subnetallocator/, test/no_fixed_ports_test.go, deploy/edge-ports.yaml, deploy/fixed-ports-ratchet.yaml): M1-PORT-LEASE-INSTANCE and M1-NO-FIXED-PORTS, then M1-STORE-CLEANUP (internal/vaultlayout, cmd/socialweb-vault, portallocator.go:200 message, versions.json rustfs/t4 entries)."
- "WT-contract-store (atproto-market lib/abc/market-bidder, lib/market-bidder, lib/market-bidder-compute, new lib/market-bidder-contract-store-fs, hono-bidder; hono-compute-provider lib/compute-provider-firecracker): M4-CONTRACT-STORE-FILE first (urgent), then M1-COMMON-EXTRACT (moves functions out of compute-provider-firecracker/mod.ts, the same file)."
- "WT-qr (did-key-associator: scripts/deploy.sh, deno.json, test/): M1-QR-DEPLOY, then M1-PROTOCOL-CHECKLIST (atproto-market docs + test/role_scope_coverage_test.ts, did-key-associator test), then M4-HOST-KEYS after M4-CONTRACT-STORE-FILE has merged (both touch atproto-market lib/market-bidder-compute)."
- "WT-userns (scratch branch + docs/research/2026-10-11-vm-lifecycle/vm-userns-spike.md; a scratch hono-compute-provider branch for the one RFP-flow guest): M0-VM-USERNS."
- "WT-docs (this repo: docs/ARCHITECTURE.md section 5, docs/compute-provider-requirements.yaml; hono-compute-provider test/requirements_inventory_test.ts): M1-DECIDE, M1-FC-INVENTORY."
- "WT-node-loss (this repo: cmd/socialweb-clusterprovider/nodes.go, internal/clusterprovider pathUpSince): M3-NODE-LOSS-CLOCK."
- "WT-claim-resume (socialweb-computer-k8s node-market + door; atproto-market lib/guest-capability-secrets; this repo api/v1alpha1 NodeClaim): M6-CLAIM-RESUME - it carries the ONE NodeClaim schema revision (status.secrets, acceptPending, phase Adopting, ContractState.hostKeys), so M4-HOST-KEYS' door half and M6-GATEWAY-REPOPULATE land on top of it, and 0007 M1-API (api/v1alpha1) rebases after it."
- "WT-denokcp (this repo: internal/denokcp/, internal/kcplibs/, deploy/denokcp/vendor.json): M2-DENOKCP-FORK, then M2-DENOPOD-LISTENERS (runner status.listeners, servicenames; typescript-helpers lib/serve)."
- "WT-iroh (socialweb-computer-ssh lib/socialweb-computer-requester-inproc; then atproto-market cloud-init-common, requester-xrpc, compute-contract-gateway-xrpc, request-vm-ssh, compute-contract-full-flow, hono-compute-provider compute-provider-firecracker): 0008 M1-SSH-GATEWAY-IROH, then M1-STRIP-LEGACY-TRANSPORTS (fifth fold, early; Live market CI green after each)."
- "WT-identity (this repo api/v1alpha1 marketidentity + internal/appcli; atproto-market lib/abc/identity-prompt and its impls, hono-bidder and request-vm-ssh prompt steps): 0007 M1-MARKET-IDENTITY (fifth fold, early)."
- "WT-bidder-lib (hono-compute-provider lib/oidc-issuer-hono; atproto-market hono-bidder, lib/atproto-helpers, new lib/hono-factory-market-bidder-app): 0007 M3-OIDC-ISSUER-INSTANCE and M3-BIDDER-ACCOUNT-PDS, then M3-BIDDER-COMPOSER after M1-MARKET-IDENTITY, the strip and M2-MARKET-NETWORK have merged."
- "WT-market-network (typescript-helpers lib/common/market-network-common, lib/market-network-fetch; the four fetch-patch call sites): 0008 M2-MARKET-NETWORK."
shared_files_to_serialize:
- "internal/versions/versions.json: M0-PG-SOURCE, M0-SIZE-BUDGET, M1-STORE-CLEANUP, M1-ASSETS (merge one at a time; entries are disjoint)."
- "deploy/check.sh: M2-HOST-CONFORMANCE (--fast proc half), M0-SIZE-BUDGET (size gate), M1-NO-FIXED-PORTS (--fast step)."
- "api/v1alpha1: M6-CLAIM-RESUME (NodeClaim revision) before M1-API (ComputeVM) and M2-HOSTPROCESS-KIND (HostProcess)."
- "atproto-market lib/market-bidder-compute: M4-CONTRACT-STORE-FILE before M4-HOST-KEYS."
- "hono-compute-provider lib/compute-provider-firecracker/mod.ts: M4-CONTRACT-STORE-FILE before M1-COMMON-EXTRACT; 0008 M1-STRIP-LEGACY-TRANSPORTS (guest JSR registry) merges with either, whichever is second keeps the stripped helpers out of compute-provider-common."
- "atproto-market lib/requester-xrpc and lib/common/cloud-init-common: 0008 M1-STRIP-LEGACY-TRANSPORTS before the requester half of 0007 M4-HOST-KEYS (known_hosts written once, on the iroh path)."
- "atproto-market hono-bidder/mod.ts: M4-CONTRACT-STORE-FILE, M1-MARKET-IDENTITY, M3-BIDDER-ACCOUNT-PDS, 0008 M2-MARKET-NETWORK, then M3-BIDDER-COMPOSER (which moves the whole composition out)."
- "api/v1alpha1 and apigen outputs: M6-CLAIM-RESUME (NodeClaim), M1-MARKET-IDENTITY, M1-API, M2-HOSTPROCESS-KIND, M2-APP-CONTROLLER - one at a time."
needs_live_hat8_market_network_rfp_flow:
- "0007 M4-CONTRACT-STORE-FILE (bidder restart with a rented guest; late vm.delete after restart)"
- "0007 M0-VM-USERNS (one guest through the chosen userns stage)"
- "0008 M1-MARKET-NETWORK-PARALLEL (two market networks at once)"
- "0007 M6-CLAIM-RESUME (TestADoorRestartMidProvisioningStillJoinsTheNode)"
- "0007 M4-HOST-KEYS (StrictHostKeyChecking=yes against the reported key; needs the QR re-sign-in)"
- "0007 M1-MARKET-IDENTITY (a market network bidder without --skip-qr, association approved through the object)"
- "0008 M1-STRIP-LEGACY-TRANSPORTS (one market network guest reached over iroh); its ssh-gateway proof is the Live market CI, not hat-8"
- "later: 0007 M4-TERMINATION-EVENT Go half, M6-*; 0006 M3-CLUSTER-UNITS, M3-APISERVER-FRONT, M3-CP-SLOTS, M3-LINK-HANDOVER, M3-CP-CONTINUITY market network tests; 0008 M4-MARKET-NETWORK-PORT"
needs_hat8_without_market: "0006 M2-HOST-CONFORMANCE (systemd half), M0-PG-SOURCE (stand-in runs), every M2-KCP-* and M2-BOOTSTRAP-LAYER livegate test (real kcp/kine/postgres, no guest); 0007 M1-QR-DEPLOY runs against the qr.fedfork.com host, not hat-8."
live_collision_rule: >
Until 0008 M1-MARKET-NETWORK-PARALLEL (wave 2) and 0006 M1-DEV-INSTANCE (wave 3) land, the market network's fixed ports
(2583-2589, kcp 6443, kine 23791) and its one-network-per-host check make every market network run on hat-8 exclusive:
run them one at a time in this order - M4-CONTRACT-STORE-FILE (stops the receipt leak that pollutes
later runs), M0-VM-USERNS's guest, then M1-MARKET-NETWORK-PARALLEL's own run of two market networks; after it, M6-CLAIM-RESUME and
M4-HOST-KEYS run in parallel market networks. M1-PORT-LEASE-INSTANCE (wave 1) is therefore first in its worktree.
open_notes:
- "Plan 0010 G4-SCHED-SINGLE-SERVER (refuse leaderElect false for more than one k3s server; GAP-7) conflicts with 0006 M3-CP-SLOTS (scheduler leaderElect true for every market cluster, user 2026-10-12). Not edited in 0010; R0-REASSESS must resolve it (see P10)."
- "Plans 0011 and 0012 reference no renamed id (none renamed); their M0-REVISIT items must re-read the HostProcess kind and the proc backend (Windows Job Objects, darwin kqueue) when they start."
- "Fifth fold (2026-10-12): no id renamed, so 0011 and 0012 were not edited. Their text predates the strip: 0012 lists the tunnel, fedproxy-ssh, fedproxy-web and web-terminal modules and forwards the guest JSR registry per VM, and 0011 names the requester's tunnel; after 0008 M1-STRIP-LEGACY-TRANSPORTS iroh is the only guest transport and there is no guest JSR registry, which their M0-REVISIT items must take in. They also must read the application model (root objects, MarketIdentity, per-application kcp)."
- "The ssh gateway's Live market CI (socialweb-computer-ssh .github/workflows/live.yml) must stay green through 0008 M1-SSH-GATEWAY-IROH and M1-STRIP-LEGACY-TRANSPORTS (user 2026-10-12)."
limits:
- title: A node cannot reach another node, so a cluster with more than one has no cluster DNS
for the Pods on the others.
body: >
CLOSED for the local substrates: plan 0005 M1-MARKET-ONLY (commit cd12808) retired them, so
there is no longer a guest of this host whose networking this could bound. Kept as a limit of
the DELETED substrates, which is what the reading below describes. On a market
cluster, built but not
yet deployed or read live: every node resolves on its own CoreDNS at 169.254.20.10 (M5-DNS),
and pods reach pods on other nodes through socialweb-meshd over iroh (M5-MESHD); the
docs/gitlab runner configs no longer carry the DNS wait. What follows is the local-substrate
reading, unchanged.
Every guest sits behind its own pasta instance, and pasta is user-mode networking: it carries
a guest's traffic out to the host's network and nothing between two guests. So the guests of
one cluster have no path to each other and k3s's flannel overlay has nothing to run over. A
Pod's resolver is cluster DNS, cluster DNS is CoreDNS, and CoreDNS runs on ONE node; measured
on a live five-node cluster, TCP to the CoreDNS Pod IP is reachable from a Pod on the node
CoreDNS runs on and unreachable from the Pods on the other four, and `nslookup` there answers
`connection timed out; no servers could be reached`. Anything that resolves a name therefore
works on a ONE-node cluster and on the node CoreDNS happens to sit on, and fails elsewhere.
This is worse than a race and it is not a startup delay: it does not go away with time. Found
by running five GitLab jobs on five freshly-booted nodes, where three died on `Could not
resolve host ... Timeout while contacting DNS servers` and were reported as `script_failure`.
The runner-side wait in `pre_get_sources_script` bounds the wait and names it, and does not
fix it. Recorded as a limit rather than an open item because the instrument is
`TestFiveGitLabJobPodsScaleToFiveNodesAndBackToZero` together with the pod-to-pod reachability
reading in `docs/plans/0004` section 6a, and the shape of the fix -- a shared L2 for one
cluster's guests, or a CoreDNS on every node -- is a change to the node network rather than a
bug in it.
- title: The node ceiling is ten, and every node is a booted guest.
body: >
`SOCIALWEB_CLUSTERPROVIDER_SCALE_MAX_NODES` defaults to 10 and
`SOCIALWEB_CLUSTERPROVIDER_SCALE_PODS_PER_NODE` to 8; the two together are the whole cost
model, because every node is a booted guest and the cap is what bounds what one cluster can
rent. A ceiling below one is refused rather than clamped. `scripts/deploy.sh` passes 10 and
one pod per node, because the GitLab runner this cluster serves uses pod anti-affinity: one
job pod occupies one node whatever `podsPerNode` says, so a default of eight would answer
five queued jobs with one node. Measured on 2026-10-07: five jobs at once, each on a distinct
market node, all green, with the count never above the ceiling (section 15).
- title: Scale-in is all-or-nothing, and that is a stated limit.
body: >
While there is work the target only grows, to the cap. When the work stops the cluster goes to
ZERO, not to an intermediate count. Scaling three nodes to two needs an answer to "which node,
and does its work move", and no instrument here measures that yet, so the branch that would do
it is not written rather than written wrong. See `docs/plans/0002` section 4.4.
- title: A tenant host needs /etc/rancher, and check-host does not test it.
body: >
The cluster's k3s server runs under nsexec, which binds `<slot data dir>/rancher` over
`/etc/rancher`. When `/etc/rancher` does not exist the bind is skipped and k3s writes on the
host instead of in the cluster's state. `check-host` has no check for it. A dev build also
needs Deno 2.9.3 on `PATH` for the policy engine: a release build extracts the embedded deno
asset, while a dev build has no embedded zip and `internal/policyengine/deno` falls back to
`exec.LookPath("deno")`. The rewrite's integration hosts hat-7 and hat-9 were prepared by hand
with linger, kubectl, `/etc/rancher`, and a subuid/subgid range for three VM slots. See
`docs/ARCHITECTURE.md` section 15, "The rewrite's readings, and the host contract's limits".
paused_2026_10_19:
- id: RC21-VERIFIED
text: >
rc21 (231a9154) is deployed to prod: gateway and Bob's bidder, contract rc21-proof-1 passed in
569 s with zero operator writes, canary green in 158 s, heartbeat filter live (found 450,
fresh 1, stale 449). The README install of rc21 from the bucket passed on a fresh host: dev up
87 s, run request 93 s, dev down 108 s, nothing left. What is missing is the GitLab double
green (see GITLAB-RC21) and a bucket re-check for any later release.
- id: SOAK-3
text: >
Plan 0013 third soak was paused at 06:28Z on 2026-10-19 with the fit window 2 h 43 min in.
dev-soak-3 is deleted; its data is in ~/.tmp/soak3 (scripts) and was summarised in
docs/ARCHITECTURE.md section 14. Resume: recreate a droplet, rebuild from main, re-run the
same setup (one-kcp network + installed bidder + chaos) and finish a full 3 h window; fix
the fit script to difference cumulative counters first.
- id: GITLAB-RC21
text: >
GitLab pipeline 255 (rc21) was running (job 635, ~20 min in) when gl-runner-1 was deleted at
teardown, so that job is lost, not failed. rc17 and rc19 were green twice; rc21 has no
integration reading. Recreate the runner (scripts/dev-droplet.sh + scripts/ci-runner-install.sh,
tunnel unit gl-runner-1-tunnel.service on hat-10) and run rc21 twice.
- id: PAGE-CONTRACT-PROFILE-LOSS
text: >
A page contract is orphaned when the browser profile changes mid-contract (2026-10-30, the
bidder released it). The page should release in-session or resume from a persisted contract
ref; drivers now use a persistent profile with in-session Release.
- id: CLI-TERMINATION-404
text: >
Requester-side termination POST returned 404 after the CLI process exited (rc118 baseline).
rc120 cli-agent-1793350170 released cleanly while the --hold process was still alive;
recheck the exit-first ordering.
- id: QR-NO-EXPECTED-ACCOUNT
text: >
An oauthQr CLI requester (run request -- --atproto-oauth-qr) that names no did or handle
takes the first sign-in qr.fedfork.com verified at the account PDS within the 5-minute QR
window; the device refuses an empty or different verifiedDid, and -- --expect-handle pins
the account. Bidders in oauthQr must name their account. Decide whether a requester with no
expected account stays allowed (section 14).