1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328
1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1350
1351
1352
1353
1354
1355
1356
1357
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377
1378
1379
1380
1381
1382
1383
1384
1385
1386
1387
1388
1389
1390
1391
1392
1393
1394
1395
1396
1397
1398
1399
1400
1401
1402
1403
1404
1405
1406
1407
1408
1409
1410
1411
1412
1413
1414
1415
1416
1417
1418
1419
1420
1421
1422
1423
1424
1425
1426
1427
1428
1429
1430
1431
1432
1433
1434
1435
1436
1437
1438
1439
1440
1441
1442
1443
1444
1445
1446
1447
1448
1449
1450
1451
1452
1453
1454
1455
1456
1457
1458
1459
1460
1461
1462
1463
1464
1465
1466
1467
1468
1469
1470
1471
1472
1473
1474
1475
1476
1477
1478
1479
1480
1481
1482
1483
1484
1485
1486
1487
1488
1489
1490
1491
1492
1493
1494
1495
1496
1497
1498
1499
1500
1501
1502
1503
1504
1505
1506
1507
1508
1509
1510
1511
1512
1513
1514
1515
1516
1517
1518
1519
1520
1521
1522
1523
1524
1525
1526
1527
1528
1529
1530
1531
1532
1533
1534
1535
1536
1537
1538
1539
1540
1541
1542
1543
1544
1545
1546
1547
1548
1549
1550
1551
1552
1553
1554
1555
1556
1557
1558
1559
1560
1561
1562
1563
1564
1565
1566
1567
1568
1569
1570
1571
1572
1573
1574
1575
1576
1577
1578
1579
1580
1581
1582
1583
1584
1585
1586
1587
1588
1589
1590
1591
1592
1593
1594
1595
1596
1597
1598
1599
1600
1601
1602
1603
1604
1605
1606
1607
1608
1609
1610
1611
1612
1613
1614
1615
1616
1617
1618
1619
1620
1621
1622
1623
1624
1625
1626
1627
1628
1629
1630
1631
1632
1633
1634
1635
1636
1637
1638
1639
1640
1641
1642
1643
1644
1645
1646
1647
1648
1649
1650
1651
1652
1653
1654
1655
1656
1657
1658
1659
1660
1661
1662
1663
1664
1665
1666
1667
1668
1669
1670
1671
1672
1673
1674
1675
1676
1677
1678
1679
1680
1681
1682
1683
1684
1685
1686
1687
1688
1689
1690
1691
1692
1693
1694
1695
1696
1697
1698
1699
1700
1701
1702
1703
1704
1705
1706
1707
1708
1709
1710
1711
1712
1713
1714
1715
1716
1717
1718
1719
1720
1721
1722
1723
1724
1725
1726
1727
1728
1729
1730
1731
1732
1733
1734
1735
1736
1737
1738
1739
1740
1741
1742
1743
1744
1745
1746
1747
1748
1749
1750
1751
1752
1753
1754
1755
1756
1757
1758
1759
1760
1761
1762
1763
1764
1765
1766
1767
1768
1769
1770
1771
1772
1773
1774
1775
1776
1777
1778
1779
1780
1781
1782
1783
1784
1785
1786
1787
1788
1789
1790
1791
1792
1793
1794
1795
1796
1797
1798
1799
1800
1801
1802
1803
1804
1805
1806
1807
1808
1809
1810
1811
1812
1813
1814
1815
1816
1817
1818
1819
1820
1821
1822
1823
1824
1825
1826
1827
1828
1829
1830
1831
1832
1833
1834
1835
1836
1837
1838
1839
1840
1841
1842
1843
1844
1845
1846
1847
1848
1849
1850
1851
1852
1853
1854
1855
1856
1857
1858
1859
1860
1861
1862
1863
1864
1865
1866
1867
1868
1869
1870
1871
1872
1873
1874
1875
1876
1877
1878
1879
1880
1881
1882
1883
1884
1885
1886
1887
1888
1889
1890
1891
1892
1893
1894
1895
1896
1897
1898
1899
1900
1901
1902
1903
1904
1905
1906
1907
1908
1909
1910
1911
1912
1913
1914
1915
1916
1917
1918
1919
1920
1921
1922
1923
1924
1925
1926
1927
1928
1929
1930
1931
1932
1933
1934
1935
1936
1937
1938
1939
1940
1941
1942
1943
1944
1945
1946
1947
1948
1949
1950
1951
1952
1953
1954
1955
1956
1957
1958
1959
1960
1961
1962
1963
1964
1965
1966
1967
1968
1969
1970
1971
1972
1973
1974
1975
1976
1977
1978
1979
1980
1981
1982
1983
1984
1985
1986
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
2027
2028
2029
2030
2031
2032
2033
2034
2035
2036
2037
2038
2039
2040
2041
2042
2043
2044
2045
2046
2047
2048
2049
2050
2051
2052
2053
2054
2055
2056
2057
2058
2059
2060
2061
2062
2063
2064
2065
2066
2067
2068
2069
2070
2071
2072
2073
2074
2075
2076
2077
2078
2079
2080
2081
2082
2083
2084
2085
2086
2087
2088
2089
2090
2091
2092
2093
2094
2095
2096
2097
2098
2099
2100
2101
2102
2103
2104
2105
2106
2107
2108
2109
2110
2111
2112
2113
2114
2115
2116
2117
2118
2119
2120
2121
2122
2123
2124
2125
2126
2127
2128
2129
2130
2131
2132
2133
2134
2135
2136
2137
2138
2139
2140
2141
2142
2143
2144
2145
2146
2147
2148
2149
2150
2151
2152
2153
2154
2155
2156
2157
2158
2159
2160
2161
2162
2163
2164
2165
2166
2167
2168
2169
2170
2171
2172
2173
2174
2175
2176
2177
2178
2179
2180
2181
2182
2183
2184
2185
2186
2187
2188
2189
2190
2191
2192
2193
2194
2195
2196
2197
2198
2199
2200
2201
2202
2203
2204
2205
2206
2207
2208
2209
2210
2211
2212
2213
2214
2215
2216
2217
2218
2219
2220
2221
2222
2223
2224
2225
2226
2227
2228
2229
2230
2231
2232
2233
2234
2235
2236
2237
2238
2239
2240
2241
2242
2243
2244
2245
2246
2247
2248
2249
2250
2251
plan: 0006-socialweb-computer-unified-binary
written: 2026-10-07
revised: >
2026-10-07, with the user's answers to the plan's questions (decisions below marked 'decided by the user
2026-10-07'). 2026-10-11, folded from docs/research/2026-10-11-vm-lifecycle/ (user-decisions.md, plan-0006-0007-linux-exploration.md,
linux-proofs.md, lifecycle-synthesis.md). Decisions from user-decisions.md are marked 'decided by the user
2026-10-11'; recommendations of the exploration not covered there are 'decided by Claude, user may veto'.
Linux-only execution with cross-OS seams; the seven known gaps fixed (kcp before the root vault, postgres to
cluster-group ordering, the gateway in a user manager, KillMode=mixed removed, vacuous kill -9 tests
replaced, DetachProcess removed, the WHPX/QEMU capability code removed); kcp route left to the M0 spike;
control-plane continuity at cutover left PENDING for the user (M3-CP-CONTINUITY). Renamed: M2-HOST-SCOPES
-> M2-HOST-UNITS, M3-CLUSTER-SCOPES -> M3-CLUSTER-UNITS, M1-KCP-PORT0 -> M2-KCP-PORT0. New: M0-XOS-GATE,
M1-INSTALL, M2-KCP-UNIT, M2-HOST-CONTRACT, M2-VM-TREE-AUDIT, M3-CP-CONTINUITY.
2026-10-12, folded from user-decisions.md (2026-10-12 lines) and control-plane-handover.md: no backwards
compatibility, migration, cutover, rollback or prod-downtime planning until the finished solution exists,
and the gateway runs no containers, so M5-CUTOVER becomes M5-FRESH-DEPLOY (a plain fresh deploy of the
finished binary) and M5-RETIRE deletes the container path with no waiting period; tenant control-plane
upgrades are blue/green behind a per-cluster front (decided by the user 2026-10-12): M3-CLUSTER-UNITS
rewritten to a systemd slice of member units, new M3-APISERVER-FRONT, M3-CP-SLOTS, M3-LINK-HANDOVER,
M2-POSTGRES amended with its upgrade paths, M3-CP-CONTINUITY rewritten to continuity for upgrades of the
finished system; new M1-STORE-CLEANUP (rustfs/t4 leftovers). Container-state tests (a state dir the
container wrote, Fedora data dirs from prod, kubectl-applied objects) removed from items this fold owns.
2026-10-12 (second fold), from user-decisions.md (the kcp, 0007 and install lines of 2026-10-12),
kcp-spike.md and kcp-blue-green.md: kcp route decided by the user = out of process as alternating
blue/green slot units of the release kcp v0.33.0 asset, no kcp patch, no vendored kcp source.
M0-KCP-SPIKE done; M1-K8S-FORK and M1-KCP-INPROC closed "route not chosen"; M2-KCP-UNIT -> M2-KCP-SLOTS;
M2-KCP-PORT0 rewritten (port 0 read back from the slot's single listening socket in /proc; the embedded
etcd and the port-file patches dropped); new M2-KCP-STORE (kine per slot -> a dedicated postgres unit),
M2-KCP-FRONT (our L4 front on 127.0.0.1:6443; an M2 item, not the M3-KCP-FRONT kcp-blue-green.md
proposed, because M2-APPLY and every kcp consumer need the front URL, and M3-APISERVER-FRONT reuses its
code), M3-KCP-ROLL (`units roll kcp`). M1-INSTALL rewritten to the user's one explicit root command
`install bidder|gateway` with `uninstall`, enumerating everything it creates and every unit the core
later starts; the bidder install reserves a subuid/subgid range for per-VM user namespaces (0007
M2-VM-RUNTIME). Answered questions removed.
2026-10-12 (third fold), from user-decisions.md (the 2026-10-12 lines, esp. "dev run mode is KCP-NATIVE
NOW"), dev-run-mode.md (its options and requirements; its phasing overridden by the user) and the user's
decisions relayed with this fold: (a) a bidder's kcp is ONE slot on kine-sqlite and its upgrade is a
short kcp restart; blue/green kcp slots on kine -> postgres are the GATEWAY's only (corrects d6f778e,
which applied kine -> postgres to every bidder handle's kcp as "user may veto": the user vetoed it);
(b) guest images come from a build-host script that runs on any Linux host, CI shells out to it later,
there is no CI service now; (c) every org repo works on main and is pushed to GitHub as work lands;
(d) the dev run mode is kcp-native now: long-lived host processes are HostProcess objects reconciled by
an in-binary supervisor on a placement backend (systemd-user for production installs, proc for dev),
with an explicit bootstrap layer (kcp, its kine and its store, its front and the core: installed
systemd units in production, devsup in dev) below kcp; (e) M0 spikes with fallbacks (M0-PG-SOURCE here,
0007 M0-VM-USERNS); (i) an execution order with a critical path and parallel worktrees (execution_order
below and NEXT.yaml). New: M0-PG-SOURCE, M1-PORT-LEASE-INSTANCE, M1-DEV-INSTANCE, M2-HOST-BACKEND-PROC,
M2-HOST-CONFORMANCE, M2-BOOTSTRAP-LAYER, M2-HOSTPROCESS-KIND, M4-DEV-PARENT-KCP, ACC6-DEV. Changed:
M1-CLI, M1-ASSETS, M1-HOST-INVENTORY, M1-INSTALL, M2-HOST-UNITS (done, now named the systemd-user
backend; evidence kept), M2-HOST-CONTRACT, M2-VM-TREE-AUDIT, M2-KCP-STORE, M2-KCP-FRONT, M2-KCP-SLOTS,
M2-KCP-PORT0, M2-POSTGRES, M2-OPENBAO-SERVER, M3-CLUSTER-UNITS, M3-APISERVER-FRONT, M3-CP-SLOTS,
M3-KCP-ROLL, M3-SERVE-GATEWAY-KCP. No item renamed.
2026-10-12 (fourth fold), the user's paradigm correction: kcp is a core part of APPLICATIONS (one kcp per
application instance, never per machine); the virtual Kubernetes clusters project is the one
traditional use. New top-level `principle` block (with the port-0 rule) and a market-network-per-workstream state
rule; host-topology wording about kcp restated per application instance (M1-PORT-LEASE-INSTANCE,
M1-DEV-INSTANCE, M2-KCP-PORT0, M2-BOOTSTRAP-LAYER - `--kcp-shared-roles` dropped - M2-OPENBAO-SERVER,
M2-HOSTPROCESS-KIND, M3-SERVE-GATEWAY-KCP, M4-DEV-PARENT-KCP, execution_order); virtual-cluster items
tagged `kcp_use: traditional-exception`. New: M1-NO-FIXED-PORTS. No item renamed.
2026-10-12 (fifth fold), from docs/research/2026-10-11-vm-lifecycle/multi-server-to-kcp.md (sections 2-6)
and the user's answers to its section 7 (user-decisions.md, the 2026-10-12 lines from "(multi-server-to-kcp.md
section 7)" on), which override the research where they differ: STRICT kcp per application, inside a
market network too (the research's nested-workspace market network is rejected; a market network is a MarketNetwork application
composing children that each run their own kcp); every hono-bidder and request-vm-ssh option keeps
working kcp-native, with sign-in as an identity object; kcp only (no standalone deno CLIs); a fresh kcp per
requester invocation with a durable state dir; OAuth sessions and keys only in the application's OpenBao;
cross-app discovery only through values the composer copies; iroh is the only guest transport. New
`principle.decomposition` (replaces `principle.pending`) and M2-APP-CONTROLLER. Changed: M1-CLI (app verb
contract, `profile` option), M1-DEV-INSTANCE (`--profile market-network` = a MarketNetwork composing child
applications with their own kcps), M3-SERVE-GATEWAY-KCP (a Gateway root object), execution_order. No item
renamed or removed. Plans 0007 and 0008 carry the rest of the fold.
queued: >
OPEN. The plan 0005 gate is satisfied: plan 0005 is complete with all 40 items done and its acceptance
checks ACC-ZERO, ACC-MESH and ACC-NOSSH done with evidence (verified 2026-10-12). Only each item's own
depends_on (within this plan and across plans) now governs when it starts.
source: >
The user's request of 2026-10-07: take the test setup (the vendored deno-kcp market network) and expand it
into a prod environment, so one Go binary, socialweb-computer, is the entrypoint for the whole stack;
it runs kcp and does Go-programmatic kubectl-apply-equivalents of templated YAML held in an embed.FS; it
supports `socialweb-computer run {request,bidder,gateway ssh}` and `socialweb-computer serve gateway kcp`;
Go and TypeScript are both first class, with the TypeScript codebase shipped inside the binary in the
style of sshai's policy_engine. The user then decided (2026-10-07): ONE binary; kcp binds port 0 and
reports the bound port; no container at runtime, with postgres and everything else the container
supplied embedded and driven by CRDs; the prod deploy is fresh with no rehearsal (run by Claude, user 2026-10-12); the core binary
and its deno processes must be upgradable while VMs keep running. On 2026-10-11 the user decided: this
plan executes on Linux only and builds the cross-OS seams (Windows is plan 0011, macOS plan 0012); VMs
survive a core crash, upgrade and logout, a reboot is Lost; no socialweb-computer code is in a running
VM's process tree; the gateway and the bidder each live in the USER manager of a lingering account; the
in-process kcp of 2026-10-07 is re-opened and decided after the M0 spike, both routes carried. On
2026-10-12 the user decided: no backwards compatibility, migration, cutover, rollback or prod-downtime
planning until the finished solution exists (the 2026-10-07 "user runs the prod cutover" becomes a fresh
deploy of the finished binary); the gateway runs no containers; tenant control-plane upgrades are
blue/green behind a per-cluster front, each cluster a systemd slice of member units; kcp runs out of
process as blue/green slot units on kine -> a dedicated postgres behind our own front on 127.0.0.1:6443;
production setup is one explicit root command, `socialweb-computer install bidder|gateway`, and systemd
is for production installs only; the dev run mode is kcp-native now (HostProcess objects placed by an
in-binary supervisor on the systemd-user or proc backend, a bootstrap layer below kcp, one kcp per
application instance; docs/research/2026-10-11-vm-lifecycle/dev-run-mode.md); a bidder's kcp is one slot on
kine-sqlite, blue/green kcp slots are the gateway's only. This plan
holds the Go binary, kcp, assets, the install and upgrade layout, host processes as units and reconciled
resources, programmatic apply, the gateway composition, control-plane continuity across upgrades, the
build and the fresh prod deploy. The embedded deno,
the TypeScript closure and DenoPods are plan 0008; the compute provider and VM lifecycle are plan 0007.
principle:
stated: >
Decided by the user 2026-10-12 (user-decisions.md): "what we are doing is a paradigm shift, we are using
kcp (the kubernetes control plane) as a core part of applications, it's probably not how it's
traditionally used. With the exception of our virtual kubernetes clusters project that is a more
traditional usage of it." This is the ORGANIZING PRINCIPLE of plans 0006, 0007 and 0008; every item is
read through it, and an item whose wording contradicts it is wrong, not the principle.
rules:
- "An application instance (a bidder handle, a requester `run request`, the gateway, a market network, a dev instance) = its own kcp + a manifest set + a profile. kcp belongs to the application instance, never to a machine: a host is only where instances run, and any number of instances run on one host."
- "Application state = kcp objects in the instance's own kcp (ComputeVM, ComputeContract, NodeClaim, OpenBaoServer, Postgres, DenoPod, HostProcess, ...). A restarted core re-reads what its instance knows from there."
- "Application components (the servers that live today inside one deno or Go process) = DenoPods and HostProcesses reconciled from those objects."
- "Modes = profiles over ONE manifest set. Example: a bidder in OAuth QR mode against a real account has no PDS pod; a bidder in the market network has its own PDS pod. A mode is never a second code path or a second manifest set."
- "Composition = applying manifest sets into an instance, or nesting instances as workspaces (M4-DEV-PARENT-KCP: a parent's workspace holds a child instance's HostProcess objects; the child keeps its own kcp). A market network or a dev instance is a composition of application instances."
- "Applications talk to each other ONLY over the atproto market protocol (records, XRPC, the relay, the ingress proxy), never by sharing a kcp: no application reads or writes another instance's objects, and no two applications are placed on one kcp."
- "The market network (deploy/market-network) is the seed of this model and the vehicle for parallel workstreams (state rule 'market network per workstream')."
- "The ONLY traditional use of kcp is the virtual Kubernetes clusters project: tenant k3s --disable-agent control planes fronted by kcp (clusterprovider, socialweb-computer-k8s, NodeClaims). Its items carry `kcp_use: traditional-exception` (here M2-POSTGRES, M3-CLUSTER-UNITS, M3-APISERVER-FRONT, M3-CP-SLOTS, M3-LINK-HANDOVER, M3-BAO-SYNC, M3-CP-CONTINUITY; plan 0007 M3-NODE-LOSS-CLOCK, M6-CLAIM-RESUME, M6-GATEWAY-REPOPULATE, M6-NOSSH-JOIN). Even there the gateway's own kcp is still one application instance's kcp."
about_the_machine: >
What stays about the machine, because it is where an instance runs and not what it is: /dev/kvm and the
kvm group, the systemd user manager and its units (production placement only), `install`, the host
contract (check-host), and the bootstrap layer that runs an instance's kcp itself (M2-BOOTSTRAP-LAYER).
port0_rule: >
Every listener behind the edge binds port 0 and reports its bound address into its own kcp object's
status: a DenoPod's runner reads the pod's port file (<runDir>/port-bound-to, written by createServe's
onListen) and records status.address (port-audit.md ranked blocker 1 = B2; 0008 M1-MARKET-NETWORK-PARALLEL); a
HostProcess's address is read back from its single listening socket (hostsvc.ListenPort, M2-KCP-STORE)
and written to its status; kcp slots, kines and fronts likewise (M2-KCP-PORT0). Peers resolve each other
by service name through kcp (the servicenames table built from status.address), so an address is
application state in kcp, never config. Reasons: N instances per machine; no allocator and no
bind-close-rebind race; nesting (a child instance needs no port plan from its parent); blue/green slots
(a new slot binds a fresh port beside the old); isolation (an application learns addresses only from
its own kcp, so it can never reach another instance's components by accident). Exception: only stable
public edge entry points keep known addresses - caddy on 443 and an application's front that caddy
proxies to (the gateway's kcp front 127.0.0.1:6443, M2-KCP-FRONT; the door's 8788, 0008
M3-DOOR-DENOPOD), each listed with its reason in deploy/edge-ports.yaml. Leases (M1-PORT-LEASE-INSTANCE)
remain only for ports that must stay stable inside one instance across restarts (a tenant cluster's
front port and link UDP ports, the virtual-clusters exception) and are recorded in that instance's kcp.
Enforced by M1-NO-FIXED-PORTS.
decomposition:
folded: >
2026-10-12 (fifth fold) from docs/research/2026-10-11-vm-lifecycle/multi-server-to-kcp.md and the user's
answers to its section 7 (user-decisions.md, 2026-10-12). Where the research and the user differ, the
user's answer is what is written here.
rules:
- "Root objects: each application is ONE root object in its own kcp - MarketBidder, MarketRequester, Gateway or MarketNetwork (M2-APP-CONTROLLER) - whose spec carries the application's options and profile and whose status aggregates its children's readiness, DID and reported listeners. The core renders the profile into owned children; a profile switch deletes what the new profile omits."
- "STRICT kcp per application (decided by the user 2026-10-12): every application instance has its own kcp everywhere, including inside a market network. A market network is itself an application, MarketNetwork: its kcp runs the shared atproto services (PLC, relay, dispatchers, PDSes, the QR relay) as its own DenoPods and holds each child application's bootstrap layer as HostProcess objects; it composes a child by applying the child's root object into the child's OWN kcp. No two applications share a kcp. The research's nested-workspace market network (apps as workspaces of one kcp) is rejected."
- "Cross-application discovery (decided by the user 2026-10-12): only through values the composer (MarketNetwork) copies onto a child's root object - the MarketNetwork value (PLC URL, ingress host, relay and firehose URLs, QR origin, guest TLS URL, CA reference) and any peer value a profile names. The DNS shim resolves only names of its own application."
- "Every CLI option stays (decided by the user 2026-10-12): every option hono-bidder and request-vm-ssh support today keeps working, transformed into a root-object field or a profile; an option may be removed only when it selects a legacy (non-iroh) guest transport, with that reason (option-coverage test in M2-APP-CONTROLLER). Interactive sign-in is an object, MarketIdentity (0007 M1-MARKET-IDENTITY): it sets the sign-in up and reports what the user must do (URL to visit, association to approve, session expired) in status, which the foreground CLI renders; tests complete it by driving hono-pds's OAuth session minting. The ephemeral-PDS mode (in-process repo + association records) stays for the bidder and the requester."
- "kcp only (decided by the user 2026-10-12): no standalone `deno run hono-bidder` or `request-vm-ssh` without kcp; every run goes through socialweb-computer and kcp. The TypeScript CLIs become DenoPod entries that refuse to start outside one, from the item that deletes the launcher they serve today (0007 M5-RUN-BIDDER, 0008 M4-RUN-REQUEST), so production keeps running until then. The TypeScript in-process harnesses stay as library tests of the same composer (decided by Claude, user may veto)."
- "Requester (decided by the user 2026-10-12): a fresh kcp per `run request` invocation; its state dir (kcp root + vault under state/<invocation>) persists, and `run request release|resume <name>` restarts that same kcp to finish the contract (0008 M4-RUN-REQUEST)."
- "Secrets (decided by the user 2026-10-12): OAuth sessions and every key (repo key, relay keypairs, OIDC JWK, requester key, door keys, per-claim keys) live in the application's own OpenBao, never in kcp Secrets; kcp objects hold references only."
- "Guest transports (decided by the user 2026-10-12): iroh only. The ssh gateway is ported to reach guests over iroh first (0008 M1-SSH-GATEWAY-IROH), then every non-iroh guest transport is stripped (0008 M1-STRIP-LEGACY-TRANSPORTS): the cloud-init-common tunnel, fedproxy-ssh, fedproxy-web and wootty modules, websocat ProxyCommand paths, relay-fqdn guest addresses and the guest JSR registry."
- "Components (decided by the user 2026-10-12): the bidder's guest-facing endpoints (OIDC issuer, /v1/on-network, behind the provider relay) become their own DenoPods after 0007 M4-BIDDER-VAULT and M3-OIDC-ISSUER-INSTANCE (0007 M5-BIDDER-COMPONENTS). The door's node fulfiller stays in the door DenoPod for now and is revisited after 0007 M6-CLAIM-RESUME (decided by Claude, user may veto; 0008 M3-DOOR-STATE-VAULT)."
- "Every listener reports into status (0008 M2-DENOPOD-LISTENERS: tcp, unix and relay ingress URLs in DenoPod.status.listeners), so no test or peer scrapes stdout for a DID, a port or a readiness line."
state_rules:
- TypeScript lives in this repository (decided by the user 2026-10-12): the closure's ten repositories were vendored with their history under ts/<repository>/ and the originals are frozen; a path this plan names under ts/ is changed here, never in the original.
- Item status (todo, doing, done, blocked) lives here; the git history is the evidence. No evidence lists are added (user, 2026-10-12).
- status is one of todo, doing, done, blocked.
- An item is done when its done_when holds on main; the commits say how.
- Open items are mirrored in NEXT.yaml (P6-*).
- A decision marked "decided by Claude, user may veto" was not confirmed by the user; one marked
"decided by the user 2026-10-07", "decided by the user 2026-10-11" or "decided by the user 2026-10-12"
was.
- >
No backwards compatibility (decided by the user 2026-10-12): no item plans a migration of container-era
state, a cutover, a rollback to the container, or prod downtime handling until the finished solution
exists. Continuity is planned only for upgrades of the finished system (M3-CP-CONTINUITY).
- >
An item names its dependencies in depends_on; an item may not start while one of them is not done.
Cross-plan dependencies are written <plan>:<item id>. Exception (user-approved 2026-10-12): an item with
a `variant_dependencies` field may start the variant that field names as unblocked before the
dependencies listed against other variants are done; the item is done only when every variant is.
Used by M2-KCP-STORE and M2-KCP-SLOTS for the `binary-base` vertical slice (execution_order).
- >
kcp route (decided by the user 2026-10-11: decide after the M0 spike, carry both paths). An item whose
decision says "route: in-process" or "route: out-of-process" is built only on the route M0-KCP-SPIKE's
recorded choice names; the item of the other route is closed done with the evidence "route not chosen:
<M0-KCP-SPIKE evidence ref>" and nothing is built for it. Items with no route tag are built on either.
Resolved 2026-10-12: the user chose out-of-process (M0-KCP-SPIKE evidence); M1-K8S-FORK and
M1-KCP-INPROC are closed under this rule, and no open item depends on them.
- >
systemd is for production installs only and the dev run mode is kcp-native now (decided by the user
2026-10-12, overriding dev-run-mode.md's phasing). Long-lived host processes are HostProcess objects in
kcp reconciled by an in-binary supervisor (M2-HOSTPROCESS-KIND) onto a placement backend: systemd-user
(production installs, M2-HOST-UNITS) or proc (dev and tests: a plain supervised process tree under the
per-instance devsup, M2-HOST-BACKEND-PROC); ComputeVMs, kcp slots and tenant cluster members are placed
through it. The BOOTSTRAP LAYER (M2-BOOTSTRAP-LAYER) is the one exception, because kcp cannot supervise
itself: kcp's store, its kine, its slot(s), its front and the core run as installed systemd units in
production and under devsup in dev, before kcp exists; once kcp is up they are mirrored into it as
read-only HostProcess records. The `host-backend` option is required with no default (install writes
systemd-user, every dev verb and test harness passes proc). Accepted dev defaults: gateway kcp store
sqlite for quick runs, postgres for the market network and prod-like profiles; the k3s node password through a
per-instance user-namespace bind-mount (no root); AppArmor in dev is report-only; one uid for every role
in dev. Every item that starts a long-lived process states which backend(s) its tests run on; an item
whose live tests run only under systemd-user says so.
- >
Every org repo works on `main` and is pushed to GitHub as work lands (decided by the user 2026-10-12).
A worktree branch is merged to its repo's main and pushed when its item (or a red-first test step of it)
lands; nothing is left committed-but-unpushed, in this repo or in a sibling (atproto-market,
hono-compute-provider, typescript-helpers, did-key-associator, socialweb-computer-k8s).
- >
Order of work: execution_order below (and NEXT.yaml execution_order) names the critical path, the
first waves of startable items, the parallel worktree split and which items need live hat-8; it is
advisory and never overrides depends_on.
- >
Market network per workstream (decided by the user 2026-10-12; principle above). Every workstream's live tests run
in the task's OWN market network: a git worktree of this repository (on its own branch) brings up
its own fully isolated atproto network - fake PLC, relay,
PDSes, ingress proxy, bidders, requesters - through 0008 M1-MARKET-NETWORK-PARALLEL now and `dev up --profile market-network`
(M1-DEV-INSTANCE, 0008 M4-MARKET-NETWORK-PORT) later. No workstream live test uses a real account or the
shared real bidder. The shared real bidder (socialweb-computer-bidder@bobvmbuilder.bsky.social on
hat-8) and alice are a FINAL SOAK only, run after the workstream's market network tests pass, never as the
workstream's market network.
- >
Linux only (decided by the user 2026-10-11). Every OS-specific body lives in a GOOS-tagged file; darwin
and windows get stubs returning ErrUnsupported until plans 0012 and 0011 replace them. check.sh proves on
every run that GOOS=darwin and GOOS=windows build and vet (M0-XOS-GATE).
- >
RFP flow is the spine (org-root CLAUDE.md). This plan boots no guest by any path of its own. Live tests
of unit and control-plane mechanics use stand-in executables in transient units (k3s, kine and postgres
processes, never a guest). A live proof that needs a market node runs on the market network
(internal/marketnetwork), where the node exists only because the market network's requester ran runComputeContract
(RFP -> bid -> accept -> cloud-init user_data) against a local bidder.
ownership:
owned: >
All code in socialweb-computer-kcp, the prod deployment on socialweb.computer, the vendored deno-kcp
market network, the atproto-market requester libraries, and the pinned release kcp v0.33.0 binary carried as
an embedded asset (the 2026-10-07 decision to vendor kcp's source, staging modules and embeddedetcd is
superseded by the user's 2026-10-12 route: no vendored kcp source and no kcp patch). hono-compute-provider
and atproto-market may be changed by commit on main, pushed to GitHub as work lands (decided by the user
2026-10-12; replaces "never pushed").
not_owned: >
The market protocol records and lexicons, and the core architecture: kcp in front of
k3s --disable-agent clusters at zero nodes, with nodes only ever born through
requester -> RFP -> bid -> accept -> cloud-init user_data.
research:
measured_on: "hat-8, 2026-10-07; repo at 13349d6 (main). Lifecycle research and proofs 2026-10-09..11 against f0460fd, see vm_lifecycle_2026_10_11"
today_prod_shape:
- "deploy/control-plane-container/Dockerfile:140-214 (target k8s): Fedora 44 base, postgres from dnf (:105-106), /etc/rancher/node/password created for uid 1000 (:108-113), kcp/k3s/kine/kubectl/bao copied from a host-staged binaries context (:117-125), external-secrets from its digest-pinned image (:127-129), deno 2.9.3 and dumbpipe 0.39.0 fetched (:142-164) with yq, iproute, socat, openssl, unzip, openssh-clients, which (:153), seven Go binaries (:172-181), the door's org-root closure at /opt/org-root (:183), credential cross-builds and meshd as door downloads (:185, :199-202), deno cache of the door (:206-210), ENTRYPOINT container-entrypoint (:212)"
- "deploy/container-entrypoint.sh: root vault bao child (:215-224), kcp start --feature-gates=WorkspaceMounts=true with the audit policy (:226-235), readyz poll (:237-247), socialweb-vault apply-root (:253-258), install-provider.sh through kubectl (:260-266), bootstrap_cluster as four kubectl heredoc applies (:129-213, :268-276), clusterprovider migrate (:278-281), socialweb-gateway (:283-285), socat TLS terminator forking per connection (:287-291), socialweb-clusterprovider (:293-295), the door as deno run with --preload undici-preload.ts (:297-317); any child exit stops all and exits 1 (:321-341). Nothing between `bao server` and apply-root needs kcp, and kcp needs nothing from the vault (exploration F8)."
- "scripts/deploy.sh: docker save | ssh | docker load (:59-67), docker rm -f the running container (:114), docker run --network host -v socialweb-k8s-state:/var/lib/socialweb (:119-124), door readiness by curl /api/config (:126-140), caddy reload (:187). Stop-then-start: there is downtime today."
- "deploy/control-plane-container/build.sh:42-64 derives the door's TypeScript closure across the org root with deno info --json; :22-39 stages and sha256-pins host binaries"
- "internal/versions/versions.json: kcp v1.36.0+kcp-v0.33.0 (MUST run --feature-gates=WorkspaceMounts=true), k3s v1.36.4+k3s1 (a --disable-agent server still needs /etc/rancher/node/password readable and writable, no flag moves it), kine v0.17.1 (CGO_ENABLED=1), openbao v2.6.2, external-secrets v2.11.0 by digest, embedded-postgres = the host's PostgreSQL 18.x binaries, postgres refuses to run as root"
- "executables named in non-test Go under internal/ and cmd/ (string mentions, 2026-10-07): k3s, firecracker, external-secrets, postgres, kine, mount, psql, bao, initdb, pasta, kubectl, mkfs.ext4, ip, busybox, dumbpipe, zstd, xz, ssh, openssl, unshare, resize2fs, e2fsck, qemu-system-x86_64; nodeboot/nodeimage also exec debugfs, cpio, du, chroot, apt-get (in a chroot) and go (exploration F3, F5)"
- "plan 0005 M6-LIMITS: every per-cluster process-group member dies with the provider (Pdeathsig SIGKILL); plan 0005 M3-NODE-LOSS: a Node NotReady for 1 minute (NodeLossGrace) is released and replaced, so a gateway restart that keeps control planes down past 1 minute releases market nodes"
- "cluster stack (exploration F7): procgroup.StackOrder = postgres, openbao, kine, k3s, ccm, eso-webhook, eso; postgres is not a group step (steps.go:91-93), StartPostgres (postgres.go:97-150) execs host initdb/postgres/psql, starts postgres as a plain child with no Pdeathsig (:306-329) and adopts a server already serving (:116-123); the cluster OpenBao stores in that postgres (clusterstack/openbao.go:32-33) and procgroup refuses openbao without PostgresUp; the root vault uses storage file (rootvault.go:43)"
kcp_in_process:
- "our go.mod: go 1.26.6, github.com/kcp-dev/sdk v0.33.1, k8s.io/{api,apimachinery,client-go} v0.36.4, sigs.k8s.io/yaml v1.6.0, klauspost/compress v1.19.2 (indirect)"
- "kcp v0.33.0 go.mod :61-95 replaces EVERY k8s.io staging module (~33, incl. api, apiserver, client-go, apimachinery, kubernetes) with github.com/kcp-dev/kubernetes/staging/... v0.0.0-20260602065202-e006560fc76a (k8s.io/kubernetes => kcp-dev/kubernetes v1.37.0-alpha.0.0.20260602065202-e006560fc76a); :98-103 replaces kcp-dev/{apimachinery,client-go,code-generator,sdk,virtual-workspace-framework} with ./staging/src paths. A dependency's replaces are ignored and ./staging paths do not resolve from a module download, so linking kcp means vendoring its source and repeating every replace in OUR go.mod, moving every package of this module onto the fork."
- "kcp v0.33.0 cmd/kcp/kcp.go:95-162: options.NewOptions(rootDir) -> Complete(ctx) -> Validate() -> server.NewConfig(ctx, opts.Server) -> Complete() -> embeddedetcd.NewServer(...).Run before NewServer -> server.NewServer(completed) -> s.Run(ctx)"
- "kcp v0.33.0 pkg/server/options/options.go:78,204 --experimental-bind-free-port; :236-240 requires --secure-port=0 with it; :355-361 creates a tcp listener on <bind>:0 and stores it in SecureServing.Listener; :387-392 adds the listener's IP to the serving cert; :411-416 relaxes SecureServing.Required. The secure port can bind 0 today; nothing reports the bound port to a caller."
- "github.com/kcp-dev/embeddedetcd 2cc5c5cce35e options/embedded.go:33-45 PeerPort and ClientPort are strings defaulting to 2380 and 2379, :51-52 flags, :70 the apiserver's etcd ServerList is formatted from ClientPort before etcd runs. Embedded etcd cannot bind 0, and two embedded kcps on one host (gateway, bidder, run request, a test) collide on 2379/2380."
- "risks of the in-process route not measured on 2026-10-07 (exploration A, M0-KCP-SPIKE): process-global singletons (klog flags, feature-gate registry, Prometheus default registry) may make two kcps in one process impossible whatever the ports; client-go 1.37.0-alpha against k3s 1.36 tenants; one heap for kcp, gateway and clusterprovider; every core upgrade restarts kcp. The out-of-process route (patched release asset in its own unit) makes core upgrades kcp-neutral."
kcp_route_2026_10_12:
files:
- "docs/research/2026-10-11-vm-lifecycle/kcp-spike.md (M0-KCP-SPIKE: route A record, route B with slots; branch kcp-spike 5f8ef8c scratch code, 3bb6d73 write-up, on main as d6e1881)"
- "docs/research/2026-10-11-vm-lifecycle/kcp-blue-green.md (two kcps on one store behind a front; sections 7 and 8 are the design and the plan changes)"
route_a_record: "in-process on kcp's fork: go build 0 compile errors, go test 1438 pass / 0 fork-caused failures, deno-kcp and kcp-libs green on the fork, +17.8 KB gateway / +34.6 KB clusterprovider, in-process kcp 8.9 s to /readyz at 415 MB RSS, two kcps in one process start, serve and stop (klog's logging registry is process-global). Not blocked by breakage; not chosen because kcp blue/green would mean two whole core processes overlapping."
route_b_measured:
- "release kcp v0.33.0 (sha256 0929a16bedf3275c..., static ELF) with --secure-port=0 --experimental-bind-free-port and an external store: /readyz 8.7-8.9 s fresh, 6.1-6.7 s restart on kine->postgres 18.6; kcp RSS 389-409 MB, kine 54-61 MB; SIGTERM -> exit 1.1 s; each kcp holds exactly one TCP listener"
- "with --shard-base-url=<front>, admin.kubeconfig and the shard URLs name the front; the slot's own port is known from its MainPID's single LISTEN socket (/proc/<pid>/net/tcp matched to /proc/<pid>/fd socket inodes, same uid) 0.17-0.30 s after exec, before any log line"
- "a new slot needs apiserver.crt, apiserver.key, sa.key, .admin-token-store and admin.kubeconfig copied in before start (without admin.kubeconfig it refuses to start); the shard-admin token and SA tokens carry over; the kcp-admin (root/base context) token is regenerated per process (authentication.go:109)"
- "handover: old slot SIGTERM -> exit 3.1 s with --shutdown-delay-duration=2s; Lease kube-system/kcp-controllers deleted through the front with the shard-admin context; the new slot led 0.46 s later. Without the delete the controller gap is 61 s (kcp never releases its lease); without leader election both slots run every controller (~120-150 write conflicts/s per process)"
- "kcp-blue-green.md section 6: 3,177 /readyz polls at 200 ms across six swaps, 0 failed; the one failing run read readiness through a copied admin.kubeconfig that still named the OLD slot (readiness must be read on the new slot's own port)"
- "caddy incident (kcp-blue-green.md section 6): a tls.automation policy with caddy's internal issuer installed a local CA into the system trust store through non-interactive sudo; reverted. Any caddy config the binary or repo renders pins install_trust false and never uses caddy's internal issuer"
dev_run_mode_2026_10_12:
file: "docs/research/2026-10-11-vm-lifecycle/dev-run-mode.md (requirements R1-R8, options a-d, design 4.1-4.9, plan deltas 5, measurement 6; its phasing of the HostProcess kind is overridden: the user decided kcp-native now)"
measured: "hat-8, stand-ins only (a 130-line Go supervisor): an adopter that is not the parent gets ECHILD from waitid(P_PIDFD) for every unit (exit status lost), Pdeathsig alone leaks a same-pgid grandchild to PID 1, a pgid sweep by recorded leader ticks kills the orphan and refuses a reused pid; hence devsup as the stable parent with PR_SET_CHILD_SUBREAPER, per-unit process groups and tree kill"
code_today: "hostsvc.Manager has one implementation (systemd_linux.go); portallocator's lease key ignores Lease.Instance (portallocator.go:145-148) and every account opens its own lease file; the market network fixes 2583-2589 (internal/marketnetwork/live_test.go:18) and the deno-kcp kcp 6443 and kine 23791 (deploy/denokcp/start-kcp.sh:5-6), so one market network per host"
server_side_apply_precedent:
- "internal/clusterprovider/tenant.go:101-102 and internal/eso/beside.go:350-351 apply unstructured objects with types.ApplyPatchType and PatchOptions{FieldManager, Force: true}; internal/clusterprovider/registry.go:293 addresses a workspace by APIPath /clusters/<logical cluster>/apis"
- "deploy/install-provider.sh:55-64 applies 5 APIResourceSchemas, 2 APIExports and 2 WorkspaceTypes; deploy/apply-permission-inventory.sh (285 lines) renders the inventory through yq; deploy/market-network/apply.sh:171 templates by sed"
binary_sizes_raw_and_zstd19:
deno_2_9_3: "106313392 / 33967519"
kcp_v0_33_0_release: "176825356 / 59189153 (zstd -19 --long=27, 0.15 s to decompress; kcp-spike.md route B section 3; the 60028199 of 2026-10-07 was plain -19). The embedded kcp asset (route decided 2026-10-12)"
bao_v2_6_2: "196404560 / 63961162"
k3s_v1_36_4: "78991522 / 72883115"
kine_v0_17_1: "57397648 / 25394980"
vm_lifecycle_2026_10_11:
files:
- "docs/research/2026-10-11-vm-lifecycle/user-decisions.md (authoritative user decisions)"
- "docs/research/2026-10-11-vm-lifecycle/plan-0006-0007-linux-exploration.md (gap list A, seam design B, revised items C, questions D)"
- "docs/research/2026-10-11-vm-lifecycle/linux-proofs.md (live proofs E0-E7, hat-8 Fedora 44, systemd 259.9, 2026-10-11 23:12-23:25Z)"
- "docs/research/2026-10-11-vm-lifecycle/lifecycle-synthesis.md (per-OS lifecycle research 2026-10-09)"
proven_on_linux:
- "E1: a transient service started in the user manager over D-Bus survives `systemctl --user restart`, `kill -9` and `stop` of its starter unit (same InvocationID and MainPID), over the session bus and over $XDG_RUNTIME_DIR/systemd/private; plain children of the starter die in all three cases with or without Pdeathsig/setsid; StartTransientUnit on an existing name fails 'already loaded' (adopt, never re-create)"
- "E1: go-systemd has no constructor for the USER private socket; a 5-line custom dialer (EXTERNAL auth, no Hello) is needed; every call takes a context with a deadline (godbus Auth otherwise blocks forever, E3)"
- "E2: go-systemd PropExecStart(argv, true) sets ignore_errors=yes: a SIGKILLed unit reports success and is garbage-collected; PropExecStart(argv, false) with CollectMode=inactive keeps `failed signal`/`oom-kill` readable until reset-failed"
- "E2: MemoryMax without MemorySwapMax=0 swaps instead of OOM-killing; MemorySwapMax=0 + OOMPolicy=stop gives Result=oom-kill; TasksMax and CPUQuota enforced; AllowedCPUs accepted but NOT enforced (no cpuset delegated); IOWeight has no effect (scheduler none) and ManagedOOMPreference=avoid|omit resets IOWeight to 100 on systemd 259.9; Fedora's user app.slice is ManagedOOMMemoryPressure=kill under systemd-oomd"
- "E0: stock systemd delegates memory+pids to user@; Fedora adds cpu/io via uresourced; linger is off by default and mandatory (without it every unit stops at the last logout); transient unit definitions live on tmpfs and vanish at reboot"
- "E3: identity check order pidfd_open -> /proc/<pid>/stat start ticks -> pidfd_send_signal(0) -> GetUnitByPIDFD (systemd >= 253) -> InvocationID; ESRCH, ticks mismatch, UnixProcessIdUnknown, NoUnitForPID, NoSuchUnit are Lost; only dial/auth/timeout errors are Unknown"
- "E6: os.Executable() after a rename-over returns the path, so re-exec runs the NEW build (version skew inside a booting chain); after a delete it is ENOENT; install to immutable content-addressed paths"
- "E7: GOOS=darwin build fails (internal/hostos has no darwin files; 23 packages blocked); GOOS=windows build fails (internal/nodevm/disk.go untagged unix.IoctlFileClone/SEEK_DATA; 10 packages blocked); GOOS=windows vet also fails on internal/apigen/live_upgrade_test.go:71 Setpgid; deploy/check.sh:30,54 builds and vets only the host GOOS"
- "code today: internal/hostos/caps_{linux,windows}.go (Firecracker/QEMU/NetworkMode/Accel incl. AccelWHPX) are dead since 9d85916 retired nodeqemu and the local substrates, read only by cmd/socialweb-computer-kcp-check-host (platform.go:14-88, checks.go:23, config.go:197); hostos.DetachProcess (sysproc_linux.go:10) is Pdeathsig SIGKILL, i.e. it ATTACHES the VMM's life to its parent, used only by internal/nodefirecracker/namespace.go:45"
- "E8 (live, RFP flow, 2026-10-12): see plan 0007 research; firecracker CtrlAltDel -> exit 0.21 s on an idle guest"
control_plane_handover_2026_10_11:
file: "docs/research/2026-10-11-vm-lifecycle/control-plane-handover.md (design, procedures, Q-H1..Q-H9)"
measured_on_stand_ins:
- "hat-8, stand-ins only (PostgreSQL 18.6, two kine v0.17.1 on one database, two k3s v1.36.4+k3s1 --disable-agent, a ~100-line L4 front): a second server with the same token file and its own data dir and kine reconciled the bootstrap data from the datastore, had byte-identical server-ca, client-ca, request-header-ca, service.key and server/token, and served the first server's admin client certificate 2.2 s after start"
- "front switched to green, blue SIGTERMed: blue exited in 0.10 s; authenticated /readyz through the front every 200 ms: 515 polls, 0 failures; fake Node lease heartbeats 0 failures, Node stayed Ready; kcm/scheduler/ccm leases moved to green 16-18 s later (15 s lease)"
- "the kubernetes EndpointSlice read (advertise address, advertise port) throughout: k3s passes --advertise-port to its apiserver, so a split advertise port (the front's) is honoured"
- "hazards: the kube-system/k3s wrangler leader lease uses the OS hostname as identity (--node-name does not change it), so two slots on one host BOTH lead k3s's own controllers during the overlap; the market kube-scheduler runs leaderElect false (internal/clusterstack/scheduler.go:51-52, 1cbf945); k3s exits 0.1 s after SIGTERM, so exec/attach/log streams through it reset on any swap"
- "two kine on one postgres: the normal k3s HA shape; unique (name, prev_revision) is the compare; no compaction leader (the apiserver compacts with a CAS); a kine schema migration takes ACCESS EXCLUSIVE, so a migrating kine bump is its own tested step"
decided_2026_10_12: "user-decisions.md: blue/green behind a front, each cluster a systemd slice of member units, new k3s+kine slot beside the old on the same postgres, front swaps, old stops; scheduler and k3s leader election fixed; idle-first swaps; link keeps its iroh key"
code_today_store: "internal/clusterprovider/stack.go:39-43 (ErrNoStore): k3s dials kine at http://127.0.0.1:<kine lease> (DaemonURL); kine's --endpoint is the cluster's own postgres DSN (cmd/socialweb-clusterprovider/main.go:370-375 storeEndpoint, unless SOCIALWEB_CLUSTERPROVIDER_STORE_ENDPOINT names an operator store); measured against k3s v1.36.4+k3s1 the S3 form is refused ('unknown driver') and a t4 endpoint makes k3s start its OWN kine. Leftover rustfs/t4 names: internal/vaultlayout PathCentralStore, PolicyCentralStore, ClassCentralStore; cmd/socialweb-vault SOCIALWEB_CENTRAL_STORE_* and its apply branch; internal/portallocator/portallocator.go:200 message; internal/versions/versions.json rustfs and t4 entries (M1-STORE-CLEANUP)"
items:
- id: M0-XOS-GATE
milestone: M0
title: darwin and windows build and vet green, and gated on every check.sh run
depends_on: []
decision: >
Decided by the user 2026-10-11: Linux-only execution with cross-OS seams built now. Decided by Claude,
user may veto, from linux-proofs E7 and exploration B7: the first commit of this plan makes
`GOOS=darwin` and `GOOS=windows` build and vet clean and keeps them so. (1) internal/hostos becomes
primitives only: lock_unix.go (`//go:build unix`, flock), proc_{linux,darwin,windows}.go,
portrange_{linux,darwin,windows}.go, bootid_{linux,darwin,windows}.go, sysproc_{linux,darwin,windows}.go;
darwin bodies are stubs returning ErrUnsupported where no portable answer exists (plan 0012 fills them),
plus internal/portallocator probe_darwin.go. (2) The capability predicates are deleted: caps_linux.go,
caps_windows.go (Firecracker, QEMU, NetworkMode, Accel, AccelWHPX); check-host stops reading them and
reads the compiled backend registry (vmruntime/hostsvc capabilities, plan 0007 M2-VM-RUNTIME and
M2-HOST-UNITS here; until those land, check-host reports "no VM runtime compiled" on non-linux).
(3) hostos.DetachProcess is deleted: it is Pdeathsig SIGKILL under a name that says the opposite; its
one caller (nodefirecracker/namespace.go:45) uses hostos.DieWithParent until plan 0007 M2-VM-RUNTIME
deletes that resident verb. (4) internal/nodevm/disk.go's reflink clone splits into clone_linux.go
(FICLONE + SEEK_DATA) and clone_other.go (plain copy). (5) internal/apigen/live_upgrade_test.go is
tagged `//go:build linux`. (6) deploy/check.sh --fast runs, before tests, for darwin/arm64,
darwin/amd64, windows/amd64, windows/arm64 and linux/arm64: `go build ./... && go vet ./...` with the
default (non-release) tags, so no embedded blob is needed (M1-ASSETS). (7) An import-boundary test
(`go list -deps -json` per GOOS) asserts the OS-neutral packages (api, kcpapply, the ComputeVM
controller core, the vmruntime and hostsvc interfaces) never import go-systemd, netlink or the
firecracker client. ARCHITECTURE section 9 "The host platform" is rewritten to the seam table in the
same change (it claims windows build and vet are clean and describes a deleted QEMU substrate).
files: ["internal/hostos/*", "internal/portallocator/probe_darwin.go", "internal/nodevm/clone_linux.go, clone_other.go (from disk.go)", "internal/apigen/live_upgrade_test.go", "cmd/socialweb-computer-kcp-check-host (platform.go, checks.go, config.go)", "internal/nodefirecracker/namespace.go", "deploy/check.sh", "internal/xosgate (import-boundary test)", "docs/ARCHITECTURE.md section 9 The host platform"]
test_first: >
check.sh's new cross-OS step run on today's tree, red with the E7 errors recorded (darwin: hostos
symbols undefined in 23 packages; windows: nodevm/disk.go in 10 packages, apigen Setpgid in vet), then
green; internal/xosgate TestOSNeutralPackagesImportNoLinuxBackend red with a deliberate go-systemd
import in a fixture, then green.
done_when: "check.sh --fast green with the five cross-OS build+vet runs; no caps*.go or DetachProcess left; ARCHITECTURE host-platform section matches the code."
risks:
- "A later item adds an untagged OS-specific file; the gate fails that commit, which is the point."
- "Plan 0008 M2-DENOKCP-FORK brings kcp-libs' execrunner (Setpgid, untagged) into the module; it must arrive build-tagged or this gate goes red."
status: done
evidence:
- "branch xos-gate: 2c83e70 (check.sh cross-OS step + internal/xosgate, committed red), bb744f4 (hostos primitives only, darwin twins; caps_*.go and DetachProcess deleted), 16a4a93 (ARCHITECTURE section 9 seam table); merged to main 2026-10-12"
- "red 2026-10-12T00:40:41Z: build:darwin/arm64, darwin/amd64, windows/amd64, windows/arm64 failed with the E7 errors (24 darwin packages via hostos/portallocator, windows nodevm/disk.go); TestOSNeutralPackagesImportNoLinuxBackend red 00:42:06Z with the go-systemd fixture, then green with TestBoundaryFlagsAGoSystemdImportInAFixture"
- "green: check.sh --fast exit 0 at 00:47:02Z and 00:49:02Z; beyond E7 the gate also fixed portscan on darwin, nodeimage buildlock on windows and five untagged linux-only test files"
- "linux go test ./... offline: before 1439 pass / 63 skip / 0 fail (00:39:52Z), after 1442 / 63 / 0; on merged main 2026-10-12 58/58 packages ok uncached, GOOS=darwin/arm64 and windows/amd64 build+vet ok"
- id: M0-KCP-SPIKE
milestone: M0
title: Measure both kcp routes - in-process on kcp's fork, or a patched release binary in its own unit - and record the route
depends_on: []
decision: >
Decided by the user 2026-10-11: the in-process fork is decided after this spike; the plan carries
both paths (in-process fork vs exec'd kcp asset with port 0 and a port-report patch), gated on the
spike's breakage count and effort. In a scratch branch, time box two executor days. In-process route:
vendor kcp v0.33.0 with its staging modules, repeat its replace set in our go.mod, add an in-process
start; record per package every compile error and every test failure of go test ./..., whether
kcp-libs and deno-kcp build on the fork, the binary size delta, cold start to /readyz, RSS on a
prod-shaped root with gateway and clusterprovider in the same process, whether two kcps start in one
process (klog flags, the feature-gate registry, the Prometheus default registry), whether server.Run
honours ctx cancel and releases etcd and the root flock within a bound, and whether tenantclient, eso
and clusterprovider apply paths still work against a k3s 1.36 tenant with client-go 1.37.0-alpha.
Out-of-process route: build the kcp release binary from the vendored source with the two port patches
(embedded etcd client/peer port 0; a port report written after bind), run it as a transient unit with
today's flags, record cold start, RSS, the port-report latency and the patch size. Gate, decided by
Claude, user may veto: in-process only if the fork's breakage is fixed within one executor day, go test
./... is green on the fork, and two in-process kcps run in one process; otherwise out-of-process. The
spike records the numbers and a recommendation; the user confirms the route (questions_for_user) and
the confirmed route is written into this item's evidence, which the route state rule reads.
files: ["scratch branch only; the breakage list, the measurements and the confirmed route are recorded in this item's evidence"]
test_first: "None (measurement). The readings are the build log, the go test summary, a timed /readyz on each route, RSS, and the two-kcps-in-one-process attempt."
done_when: "both routes measured; the gate's three conditions recorded with numbers; the user-confirmed route recorded."
risks:
- "A partial in-process breakage list is still recorded; a time-boxed spike that cannot finish the fork counts against the in-process route."
- "The in-process route restarts kcp on every core upgrade (kcp.socialweb.computer down for the restart window); the out-of-process route does not. Recorded as part of the comparison."
status: done
evidence:
- "docs/research/2026-10-11-vm-lifecycle/kcp-spike.md, host hat-8 (16 cores, 46 GiB, Fedora, go 1.26.6), 2026-10-12; scratch code on branch kcp-spike 5f8ef8c (scripts/kcp-spike/*, cmd/socialweb-kcp-spike), write-up 3bb6d73 cherry-picked to main as d6e1881. Stand-in processes only (kcp, kine, postgres, socat on 127.0.0.1); no guest, no container."
- "route A (in-process, kcp v0.33.0 1a4cb725 + embeddedetcd 2cc5c5cce35e vendored, kcp's 33 k8s.io replaces in our go.mod): go build 0 compile errors (37 s cold), go vet ok, go test 1438 pass / 0 fork-caused failures, deno-kcp 25d10f92 and kcp-libs f6911d2 green on the fork; darwin/windows failures identical to main (M0-XOS-GATE's, since fixed); gateway +17.8 KB, clusterprovider +34.6 KB, in-process kcp binary 177,440,498 B; cold start 8.9 s to /readyz, RSS 415 MB, ctx cancel returns in 8 ms-1.1 s; two kcps in one process both ready (9.0 s, 9.3 s, RSS 656 MB), the second inherits the first's logging config. Gate's three conditions met (no breakage, go test green, two in-process kcps)."
- "route B (release kcp in its own process, kine sqlite and postgres 18.6, port 0): /readyz 8.5-8.9 s fresh, 5.9-6.7 s restart; kcp RSS 378-409 MB, kine 54-62 MB; SIGTERM -> exit 0.05-1.2 s; the bound port is read from the slot MainPID's single LISTEN socket 0.17-0.30 s after exec with no kcp patch; five root files copied make a second slot accept the first's shard-admin and SA tokens; lease delete after the old slot's exit hands controllers over in 0.46 s; asset 176,825,356 B raw / 59,189,153 B zstd -19 --long=27."
- "route recorded: OUT-OF-PROCESS, decided by the user 2026-10-12 (user-decisions.md kcp line, from kcp-blue-green.md section 9): release kcp v0.33.0 as an embedded asset, no kcp patch, alternating blue/green slot units on kine -> a dedicated postgres behind our own L4 front on 127.0.0.1:6443. Route A is feasible and not chosen because in-process ties kcp's lifetime to the core process (kcp blue/green would need two whole core processes overlapping). Built by M2-KCP-STORE, M2-KCP-FRONT, M2-KCP-SLOTS, M2-KCP-PORT0, M3-KCP-ROLL."
- "not measured (capped when the route was decided): client-go 1.37.0-alpha against a live k3s 1.36 tenant and RSS on a prod-shaped root in one process (route A only, moot); prod-shaped object-count timings for route B are taken by M2-KCP-SLOTS and M3-KCP-ROLL."
- id: M0-SIZE-BUDGET
milestone: M0
title: One binary, a size budget for linux/amd64 and linux/arm64, darwin and windows as seams only
depends_on: [M0-KCP-SPIKE]
decision: >
ONE release binary (decided by the user 2026-10-07). Linux only in this plan (decided by the user
2026-10-11): the platform matrix in versions.json lists linux/amd64 and linux/arm64 with their asset
sets and budgets; darwin/arm64, darwin/amd64, windows/amd64 and windows/arm64 are listed as
"seam: builds and vets, no assets, owned by plan 0012 (darwin) / plan 0011 (windows)", which adds
their own budget lines. Decided by Claude, user may veto: the budget per platform is the measured
asset sizes (the per-OS set of M1-ASSETS), the TypeScript closure (0008:M1-TS-CLOSURE), plus the kcp
release asset (route decided by the user 2026-10-12: out of process; M0-KCP-SPIKE measured
176,825,356 B raw and 59,189,153 B as zstd -19 --long=27 for linux/amd64, extracted once to the
content-addressed runtime dir; linux/arm64 measured in this item); guest images
are NOT embedded (fetched and verified, plan 0007 M1-GUEST-IMAGE) and get a reserved zero line so the
binary never grows by a rootfs; enforced by check.sh. socialweb-credential is still built separately
for four platforms, but only as a download asset carried inside the binary (today's web/downloads,
Dockerfile:47-53, :185).
files: ["internal/versions/versions.json (budget, platform matrix, per-asset sha256)", "deploy/check.sh (size gate)"]
test_first: "internal/versions TestTheReleaseBinaryFitsItsSizeBudget, red with a 1-byte budget, then green; TestTheMatrixListsDarwinAndWindowsAsSeamsWithNoAssets."
done_when: "budget and matrix recorded and enforced."
risks:
- "k3s barely compresses (79 MB -> 73 MB); it is linux-only and stays in the linux builds regardless."
- "kcp is the largest asset after compression with bao (~59 MB and ~64 MB zstd); the gateway (blue/green slots) and every bidder (one slot per handle on kine-sqlite, decided by the user 2026-10-12) carry it."
- "The relocatable PostgreSQL 18 set (M0-PG-SOURCE) is a budget line of its own; its size is recorded there per platform."
status: done
evidence:
- id: M0-PG-SOURCE
milestone: M0
title: "Spike: where a pinned relocatable PostgreSQL 18 per linux platform comes from, for the gateway's kcp store and the tenant stores; choice and sha256 pins recorded"
depends_on: []
decision: >
Decided by the user 2026-10-12: an M0 spike with a fallback. The gateway's kcp store (M2-KCP-STORE; a
bidder's kcp is kine-sqlite and needs no postgres) and every tenant cluster store (M2-POSTGRES) run one
pinned relocatable PostgreSQL 18 (postgres, initdb, pg_dump; no psql, M2-POSTGRES uses pgx) for
linux/amd64 and linux/arm64, never the host's package. Candidates measured in a scratch branch on hat-8
(stand-ins only: initdb + postgres on a unix socket, kine on top, no kcp, no guest): (1) zonky
embedded-postgres binaries (the io.zonky.test.postgres embedded-postgres-binaries jars for linux-amd64
and linux-arm64v8: relocatable txz inside a maven jar); (2) theseus-rs/postgresql-binaries (GitHub
release tarballs per target triple); (3) building from source with the build-host script
(scripts/build-postgres.sh, the same any-Linux-host script shape as 0007 M1-GUEST-IMAGE; CI calls it
later). Recorded per candidate: provenance and how upstream builds it, whether a sha256 or signature is
published, the 18.x minor available, size raw and zstd -19 (feeds M0-SIZE-BUDGET), the glibc floor
(oldest distro it runs on: Ubuntu 22.04/24.04, Debian 12, Fedora 44 checked by running initdb + postgres
+ pg_dump), ICU and collation provider versions (M2-POSTGRES treats a collation change as major-class),
whether it runs from any extraction path (relocatable: no baked prefix, share/ and lib/ found relative
to bin/), --with-systemd absent (readiness by our pgx probe), and the time to initdb + first SELECT 1.
Decided by Claude, user may veto: prefer a published upstream binary set with a published checksum and
a glibc floor at or below Ubuntu 22.04; fallback = build from source with scripts/build-postgres.sh on
any Linux host (pinned source tarball sha256, reproducible flags, ICU vendored or disabled, recorded),
which is always available. The choice and the per-platform sha256 pins go into versions.json and this
item's evidence; M1-ASSETS embeds the chosen set.
files: ["scratch branch only (measurements)", "internal/versions/versions.json (postgres source, version, per-platform sha256)", "scripts/build-postgres.sh (written only if the fallback is chosen)"]
test_first: "None (measurement). Readings: per candidate the initdb + postgres + pg_dump run on each listed distro (or container image of it, stand-in only), sizes, glibc floor, ICU version, checksum provenance."
done_when: "the choice recorded with its sha256 pins for linux/amd64 and linux/arm64, the fallback's status recorded, and the user told the choice."
risks:
- "A prebuilt set whose glibc floor is newer than a target distro is refused there by check-host; the fallback build sets the floor."
- "zonky jars are a Java ecosystem artefact; their checksums are maven's, recorded as such."
status: done
evidence:
- id: M1-CLI
milestone: M1
title: cmd/socialweb-computer, a multi-call CLI whose subcommand tree is the stack
depends_on: []
decision: >
The four subcommands the user named: serve gateway kcp, run bidder, run gateway ssh, run request.
The rest decided by Claude, user may veto: serve provider, vault prepare-root|apply-root, migrate,
apply <set> [--set k=v] [--dry-run], manifests list|render <set>, check-host, version [--json],
install gateway|bidder and uninstall gateway|bidder (M1-INSTALL), upgrade, rollback (M1-INSTALL),
units list|audit|gc|roll (M2-HOST-UNITS, M2-VM-TREE-AUDIT, M3-CLUSTER-UNITS, M3-CP-SLOTS, `units roll
kcp` M3-KCP-ROLL), internal <verb> (re-exec targets: cloud-controller-manager, tenantcredential,
apiserver-front (M2-KCP-FRONT, reused by M3-APISERVER-FRONT); there is no
cluster-group supervisor verb, M3-CLUSTER-UNITS). There is no nodeboot or nodeboot-namespace
verb (deleted, M3-FOLD; plan 0007 M2-VM-RUNTIME) and nodeimage exists only under the `dev` tag (plan
0007 M1-GUEST-IMAGE). The CLI tree is identical on every OS; a verb whose backend is not compiled for
this GOOS exits 2 naming the plan that builds it ("run bidder: no VM runtime on darwin/arm64 in this
build (plan 0012)"); an unbuilt subcommand exits 2 naming the item that builds it. Config resolution
mirrors the TypeScript cli-args-env chain (flag, env, config file, default) with
internal/clicfg/options.json; every SOCIALWEB_* variable today's entrypoint reads stays the env name
of its option. Stdlib flag only. Dev run mode (decided by the user 2026-10-12: kcp-native now; the
host-backend option is required with no default): the tree gains `dev up|down|status|ls|logs|env|check|
restart|roll|gc` (M1-DEV-INSTANCE; `dev roll` lands with M3-KCP-ROLL, `dev gc` with M1-ASSETS) and
`internal devsup` (M2-HOST-BACKEND-PROC); options.json gains `host-backend` (systemd-user | proc, env
SOCIALWEB_HOST_BACKEND, NO default: every verb that places a host process exits 2 naming the option when
it is unset, never probes), `instance`, `kcp-base-workspace` (default root), `state-dir` (resolved by
install or dev, no hardcoded default in the CLI) and `asset-cache`.
Fifth fold 2026-10-12 (multi-server-to-kcp.md 5.1; decided by the user: every hono-bidder and
request-vm-ssh option keeps working kcp-native, kcp only): every APPLICATION verb (`run bidder`, `run
request`, `serve gateway kcp`, `dev up`) follows one contract - resolve the application's own kcp,
server-side apply its root object (M2-APP-CONTROLLER) with --profile, watch the root status, render
pending actions (MarketIdentity, 0007 M1-MARKET-IDENTITY) and print outputs; no application verb opens
a listener or spawns deno itself. options.json gains `profile`. The contract's scan test
(TestNoAppVerbOpensAListenerOrSpawnsDeno) lands with M2-APP-CONTROLLER, so this item's scope (and the
binary-base slice) is not widened.
files: ["cmd/socialweb-computer/main.go", "internal/clicfg/", "internal/clicfg/options.json", "internal/capabilities (verb -> required backends)"]
test_first: >
internal/clicfg TestTheFlagBeatsTheEnvBeatsTheConfigBeatsTheDefault and
TestEveryEnvVariableTheEntrypointReadsIsAnOptionsEnvName (parses container-entrypoint.sh and
container.env.example); cmd/socialweb-computer TestAVerbWithNoBackendOnThisOSExits2NamingItsPlan (run
as a GOOS=windows vet-compiled table test of the capability registry),
TestServeWithoutAHostBackendExits2NamingTheOption and TestTheHostBackendOptionHasNoDefault; red before
options.json exists.
done_when: "--help lists the tree and marks verbs with no backend on this OS; unbuilt subcommands exit 2 naming the item; check.sh --fast green."
risks: ["A second config system beside cli-args-env.json; options.json mirrors its JSON shape exactly."]
status: done
evidence:
- "task/binary-base merged to main a107b5a 2026-10-12 (socialweb-computer-kcp f2d6b8c CLI + dev up|status|down, 7e3e54f readings, 5fb7cb2): cmd/socialweb-computer multi-call CLI over internal/clicfg (flag > env > config > default, options.json mirroring cli-args-env), host-backend required with no default; --help lists the tree marking unbuilt verbs with the item that builds them; unbuilt verbs exit 2 naming it"
- "clicfg red 03:07:51Z (empty options.json) green 03:07:52Z; CLI exit-code tests; merged main go test ./... 66/66 ok, check.sh --fast exit 0 (darwin/windows build+vet green)"
- "deviations: version --json shows (devel) without vcs in a task tree and assets empty until M1-ASSETS; the app-verb contract scan lands with M2-APP-CONTROLLER"
- id: M1-K8S-FORK
milestone: M1
title: "route: in-process - move this module onto kcp's Kubernetes fork and own the vendored kcp source"
depends_on: [M0-KCP-SPIKE]
decision: >
Route: in-process (built only if M0-KCP-SPIKE's confirmed route is in-process; otherwise closed per the
route state rule). Real work, not a side effect (decided by the user 2026-10-07). Vendor kcp v0.33.0
(source tree incl. staging/src/github.com/kcp-dev/*) to third_party/kcp and
github.com/kcp-dev/embeddedetcd at 2cc5c5cce35e to third_party/embeddedetcd, each recorded (upstream,
commit, sha256 per file) in third_party/vendor.json and checked offline like internal/marketnetwork checks
deploy/denokcp; go.mod replaces github.com/kcp-dev/kcp, the five kcp-dev staging modules,
embeddedetcd and every k8s.io module exactly as kcp's go.mod :61-103 does. github.com/kcp-dev/sdk moves
from v0.33.1 to kcp's staging sdk. Every package that fails (M0's list) is fixed in this item; go test
./... and deploy/check.sh --full must be as green as before, including M0-XOS-GATE's darwin and windows
runs. A later kcp upgrade is a re-vendor through one script (scripts/vendor-kcp.sh) and a diff of the
recorded patches, never a hand edit.
files: ["third_party/kcp/, third_party/embeddedetcd/ (generated, recorded)", "third_party/vendor.json", "scripts/vendor-kcp.sh", "go.mod, go.sum", "every package in M0's breakage list"]
test_first: >
None: closed, route not chosen. (Had it been built: internal/thirdparty TestTheVendoredKcpTreeIsExactlyItsRecord,
then go test ./... green on the fork with equal before/after counts.)
done_when: "Closed under the kcp route state rule: the user chose out-of-process on 2026-10-12."
risks:
- "None open. Recorded for a future revisit: the fork's client-go is 1.37.0-alpha (M0 measured no breakage offline; live apply paths against a k3s 1.36 tenant were not measured)."
status: done
evidence:
- "route not chosen: M0-KCP-SPIKE evidence (kcp-spike.md; branch kcp-spike 5f8ef8c, main d6e1881), user decision 2026-10-12 (out-of-process). Nothing built; the module stays on k8s.io v0.36.4 and github.com/kcp-dev/sdk v0.33.1. The spike's fork measurements are the record of this route."
- id: M1-KCP-INPROC
milestone: M1
title: "route: in-process - kcp served from inside socialweb-computer with today's flags"
depends_on: [M1-K8S-FORK]
decision: >
Route: in-process. internal/kcpserver (interface Server {Start(ctx) (Endpoints, error); Ready();
Stop(ctx)} shared with M2-KCP-SLOTS) gets its in-process implementation: the cmd/kcp/kcp.go:95-162
sequence with the options today's entrypoint passes (container-entrypoint.sh:227-234): root directory,
bind address, secure port, WorkspaceMounts=true set on kcp's feature gate (refused if absent;
versions.json kcp invariant), the embedded audit policy, audit log rotation. Readiness is the server's
own readyz. Stop is context cancellation with a bounded drain. One kcp per root directory: an flock on
<root>/.socialweb-computer.lock taken before options complete.
files: ["none (closed, route not chosen)"]
test_first: >
None: closed, route not chosen. Its earlier live test opened a root directory the container-era release
binary wrote; no such state exists on a fresh deploy (decided by the user 2026-10-12), so that test is
void on any route. The WorkspaceMounts gate check moves to M2-KCP-SLOTS' argv builder.
done_when: "Closed under the kcp route state rule: the user chose out-of-process on 2026-10-12."
risks: []
status: done
evidence:
- "route not chosen: M0-KCP-SPIKE evidence (kcp-spike.md route A: in-process kcp ready in 8.9 s at 415 MB RSS; two in one process work), user decision 2026-10-12 (out-of-process, because kcp blue/green cannot be done in-process without overlapping two core processes). Nothing built."
- id: M1-ASSETS
milestone: M1
title: Per-GOOS_GOARCH embedded executables, content-addressed, verified on every start, collected only when unreferenced
depends_on: [M0-SIZE-BUDGET, M0-XOS-GATE, M0-PG-SOURCE]
decision: >
Dev run mode (decided by the user 2026-10-12, folded here 2026-10-12 third fold; details decided by
Claude, user may veto, dev-run-mode.md R2 and 5): the extraction root is an option (asset-cache):
install points it at <state>/runtime/, dev at ONE per-user cache
$XDG_CACHE_HOME/socialweb-computer/runtime/<sha256>/ shared by every dev instance (content-addressed, so
sharing is safe; re-hashed before every use as below), so N worktrees do not extract ~400 MB each. The
dev cache is collected only by `dev gc`, and only while no devsup of this user runs. "No loaded unit
names it" is read through hostsvc.Manager on either backend (systemd-user: ExecStart/Environment over
D-Bus; proc: devsup's unit records). The postgres set is M0-PG-SOURCE's choice.
Decided by the user 2026-10-11: per-OS embedded assets compiled in only for the target OS. Decided by
Claude, user may veto (exploration B6): internal/assets/assets_<goos>_<goarch>.go with
`//go:build release && <goos> && <goarch>` and `//go:embed blobs/<goos>_<goarch>/*.zst` plus its
manifest (one file per platform, the sshai deno_embed_linux_amd64.go pattern); assets_dev.go
(`//go:build !release`) resolves each asset from an explicit path option, never PATH, so cross-OS
builds and vets need no blobs; assets_release_unsupported.go
(`//go:build release && !(linux && (amd64 || arm64))`) is an empty manifest that plans 0011 and 0012
replace. Extraction to <state-dir>/runtime/<sha256>/<name> with temp-file-plus-rename; an existing
extraction is re-hashed before every use (sshai reuses an extracted file with no check,
deno_manager.go:112-115); no PATH or download fallback in a release build. Linux set: firecracker,
pasta, socialweb-vmexec (plan 0007 M2-VM-RUNTIME), k3s, kine, bao, postgres + initdb + pg_dump, external-secrets,
dumbpipe, deno (0008), and kcp (the unpatched release v0.33.0, route decided by the user 2026-10-12).
NOT in the set: guest images (fetched and
verified, plan 0007 M1-GUEST-IMAGE), psql (replaced by pgx, M2-POSTGRES), e2fsprogs, debugfs, cpio,
xz, zstd, ip, socat, yq, kubectl, openssl (deleted by design, M1-HOST-INVENTORY). GC rule: a
runtime/<sha256>/ is removed only when it is not the current or previous binary's set, no loaded unit's
ExecStart or Environment names a path under it (read over D-Bus, M2-HOST-UNITS), and it is older than
the rollback window. Pins live in versions.json.
files: ["internal/assets/ (assets_linux_amd64.go, assets_linux_arm64.go, assets_dev.go, assets_release_unsupported.go, gc.go)", "internal/assets/blobs/ (gitignored, filled by scripts/stage-assets.sh)", "scripts/stage-assets.sh", "internal/versions/versions.json"]
test_first: >
internal/assets TestAnExtractedAssetThatNoLongerMatchesItsPinIsReplacedNotRun,
TestAReleaseBuildWithAMissingAssetRefusesToStart,
TestEveryReleaseEmbedFileNamesExactlyItsPlatformsAssetSet (reads versions.json) and
TestARuntimeDirNamedByALoadedUnitIsNeverCollected (fake unit lister, both backends' listers),
TestTwoDevInstancesShareOneExtractionOfAnAsset and TestDevGcRefusesWhileADevsupOfThisUserRuns; red
against a naive extractor.
done_when: "every inventoried linux asset extracts and verifies; `version --json` lists each with version and sha256; cross-OS gate still green with no blobs present."
risks: ["The state dir, never /tmp (a 24 GiB tmpfs on hat-8, nosuid), holds the runtime."]
status: done
evidence:
- id: M1-HOST-INVENTORY
milestone: M1
title: Everything the container and the bidder host supplied, each given a fate on a bare Linux host
depends_on: []
decision: >
Decided by the user 2026-10-07: no container at runtime; postgres and everything else the container
supplied is embedded and, when it is a long-running server, driven by a CRD. This item writes
docs/host-inventory.yaml: every package and binary of the Dockerfile's controlplane and k8s stages and
of the bidder's runner image, and every executable the Go and TypeScript code execs (the list above
plus each Deno.Command target), each with one fate: embedded server in
its own unit driven by a CRD (postgres -> M2-POSTGRES, the root and bidder vaults -> M2-OPENBAO-SERVER),
embedded server in its own unit started by the core from local state (kcp slots -> M2-KCP-SLOTS; kcp's
own kine and postgres -> M2-KCP-STORE), embedded asset run as a member unit of a cluster's
slice (k3s, kine, the per-cluster bao, external-secrets, dumbpipe; M3-CLUSTER-UNITS), core-binary verb
run as a member unit (the apiserver front, the CCM), embedded VM-unit asset (firecracker, pasta,
socialweb-vmexec), embedded child of the core unit (deno), replaced by Go (socat -> crypto/tls proxy,
yq and kubectl -> kcpapply, openssl -> crypto/x509, psql -> pgx, ip -> netlink in vmexec,
which/unzip/curl -> gone), deleted by design (debugfs, e2fsck, resize2fs, mkfs.ext4, cpio, du, xz,
zstd, chroot, mount, apt-get, go: the guest seed is a separate pure-Go NoCloud cidata drive, the rootfs
is copied as-is and the guest grows its own root, decided by the user 2026-10-12; images are built by
the build-host script on any Linux host, CI shells out to the same script later, decided by the user
2026-10-12 - plan 0007 M1-GUEST-IMAGE, M2-VM-RUNTIME), build-host only (cmd/socialweb-nodeimage
and its tools, run by scripts/build-guest-image.sh), download asset (socialweb-credential x4, socialweb-meshd x2), host file written by
`install` (/etc/rancher/node/password, the k3s invariant; the subuid/subgid lines, M1-INSTALL), or a
stated host prerequisite (Linux kernel, systemd 253+, cgroup v2, linger, /dev/kvm and /dev/net/tun for
bidders; for bidders also the distro's setuid newuidmap and newgidmap, the helpers rootless podman uses,
for the per-VM user namespaces of 0007 M2-VM-RUNTIME - a setuid helper is never shipped as an embedded
asset - and util-linux unshare >= 2.38 only if 0007 M0-VM-USERNS chooses it for the userns stage;
M2-HOST-CONTRACT). A test holds the inventory complete.
files: ["docs/host-inventory.yaml", "internal/versions (inventory test)"]
test_first: >
internal/versions TestEveryExecutableTheCodeExecsHasAFateInTheHostInventory (scans exec.Command,
exec.LookPath and Deno.Command call sites); red before the file exists.
done_when: "inventory committed, test green; no fate is 'from the host's package manager' except the stated host prerequisites (systemd, newuidmap/newgidmap), each checked by check-host."
risks: ["The relocatable PostgreSQL 18 build is chosen by M0-PG-SOURCE (moved there 2026-10-12, third fold) and recorded here as an embedded asset; no container-era data dir has to open with it (fresh deploy, decided by the user 2026-10-12)."]
status: done
evidence:
- "socialweb-computer-kcp df10e7c (task/inventory-cleanup), merged to main 77cb5cf 2026-10-12: docs/host-inventory.yaml, 114 entries + 27 resolved call sites; fates removed-legacy 25, build-host 23, replaced-by-go 18, embedded-server 11, host-prerequisite 11, linked 9, host-file 6, download-asset 5, embedded-child 3, embedded-tool 3"
- "TestEveryExecutableTheCodeExecsHasAFateInTheHostInventory red (no file) then green; scans exec call sites in non-test Go plus Deno.Command in this repo, atproto-market and hono-compute-provider, and both Dockerfiles; stale entries fail it"
- "host prerequisites beyond systemd/newuidmap/newgidmap are kernel features or requester-side tools (/dev/kvm, /dev/net/tun, cgroup v2; the requester's own ssh client; util-linux unshare only if M0-VM-USERNS picks it); each is checked by M2-HOST-CONTRACT's check-host, decided by Claude, user may veto. Also decided by Claude: ssh-keygen replaced by Go key generation (keys in the app's OpenBao), ESO bundle embedded, docker/wsl/Apple container CLI and websocat removed-legacy (2026-10-12 strip decision)"
- "go test ./... 58/58 ok and deploy/check.sh --fast exit 0 on merged main"
- id: M1-STORE-CLEANUP
milestone: M1
title: Remove the rustfs/t4 store design's leftovers; the tenant store is kine over the cluster's own postgres
depends_on: []
decision: >
Decided by Claude, user may veto (2026-10-12 fold): the code stores every tenant cluster as kine over
that cluster's own postgres (k3s dials kine; internal/clusterprovider/stack.go:39-43 records that k3s
refuses the S3 form and that a t4 endpoint makes k3s start its own kine), and ARCHITECTURE section 4 now
says so with the rustfs/t4 design marked historical. The leftovers that still name it are deleted:
internal/vaultlayout PathCentralStore, PolicyCentralStore, ClassCentralStore and every entry using them
(the policy, the root consumer's policy list, the secret-class table); cmd/socialweb-vault's
SOCIALWEB_CENTRAL_STORE_{ENDPOINT,BUCKET,ACCESS_KEY,SECRET_KEY} options, centralStoreValues and the
ClassCentralStore branch of apply.go; the rustfs wording in internal/portallocator/portallocator.go:200;
the rustfs and t4 entries of internal/versions/versions.json and the 'bucket' wording of its kine
invariant. stack.go's ErrNoStore text stays (it is the measurement that rules the S3 form out).
ARCHITECTURE sections 4, 6 (the Kine row) and 7 (the root vault's central-store paragraph) were
corrected by the 2026-10-12 fold and are re-read in this change; section 5's "Not building ... a
registry plus volume-commit machinery" line, whose reason was S3-backed PVs on the rustfs, gets a reason
that holds or is struck.
files: ["internal/vaultlayout/vaultlayout.go", "cmd/socialweb-vault (config.go, sources.go, apply.go)", "internal/portallocator/portallocator.go", "internal/versions/versions.json", "docs/ARCHITECTURE.md sections 5, 6, 7"]
test_first: >
internal/versions TestNoRustfsOrT4NameOutsideTheStoreMeasurement (scans non-test Go and versions.json;
allows only stack.go's ErrNoStore), red on today's tree; vaultlayout's existing Validate test stays green
with the class removed.
done_when: "the scan test green; `socialweb-vault` help lists no central-store option; check.sh --fast green."
risks: ["A root vault that already holds secret/data/socialweb/store/rustfs keeps it until M3-BAO-SYNC's report lists it; no container-era vault is carried over (fresh deploy), so none is expected."]
status: done
evidence:
- "socialweb-computer-kcp 8a43224 (task/inventory-cleanup), merged to main 77cb5cf 2026-10-12: vaultlayout central-store path/policy/class and the cluster-provider role's use removed; socialweb-vault SOCIALWEB_CENTRAL_STORE_* options, apply branch, centralStoreValues, externalStoreNamed, envClusterStore and their 4 tests removed; versions.json rustfs/t4 entries and kine bucket wording removed; portallocator wording fixed (the 9100-9103 reservation stays, origin untraced)"
- "TestNoRustfsOrT4NameOutsideTheStoreMeasurement and a kine-pin test red (~40 hits) then green; ARCHITECTURE sections 4 and 15 updated; section 5's registry/volume-commit 'not building' line struck (its only reason was the never-built store)"
- "go test ./... 58/58 ok and deploy/check.sh --fast exit 0 on merged main"
- id: M1-PORT-LEASE-INSTANCE
milestone: M1
title: Port and subnet leases keyed by application instance, in one per-user lease file in dev, for the few ports the port-0 rule allows to be stable
depends_on: []
decision: >
Principle (user 2026-10-12): leases are the exception to principle.port0_rule, not the way ports are
found; every lease names the application instance that owns it and is recorded in that instance's kcp.
Decided by the user 2026-10-12: dev run mode now, any number of application instances on one machine
(dev-run-mode.md R2, 4.5).
Decided by Claude, user may veto: portallocator's lease key becomes (instance, cluster, purpose) - the
Lease.Instance field exists (portallocator.go:106) and the key ignores it (:145-148); prod keeps one
lease file per account with instance "" (unchanged behaviour); dev and tests open ONE per-user lease file
$XDG_STATE_HOME/socialweb-computer/ports/leases.json under its existing flock (:227-243), so instances
see each other's leases and the probe-then-bind race between two lease files is gone; the dev carve
does not overlap the live-test blocks of internal/liveports (liveports.go:33-56). subnetallocator gets
the same key change. Only ports that must stay stable inside an instance across restarts are leased
(a cluster's front port, link UDP ports); everything else binds 0 and reports into its kcp object's
status (principle.port0_rule, M2-KCP-PORT0, M1-NO-FIXED-PORTS).
files: ["internal/portallocator/ (lease key, lease-file path option)", "internal/subnetallocator/", "internal/liveports (carve boundary test)"]
test_first: >
internal/portallocator TestALeaseIsKeyedByInstance (two instances, same cluster and purpose, distinct
ports), TestTwoInstancesSharingOneLeaseFileNeverLeaseTheSamePort (two processes, one file, concurrent),
TestTheDevCarveDoesNotOverlapTheLiveTestBlocks, TestAProdLeaseFileWithoutInstancesReadsUnchanged; red
first.
done_when: "tests green; check.sh --fast green."
risks: ["A lease file shared by every instance of a user is a single point of contention; flock hold times are recorded under 50 concurrent leases."]
status: done
evidence:
- id: M1-NO-FIXED-PORTS
milestone: M1
title: "Check: a test fails on any fixed port in a manifest or default behind the edge - a ratchet that only shrinks"
depends_on: []
decision: >
Decided by the user 2026-10-12: principle.port0_rule. Decided by Claude, user may veto: one check test
in check.sh --fast scans this repo for a TCP or UDP port that is a literal other than 0 behind the edge
- every YAML under deploy/ (the vendored market network and deno-kcp manifests included), deploy/**/*.sh, every
config.json under deploy/, and the option defaults (internal/clicfg/options.json once M1-CLI lands) -
in args (--port N, --*-port N), env (PORT, *_PORT), loopback URLs (127.0.0.1:N, localhost:N,
*.localhost:N) and defaults. A DenoPod or HostProcess manifest that names no port inherits its CLI's
default, so it must pass its port explicitly (0). Two files sit beside the test:
deploy/edge-ports.yaml, the exceptions (caddy 443, the gateway's kcp front 6443 in its prod config, the
door's 8788), each with its reason, extended only by a user decision; and deploy/fixed-ports-ratchet.yaml,
today's violations as file + port (the market network's 2583-2589, deno-kcp kcp 6443 and kine 23791 and the rest of
port-audit.md sections 1 and 5). The test fails on a fixed port in neither file AND on a ratchet entry
that no longer matches the tree, so the ratchet only shrinks and every fix deletes its line; it can land
in wave 1 and gate everything after. Once M2-TEMPLATE lands it also scans every set rendered with every
profile (quick, market-network, prodlike, production), because a template value can render a literal.
files: ["test/no_fixed_ports_test.go", "deploy/edge-ports.yaml", "deploy/fixed-ports-ratchet.yaml", "deploy/check.sh (--fast step)"]
test_first: >
TestNoFixedPortBehindTheEdge (red first on a fixture tree with one literal beyond the ratchet, green on
today's tree with the ratchet), TestTheRatchetListsOnlyViolationsThatStillExist,
TestEveryEdgePortHasAReason, TestAServiceManifestWithoutAnExplicitPortIsAViolation.
done_when: "in check.sh --fast and green; the ratchet's starting entry count recorded; 0008 M1-MARKET-NETWORK-PARALLEL, M4-MARKET-NETWORK-PORT and M2-KCP-PORT0 each name the ratchet lines they delete in their evidence; the ratchet is empty when they are done."
risks: ["A pattern scan misses a port built at runtime (string concatenation); the rendered-set scan after M2-TEMPLATE and each item's live two-instance test cover what the scan cannot."]
status: done
evidence:
- id: M1-DEV-INSTANCE
milestone: M1
title: "`socialweb-computer dev up|down|status` per worktree - instance identity, XDG state keyed by instance, devsup per instance, port 0"
depends_on: [M1-CLI, M2-HOST-BACKEND-PROC, M1-PORT-LEASE-INSTANCE]
decision: >
Principle (user 2026-10-12): a dev instance is a composition of application instances (its profile
names which: the market-network profile = the market network's atproto network, bidders and requesters), each with
its own kcp, manifest set and profile; the host is only where it runs. Decided by the user 2026-10-12:
the dev run mode is kcp-native now; no root, no install, no systemd,
parallel instances per worktree; the dev defaults accepted (host-backend required, no default; dev
passes proc; AppArmor in dev report-only; one uid for every role in dev). Decided by Claude, user may
veto (dev-run-mode.md R1-R4, R8, 4.1, 4.7): instance id = <worktree basename, 16 chars>-<first 8 hex of
sha256(realpath(worktree))>, or --instance <name>; state $XDG_STATE_HOME/socialweb-computer/dev/<id>/
(default ~/.local/state), sockets $XDG_RUNTIME_DIR/swc/<id8>/ (0700, every socket path < 108 bytes,
checked), a gitignored symlink <worktree>/.swc-dev -> <state> for discovery, never state inside the
worktree (git clean, worktree remove and the 108-byte limit). `dev up [--profile quick|market network|prodlike]
[--instance x] [--fg]`: resolve instance -> flock <state>/instance.lock -> start or adopt the
instance's devsup (`internal devsup`, setsid so it survives the terminal; --fg keeps it in the
foreground and Ctrl-C is `dev down`) -> place the bootstrap layer (M2-BOOTSTRAP-LAYER; before it lands,
`dev up` places only what the caller names, stand-ins in tests) -> wait readiness -> print `dev env`.
`dev down [--keep-state]` stops everything in reverse order within the largest stop timeout. `dev
status` and `dev ls` read devsup or, with devsup down, the on-disk unit records. `dev logs <unit>`,
`dev env` (KUBECONFIG, base workspace, ports, for shells and TS tools), `dev restart core [--binary
path]` (rebuild loop: the core is replaced, every other unit adopted), `dev check` (R8 host
prerequisites - /dev/kvm rw, unprivileged userns for pasta, the AppArmor userns restriction on Ubuntu
24.04+, /dev/net/tun, newuidmap/newgidmap, logind KillUserProcesses - reported, never fixed). Every
port is 0 except leases from M1-PORT-LEASE-INSTANCE. `dev` refuses uid 0; `install` refuses
host-backend=proc (M1-INSTALL). Survival (user 2026-10-12, devsup runs the dev tree): units survive a
core restart, crash or rebuild and are adopted; every unit dies with devsup (`dev down`, crash, kill
-9, --die-with test pid); a host reboot is Lost(HostRebooted) as in prod. Dev adds no provisioning
path: guests come up only through runComputeContract against the instance's bidder (RFP spine).
Fifth fold 2026-10-12 (decided by the user: STRICT - every application instance has its own kcp,
inside a market network too; the research's nested-workspace market network is rejected): the dev instance's own kcp is ONE
application's kcp. `dev up --profile market-network` makes the instance a MarketNetwork application
(M2-APP-CONTROLLER) that composes child applications, each with its OWN kcp whose bootstrap layer is
HostProcess objects in the instance's kcp: `--bidders N` (default 1) MarketBidder children, an
optional Gateway child (`--with-gateway`), and requester invocations added by `run request --market-network
<instance>` (0008 M4-RUN-REQUEST); the network copies its status URLs onto each child's root object
and no child reads another's kcp. Until M2-APP-CONTROLLER lands, `dev up --profile market-network` exits 2 naming
it (today's market network is 0008 M1-MARKET-NETWORK-PARALLEL). The market network composition tests are M2-APP-CONTROLLER's
(TestEveryChildApplicationInAMarketNetworkHasItsOwnKcp, TestANetworkBidderLearnsThePlcOnlyFromItsNetworkValue),
because this item lands before that controller exists.
files: ["cmd/socialweb-computer dev", "internal/devinstance/ (id, layout, lock, env)", "internal/devstack (Start(t, profile) test helper)", ".gitignore (.swc-dev)"]
test_first: >
internal/devinstance TestTwoWorktreesGetDistinctInstanceIDsAndShortSocketPaths,
TestTheSocketPathOfTheLongestWorktreeIsUnder108Bytes, TestDevRefusesRoot,
TestASecondDevUpOfOneInstanceAdoptsItsDevsup; behind livegate (proc backend, stand-ins only, no
systemd): TestTwoInstancesOnOneHostShareNothing (two devstacks in one test: distinct ports, dirs,
sockets; stopping one leaves the other's InvocationIDs unchanged), TestDevDownLeavesNoProcess,
TestACoreRestartUnderDevAdoptsEveryUnit. Red first.
done_when: "tests green; two worktrees on hat-8 each ran dev up / status / down in parallel with readings (ports, socket paths, unit lists) recorded."
risks:
- "logind KillUserProcesses=yes on a host kills a detached devsup at logout; dev check reports it and --fg is the documented alternative."
- "Each application instance's kcp costs ~0.4-0.5 GB and ~6-9 s cold start (kcp-spike.md); measured per profile once M2-BOOTSTRAP-LAYER lands."
status: done
evidence:
- id: M1-INSTALL
milestone: M1
title: "`socialweb-computer install bidder|gateway` - the one explicit, idempotent root step, with `uninstall`; content-addressed upgrade and rollback of the core binary"
depends_on: [M1-ASSETS, M1-CLI]
decision: >
Decided by the user 2026-10-11: VMs survive a core upgrade; the gateway and bidders each live in the
user manager of a lingering account; ONE shared lingering account (swc-bidder) for all bidder handles.
Decided by the user 2026-10-12: production setup is ONE explicit command, `socialweb-computer install
bidder|gateway` (the root step: account, subuid range, kvm group, linger, delegation drop-in, AppArmor
profile, node password, every user unit); Unix accounts are never created on demand - the only
account is the one this step creates; systemd is for production installs only (the dev run mode,
M1-DEV-INSTANCE, needs nothing from this item; `install` refuses host-backend=proc and writes
`host-backend: systemd-user` into the account's config.json, the only place a production default is
set, decided by the user 2026-10-12). Decided by Claude, user may veto (dev-run-mode.md 5.4): `install
--dry-run` prints the computed step list and changes nothing; an account that exists but has no
install record is refused unless --adopt-account, so a typo never takes over a human's account; if
user@<uid> already runs without kvm, install refuses and names the restart (it would stop every VM),
never restarting it itself. Decided by Claude, user may veto (exploration A-x-upgrade,
linux-proofs E6): (1) Layout under the
account's state dir: bin/<sha256>/socialweb-computer, bin/current and bin/previous symlinks,
runtime/<sha256>/ (M1-ASSETS). A unit's ExecStart names bin/current only for the core unit; anything
the core starts (cluster member units) names the resolved bin/<sha256>/ path
(filepath.EvalSymlinks(os.Executable()) once at start), so no rename-over ever changes code inside a
running chain. (2) `upgrade --binary <path>`: verify the manifest and sha256 (signature later), copy to
bin/<sha>/, run `<new> self-check` (version --json, asset extract + verify, check-host), refuse on any
failure, retarget current atomically (symlink rename), record previous, then `systemctl --user
restart` the core unit (gateway or bidder@<handle>). (3) `rollback` retargets current to previous and
restarts. (4) GC of bin/<sha>/ follows M1-ASSETS' rule (never current, previous, or a path a loaded
unit names). (5) `install gateway|bidder [--handle <h>]...` is the ONLY root step. It is idempotent:
every step reads the current host state, compares it with the desired state and writes only on a
difference, so a second run writes nothing, runs no side-effecting command and restarts nothing. It
creates, in this order (each listed in a root-owned install record
/var/lib/socialweb-computer/install-<account>.json as "created" or "found present", so uninstall
removes only what install created):
(a) the account - socialweb for the gateway, swc-bidder for bidders (one shared account for every
handle) - a system user with no password and no login shell, home = its state dir; no other code path
creates an account (run bidder, serve gateway kcp and the controllers never call useradd and never
escalate);
(b) bidders: ONE contiguous subordinate range for the account in BOTH /etc/subuid and /etc/subgid,
65536 ids per VM slot x 256 slots (16,777,216 ids; --vm-slots), placed above every existing entry of
either file, from which 0007 M2-VM-RUNTIME maps each VM unit into its own 65536-id sub-range
(decided by the user 2026-10-12: per-VM subuid user namespaces);
(c) bidders: membership of group kvm, BEFORE linger and before user@<uid> first starts (supplementary
groups are fixed when the user manager starts; adding kvm later needs a user@<uid> restart, which
kills every VM);
(d) /etc/systemd/system/user@<uid>.service.d/socialweb-delegate.conf `Delegate=cpu memory pids`
(written on every host - stock systemd delegates memory+pids only, linux-proofs E0 - so CPUQuota is
enforced) and a system `systemctl daemon-reload`, before linger;
(e) where AppArmor restricts unprivileged user namespaces (Ubuntu 24.04+:
kernel.apparmor_restrict_unprivileged_userns=1), /etc/apparmor.d/socialweb-pasta granting userns to
<state>/runtime/*/pasta, loaded with apparmor_parser -r (decided by the user 2026-10-11);
(f) gateway: /etc/rancher/node/password 0600 owned by the account (the k3s invariant);
(g) the state dir layout and the binary at bin/<sha256>/socialweb-computer with bin/current (and
bin/previous once an upgrade happened), owned by the account;
(h) the install-time USER unit files under the account's ~/.config/systemd/user: socialweb.slice (the
top slice every other unit of the account hangs below by name, outside the app.slice Fedora's
systemd-oomd watches) and the core unit - gateway: socialweb-computer-gateway.service
(ExecStart=<state>/bin/current/socialweb-computer serve gateway kcp, Restart=always, default
KillMode=control-group; finalised by M3-HOST-INSTALL); bidder: the template
socialweb-computer-bidder@.service (0007 M5-RUN-BIDDER) with one enabled instance per --handle (a
re-run with another handle adds that instance and changes nothing else); plus the BOOTSTRAP-LAYER unit
files (decided by the user 2026-10-12: kcp, its kine and its store run as installed systemd units in
production; the table and the files come from M2-BOOTSTRAP-LAYER, which extends this step) - gateway:
socialweb-kcp-gw-pg.service, socialweb-kcp-gw-kine@.service and socialweb-kcp-gw@.service (instances
a and b), socialweb-kcp-gw-front.service; bidder, per handle: socialweb-kcp-<hid>-kine@.service (one
instance a, kine on sqlite), socialweb-kcp-<hid>@.service (one instance a) and
socialweb-kcp-<hid>-front.service (decided by the user 2026-10-12: a bidder's kcp is one slot on
kine-sqlite);
(i) `loginctl enable-linger <account>` (user@<uid> first starts here, after (b)-(d));
(j) as the account (`systemctl --user -M <account>@`): daemon-reload, enable --now the core unit(s);
(k) check-host as the account (M2-HOST-CONTRACT); a failure exits non-zero naming the check, and a
re-run after the fix completes from there.
Every OTHER unit the gateway or a bidder runs is a RUNTIME transient unit placed by the in-binary
supervisor from a HostProcess object (M2-HOSTPROCESS-KIND, decided by the user 2026-10-12) on the
systemd-user backend over the user manager's private socket (M2-HOST-UNITS), adopted by name, never
written as a file, gone at reboot or a user-manager restart and re-placed from its object on the core's
next start; install creates none of them, uninstall stops all of them by the socialweb- prefix.
Gateway: socialweb-bao-root.service (M2-OPENBAO-SERVER); per cluster socialweb-pg-<key>.service
(M2-POSTGRES) and socialweb-cluster-<key>.slice with its socialweb-c-<key>-<member>.service members
(M3-CLUSTER-UNITS). Bidder, per handle (hid = first 10 hex of sha256(handle)): socialweb-bao-<hid>.service
(0007 M4-BIDDER-VAULT); socialweb-b-<hid>.slice holding socialweb-vm-<hid>-<name>.service per VM (0007
M2-VM-RUNTIME, a HostProcess child of its ComputeVM). The bootstrap-layer units above are installed
files, in socialweb-kcp-<owner>.slice, started at user-manager start (so kcp is back after a reboot
before the core) and started or stopped by instance by the core during a roll. Both: check-host's
socialweb-probe-<rand>.service, stopped when the check ends. Not units at all: deno processes and the
TLS proxy, plain children of the core unit (0008 M2-DENOKCP-INPROC). (6) `uninstall gateway|bidder
[--purge] [--force]` (root): refuses while a VM unit of the account is active unless --force (stopping
one makes its VM Lost); stops and disables the core unit(s); stops every runtime unit and slice of the
account by prefix and waits for inactive; removes the unit files; disables linger; removes the drop-in
(daemon-reload), the AppArmor profile (apparmor_parser -R), the subuid/subgid lines, the kvm membership
and the node password where the install record says install created them; keeps the state dir and the
account (the durable state: vault, kcp store, keys) unless --purge, which removes both.
files: ["internal/install/ (install_linux.go, uninstall_linux.go, record.go, subids.go; windows/darwin stubs ErrUnsupported)", "cmd/socialweb-computer install|uninstall|upgrade|rollback", "deploy/host/apparmor/socialweb-pasta (profile template)", "deploy/host/user@.service.d/socialweb-delegate.conf", "deploy/host/socialweb.slice", "deploy/host/socialweb-computer-gateway.service (first written here, finalised by M3-HOST-INSTALL)", "deploy/bidder/socialweb-computer-bidder@.service (first written here, finalised by 0007 M5-RUN-BIDDER)"]
test_first: >
internal/install (fake root filesystem + fake command runner): TestInstallTwiceChangesNothingTheSecondTime
(the second run records zero writes and zero side-effecting commands, for gateway and bidder),
TestUninstallPurgeRestoresEveryHostFileInstallTouched (passwd, group, subuid, subgid, the drop-in, the
AppArmor profile, the linger file, /etc/rancher/node/password byte-identical to before install),
TestUninstallKeepsWhatInstallFoundPresent (a pre-existing kvm membership and subuid entry survive),
TestUninstallRefusesWhileAVMUnitIsActive, TestInstallBidderAddsKvmAndTheDelegationDropInBeforeEnablingLinger
(the runner records the order), TestTheSubuidRangeIsAboveEveryExistingEntryOfBothFiles,
TestEveryInstalledUnitFileVerifies (`systemd-analyze --user verify` clean on socialweb.slice and the core
units, `systemd-analyze verify` on the drop-in), TestNoCodePathOutsideInstallCreatesAnAccount (scans
non-test Go for useradd/adduser/groupadd/usermod execs outside internal/install),
TestEveryUnitPrefixTheCodeStartsIsEnumeratedForUninstall (every unit-name prefix hostsvc.Start is called
with is in install's runtime-unit table), TestUpgradeRefusesABinaryWhoseManifestDoesNotMatch,
TestUpgradeRefusesABinaryWhoseSelfCheckFails, TestRollbackRetargetsCurrentToPrevious,
TestABinaryDirNamedByALoadedUnitIsNeverCollected, TestInstallRefusesTheProcBackend,
TestInstallWritesSystemdUserAsTheHostBackend, TestInstallRefusesAnExistingAccountItDidNotCreate,
TestInstallDryRunChangesNothing; red first.
done_when: "tests green; on hat-8 a throwaway gateway account and a throwaway bidder account each installed twice (second run: no change, reading recorded), upgraded twice, rolled back, then `uninstall --purge` (diff of the recorded host files against the pre-install snapshot empty), with `ls bin/`, the unit's ExecStart and `systemctl --user list-units 'socialweb-*'` readings recorded."
risks:
- "The AppArmor profile must match the runtime path glob; check-host (M2-HOST-CONTRACT) verifies it is loaded and that the embedded pasta creates a user namespace."
- "A subuid range another tool (podman, lxd) later claims overlaps ours; check-host re-reads both files and refuses an overlap by name."
- "uninstall --purge deletes the bidder's vault and kcp store; it is the only command that does, and it prints what it will remove before acting."
status: done
evidence:
- id: M2-HOST-UNITS
milestone: M2
title: "The systemd-user placement backend - long-lived children as transient SERVICES in the account's user manager, adopted never re-created"
depends_on: [M1-ASSETS]
decision: >
Reconciled 2026-10-12 (third fold, the user's kcp-native dev run mode): this done item IS the
systemd-user backend of hostsvc.Manager, the production placement; its evidence stands. The proc
backend for dev and tests is M2-HOST-BACKEND-PROC, which also adds Open(Config) with a required
Backend, Capabilities, Watch, StopSlice, UnitSpec.Restart/PrivateHostname/FDStore and Identity.Manager
to BOTH backends; one contract suite runs against both (M2-HOST-CONFORMANCE, which re-homes this
item's live tests); callers place units only through the HostProcess supervisor (M2-HOSTPROCESS-KIND)
or the bootstrap layer (M2-BOOTSTRAP-LAYER). The text below is the item as built.
Decided by the user 2026-10-07: the core binary and its deno processes are upgradable while VMs keep
running. Decided by the user 2026-10-11: the gateway and bidders use the USER manager of a lingering
account (the gateway too; cluster groups are transient services there); only cpu, memory and pids
delegation is relied on; minimum systemd 253. Decided by Claude, user may veto, from linux-proofs E1-E3:
internal/hostsvc (interface Manager {Start, Inspect, Stop, ResetFailed, List(prefix), Close};
UnitSpec, UnitStatus; ErrUnsupported, ErrUnreachable) with systemd_linux.go over
github.com/coreos/go-systemd/v22/dbus (pure Go, CGO_ENABLED=0; sdjournal is cgo and not used) and
unsupported_other.go (`//go:build !linux`) returning ErrUnsupported (plan 0011 adds scm_windows.go, plan
0012 launchd_darwin.go). Each long-lived child that must outlive the core (a VM, plan 0007; a postgres;
an OpenBao; each member of a cluster's slice, M3-CLUSTER-UNITS; kcp on the out-of-process route) is a transient SERVICE
started by StartTransientUnit - never a scope, because a scope adopts a pid the starter forked and makes
our code its parent. Rules: connect over $XDG_RUNTIME_DIR/systemd/private with a custom dialer (EXTERNAL
auth, no Hello; go-systemd has no user private-socket constructor); every call carries a context with a
deadline; only dial, auth and timeout errors are ErrUnreachable (Unknown); ExecStart is set with
PropExecStart(argv, false) (go-systemd's bool is ignore-failure: true makes a crash report success and
the unit vanish); CollectMode=inactive so Result, ExecMainCode and ExecMainStatus stay readable until
the owner records them and calls ResetFailedUnit; a unit whose name already exists is ADOPTED, never
re-created (StartTransientUnit fails "already loaded"); ownership is the unit-name prefix
(socialweb-<kind>-<owner>-<name>), and List never returns another prefix. Limits offered: MemoryMax,
MemorySwapMax (0 whenever MemoryMax is set), OOMPolicy, TasksMax, CPUQuota, CPUWeight; NOT offered:
AllowedCPUs (no cpuset delegated, silently unenforced), IOWeight (no effect without BFQ/io.cost, and
ManagedOOMPreference resets it on 259.9). Units go in a socialweb slice of their own, never app.slice
(Fedora's systemd-oomd watches the user app.slice with ManagedOOMMemoryPressure=kill). Adoption identity
read FROM systemd, never from a file the child writes: unit name, InvocationID, MainPID + start ticks,
boot_id, checked pidfd_open -> stat ticks -> pidfd_send_signal(0) -> GetUnitByPIDFD. Children that must
die with the core (deno, the TLS proxy) stay plain children in the core unit's cgroup, which systemd
kills on any stop, restart or crash of the core (default KillMode=control-group).
files: ["internal/hostsvc/ (hostsvc.go, busdial_linux.go, systemd_linux.go, identity_linux.go, unsupported_other.go)", "internal/hostos (ProcIdentity, BootID)", "go.mod (go-systemd, godbus)"]
test_first: >
Offline (fake bus): TestOnlyDialAuthAndTimeoutAreUnreachable, TestAnExistingUnitNameIsAdoptedNotRestarted,
TestEveryCallCarriesADeadline, TestExecStartIsNeverIgnoreFailure. Behind livegate on hat-8 under the real
user manager, with stand-in executables (sleep/a Go stand-in, never a guest):
TestAUnitSurvivesSystemctlRestartStopAndKill9OfItsStarter, TestAUnitSurvivesDaemonReloadAndDaemonReexec,
TestAFailedUnitsResultIsReadableUntilReset (kill -9 the stand-in: failed/signal read, then reset),
TestMemoryMaxWithSwapZeroIsAnOomKillNotSwap, TestUnitsOfAnotherPrefixAreNeverListed. Red first.
done_when: "green on hat-8; `systemctl --user list-units 'socialweb-*'` readings before and after each restart/reexec recorded with UTC times."
risks:
- "Transient units vanish at reboot and when the user manager stops (linger removed, `loginctl terminate-user`, user@<uid> restart); each is treated as a reboot - every child Lost - and documented as an operator action of that weight."
- "Whether systemd-oomd acts under user@<uid>.service outside app.slice is not stress-tested; check-host records which cgroups oomd monitors."
status: done
evidence:
- "started before M1-ASSETS with the user's approval (2026-10-12): the seam and backend need no embedded asset"
- "branch host-units: 07209da (internal/hostsvc Manager {Start, Inspect, Stop, ResetFailed, List, Identify, Check, Close}; go-systemd v22.7.0 + godbus v5.1.0 from the offline module cache), c6698f9 (live tests); merged to main 2026-10-12"
- "offline fake-bus tests red against a stub 2026-10-12T00:46:20Z, green 00:47:15Z: the four named tests, a 15-row verdict table, a 300ms deadline dial, the 252-vs-253 version refusal"
- "live on hat-8 00:49:35-00:49:51Z, stand-ins only (sleep), units socialweb-test-<rand>-*: child InvocationID+pid unchanged across systemctl --user restart, kill -9 and stop of its starter (adopted, list-units read each step); survived daemon-reload and daemon-reexec; kill -9 reads failed/signal/2/9 until reset; MemoryMax=64M with 256MiB alloc is an oom-kill with memory.swap.max=0; prefixes never cross-listed; a recorded pid systemd placed in another unit reads Lost and is stopped"
- "gaps: live tests were written after the implementation (only the mismatch test went red); this host has Linger=no so the runs used the login session's user manager, not a lingering account (proved again under M2-HOST-CONTRACT/M5); a SIGKILL stop leaves the unit failed/signal until ResetFailed, which 0007 M2-CONTROLLER must call after every stop it issues"
- id: M2-HOST-BACKEND-PROC
milestone: M2
title: "The proc placement backend - devsup, a per-instance parent of every unit, for dev and tests"
depends_on: [M2-HOST-UNITS, M2-HOST-CONFORMANCE]
decision: >
Decided by the user 2026-10-12: dev run mode now; placement backends systemd-user (production installs)
and proc (dev: a plain supervised process tree); in dev the core's devsup runs the bootstrap layer
before kcp exists; the host-backend option is required with no default. Decided by Claude, user may
veto (dev-run-mode.md 3(a2), 4.6; its measurement section 6): `socialweb-computer internal devsup
--state <state>/sup --socket <rt>/sup.sock [--die-with <pid>]` is the real parent of every unit (reads
each exit by wait4, so Result/ExecMainCode/ExecMainStatus, CollectMode=inactive, ResetFailed and
Restart=on-failure|always behave like systemd's), sets PR_SET_CHILD_SUBREAPER (a descendant that leaves
its unit's process group is reparented to devsup and attributed by the inherited SWC_UNIT=<name>; killed
if its unit is not active - the KillMode=control-group emulation), starts each unit in its own process
group with Pdeathsig SIGKILL from a locked OS thread, stdout/stderr to the Output files, Env, WorkDir
and a fresh 128-bit InvocationID. Stop = KillSignal to -pgid and every attributed descendant (snapshot
from /proc/<pid>/task/*/children first), wait StopTimeout, then SIGKILL the same set. Identity/Check use
the existing pin sequence; GetUnitByPIDFD is replaced by "pid in the unit's pgid or attributed to it",
answered by devsup. Slice is a label and StopSlice stops nested slices by dash prefix; After/Wants are
validated and advisory; Watch streams unit state changes; an fd store holds a listening fd a unit
hands it and passes it back with LISTEN_FDS (Capabilities().FDStore); PrivateHostname is the
`internal ns-exec` one-shot (user+UTS namespace, sethostname, optional bind of <state>/rancher over
/etc/rancher, execve; M3-CP-SLOTS). State <state>/sup/units/<name>.json {spec, InvocationID, pid, pgid,
leader ticks, bootID, devsup instance, last Result}; after a devsup crash the next devsup sweeps
leftovers by the safe pgid rule (signal -pgid only when the leader pid has the recorded ticks, or no
process has pid == pgid but the group still has members) before accepting calls. Client:
internal/hostsvc/proc_linux.go over the socket. Interface changes on hostsvc (both backends): Open(ctx,
Config{Backend, UnitPrefix, SystemdPrivateSocket, ProcSocket}) refuses an empty Backend (no default,
no probing); Capabilities {Backend, EnforcesLimits, OOMKillReported, FDStore, PrivateHostname, Journal};
Watch(prefix); StopSlice(slice); UnitSpec.Restart, PrivateHostname, FDStore; Identity.Manager (systemd:
the user manager's InvocationID; proc: devsup's instance id) so a manager restart is told apart from a
unit restart; every status records runtime.backend, and a unit placed by one backend is never adopted
by the other (Lost(IdentityMismatch) naming both). Not enforced on proc and declared by
Capabilities().EnforcesLimits=false: MemoryMax, MemorySwapMax, TasksMax, CPUQuota, CPUWeight, OOMPolicy
(a VM's memory stays capped by firecracker's machine-config). Cross-OS: proc_linux.go first; stubs
return ErrUnsupported (Windows Job Objects, darwin kqueue NOTE_EXIT are plans 0011/0012).
files: ["internal/devsup/ (server)", "internal/hostsvc/proc_linux.go (client)", "internal/hostsvc/hostsvc.go (Backend, Capabilities, Config, Open, Watch, StopSlice, Restart, PrivateHostname, FDStore, Identity.Manager)", "internal/hostsvc/systemd_linux.go (the same additions on systemd-user)", "cmd/socialweb-computer internal devsup, internal ns-exec"]
test_first: >
The proc half of M2-HOST-CONFORMANCE red against a stub, plus
TestADevsupCrashLeavesNoProcessAndTheNextDevsupSweepsByLeaderTicks,
TestAPgidWhoseLeaderPidWasReusedIsNeverSignalled, TestDieWithStopsEverythingWhenTheWatchedPidExits,
TestAnOrphanReparentedToDevsupIsKilledWhenItsUnitIsInactive, TestOpenRefusesAnEmptyBackend,
TestAUnitPlacedByOneBackendIsLostOnTheOther. Stand-ins only, no root, no systemd.
done_when: "the proc half of the contract suite green in check.sh --fast; readings of a devsup crash and sweep recorded."
risks:
- "devsup is a resident process of ours that is pasta's parent in dev; it is outside every unit's process group (as systemd is outside a unit's cgroup), so the VM tree audit stays meaningful (M2-VM-TREE-AUDIT is made backend-aware)."
- "Pdeathsig fires when the creating THREAD exits; devsup keeps the locked-thread trick of processrunner.go:138-147."
status: done
evidence:
- "task/binary-base 6078bc8 merged to main a107b5a 2026-10-12: internal/hostsvc proc backend + `internal devsup` (per-instance supervisor over a unix socket under $XDG_RUNTIME_DIR/swc/<id8>/, real parent of every unit, adoption by pid + start ticks across a core restart, everything dies with devsup or dev down)"
- "proc conformance: 12/15 red against a stub devsup 03:00:50Z, green 14 + 1 skipped (limits: proc does not enforce them, declared in Capabilities) 03:02:30Z, stable x5; wired into check.sh --fast (exit 0 on merged main)"
- "live hat-8 03:20:42Z: two instances (kine + one kcp slot each) up in parallel, ready 11.2 s; kine/kcp bound ports read back from /proc 55/82 ms after start; core kill -9 at 03:21:07.729Z re-adopted kine and kcp 120 ms later (same pids/InvocationIDs); dev up on a running instance adopts in 0.2 s; dev down 3.4 s leaves no process/listener/socket/dir; killing devsup takes its units with it and dev down sweeps the records (TestTwoInstancesOnOneHostShareNothing 17.4 s); readings in docs/research/2026-10-11-vm-lifecycle/binary-base-readings.md"
- "known gap: a grandchild in a unit's group survives a devsup kill -9 until the next devsup or dev down sweeps it"
- id: M2-HOST-CONFORMANCE
milestone: M2
title: One hostsvc.Manager contract suite, run against proc on every check.sh --fast and against systemd-user behind livegate
depends_on: [M2-HOST-UNITS]
decision: >
Decided by the user 2026-10-12 (fold item list): one contract suite run against both backends.
Decided by Claude, user may veto (dev-run-mode.md 4.8 tier 1): internal/hostsvc/hostsvctest.Run(t,
open func(t) Manager), stand-ins only. proc runs on every check.sh --fast with --die-with the test pid
(no root, no systemd, CI-able); systemd-user runs behind livegate on hat-8 with a socialweb-test-<rand>-
prefix (the login session's manager is acceptable here; the lingering-account proof stays with
M2-HOST-CONTRACT). Cases: start/inspect/stop; adopt-not-recreate on a second Start; a failed unit
readable until ResetFailed; the exit status of a killed unit (signal 9); survives the starter's restart
(proc: a second client process; systemd: E1); stop kills a descendant that called setsid (`sh -c 'setsid
sleep 1000 & exec sleep 1001'`); Restart=on-failure restarts with a new InvocationID; Watch reports an
exit within 2 s; StopSlice stops nested slices and nothing else; List never crosses prefixes; identity
mismatch on a reused pid; a manager restart makes every unit Lost(UnitGone); limits gated on
EnforcesLimits; fd store gated on FDStore; private hostname gated on PrivateHostname. Each
capability-gated case states which backend skips it. M2-HOST-UNITS' live tests (live_linux_test.go)
are re-homed into the suite's systemd half, keeping their names in the evidence.
files: ["internal/hostsvc/hostsvctest/", "internal/hostsvc/live_linux_test.go (re-homed)", "deploy/check.sh (--fast runs the proc half)"]
test_first: "TestEveryManagerMethodHasAContractCase (reflects over the Manager interface) red first; then each case red against a stub backend."
done_when: "the suite exists and is green on systemd-user on hat-8 (readings with UTC times); no Manager method without a case. The proc half is wired into check.sh --fast and turned green by M2-HOST-BACKEND-PROC, which depends on this item."
risks: ["A case that passes on proc and fails on systemd-user (or the reverse) is a backend difference to declare in Capabilities, never a skipped assertion."]
status: done
evidence:
- "task/binary-base 6078bc8 merged to main a107b5a 2026-10-12: one contract suite for hostsvc.Manager run against any backend; no Manager method without a case"
- "systemd-user behind livegate on hat-8 2026-10-12T03:05:44Z: 14 passed, manager-restart case skipped (restarting user@ ends the login session); proc half green in check.sh --fast (M2-HOST-BACKEND-PROC)"
- "backend differences declared in Capabilities, not skipped: FDStore and PrivateHostname false on both (ns-exec not built yet), proc does not enforce limits; systemd Watch polls every 250 ms; systemd manager identity = boot id + manager start timestamp (init.scope has no InvocationID)"
- id: M2-HOST-CONTRACT
milestone: M2
title: The Linux host contract, probed by check-host, refused by name
depends_on: [M1-ASSETS, M2-HOST-UNITS]
decision: >
Decided by the user 2026-10-11: minimum systemd 253 (GetUnitByPIDFD), check-host refuses older, and
features are also probed directly; Ubuntu 24.04+ gets an install-time AppArmor profile for the pinned
pasta path that check-host verifies. Decided by Claude, user may veto: check-host (as the account)
checks and names each failure: systemd Manager.Version >= 253 over the private socket; a throwaway
transient service socialweb-probe-<rand> started with every property the runtime uses (MemoryMax,
MemorySwapMax=0, OOMPolicy, TasksMax, CPUQuota, CPUWeight, CollectMode, After/Wants, Slice, KillSignal,
TimeoutStopSec, StandardOutput=append:) and each read back; cgroup v2 unified; the user@<uid>
cgroup.controllers include cpu, memory, pids (else CPUQuota is accepted but unenforced); linger on;
$XDG_RUNTIME_DIR short enough that the longest socket path the runtime builds is < 108 bytes; boot_id
readable; for bidders: /dev/kvm read-write for the account, the user manager's own Groups (from
/proc/<user@ pid>/status) include kvm (else "the account is in kvm in /etc/group but its user manager
started before; restart user@<uid> - this stops every VM - or reboot"; never "chmod /dev/kvm" and never
"log in again", which does not refresh a lingering manager), /dev/net/tun usable, the embedded
runtime/<sha>/pasta creates a user+net namespace and execs through the /bin/sh trampoline (SELinux
pasta_t may not exec our binaries directly, linux-proofs E4-sel), the AppArmor profile loaded where
AppArmor restricts userns, the account's subuid AND subgid range present (M1-INSTALL), overlapping no
other entry of either file, newuidmap and newgidmap present and setuid root (or with their file
capabilities), a throwaway user namespace mapped through them into the first VM sub-range, and POSIX
ACL support on the state dir's filesystem (0007 M2-VM-RUNTIME grants each VM's sub-uid access to its
own work files by ACL, decided by the user 2026-10-12: per-VM subuid user namespaces); oomd's
monitored cgroups reported; for the gateway, the user manager's fd
store (M3-APISERVER-FRONT) and the k3s slot hostname mechanism (M3-CP-SLOTS). The written contract is
docs/host-contract.md. check-host is the production check (host-backend=systemd-user); the dev run
mode's `dev check` (M1-DEV-INSTANCE) reports the dev subset (kvm, userns for pasta, AppArmor, tun,
newuidmap/newgidmap) and fixes nothing (decided by the user 2026-10-12: AppArmor in dev report-only).
The userns-stage prerequisite (util-linux unshare >= 2.38, or none) follows 0007 M0-VM-USERNS's choice.
files: ["cmd/socialweb-computer check-host (host contract checks)", "docs/host-contract.md"]
test_first: >
Offline: TestASystemdOlderThan253IsRefused (fake bus reporting 252), TestEachRuntimePropertyIsReadBackFromAProbeUnit
(fake bus drops one property: refused naming it), TestAManagerWithoutKvmInItsGroupsIsRefusedNamingTheRestart,
TestASubuidRangeOverlappingAnotherEntryIsRefused, TestAMissingSubgidRangeIsRefusedEvenWithASubuidRange.
Behind livegate on hat-8: TestTheEmbeddedPastaCreatesAUserNamespaceHere,
TestNewuidmapMapsAThrowawayNamespaceIntoTheFirstVMSubRange. Red first.
done_when: "green; check-host readings on hat-8 and on the prod gateway host recorded (prod's user@ Delegate value read, not assumed)."
risks:
- "systemd 253 excludes Debian 12 and RHEL 9 (252, from memory; verified in this item) - see questions_for_user."
- "Ubuntu 22.04 and older are unmeasured and refused until measured."
status: done
evidence:
- id: M2-VM-TREE-AUDIT
milestone: M2
title: No socialweb-computer code in a running VM's process tree, audited from the unit's cgroup (systemd-user) or process group (proc)
depends_on: [M1-ASSETS, M2-HOST-UNITS, M2-HOST-BACKEND-PROC]
decision: >
Backend-aware since 2026-10-12 (third fold; decided by Claude, user may veto, dev-run-mode.md 5): the
audit lists a unit's processes through the Manager - systemd-user: cgroup.procs; proc: the unit's
process-group members plus the descendants devsup attributes to it - and devsup itself, outside every
unit's group, is not a member (as systemd is not in a unit's cgroup).
Decided by the user 2026-10-11: no socialweb-computer code in a running VM's process tree; the VMM is
launched by the service manager and our code only does one-shot prep (the tiny exec helper execve's
away). This item builds the instrument, decided by Claude, user may veto: internal/hostsvc/treeaudit
lists every unit with the VM prefix (socialweb-vm-* and the market network prefix socialweb-market-network-*-vm-*), reads its
cgroup.procs, opens /proc/<pid>/exe for each process and hashes it, and passes only when the unit is
active and every process's executable sha256 is one of the pinned VM-unit assets of the current or a
still-referenced runtime set (pasta, firecracker); socialweb-vmexec, /bin/sh (the trampoline) and the
core binary are failures once the unit is past Activating. Exposed as `units audit` (exit 1 naming the
unit and the offending exe) and as a Go helper the live tests of plan 0007 call after every boot,
restart and upgrade.
files: ["internal/hostsvc/treeaudit/", "cmd/socialweb-computer units audit"]
test_first: >
Offline over a fake /proc and cgroup tree: TestAVMUnitRunningOnlyPinnedPastaAndFirecrackerPasses,
TestACoreBinaryInAVMUnitFails, TestAVmexecLeftAfterActivatingFails. Behind livegate on hat-8 with
stand-in pinned assets in a transient unit (not a guest): TestTheAuditReadsTheRealCgroupOfAStandInUnit;
on proc (check.sh --fast, stand-ins): TestTheAuditPassesOnTheProcBackendWithDevsupOutsideTheGroup and
TestAnAttributedDescendantRunningTheCoreBinaryFailsOnProc. Red first.
done_when: "green; plan 0007's live tests call it (0007 ACC7-NO-RESIDENT-CODE)."
risks: ["Hashing a large VMM exe on every pass is cost, not risk; the audit runs in tests and on demand, not every controller pass."]
status: done
evidence:
- id: M2-KCP-STORE
milestone: M2
title: "kcp's own store - the gateway: one kine per slot over a dedicated postgres unit; a bidder: one kine on sqlite; both in the bootstrap layer, before kcp"
depends_on: [M0-KCP-SPIKE, M1-ASSETS, M0-PG-SOURCE, M2-HOST-UNITS, M2-HOST-BACKEND-PROC]
variant_dependencies:
embedded_asset_and_postgres: "M1-ASSETS (embedded kine asset), M0-PG-SOURCE (the gateway's postgres unit), M2-HOST-UNITS (systemd-user placement)"
dev_and_bidder_kine_sqlite: "M0-KCP-SPIKE, M2-HOST-BACKEND-PROC only - kine on sqlite with kine resolved from an explicit path in a dev build; not gated on M1-ASSETS, M0-PG-SOURCE or M2-HOST-UNITS"
decision: >
Dependencies relaxed (user-approved 2026-10-12, state_rules exception): M1-ASSETS and M0-PG-SOURCE gate
only the embedded-asset and postgres variants. The dev/bidder variant - one kine on sqlite, kine taken
from an explicit path in a dev build, placed on proc - starts without them as part of the `binary-base`
vertical slice (execution_order); the item is done when both variants are.
Decided by the user 2026-10-12: the GATEWAY's kcp store is kine -> a dedicated postgres unit started
before kcp, NOT a Postgres CRD object (the Postgres objects live in kcp, so kcp's own store cannot be
reconciled from kcp; kcp-blue-green.md section 3), one kine per slot, and a pg_dump before each kcp
upgrade. Decided by the user 2026-10-12 (third fold), correcting d6f778e: a BIDDER's kcp is a single
slot on kine-sqlite (one kine process on <state>/handles/<hid>/kcp/kine.db; sqlite allows one kine), its
upgrade is a short kcp restart, VMs are unaffected and the controller re-adopts them; d6f778e's "every
production kcp on kine -> postgres, user may veto" was vetoed by the user. Before a bidder kcp restart
that changes the kcp or kine asset, the core stops the slot and the kine, copies kine.db (with -wal and
-shm) to <state>/handles/<hid>/kcp/backups/<ts>/ (size > 0 checked), and starts both again (decided by
Claude, user may veto). Dev (decided by the user 2026-10-12): a dev instance's gateway-role kcp store is
kine-sqlite in the quick profile and kine -> postgres in the market network and prodlike profiles; the bidder role
is always kine-sqlite, as in production. Placement (decided by the user 2026-10-12): every store unit is
in the BOOTSTRAP LAYER (M2-BOOTSTRAP-LAYER) - an installed user unit on systemd-user, a devsup unit on
proc - never a HostProcess placement; this item builds the store code and its unit specs, the bootstrap
layer installs and orders them.
Gateway units (in socialweb-kcp-gw.slice, adopted by name):
socialweb-kcp-gw-pg.service runs the pinned relocatable postgres (M0-PG-SOURCE's choice, the
same asset M2-POSTGRES uses) on <state>/kcp/pg with listen_addresses='' and one unix socket directory
<state>/kcp/pg/run (no TCP port at all, path < 108 bytes checked; the directory is 0700 to the
account), KillSignal=SIGINT, Restart=on-failure, readiness by the core's own pgx `SELECT 1` over the
socket (relocatable builds are not --with-systemd); initdb runs one-shot in the core, never
ExecStartPre. socialweb-kcp-gw-kine@<slot>.service (slot a|b) runs the kine asset with
--listen-address 127.0.0.1:0 and the socket DSN, After= and Wants= (not Requires=) the pg unit, so a
postgres restart does not cascade-stop kcp; its port is read back from its MainPID's single LISTEN
socket by hostsvc.ListenPort, written here because kine is the first port-0 unit (MainPID from
systemd, then /proc/<pid>/net/tcp and tcp6 LISTEN rows matched against the socket inodes of
/proc/<pid>/fd, same uid, no privilege; polled until exactly one LISTEN socket exists, two or more
refused by name; kine also logs `Kine available at`). Schema-creation race on a fresh
database (two kines measured to FATA on the first create): the first kine of a fresh store is started
alone and the second only after the first answers; a kine that still exits on SQLSTATE 23505 is
restarted once by the core. The small host-postgres library this item writes (internal/pgunit:
initdb, pg_hba for the socket only, the unit spec, readiness, pg_dump to <state>/kcp/backups/<ts>.sql
with exit 0 and size > 0 checked) is reused by M2-POSTGRES' controller, so there is one way to run a
host postgres. pg_dump joins the postgres asset set (M1-ASSETS; M3-CP-SLOTS needs it too).
Bidder units, per handle (in socialweb-kcp-<hid>.slice): socialweb-kcp-<hid>-kine@a.service runs kine
with --endpoint sqlite://<state>/handles/<hid>/kcp/kine.db and --listen-address 127.0.0.1:0, port read
back by ListenPort; no postgres unit, no second kine. ListenPort reads the MainPID from the Manager, so
it works on both backends (devsup reports the pid on proc).
files: ["internal/pgunit/ (initdb, hba, unit spec, readiness, dump)", "internal/kcpstore/ (gateway: pg + kine per slot; bidder: one kine on sqlite; unit specs, start order, adoption, sqlite backup copy)", "internal/hostsvc (ListenPort)", "internal/versions/versions.json (pg_dump in the postgres asset set)"]
test_first: >
Offline (fake hostsvc): TestKcpStoreStartsBeforeKcpAndOutsideTheCRDReconciler (the start order pg ->
kine -> slot, and no Postgres object is read or written), TestKineIsWantsNotRequiresOnItsPostgres,
TestThePgUnitHasNoTcpListener (argv: listen_addresses empty, socket dir under 108 bytes); over a fake
/proc: TestListenPortReadsTheOneListenSocketOfTheMainPID, TestListenPortRefusesAUnitWithTwoListenSockets,
TestListenPortIgnoresAnotherProcessesSockets. Behind
livegate on hat-8 with stand-ins (real postgres and kine processes in transient units, no kcp, no guest):
TestTwoKinesOnOneFreshDatabaseBothComeUp (red first: the measured FATA when both start at once),
TestAPostgresRestartDoesNotStopItsKines, TestAPgDumpOfTheKcpStoreIsNonEmptyAndRestores. Bidder and dev
(offline): TestTheBidderKcpStoreIsOneKineOnSqliteAndNoPostgres, TestTheBidderStoreIsCopiedBeforeAnAssetChangingRestart,
TestTheDevQuickProfileUsesSqliteAndTheMarketNetworkProfilePostgres; the live stand-in tests also run on proc
(devsup, check.sh --fast where no postgres asset is needed, behind livegate otherwise).
done_when: "green on hat-8 on both backends; two kines on one fresh database come up 10 times in a row; a bidder sqlite store survives a slot+kine restart with its workspaces listed; readings recorded."
risks:
- "A kine schema migration takes ACCESS EXCLUSIVE (control-plane-handover.md); a kine bump for the gateway's kcp is rolled like a kcp slot (M3-KCP-ROLL) with old and new kine on one database tested first; on a bidder it is part of the short restart."
- "Per-handle stores on a bidder: kine + one kcp + front per handle, ~0.45 GB (kcp-spike.md: kcp 378-409 MB, kine 54-62 MB); recorded per handle in M2-KCP-PORT0's readings."
- "A sqlite kine restart is a store outage for the bidder's kcp; accepted, because the bidder's kcp restart is already a short outage that VMs never notice (decided by the user 2026-10-12)."
status: done
evidence:
- id: M2-KCP-FRONT
milestone: M2
title: Our own L4 front holds kcp's address (127.0.0.1:6443 on the gateway, port 0 elsewhere) and swaps its backend between slots; caddy is never touched
depends_on: [M1-CLI, M2-HOST-UNITS, M2-HOST-BACKEND-PROC]
decision: >
Third fold 2026-10-12 (decided by Claude, user may veto): the front is a bootstrap-layer unit
(M2-BOOTSTRAP-LAYER: an installed user unit on systemd-user, a devsup unit on proc), so this item no
longer waits for M1-INSTALL; the installed unit file is written by M2-BOOTSTRAP-LAYER. A bidder keeps a
port-0 front for its single slot too: the slot binds a new port on every start (port 0), and the front
is the stable --shard-base-url and consumer address across the bidder's short kcp restarts. Every
front outside the production gateway binds 127.0.0.1:0 and reports its port through OnListen
(M2-KCP-PORT0); the fd store is UnitSpec.FDStore on both backends (devsup holds it on proc).
Decided by the user 2026-10-12: our own L4 front holds 127.0.0.1:6443 (the same code as the tenant k3s
fronts) and does the swap after the new slot's own-port /readyz; caddy stays public TLS only, is never
reconfigured by a swap, and any generated caddy config sets install_trust false. Decided by Claude, user
may veto (kcp-blue-green.md 5 and 7.3; control-plane-handover.md 4): internal/apiserverfront, pure TCP
passthrough (TLS stays end to end with kcp, whose serving cert is the shared apiserver.crt), run as
`internal apiserver-front --owner <owner>` in the bootstrap-layer unit socialweb-kcp-<owner>-front.service
naming the resolved bin/<sha256>/ path (rewritten only by an explicit front roll), so a core upgrade
never changes the running front.
The backend is chosen per NEW connection from a state the core sets over a control socket in
<state>/kcp/ (path < 108 bytes); open connections stay on their backend until it closes them; the front
never fails over by itself. Its listening socket is kept in the user manager's fd store (FDSTORE=1,
FileDescriptorStoreMax=1), so a front restart refuses no connection. Address: the option kcp-front-address
(default 127.0.0.1:0; the gateway's prod config.json sets 127.0.0.1:6443); a port-0 front's bound port
is read back like a slot's (M2-KCP-PORT0) and written to kcp-slots.json. caddy keeps
kcp.socialweb.computer -> https://127.0.0.1:6443 unchanged: no caddy admin access for the core, no
caddy reload by a swap (a reload cuts every upgraded stream, exec/attach included), and kcp's private key
never leaves kcp's files. Any caddy config this repo renders or ships (scripts/deploy.sh's site, the
M5-FRESH-DEPLOY config) pins `pki.certificate_authorities.local.install_trust: false` (Caddyfile:
`skip_install_trust`) and never uses caddy's internal issuer (the kcp-blue-green.md section 6 incident:
caddy installed a local CA into the system trust store through sudo). This item builds the one front
type; M3-APISERVER-FRONT reuses it per tenant cluster.
files: ["internal/apiserverfront/ (front, control socket, fd store)", "cmd/socialweb-computer internal apiserver-front", "internal/kcpserver (front unit, backend flip)", "internal/clicfg/options.json (kcp-front-address)", "scripts/deploy.sh (caddy site: install_trust false)"]
test_first: >
Offline: TestTheFrontPicksTheBackendPerNewConnectionAndLeavesOpenOnesAlone,
TestTheFrontNeverSwitchesByItselfWhenItsBackendDies, TestEveryCaddyConfigInTheRepoPinsInstallTrustFalse
(scans deploy/ and scripts/ for caddy JSON and Caddyfiles; red on today's tree if any lacks it).
Behind livegate on hat-8 with stand-ins (two real kcp slots on kine -> postgres from M2-KCP-STORE, no
guest): TestTheKcpFrontFlipsWithZeroFailedReadyzPolls (authenticated /readyz with the shard-admin token
every 200 ms through the front across a flip and SIGTERM of the old slot: 0 failures),
TestAFrontRestartRefusesNoConnection (fd store), each run on systemd-user and on proc (devsup holds the
fd), plus TestABidderFrontKeepsItsAddressAcrossAKcpRestart. Red first.
done_when: "green on hat-8 on both backends; the flip's poll count and failures recorded with UTC times."
risks:
- "User-manager fd-store support is unverified on the target systemd; without it a front restart has a refused-connection window of its restart time (M2-HOST-CONTRACT probes it)."
- "kubectl exec/attach/port-forward through a mount are upgraded streams; they survive a flip (open connections stay) but end when the old slot stops - roll idle-first (M3-KCP-ROLL)."
status: done
evidence:
- id: M2-KCP-SLOTS
milestone: M2
title: "kcp slot units of the release asset behind the front - the gateway: alternating blue/green slots on a shared kine store, leader-elected; a bidder: one slot on kine-sqlite"
depends_on: [M0-KCP-SPIKE, M1-ASSETS, M2-HOST-UNITS, M2-HOST-BACKEND-PROC, M2-KCP-STORE, M2-KCP-FRONT]
variant_dependencies:
embedded_asset_and_postgres: "M1-ASSETS (embedded kcp asset), M2-HOST-UNITS, and M2-KCP-STORE's postgres variant"
multi_slot_blue_green: "M2-KCP-FRONT (the front a slot flip needs; the gateway's blue/green slots)"
dev_single_slot: "M0-KCP-SPIKE, M2-HOST-BACKEND-PROC and M2-KCP-STORE's dev_and_bidder_kine_sqlite variant only - one slot on kine-sqlite, kcp resolved from an explicit path in a dev build, port 0 with the /proc read-back; not gated on M1-ASSETS, M2-HOST-UNITS or M2-KCP-FRONT"
decision: >
Dependencies relaxed (user-approved 2026-10-12, state_rules exception): M1-ASSETS (and, through
M2-KCP-STORE, M0-PG-SOURCE) gate only the embedded-asset and postgres variants; M2-KCP-FRONT gates only
multi-slot blue/green, not a single dev slot. The dev single-slot variant (kine-sqlite, one slot, kcp and
kine from explicit paths in a dev build) starts without them as part of the `binary-base` vertical
slice (execution_order); the item is done when every variant is.
Third fold 2026-10-12, decided by the user: a BIDDER's kcp is a single slot (a) on kine-sqlite; its
upgrade is a short kcp restart (stop the slot, start it on the new asset, the same root and store); VM
units are separate units and never notice; the ComputeVM controller re-adopts them by the Verdict once
kcp answers (0007 M2-CONTROLLER). Blue/green slots on kine -> postgres are the GATEWAY's only.
Decided by Claude, user may veto: a bidder slot runs WITHOUT --enable-leader-election (the core never
starts a second process for that root, so no lease can strand the controllers for 60 s after a
restart), and the argv builder requires leader election only for a blue/green owner; the shared
material copy is skipped (one slot root). Placement (decided by the user 2026-10-12): slot units are in
the bootstrap layer (M2-BOOTSTRAP-LAYER: installed template units socialweb-kcp-<owner>@<slot>.service
on systemd-user, devsup units on proc), started and stopped by instance; the core writes each
instance's EnvironmentFile (kine port, front URL, root dir) before starting it. Everything below holds
for the gateway; for a bidder read "the one slot" for "the active slot".
Route: out-of-process (decided by the user 2026-10-12, M0-KCP-SPIKE evidence): kcp runs OUT OF PROCESS as
alternating slot units; the release kcp v0.33.0 is an embedded asset (M1-ASSETS), unpatched, with no
vendored source; slots run --enable-leader-election and --shard-base-url=<front>; five root files are
copied into each new slot; the root admin token is regenerated per start, so consumers use shard-admin
or SA tokens. Decided by Claude, user may veto (kcp-blue-green.md 7.2, kcp-spike.md B2):
socialweb-kcp-<owner>-<slot>.service (slot a|b, in socialweb-kcp-<owner>.slice), Type=exec,
Restart=on-failure, PropExecStart(argv, false), CollectMode=inactive, After=/Wants= its kine unit,
argv `kcp start --root-directory=<state>/kcp/slot-<slot> --bind-address=127.0.0.1 --secure-port=0
--experimental-bind-free-port --etcd-servers=http://127.0.0.1:<its kine port> --enable-leader-election
--shard-base-url=https://127.0.0.1:<front port> --shutdown-delay-duration=2s
--feature-gates=WorkspaceMounts=true --audit-policy-file=<embedded policy written to state>
--audit-log-path=<slot root>/audit.log` plus today's rotation flags; the argv builder refuses an argv
without WorkspaceMounts=true (versions.json kcp invariant), leader election or the front URL.
Shared material: <state>/kcp/shared/ (0700) holds apiserver.crt, apiserver.key, sa.key,
.admin-token-store and admin.kubeconfig; the very first slot starts on a fresh root and, once its
/readyz answers, the core copies the five files into shared/; every later slot root gets all five
copied in BEFORE its unit starts (without admin.kubeconfig kcp refuses to start when
.admin-token-store is present). Consumers (kcpapply, the gateway, the clusterprovider, deno-kcp, the
DenoPods' token minting) take a rest.Config from internal/kcpserver = the front URL + the shared CA +
the shard-admin token (the system:admin / shard-base contexts, persistent) or a minted SA token; nothing
holds the kcp-admin root/base context, whose token is regenerated by every kcp start. State:
<state>/kcp/kcp-slots.json {active, slots {a, b: asset sha256, kcp version, port, kine port, unit
names, InvocationIDs}, front {address, port}}, written before any front flip. One core per kcp state:
an flock on <state>/kcp/ held by the core. Adoption on a core restart or upgrade: units by name +
InvocationID + kcp-slots.json; the active slot active -> adopt, no write; missing -> start it on its
existing root (the store keeps the data). A slot restarted in place by systemd (new InvocationID, no
other slot running) gets the Lease kube-system/kcp-controllers deleted through the front with the
shard-admin context once its predecessor process is gone, so a crash costs no 60 s controller gap. A
core upgrade restarts none of the kcp units; a kcp version change is M3-KCP-ROLL.
files: ["internal/kcpserver/ (Server interface, slots_linux.go, argv.go, shared material, kcp-slots.json, rest.Config)", "internal/assets (kcp)", "internal/versions/versions.json (kcp release pin and sha256, WorkspaceMounts invariant)"]
test_first: >
Offline: TestTheSlotArgvCarriesLeaderElectionAndTheFrontURL (red without the flags),
TestKcpIsRefusedWithoutTheWorkspaceMountsGate (argv builder), TestEverySlotRootGetsTheSharedKeyMaterialBeforeStart
(all five files, admin.kubeconfig included, before StartTransientUnit is called),
TestNoConsumerUsesTheVolatileKcpAdminContext (every rest.Config kcpserver hands out names the front and
the shard-admin or an SA identity). Behind livegate on hat-8 with stand-ins (real kcp, kine and
postgres processes in transient units, no guest): TestTwoKcpSlotsOnOneStoreServeTheSameWorkspaces,
TestACoreUpgradeLeavesEveryKcpUnitsInvocationIDUnchanged (slot, kines, pg, front),
TestASlotRestartedOnItsOwnStoreListsTheSameWorkspaces (workspaces created through this item's own slot
on a fresh state dir, the slot stopped and started: listed equal - replaces the 2026-10-11 test that
opened a root the container-era release binary wrote; fresh deploy, decided by the user 2026-10-12),
TestASlotRestartedBySystemdHasNoControllerGapLongerThan5s (kill -9 of the slot's MainPID; a probe
workspace Ready within 5 s of the restarted slot's /readyz). Bidder single slot: offline
TestABidderSlotArgvHasNoLeaderElectionAndThereIsNoSecondSlot; behind livegate (stand-ins, both
backends): TestABidderKcpRestartOnANewAssetKeepsItsWorkspacesAndLeavesStandInVMUnitsUntouched (stand-in
socialweb-vm-<hid>- units keep their InvocationIDs across the slot restart; the restart's kcp-down
seconds recorded). The live stand-in tests run on systemd-user and on proc. Red first.
done_when: "green on hat-8 on both backends; cold start, restart and RSS readings for slot, kine and pg recorded on a prod-shaped object count (UTC); the bidder slot's restart-to-/readyz seconds recorded."
risks:
- "kine serves watches written through ANOTHER kine on its next poll (1 s ticker; 0.27-0.42 s measured); during an overlap a client may see a sub-second lag, recorded, not a failure."
- "Both slots run the apiserver's compactor; the compare-and-swap on compact_rev_key makes one win per interval (kcp-blue-green.md 3); nothing to coordinate."
status: done
evidence:
- id: M2-KCP-PORT0
milestone: M2
title: Every kcp, kine and front binds port 0 and the core reads the bound port from the unit's single listening socket - no kcp patch
depends_on: [M2-KCP-SLOTS]
decision: >
Decided by the user 2026-10-07: kcp supports binding port 0 and the caller learns the bound port, the
Go twin of createServe's onListen. Decided by the user 2026-10-12: no kcp patch; the core reads the
slot's single listening socket from /proc (kcp-spike.md route B section 1). There is no embedded etcd
(the store is M2-KCP-STORE's kine), so the 2026-10-07 embedded-etcd port-0 patch, the etcd-client and
etcd-peer listeners and the port-file patch are all dropped. Decided by Claude, user may veto:
slots run `--secure-port=0 --experimental-bind-free-port` (options.go:355-361 creates the listener in
Options.Complete, before anything serves), and the core reads each slot's port with hostsvc.ListenPort
(M2-KCP-STORE; 0.17-0.30 s after exec measured for kcp). kcpserver.Config gains OnListen(func(name string,
addr net.Addr)), called once per listener (kcp-<slot>, kine-<slot>, front) with the address read back,
before readiness. Readiness is /readyz == 200 on THAT port with the shard-admin token - never through a
copied admin.kubeconfig, which names the front (and named the OLD slot in the spike's one failing run).
Only the gateway's front has a fixed address (127.0.0.1:6443 from its prod config, M2-KCP-FRONT); every
slot, kine and every other front, and every test, use 0, so any number of application instances, each
with its own kcp (the gateway, each bidder handle, run request, tests, every dev instance), run on one
machine with no port allocation (principle.port0_rule; the gateway front is its edge exception). Fallback
kept on record (not built): fixed alternating A/B slot ports, if the /proc read-back is ever rejected.
Both backends (decided by the user 2026-10-12, dev run mode now): ListenPort takes the MainPID from the
Manager, so the read-back works under devsup; the front binds before the first slot starts (its port is
in the slot's --shard-base-url), in production and in dev.
files: ["internal/kcpserver (OnListen, readiness on the read-back port)"]
test_first: >
Behind livegate on hat-8 with stand-ins (real kcp, kine, postgres, no guest):
TestTwoKcpStacksOnPortZeroGetDistinctPortsAndTheShardURLIsTheFront (a gateway-shaped owner with pg,
kine, slot and a port-0 front, and a bidder-shaped owner with sqlite kine, one slot and a port-0 front,
on one host: OnListen reports distinct ports for every listener; each owner's shard object and
admin.kubeconfig name its own front, never a slot port), run on systemd-user and on proc,
TestOnListenFiresBeforeReadyWithTheSlotsOwnPort, TestReadinessIsNeverReadThroughACopiedAdminKubeconfig
(a stale copied admin.kubeconfig naming a dead port does not make a slot Ready). Red first.
done_when: "green on hat-8 on both backends; no fixed kcp, kine or etcd port left in any socialweb-computer mode except the gateway front's prod 6443 (M1-NO-FIXED-PORTS' ratchet holds no kcp, kine or front entry); per-instance process count and RSS recorded (gateway and bidder shapes)."
risks: ["A kcp release that opens a second TCP listener breaks the single-socket rule; ListenPort refuses by name and the asset pin's self-check (M1-INSTALL upgrade) runs a slot once to catch it."]
status: done
evidence:
- id: M2-BOOTSTRAP-LAYER
milestone: M2
title: "The bootstrap layer - kcp's store, kine, slot(s), front and the core: installed systemd units in production, devsup units in dev, up before kcp exists"
depends_on: [M2-KCP-PORT0, M1-DEV-INSTANCE, M1-INSTALL]
decision: >
Decided by the user 2026-10-12: kcp, its kine and its store cannot be supervised by kcp itself, so in
production systemd runs them as installed units and in dev the core's devsup runs them before kcp
exists; everything else goes through kcp (HostProcess, M2-HOSTPROCESS-KIND); instances nest as kcp
workspaces (one kcp per application instance; principle). The bootstrap layer is per APPLICATION
INSTANCE, not per machine: each instance brings up its own kcp's store, kine, slot(s) and front, and
several instances' bootstrap layers sit side by side on one host sharing nothing but the machine.
Decided by Claude, user may veto: the bootstrap layer is exactly
{kcp store (gateway: the pg unit; bidder: none), kine unit(s), kcp slot unit(s), the kcp front, the
core unit itself} and is defined by ONE table, internal/bootstrap, per role: gateway = socialweb-kcp-gw-pg,
socialweb-kcp-gw-kine@a|b, socialweb-kcp-gw@a|b, socialweb-kcp-gw-front, socialweb-computer-gateway;
bidder handle = socialweb-kcp-<hid>-kine@a (sqlite), socialweb-kcp-<hid>@a, socialweb-kcp-<hid>-front,
socialweb-computer-bidder@<handle>. The table renders (a) on systemd-user to installed user unit files
(written and removed by install/uninstall, extending M1-INSTALL's unit table; WantedBy=default.target
so a lingering user manager brings kcp up at boot before the core; ordering store -> kine -> front ->
slot -> core by After=/Wants=; per-instance parameters - kine port, front URL, root dir, asset paths -
from an EnvironmentFile <state>/kcp/<unit>.env the core writes before it starts an instance, the first
one written by install), started and stopped by name (hostsvc StartInstalled/StopUnit, never
StartTransientUnit); and (b) on proc to devsup unit specs that `dev up` places in the same order before
the core (devsup is the bootstrap supervisor in dev; the core is itself a devsup unit, so `dev restart
core` replaces it while kcp keeps running). Dev profiles: quick (gateway-role kcp on kine-sqlite), market network
and prodlike (kine -> postgres); a dev instance whose profile composes a gateway and a bidder runs two
application instances, each with its own kcp (~1 GB together), talking to each other only over the
atproto market protocol. The earlier `--kcp-shared-roles` (both roles on one kcp under <base>:gateway
and <base>:bidder) is dropped: two applications never share a kcp (principle, user 2026-10-12); the
lighter shape is nesting under a parent (M4-DEV-PARENT-KCP), where each child keeps its own kcp. Every role addresses its workspaces under the kcp-base-workspace option (root in
production), so no role assumes it owns root and instances can later nest (M4-DEV-PARENT-KCP). Once kcp
answers, the core writes one read-only HostProcess MIRROR per bootstrap unit (spec.placement bootstrap;
M2-HOSTPROCESS-KIND), the kubelet static-pod/mirror-pod pattern: deleting a mirror never stops its
unit and the supervisor never places a bootstrap unit.
files: ["internal/bootstrap/ (role tables, unit-file renderer, devsup specs, env files)", "internal/hostsvc (StartInstalled on both backends)", "internal/install (bootstrap unit files in the install table)", "cmd/socialweb-computer dev up (bootstrap placement per application instance, profiles)"]
test_first: >
Offline: TestTheBootstrapTableIsTheOnlySourceOfBootstrapUnitsOnBothBackends,
TestEveryRenderedBootstrapUnitFileVerifies (`systemd-analyze --user verify`),
TestInstallWritesEveryBootstrapUnitFileAndUninstallRemovesThem (fake root fs),
TestTheBidderBootstrapHasNoPostgresAndOneSlot, TestTheBootstrapOrderIsStoreKineFrontSlotCore,
TestNoBootstrapUnitIsEverPlacedByTheHostProcessSupervisor,
TestNoTwoApplicationInstancesAreGivenOneKcp (a profile composing a gateway and a bidder yields two
bootstrap tables with distinct kcp roots and fronts). Behind livegate, proc (stand-ins plus real
kcp/kine assets, no guest): TestTwoDevInstancesEachServeTheirOwnKcpOnPortZero,
TestADevCoreRestartLeavesKcpKineAndFrontRunning (InvocationIDs unchanged); on hat-8 under a throwaway
installed account (systemd-user): TestInstalledBootstrapUnitsBringKcpUpBeforeTheCore and
TestAUserManagerRestartBringsKcpBackWithoutTheCore. Red first.
done_when: "green on both backends; on hat-8 two dev instances from two worktrees each `dev up` to a Ready kcp in parallel (cold and warm seconds, RSS per instance recorded), and a throwaway installed account brings kcp up from its installed units."
risks:
- "systemd's EnvironmentFile cannot change ExecStart's executable path; the slot units name the content-addressed kcp asset path through a stable per-owner symlink retargeted only by a roll, measured and recorded."
- "Installed units started at boot make kcp come back after a reboot while every VM is Lost (reboot = Lost, user 2026-10-11); that is the intended order: kcp first, then the core re-populates and marks VMs Lost(HostRebooted)."
status: done
evidence:
- id: M2-POSTGRES
milestone: M2
kcp_use: "traditional-exception (virtual Kubernetes clusters project: tenant cluster stores; principle)"
title: An embedded PostgreSQL driven by a Postgres CRD, one unit per server placed through a HostProcess
depends_on: [M0-PG-SOURCE, M2-HOST-UNITS, M2-APPLY, M2-KCP-STORE, M2-HOSTPROCESS-KIND]
decision: >
Placement (decided by the user 2026-10-12, kcp-native now): the Postgres controller owns one HostProcess
child per server and never calls hostsvc itself; the supervisor places it on systemd-user or proc and
status.runtime.backend names which. Decided by the user 2026-10-07: postgres is embedded and is a reconciled resource, not a container
package. Decided by the user 2026-10-11: KEEP the Postgres CRD with its own unit. Decided by Claude,
user may veto: a pinned relocatable PostgreSQL 18 linux build (amd64, arm64; source chosen by
M0-PG-SOURCE between prebuilt relocatable sets and a build from source with the build-host script, its
locale/ICU and glibc collation versions recorded in versions.json), extracted by internal/assets.
Kind Postgres (socialweb.computer/v1alpha1, export hostprocesses, apigen-generated) with spec {version,
port (0 = leased), dataDir, listen 127.0.0.1, deleteData} and an OS-neutral status {phase, port,
runtime {backend, ref, instance, pid, startTime, hostBootID}, message}. The controller (built on
internal/pgunit from M2-KCP-STORE, so kcp's store and every cluster store run postgres one way) runs initdb
one-shot in-process (never ExecStartPre), writes pg_hba (127.0.0.1/32 only, today's rule), starts
socialweb-pg-<clusterKey>.service (ExecStart runtime/<pg-sha>/bin/postgres -D ... -p <lease> -c
listen_addresses=127.0.0.1 -c unix_socket_directories=; Restart=on-failure; KillSignal=SIGINT, fast
shutdown; TimeoutStopSec=60), and reports Ready only after its own pgx `SELECT 1` on the leased port
(relocatable builds are not --with-systemd, so systemd cannot see readiness). Adoption on a core restart
by unit name + identity (M2-HOST-UNITS); a unit whose InvocationID changed (systemd restarted it) is
re-probed and status bumped. On deletion it stops the unit and keeps the data dir unless
spec.deleteData. The clusterprovider's per-cluster store (postgres.go:286-364 today execs initdb,
postgres and psql from PATH) creates a Postgres object in the cluster's home workspace and waits for
Ready; psql calls (bootstraprows.go:28, postgres.go:364) become pgx; clearStalePid/pidIsOurPostmaster
(postgres.go:222-257) are replaced by unit identity. Postgres refuses root; the gateway account is not
root (M1-INSTALL). Upgrades of the finished system (amended 2026-10-12 from control-plane-handover.md
5(e), decided by Claude, user may veto): a MINOR upgrade (new pin, same major, same collation versions)
is a unit restart with the new ExecStart (fast shutdown + start, seconds; kine and bao reconnect); a
MAJOR upgrade is logical replication into a NEW Postgres object of the new major, then a write freeze
of 1-2 s: stop the cluster's kine slot units (M3-CP-SLOTS), wait for catch-up, setval('kine_id_seq',
max(id)) on the new primary (sequences are not replicated by logical replication before PG 19, verified
for the target major in this item), switch the DSN of kine and bao, start kine; below a size bound
measured here, dump/restore under the same write freeze instead. A pin whose collation provider version
differs from the one a data dir was initdb'd with is treated as a major-class change (reindex or
replication), never opened silently. Before every k3s MINOR bump M3-CP-SLOTS takes a pg_dump of the
k3s and openbao databases. Readiness and ordering against the cluster's member units:
M3-CLUSTER-UNITS.
files: ["api/v1alpha1/postgres_types.go", "internal/postgresctl/", "internal/clusterprovider/postgres.go, bootstraprows.go", "deploy/sets.yaml (hostprocesses)"]
test_first: >
Behind livegate: internal/postgresctl TestAPostgresObjectBecomesAServerOnItsReportedPortAndSurvivesItsControllersRestart,
TestAPostgresUnitRestartedBySystemdIsReprobedAndStaysReady,
TestAMinorUpgradeIsAUnitRestartAndKineReconnects (a stand-in kine polled through the restart; SELECT 1
and a kine write succeed after it), TestAMajorUpgradeByLogicalReplicationKeepsEveryKineRowAndTheSequence
(stand-in data: rows and kine_id_seq equal after the switch, writes resume past max(id)),
TestADataDirFromAnotherCollationVersionIsRefusedNotOpened; internal/clusterprovider
TestTheClusterStoreIsAPostgresObjectNotAProcessGroupMember. Red first.
done_when: "a market cluster reaches Ready on hat-8 with its store as a Postgres object; no postgres or psql binary from the host is on the path; the minor and major upgrade tests green with their write-freeze durations recorded."
risks:
- "A cluster's store is adopted, never re-initialised (refused if PG_VERSION differs from the pin's major)."
- "Logical replication needs wal_level=logical on the old server; set from the first start so a major upgrade needs no restart of the old server."
status: done
evidence:
- id: M2-OPENBAO-SERVER
milestone: M2
title: An OpenBao server as a reconciled resource in its own unit (placed through a HostProcess), for the gateway's root vault and the bidder's vault
depends_on: [M1-ASSETS, M2-HOST-UNITS, M2-APPLY, M2-HOSTPROCESS-KIND]
decision: >
Placement (decided by the user 2026-10-12, kcp-native now): the OpenBaoServer controller owns one
HostProcess child per server; the supervisor places it on either backend; "a new InvocationID" below is
read from the child's status.runtime.
Decided by the user 2026-10-11: KEEP the OpenBaoServer CRD with its own unit. Kind OpenBaoServer
(export hostprocesses) with spec {listen, tls, storageDir, init: {shares, threshold, keysSecret}} and
status {phase, address, initialized, sealed, runtime {backend, ref, instance, pid, startTime,
hostBootID}}; the controller reuses internal/rootvault for init, unseal and engine layout and runs the
extracted bao as socialweb-bao-root.service (gateway) or socialweb-bao-<hid>.service (bidder, plan 0007
M4-BIDDER-VAULT), Restart=on-failure. A bao restarted by systemd comes up SEALED: the controller watches
each unit's InvocationID and unseals on every new one, including one found at adoption after the core
was down. The gateway's root vault (container-entrypoint.sh:215-224, :253-258) becomes one such object;
plan 0007's bidder vault is another (decided by the user 2026-10-07: bidder identity keys live in an
OpenBao beside the bidder). Because the object lives in the instance's own kcp, a fresh instance brings
up its kcp before its vault (M3-SERVE-GATEWAY-KCP); in a running instance the vault unit is already up
and outlives kcp and core restarts. Unseal material stays where internal/rootvault keeps it today (state dir, 0600). Distinct
from deno-kcp's OpenBao kind, which provisions a namespace and PKI inside a running server.
files: ["api/v1alpha1/openbaoserver_types.go", "internal/openbaoctl/", "internal/rootvault (callers)"]
test_first: >
Behind livegate: internal/openbaoctl TestAnOpenBaoServerIsInitialisedOnceUnsealedOnEveryStartAndKeepsItsKV
(restart the controller and the server; a KV value written before reads back) and
TestAVaultRestartedBySystemdIsUnsealedOnTheNextPass (kill -9 the bao main pid; Restart=on-failure brings
it back sealed; one pass later it is unsealed). Red first.
done_when: "green; the gateway's root vault comes up as an object on hat-8."
risks: ["One writer of the root vault layout: the controller calls apply-root's code, never a copy."]
status: done
evidence:
- id: M2-MANIFESTS
milestone: M2
title: Every YAML the stack applies lives in one embed.FS, grouped into named, ordered sets
depends_on: [M1-CLI]
decision: >
deploy/embed.go (package deploy) embeds deploy/*.yaml, deploy/denokcp/*.yaml, deploy/eso/*.yaml and
deploy/market-network/*.yaml, and deploy/sets.yaml names ordered sets of phases with wait
conditions: provider, permission-inventory, denoruntime, hostprocesses, bootstrap-cluster,
market-network, plus the sets plans 0007 and 0008 add. Decided by Claude, user may veto.
files: ["deploy/embed.go", "deploy/sets.yaml", "internal/marketnetwork (sets.yaml recorded as ours)"]
test_first: "deploy TestEveryYAMLUnderDeployIsInExactlyOneSetOrListedAsUnapplied and TestEverySetDecodesStrictlyIntoKcpAndKubernetesTypes; red first."
done_when: "`manifests list` prints every set; tests green."
risks: ["deploy becomes a Go package; check.sh keeps working."]
status: done
evidence:
- id: M2-TEMPLATE
milestone: M2
title: Templating with typed values and no silent blanks
depends_on: [M2-MANIFESTS]
decision: >
text/template with missingkey=error over a typed values struct per set; strict decode after render.
The market network's twelve __PLACEHOLDER__ tokens and apply.sh:171's sed org-root rewrite become fields.
Only files recorded as adapted in deploy/denokcp/vendor.json carry template fields. Decided by
Claude, user may veto.
files: ["internal/manifests/", "deploy/denokcp/vendor.json"]
test_first: "internal/manifests TestAMissingValueIsARenderErrorNotAnEmptyField and TestTheRenderedMarketNetworkEqualsWhatApplyShProduced; red first."
done_when: "golden equal; tests green."
risks: []
status: done
evidence:
- id: M2-APPLY
milestone: M2
title: Server-side apply into kcp workspaces from Go, phase by phase, never kubectl
depends_on: [M2-TEMPLATE, M2-KCP-PORT0]
decision: >
internal/kcpapply: dynamic client, ApplyPatchType, field manager socialweb-computer, Force (tenant.go
precedent); workspace by /clusters/<path>; fresh discovery per phase; typed waits; APIResourceSchemas
create-only and compared (immutable spec); no kubectl-applied object is adopted (fresh deploy, decided
by the user 2026-10-12); `apply --dry-run` reports every field whose manager changes. It
takes a rest.Config from internal/kcpserver (the front URL with the shard-admin context, M2-KCP-SLOTS;
never the per-start kcp-admin token), no kubeconfig path in its API.
files: ["internal/kcpapply/", "cmd/socialweb-computer apply"]
test_first: >
Behind livegate: TestApplyingTheProviderSetTwiceChangesNothingTheSecondTime,
TestAChangedAPIResourceSchemaIsRefusedNotPatched.
done_when: "install-provider.sh, apply-permission-inventory.sh, the denokcp install and apply.sh each reproduced object for object; no kubectl or yq anywhere."
risks: ["apply-permission-inventory.sh's yq subject/grant split (:52-53) and DRY_RUN server mode must be kept."]
status: done
evidence:
- id: M2-HOSTPROCESS-KIND
milestone: M2
title: "HostProcess objects reconciled by the in-binary supervisor onto a placement backend - each application instance's kcp as its own service manager"
depends_on: [M2-HOST-BACKEND-PROC, M2-HOST-CONFORMANCE, M2-APPLY, M2-BOOTSTRAP-LAYER]
decision: >
Principle (user 2026-10-12): HostProcess is how an application's components become objects of that
application's own kcp; a supervisor places only its own instance's HostProcesses and never reads another
instance's. Decided by the user 2026-10-12 (dev run mode is KCP-NATIVE NOW, overriding dev-run-mode.md's "phase 2"):
long-lived host processes are kcp objects - a HostProcess kind that generalises DenoPod - reconciled
by an in-binary supervisor; ComputeVMs, kcp slots and tenant cluster members are placed through it;
systemd-user is the production placement backend and proc the dev one. Decided by Claude, user may
veto (dev-run-mode.md 3(b)): Kind HostProcess (socialweb.computer/v1alpha1, export hostprocesses,
apigen-generated, OS-neutral) spec {asset (name + sha256 from versions.json) or exec path, argv, env,
workDir, output {stdout, stderr files, cap}, limits, restartPolicy (Never | OnFailure | Always),
stop {signal, graceSeconds}, after/wants (other HostProcess names), slice, privateHostname, fdStore,
readiness {exec | tcp | http | pgx}, placement (supervisor | bootstrap)} and status {phase (Pending,
Starting, Running, Stopping, Stopped, Failed, Lost), lostReason (HostRebooted, UnitGone, UnitFailed,
OOMKilled, IdentityMismatch, ManagerRestarted), runtime {backend, unit, invocationID, pid, startTicks,
hostBootID, manager}, lastResult {result, exitCode, signal, at}, conditions (Ready, LimitsEnforced)}.
The supervisor is a pure reconciler in the deno-kcp denopod.go:105-140 style (Observed -> Result{Phase,
Ops}) with hostsvc.Manager as its placement port, run inside the core (serve gateway kcp, run bidder,
the dev core); it is the ONLY caller of Manager.Start outside the bootstrap layer. Rules: adopt by unit
name + InvocationID, never re-create; identity mismatch on an active unit = Lost and stop the unit at
once (the user's 2026-10-11 VM rule, generalised); a dial/auth/timeout error is Unknown, never Lost; an
object whose status.runtime.backend differs from the running backend is Lost(IdentityMismatch) naming
both, never adopted; LimitsEnforced=False on proc. placement bootstrap = a read-only MIRROR of a
bootstrap-layer unit (M2-BOOTSTRAP-LAYER): status only, deleting it never stops the unit. Owners:
Postgres (M2-POSTGRES), OpenBaoServer (M2-OPENBAO-SERVER), each cluster member (M3-CLUSTER-UNITS) and
each ComputeVM (0007 M2-CONTROLLER) own HostProcess children (ownerReferences, created in the owner's
workspace); the kcp slots are bootstrap mirrors, and a nested instance's kcp slots are ordinary
HostProcess objects in its parent's kcp (M4-DEV-PARENT-KCP). DenoPod stays its own kind for TypeScript
services (children of the core that die with it, plan 0008) and HostProcess reuses its reconciler
shape; whether DenoPods become HostProcesses is not decided here.
files: ["api/v1alpha1/hostprocess_types.go", "internal/apigen (schema, export)", "deploy/sets.yaml (hostprocesses set)", "internal/hostprocess/ (pure reconciler, supervisor loop, mirrors)", "internal/stack/* (supervisor started in every core role)"]
test_first: >
Offline (fake kcp, fake Manager): one test per reconciler row, plus
TestAHostProcessIsPlacedOnTheBackendItsStatusNames, TestAnObjectPlacedByAnotherBackendIsLostNotAdopted,
TestIdentityMismatchOnAnActiveUnitStopsItAtOnce, TestAnUnreachableManagerIsUnknownNeverLost,
TestDeletingAMirrorRecordDoesNotStopTheBootstrapUnit, TestOnlyTheSupervisorAndTheBootstrapLayerCallManagerStart
(scans non-test Go), TestTheHostProcessSchemaHasNoBackendEnum. Through the conformance harness on both
backends (stand-ins): TestTheSameObjectsReconcileIdenticallyOnBothBackends,
TestACoreRestartAdoptsEveryHostProcessUnit (InvocationIDs unchanged). Red first.
done_when: "tests green on proc (check.sh --fast) and systemd-user (hat-8); `kubectl get hostprocesses` on a dev instance lists the bootstrap mirrors and a stand-in object's runtime."
risks:
- "kcp down = no reconcile; units keep running (they are not children of the supervisor) and are adopted when kcp returns. A crash-looping unit with restartPolicy Always is restarted by the backend (systemd Restart= / devsup), not by the reconciler."
- "APIResourceSchema.spec is immutable; the first HostProcess revision is reviewed field by field against the two working backends before it is applied."
status: done
evidence:
- id: M2-APP-CONTROLLER
milestone: M2
title: "Each application is ONE root object in its own kcp, rendered by the core into children; a MarketNetwork composes child applications, each with its OWN kcp"
depends_on: [M2-APPLY, M2-TEMPLATE, M2-HOSTPROCESS-KIND, M2-BOOTSTRAP-LAYER, "0008:M2-SERVICE-TEMPLATE", "0008:M2-DENOPOD-LISTENERS", "0008:M2-MARKET-NETWORK", "0007:M1-MARKET-IDENTITY"]
decision: >
Fifth fold 2026-10-12, from multi-server-to-kcp.md (sections 4-6, gap G1) with the user's answers to its
section 7 (user-decisions.md 2026-10-12; principle.decomposition). An application instance (a bidder
handle, a requester invocation, the gateway, a market network / market network) = its own kcp + manifest set +
profile; its state is objects and its components are children. Kinds MarketBidder, MarketRequester,
Gateway and MarketNetwork (socialweb.computer/v1alpha1, apigen-generated) with spec = every option
of the application's CLI today (hono-bidder's 54 cli-args-env options, request-vm-ssh's 40, the door's)
except the network options, which arrive as the MarketNetwork value (0008 M2-MARKET-NETWORK) + profile
+ an identity reference (MarketIdentity, 0007 M1-MARKET-IDENTITY), and status = phase, conditions,
did, and the children's reported listeners aggregated (0008 M2-DENOPOD-LISTENERS). internal/appctl is a
pure reconciler (the deno-kcp denopod.go Observed -> Result style) that renders
deploy/apps/<app>/<profile>.values.yaml through M2-TEMPLATE and the service DenoPod template (0008
M2-SERVICE-TEMPLATE) into owned children (DenoPods, HostProcesses, an OpenBaoServer, MarketIdentity;
ownerReferences, field manager appctl); a child the active profile omits is deleted (market network -> prod-qr
deletes the bidder's PDS DenoPod). It runs in every core (run bidder, run request, serve gateway kcp,
dev). Option coverage (decided by the user 2026-10-12: every option stays): deploy/apps/option-map.yaml
maps every cli-args-env option of hono-bidder and request-vm-ssh to a root-object field or a profile;
the only accepted removal reason is legacy-transport (a non-iroh guest transport, 0008
M1-STRIP-LEGACY-TRANSPORTS, e.g. request-vm-ssh --user-data-transport); state-path options (key file,
repo KV, session file, port file) map to the instance's state dir or a one-time import into its vault
(keys and sessions are never kcp objects, decided by the user 2026-10-12); port options map to
status.listeners and accept only 0 or an entry of deploy/edge-ports.yaml (principle.port0_rule).
STRICT composition (decided by the user 2026-10-12, overriding the research's nested-workspace market network):
the MarketNetwork reconciler runs the shared atproto services as its own DenoPods - plc, relay,
dispatcher, dispatcher-tls, pds-<account> where a profile names one, qr (did-key-associator's
hono-backend, so QR sign-in runs inside a market network with nothing external) and the verifier DenoJob - and
composes child applications from spec.apps[] {name, kind, profile, values}: for each child it renders
the child's bootstrap layer (M2-BOOTSTRAP-LAYER's table for that role, rendered as HostProcess objects
in the network's kcp: kine-sqlite, one kcp slot on port 0, the child's core), so every child runs its
OWN kcp; it waits for that kcp's /readyz and server-side applies the child's root object into the
child's base workspace with the MarketNetwork value (the network's own status URLs) and any peer value
the network profile names (a bidder child's DID onto a requester's allow list) copied onto it; it reads
back only the child's root status. That copy is the ONLY cross-application discovery (decided by the
user 2026-10-12): no application reads another's objects, and the DNS shim resolves only its own
application's names. Guests still come up only through a requester child's ComputeContract ->
runComputeContract -> RFP -> bid -> accept -> cloud-init against a bidder child (RFP spine). Decided by
Claude, user may veto: the option-map shape, rendering a child's bootstrap layer as HostProcess
objects in its composer's kcp (the shape M4-DEV-PARENT-KCP later generalises for worktrees), the qr
DenoPod.
files: ["api/v1alpha1/app_types.go (MarketBidder, MarketRequester, Gateway, MarketNetwork)", "internal/apigen (schemas, exports, TS wire types)", "internal/appctl/ (reconciler, MarketNetwork composer)", "internal/bootstrap (a role's table rendered as HostProcess objects)", "deploy/apps/<app>/<profile>.values.yaml", "deploy/apps/option-map.yaml", "deploy/sets.yaml"]
test_first: >
Offline (fake kcp): TestSwitchingABidderFromNetworkToProdQrDeletesItsPdsDenoPod,
TestDeletingTheRootObjectRemovesEveryComponent, TestRootStatusCarriesEveryChildsReportedListener,
TestAnAppsCredentialIsRefusedByAnotherAppsKcp,
TestEveryHonoBidderAndRequestVmSshOptionMapsToAFieldAProfileOrALegacyTransportRemoval (reads both
cli-args-env.ts through deno when the org root is present, else a committed snapshot that check.sh
drift-checks), TestNoAppVerbOpensAListenerOrSpawnsDeno (scans the application verbs' Go: only the
core's reconcilers start listeners or deno; M1-CLI's contract). Behind livegate on proc (stand-in
children, real kcp/kine assets, no guest): TestEveryChildApplicationInAMarketNetworkHasItsOwnKcp (no
two children share a kcp root, slot or front - the "no two applications share a kcp" test),
TestANetworkBidderLearnsThePlcOnlyFromItsNetworkValue, TestTheShimNeverResolvesAnotherApplicationsName.
Red first.
done_when: "tests green; `kubectl get marketbidders` on a dev instance shows did and listeners; a MarketNetwork with two bidder children shows two distinct child kcps (roots, ports) with readings recorded."
risks:
- "Each child kcp costs ~0.4-0.5 GB and 6-9 s cold start (kcp-spike.md); the user chose strict over the cheaper nested shape. A market network's memory (network + N bidders + gateway + requester invocations) is measured here and bounds how many market networks one host runs (0008 M1-MARKET-NETWORK-PARALLEL's free-memory refusal)."
- "APIResourceSchema.spec is immutable; the four kinds' first revision is reviewed against option-map.yaml before it is applied."
status: done
evidence:
- id: M3-FOLD
milestone: M3
title: Fold every host-side Go binary into verbs; delete the resident VM wrapper
depends_on: [M1-CLI]
decision: >
Fold: socialweb-gateway, socialweb-clusterprovider, socialweb-vault, socialweb-computer-kcp-check-host;
socialweb-cloud-controller-manager and socialweb-tenantcredential as `internal <verb>` re-execs of the
resolved bin/<sha>/ path (M1-INSTALL). Decided by the user 2026-10-11 (no socialweb code in a VM's
process tree): socialweb-nodeboot and its namespace verb are DELETED, not folded; their one-shot prep
moves into the ComputeVM controller and the tiny socialweb-vmexec asset (plan 0007 M2-VM-RUNTIME).
Decided by the user 2026-10-11 (guest images built off the bidder) and 2026-10-12 (by a build-host
script on any Linux host; CI shells out to it later, no CI service now): socialweb-nodeimage leaves the
host binary; it is run by scripts/build-guest-image.sh (plan 0007 M1-GUEST-IMAGE) with a `dev`-tag
verb for local builds.
socialweb-nodeinit is a GUEST binary and belongs to the image pipeline, not the host asset set. Built
separately but carried inside the binary as download assets: socialweb-credential (four platforms),
socialweb-meshd (two arches). socialweb-apigen stays a go generate tool. Folded cmd/ keep a one-line
main only while the container build still compiles them; M5-RETIRE deletes both together.
files: ["cmd/socialweb-computer", "cmd/socialweb-*/main.go (shims)", "internal/clusterprovider (exec path defaults)", "cmd/socialweb-nodeboot (deleted with plan 0007 M2-VM-RUNTIME)"]
test_first: >
TestEveryFoldedBinaryIsAVerbWithTheSameUsageAndExitCodes, TestTheClusterProviderSpawnsTheCCMAsAReExecOfItsResolvedPath,
TestNoNodebootOrNodeimageVerbInAReleaseBuild; red first.
done_when: "the host install carries socialweb-computer alone; tests green."
risks: ["A re-exec'd verb inherits the parent's env; the CCM's curated env is rebuilt explicitly."]
status: done
evidence:
- id: M3-CLUSTER-UNITS
milestone: M3
kcp_use: "traditional-exception (virtual Kubernetes clusters project; principle)"
title: Each cluster is a systemd slice of member units in the gateway's user manager, ordered after its postgres unit; a core upgrade restarts none of them
depends_on: [M2-HOST-UNITS, M2-POSTGRES, M1-INSTALL, M2-HOSTPROCESS-KIND]
decision: >
Third fold 2026-10-12 (decided by the user: tenant cluster members are placed through HostProcess; dev
is kcp-native now): every member below is a HostProcess child of the cluster (placement supervisor)
reconciled onto systemd-user in production and proc in dev; the slice is a systemd slice on
systemd-user and a label on proc, and teardown uses Manager.StopSlice on both; port attribution reads
unit membership through the Manager (cgroup.procs or devsup's process groups), never cgroup files
directly. Decided by Claude, user may veto (dev-run-mode.md 5).
Decided by the user 2026-10-11: cluster processes live in the gateway account's user manager; KEEP the
Postgres CRD and add explicit ordering and readiness between a cluster's postgres unit and the rest of
the cluster for start, adoption and restart. Decided by the user 2026-10-12 (replaces the 2026-10-11
"one cluster-group unit with a resident supervisor"): each cluster is a systemd SLICE of member units,
so blue/green control-plane upgrades (M3-APISERVER-FRONT, M3-CP-SLOTS) need no resident supervisor and
a core upgrade restarts nothing. Decided by Claude, user may veto (control-plane-handover.md 8.1): the
slice is socialweb-cluster-<key>.slice (the key unit-name-escaped, so a dash in it adds no slice level)
holding one transient SERVICE per member, named socialweb-c-<key>-<member>.service: front
(M3-APISERVER-FRONT), kine@<slot> and k3s@<slot> (slot a until M3-CP-SLOTS adds b), bao, link-apiserver,
link-openbao, ccm, eso, esowh. Ordering: kine@<slot> and bao are After= and Wants=
socialweb-pg-<key>.service - NOT Requires or BindsTo, so a postgres crash or restart does not
cascade-stop the control plane; k3s@<slot> is After=/Wants= its kine@<slot>; ccm, eso and esowh
After=/Wants= the front (or k3s@a before the front exists). Every member Restart=on-failure (links
Restart=always, M3-LINK-HANDOVER), PropExecStart(argv, false), CollectMode=inactive. Members whose
code is the core binary (ccm, front) name the resolved bin/<sha>/ path, so a core upgrade never
changes code inside a running member. The reconcile loop is the supervisor (ARCHITECTURE section 5
already says so); there is no `internal cluster-group` verb and no Pdeathsig inside the cluster (units
own lifetime); plan 0005 M6-LIMITS' guarantee is restated as "every member is a unit of its cluster's
slice". A host-local record <dataDir>/controlplane.json {activeSlot, slots {a, b: k3s version, kine
version, ports, unit names, InvocationIDs}, front {port, backend}} is written by the reconciler and read
at adoption before kcp is consulted. Start (fresh): clusterprovider ensures the Postgres object ->
Ready{port, instance} -> starts the members in order with the pg port in their params; kine and bao
still gate on the PostgresUp probe (ErrPostgresUnavailable kept). Adoption (core restart or upgrade)
per member, by unit name + InvocationID + controlplane.json: active -> adopt, no write; missing or
failed -> start that member only (record Result first, then ResetFailedUnit); pg missing -> start pg,
members reconnect; everything missing (reboot, or the user manager restarted) -> cold start in order.
A bao whose InvocationID changed is unsealed on the next pass (M2-OPENBAO-SERVER rule); its behaviour
on losing its postgresql storage connection is measured here. Teardown: stop the slice, wait every
member inactive, then stop pg. Port leases after adoption are attributed from each member unit's
cgroup.procs, not pid cmdlines (internal/portscan), so a restarted provider never re-leases a live
member's port. Version changes of members are M3-CP-SLOTS (k3s, kine), M3-LINK-HANDOVER (dumbpipe),
M2-POSTGRES (postgres); ccm, eso and esowh roll by stop/start (no election, seconds without their
loops). ARCHITECTURE section 5 "The per-cluster process group" is rewritten in the same change.
files: ["internal/clusterprovider (member units, adoption, controlplane.json)", "internal/procgroup (retired for clusters or reduced to the order table)", "internal/hostsvc (slices)", "internal/portscan (cgroup attribution)", "cmd/socialweb-computer units roll", "docs/ARCHITECTURE.md section 5"]
test_first: >
Offline (fake hostsvc): TestEveryMemberIsAUnitOfItsClustersSliceWithWantsNotRequiresOnPostgres,
TestAdoptionStartsOnlyTheMissingMembers (all active; pg missing; one member failed; everything missing),
TestTeardownStopsTheSliceBeforeItsPostgres, TestALeaseHeldByALiveMemberUnitIsNeverReleased,
TestNoClusterMemberIsSpawnedWithPdeathsig, TestADashInTheClusterKeyAddsNoSliceLevel,
TestEveryMemberIsAHostProcessChildOfItsCluster; the stand-in live tests below run on both backends
(proc in a dev instance, systemd-user on hat-8). Behind livegate
on hat-8 with stand-ins (real postgres, kine, k3s --disable-agent, bao; no guest):
TestACoreRestartAndUpgradeLeaveEveryClusterMemberWithItsInvocationID (a zero-node market-shaped
cluster's /readyz on status.URL polled every 200 ms across `systemctl --user restart` of the gateway
unit AND an `upgrade` to a different binary: no failed poll), TestAPostgresRestartDoesNotStopItsClusterMembers,
TestAKilledMemberIsRestartedBySystemdAndAdoptedNotDuplicated. On the market network (node joined via
runComputeContract): TestAGatewayUpgradeKeepsAMarketNodeReady (no Ready transition, no claim write, no
vm.delete). Red first.
done_when: "green on hat-8 and the market network; across the restart and the upgrade the provider logs adoption, not respawn, for every member of every cluster."
risks:
- "Changes plan 0005 M6-LIMITS's 'every member dies with the provider' to 'every member is a unit of its cluster's slice'."
- "A user-manager restart, linger removal or reboot removes every transient unit: every cluster cold-starts; 0007:M3-NODE-LOSS-CLOCK keeps that outage off tenants' nodes."
- "The market network's gateway must run this tree's clusterprovider in the user manager; if the market network cannot yet host the member units, the market network test waits for 0008:M4-MARKET-NETWORK-PORT and that is recorded."
status: done
evidence:
- id: M3-APISERVER-FRONT
milestone: M3
kcp_use: "traditional-exception (virtual Kubernetes clusters project; principle)"
title: A per-cluster L4 front owns the apiserver port; k3s moves behind it on slot ports
depends_on: [M3-CLUSTER-UNITS, M2-KCP-FRONT]
decision: >
Decided by the user 2026-10-12: a per-cluster TCP front owns the existing apiserver port, so guest
tickets, node secrets, kubeconfigs and status.URL are unchanged across control-plane swaps; kcp's front
and the tenant fronts are the same code. Decided by Claude, user may veto (control-plane-handover.md 4,
Q-H2): the front type, its control socket and fd-store handling are M2-KCP-FRONT's
internal/apiserverfront; this item adds the tenant use: `internal apiserver-front --cluster <key>`
runs as member unit socialweb-c-<key>-front and binds
BindAddress:<k3s-secure lease port> - the number k3s binds today - so dumbpipe's --host, every node
secret's apiserverPort, status.URL, every kubeconfig and the advertise port keep their value. k3s slots
take new leases k3s-slot-a and k3s-slot-b (width 2 each) plus per-slot controller ports and a per-slot
kine lease. The backend is chosen per NEW connection from a state the reconciler sets over a control
socket in the cluster's data dir (path < 108 bytes); open connections stay on their backend until it
closes them (that is the drain). Pure TCP passthrough: TLS stays end to end with k3s (its serving SANs
already cover 127.0.0.1 and the advertise address). The front never fails over by itself (auto-failover
would hide a dead green). Its listening socket is kept in the backend's fd store through
UnitSpec.FDStore (systemd: FDSTORE=1, FileDescriptorStoreMax=1; proc: devsup holds it, decided by
Claude 2026-10-12, user may veto) so a front restart refuses no connection and only resets open ones;
check-host probes fd-store support in the user manager (M2-HOST-CONTRACT). Every slot runs --advertise-address
<identical> --advertise-port <front port>; slot listen ports are never advertised; the provider's admin
client, the CCM, ESO and the kcp mount (status.URL) name the front port. AssertMarketAdvertiseInvariant
(internal/clusterprovider/steps.go:164-174) is rewritten: advertise port == front port, advertise
address identical across slots, no slot listen port in the kubernetes EndpointSlice. ARCHITECTURE
section 8 (Ports) is updated in the same change.
files: ["cmd/socialweb-computer internal apiserver-front (--cluster)", "internal/clusterprovider (front member, slot leases, AssertMarketAdvertiseInvariant)", "internal/portallocator (k3s-slot-a/b, per-slot purposes)", "docs/ARCHITECTURE.md section 8"]
test_first: >
Offline: TestTheAdvertiseInvariantRefusesASlotListenPort (the front's own per-connection and
no-self-failover tests are M2-KCP-FRONT's).
Behind livegate on hat-8 with stand-ins (two k3s --disable-agent on their own kine over one postgres; no
guest): TestTheFrontSwitchFailsNoPoll (authenticated /readyz every 200 ms through the front across
switch + SIGTERM of the old slot: 0 failures - control-plane-handover.md 2.1 as a test),
TestEverySlotAdvertisesTheFrontPair (EndpointSlice == (advertise address, front port) with one, two and
the remaining slot), TestAFrontRestartRefusesNoConnection (fd store). On the market network (node via
runComputeContract): TestAMarketNodeStaysReadyAcrossAFrontSwitch (the EndpointSlice never empties, no
Ready transition). Red first.
done_when: "green on hat-8 and the market network; the front unit's port equals the k3s-secure lease the cluster had before the change."
risks:
- "User-manager fd-store support is unverified on the target systemd; without it a front restart has a refused-connection window of its restart time, recorded."
- "On graceful stop the lease endpoint reconciler may RemoveEndpoints for the shared IP; not observed (k3s exits in 0.1 s), asserted on the market network."
status: done
evidence:
- id: M3-CP-SLOTS
milestone: M3
kcp_use: "traditional-exception (virtual Kubernetes clusters project; principle)"
title: Blue/green k3s+kine slots per cluster on the same postgres, swapped behind the front, idle first
depends_on: [M3-APISERVER-FRONT, M2-POSTGRES]
decision: >
Decided by the user 2026-10-12: a new k3s+kine slot starts beside the old on the same postgres, the
front swaps, the old stops; market kube-scheduler leader election ON with a short lease; the k3s
hostname leader-lease problem solved; idle-first swaps. Procedure (control-plane-handover.md 5(a)),
decided by Claude, user may veto: (1) preconditions: target k3s minor in {active, active+1}; target >=
every joined guest's minor and <= guest minor + 3; cluster Ready; EndpointSlice == (advertise address,
front port). (2) Idle first (Q-H4, decided by Claude, user may veto): the swap waits while any
non-kube-system pod is Running on a market node, up to a bounded deferral (default 30 min), then
proceeds as blue/green; a cluster at zero nodes swaps at once. (3) Minor bump: pg_dump of the k3s and
openbao databases to <dataDir>/backups/<ts> first (exit 0, size > 0); minors one at a time; rollback
only within the same minor (Q-H8, decided by Claude, user may veto). (4) Lease the green ports; start
kine@green (target kine) on the same DSN (log 'tables and indexes are up to date'; no lock wait > 2 s
on the active kine); start k3s@green with identical cluster flags, the same token file, the same
advertise pair and its own data dir; gate: green CA hash == active CA hash, /readyz 200 on the green
port with the admin cert, first kube-system Deployment listed. (5) Persist controlplane.json
(activeSlot=green) BEFORE the front switch is acknowledged; switch; immediately stop k3s@blue (exits in
~0.1 s) and kine@blue. (6) Watch 60 s: kcm/scheduler/ccm leases held by green, agent tunnels back (an
exec into one pod per node), ClusterLinks Listening, every held Node Ready with no transition. (7) Keep
blue's data dir and unit definition for a same-minor rollback (the same procedure reversed). Scheduler:
internal/clusterstack/scheduler.go renders leaderElect true with leaseDuration 4s, renewDeadline 3s,
retryPeriod 1s (reverts 1cbf945's false; first-pod delay after scale from zero up to ~4 s, measured).
k3s hostname lease: wrangler's kube-system/k3s lease identity is os.Hostname(), so each k3s slot runs
with its own UTS hostname swc-<key12>-<slot>, through UnitSpec.PrivateHostname: systemd
ProtectHostname=private:<name> on the slot unit where the host's systemd supports it in a user manager,
else - and always on proc - the one-shot `internal ns-exec` (M2-HOST-BACKEND-PROC) that enters a
user+UTS(+mount) namespace, sethostname()s, optionally bind-mounts <state>/rancher over /etc/rancher
and execve's k3s (no resident code); the mechanism is measured in this item and probed by check-host.
k3s's /etc/rancher/node/password (the --disable-agent invariant): production gets the root-owned file
from `install gateway` (M1-INSTALL); dev gets a per-instance file bind-mounted by ns-exec in the slot's
user+mount namespace, no root (decided by the user 2026-10-12). Leader election here (scheduler and
k3s) conflicts with plan 0010 G4-SCHED-SINGLE-SERVER's proposed refusal of leaderElect false; flagged in
NEXT.yaml, 0010 R0-REASSESS resolves it. A kine bump is its own green slot; one whose release
notes name a schema migration (ACCESS EXCLUSIVE) gets a stand-in test with old+new kine on one database
before it ships. Triggered by `units roll cluster <key> --k3s <version> [--kine <version>]` and by the
reconciler when versions.json pins change.
files: ["internal/clusterprovider (slots, swap procedure, idle check)", "internal/clusterstack/scheduler.go (leaderElect)", "internal/clusterstack (per-slot argv, hostname)", "cmd/socialweb-computer units roll cluster", "internal/versions/versions.json (k3s and kine pins, skew rule)"]
test_first: >
Offline: TestASwapIsDeferredWhileWorkRunsUpToTheBoundThenProceeds, TestAZeroNodeClusterSwapsAtOnce,
TestAMinorSkipIsRefused, TestATargetOlderThanAGuestOrMoreThanThreeMinorsNewerIsRefused,
TestControlplaneJsonIsPersistedBeforeTheFrontSwitchIsAcknowledged, TestTheMarketSchedulerElectsWithAShortLease,
TestAMinorBumpTakesAPgDumpFirst. Behind livegate on hat-8 with stand-ins (no guest):
TestAGreenSlotSharesCAAndTokenWithBlue, TestTwoSlotsOnOneHostNeverBothHoldTheK3sLease,
TestTheSchedulerIsNotDoubledDuringOverlap (two slots and a fake Node: one kube-scheduler lease holder,
one bind per pod), TestAPatchSwapFailsNoPollThroughTheFront. On the market network (node via
runComputeContract): TestABlueGreenSwapKeepsAMarketNodeReadyAndExecWorksAfter (no NotReady, no claim
change, no vm.delete; EndpointSlice never empties; exec into the node's pod succeeds within N s of the
swap, N = the measured agent tunnel reconnect, recorded). Red first.
done_when: "green on hat-8 and the market network; a k3s patch swap's readings recorded (poll failures 0, lease handover seconds, tunnel reconnect seconds, first-pod delay with the 4 s lease)."
risks:
- "k3s's own wrangler lease is 45 s; after blue stops, green's k3s controllers (addons, helm) may wait up to 45 s unless blue releases on cancel - measured."
- "exec/attach/log streams through blue reset on every swap; that is why swaps are idle-first."
- "If neither hostname mechanism works in the user manager, this item is blocked and the user is asked; switch-then-stop alone is not accepted as solving the lease collision."
status: done
evidence:
- id: M3-LINK-HANDOVER
milestone: M3
kcp_use: "traditional-exception (virtual Kubernetes clusters project; principle)"
title: A cluster link keeps its iroh key across restarts and replacements; a dead link is restarted
depends_on: [M3-CLUSTER-UNITS]
decision: >
Decided by the user 2026-10-12: the link keeps its iroh key (cluster vault KV socialweb-links/<service>,
CAS-0, never regenerated); the dumbpipe duplicate-endpoint behaviour is measured. Decided by Claude, user
may veto (control-plane-handover.md 4): a link is never re-pointed for a k3s swap (its target is the
front). Link units Restart=always with the same IROH_SECRET, closing ARCHITECTURE section 5's stated
limit that a link that dies is not restarted. A dumbpipe pin bump starts the new listener with the same
key on the link's second UDP lease (each link gets two), waits for its printed ticket naming the key,
then stops the old one. Measured on the market network for the pinned dumbpipe v0.39.0: whether iroh-relay promotes
the newer connection of a duplicated endpoint id (SameEndpointIdConnected), how long a guest's
connect-tcp takes to reach the new process, and the discovery record's last-writer-wins delay. If the
measurement shows dials lost, the replacement falls back to stop-then-start (seconds; the loss clock is
rebased by 0007:M3-NODE-LOSS-CLOCK) and that is recorded here.
files: ["internal/clusterprovider/links.go (link units, second UDP lease, replacement)", "internal/clusterlink", "internal/portallocator (second link lease)"]
test_first: >
Offline (fake vault, fake hostsvc): TestALinkUnitRestartsWithTheSameKey, TestTheReplacementListenerTakesTheOtherUDPLease.
Behind livegate on hat-8 with two real dumbpipe listeners and a dumbpipe connect-tcp client process (no
guest): TestTwoListenersWithOneKeyServeAndTheNewerTakesNewDials. On the market network (node via
runComputeContract): TestALinkReplacedWithTheSameKeyKeepsTheNodeReady (ClusterLink stays Listening,
ticket byte-identical, the node's Lease renewal gap recorded). Red first.
done_when: "green; the relay-promotion and discovery timings for the pinned dumbpipe recorded with UTC timestamps."
risks: ["n0's public relays must run the promotion behaviour; the measurement names the relay used."]
status: done
evidence:
- id: M3-BAO-SYNC
milestone: M3
kcp_use: "traditional-exception for the cluster OpenBaos (virtual Kubernetes clusters project); the root vault half is the gateway application's own state (principle)"
title: Every OpenBao this binary runs holds exactly the backends, roles, policies and KV entries live objects ask for
depends_on: [M2-OPENBAO-SERVER, M3-CLUSTER-UNITS, "0005:M3-BAO-SYNC"]
decision: >
User asked 2026-10-07 that the plans keep OpenBao backends and roles in sync with what is active;
plan 0005 M3-BAO-SYNC found 5 jwt-* mounts in one cluster OpenBao, one per issuer a bidder ever had,
only one serving a live node. Carried to every OpenBao this plan introduces or moves, decided by
Claude, user may veto. One reconcile library, internal/baosync: desired state is a pure function of
live objects; the reconciler lists actual auth mounts, roles, ACL policies and KV paths under the
prefixes it owns, creates what is missing, deletes what no live object asks for, and disables a
jwt-* auth mount left with no role. It runs every pass, at controller startup, at adoption of a
server or cluster's member units after a restart or upgrade, and whenever a vault unit's InvocationID changed
(it restarted while the core was down; M2-OPENBAO-SERVER unseals first), so a crash between a release
and its revoke is repaired by the next pass. Owned prefixes only: a mount, policy or path outside them
is reported, never touched. Applied to: (1) each cluster's OpenBao, a member unit of the cluster's slice
(M3-CLUSTER-UNITS): the plan 0005 M3-BAO-SYNC rule moves into internal/baosync unchanged (per live
claim with a grant: mount jwt-<sha12(issuer)>, role <node>, policy node-<node>, KV
socialweb-nodes/<node>; plus the ClusterLink keys in socialweb-links and the ESO cert role, derived
from the cluster's live ClusterLinks and its ESO pair; plus each live claim's door secrets-ingress key
socialweb-door/claims/<claim>, 0007:M6-CLAIM-RESUME); (2) the gateway's root vault (OpenBaoServer
object): internal/vaultlayout's static layout plus per-cluster and per-account entries derived from
live Cluster and AtprotoIdentity objects; an entry for a deleted cluster or revoked identity is
removed. The layout's static paths (vaultlayout.go Path* constants) are owned too: a path the layout
no longer names is deleted only by a layout change, and the reconcile's report lists it until then.
files: ["internal/baosync/ (Desired, Actual, Diff, Apply; fake OpenBao for tests)", "internal/clusterprovider (cluster vault reconcile moves onto baosync)", "internal/openbaoctl (root vault reconcile)", "internal/vaultlayout (owned prefixes declared)"]
test_first: >
internal/baosync TestAVaultSeededWithStaleMountsRolesPoliciesAndKVIsReducedToExactlyTheLiveSet (fake
OpenBao seeded with five jwt-* mounts, roles and policies for released nodes, a deleted cluster's
root-vault entries, and one foreign mount: after one pass only the live set remains, the foreign
mount untouched), TestAJwtMountWithNoRoleLeftIsDisabled, TestTheReconcileRunsOnAdoptionBeforeTheFirstPass,
TestTheReconcileRunsWhenAVaultUnitsInvocationIDChanged; behind livegate on a real bao:
TestARestartBetweenReleaseAndRevokeIsRepairedByTheNextPass. Red first.
done_when: >
Tests green; on hat-8 after a release and after an upgrade of the binary, `bao auth list`, `bao list
auth/<m>/role`, `bao policy list` and the owned KV listings in every cluster vault and the root
vault name exactly the live set (reading recorded).
risks:
- "A reconcile that deletes from a wrong desired set (for example a claim list that failed to load) would revoke live nodes; an empty or failed listing of live objects aborts the pass instead of reading as 'nothing is live'."
- "Plan 0005 M3-BAO-SYNC landed first; this item moves its code into internal/baosync without changing behaviour, proved by its existing test."
status: done
evidence:
- id: M3-KCP-ROLL
milestone: M3
title: "`units roll kcp` - a new kcp slot beside the old on the same store, the front flips, the old stops, the controller lease handed over"
depends_on: [M2-KCP-SLOTS, M2-KCP-PORT0, M2-KCP-FRONT, M2-KCP-STORE]
decision: >
Scope (decided by the user 2026-10-12, third fold): the blue/green procedure below is the GATEWAY's. On
a bidder, `units roll kcp` is the short restart of its single slot: copy the sqlite store (M2-KCP-STORE),
stop the slot (and the kine when its asset changes), retarget the per-owner asset link, start them on
the same root, wait /readyz on the slot's own port, then the ComputeVM controller re-adopts every VM by
the Verdict; VM units are never touched and no version-pair rule applies (a restart cannot overlap two
versions). `dev roll kcp` runs the gateway procedure inside a dev instance (proc backend).
Decided by the user 2026-10-12: kcp upgrades follow the tenant pattern - start the new kcp, switch our
own front over, stop the old; pg_dump before each kcp upgrade; lease handover = delete
kube-system/kcp-controllers after the old slot exits (no ReleaseOnCancel patch). Procedure, decided by
Claude, user may veto (kcp-blue-green.md 7.4): (1) refuse unless the version rule below allows the pair;
(2) pg_dump the kcp store (internal/pgunit; exit 0, size > 0); (3) copy the five shared files into the
idle slot's root; (4) start socialweb-kcp-gw-kine@<new> then socialweb-kcp-gw@<new> with the
target asset, read both ports (ListenPort); (5) wait /readyz == 200 on the NEW slot's own port with the
shard-admin token; (6) persist kcp-slots.json (active = new) BEFORE the front flip is acknowledged, then
flip; (7) SIGTERM the old slot (--shutdown-delay-duration=2s keeps it serving non-watch requests while
clients move) and wait for its exit; (8) delete Lease kube-system/kcp-controllers in system:admin with
the shard-admin context through the front; (9) check that a probe object reconciles (a throwaway
workspace Ready) within 10 s; (10) stop the old kine; keep the old root for a same-pair rollback.
Idle first: prefer a moment with no exec/attach stream through a mount (a bounded deferral, the
M3-CP-SLOTS rule). Version rule (kcp-blue-green.md 4 and 9 Q4, decided by Claude 2026-10-12, user may
veto): a same-asset roll (flags changed, binary rebuilt) is always blue/green; a roll between two kcp
releases is blue/green only when versions.json records the pair as storage-compatible - equal
served/storage versions of every kcp CRD and APIResourceSchema and equal phase-0/1 bootstrap schemas
(config/root-phase0, config/root), computed by scripts/kcp-compat.sh from the two release tags' source
on the build host when the new asset is pinned (kcp source is read there, never vendored) and recorded
as kcpCompat {from, to, storageCompatible, diff sha256}; any other pair is refused, and the only path is
`units roll kcp --planned-outage`: pg_dump, stop the old slot, start the new slot on the same store,
wait /readyz (kcp down for one start, ~9 s measured on a near-empty store), with the outage recorded.
Rollback: the same procedure toward the previous asset, allowed only for a storage-compatible pair;
otherwise restoring the pg_dump is the rollback of last resort. A kine bump for kcp rolls the same way
(new kine with the new slot).
files: ["internal/kcpserver/roll.go", "cmd/socialweb-computer units roll kcp", "scripts/kcp-compat.sh (build host, at pin time)", "internal/versions/versions.json (kcpCompat records)"]
test_first: >
Offline (fake hostsvc, fake kcp client): TestRollKcpRefusesAPairWithDifferentStorageVersions,
TestRollKcpTakesAPgDumpFirst, TestKcpSlotsJsonIsPersistedBeforeTheFrontFlip,
TestTheLeaseIsDeletedOnlyAfterTheOldSlotExited, TestReadinessIsReadOnTheNewSlotsOwnPortNeverThroughACopiedKubeconfig
(the spike's run-1 hazard), TestAPlannedOutageRollIsTheOnlyPathForAnIncompatiblePair. Behind livegate
on hat-8 with stand-ins (real kcp slots on kine -> postgres behind the front, no guest):
TestRollKcpLeavesNoControllerGapLongerThan5s (red first with step 8 removed: the measured 61 s),
TestRollKcpFailsNoReadyzPollThroughTheFront (shard-admin /readyz every 200 ms: 0 failures),
TestAnInformerThroughTheFrontSeesEveryEventAcrossARoll (watch reconnects to the new slot, no event
missed); bidder: TestABidderKcpRollIsARestartThatTouchesNoVMUnit (offline, fake Manager) and, behind
livegate with stand-in VM units, the restart's kcp-down seconds recorded. The gateway live tests run on
systemd-user and on proc (dev roll kcp). Red first.
done_when: "green on hat-8; a same-asset roll's readings on a prod-shaped object count recorded (poll failures 0, old-slot exit seconds, lease handover seconds, probe Ready seconds, UTC)."
risks:
- "No published kcp version-skew policy for replicas of one shard; the storage-compatibility record is our own check, and a pair it misjudges is caught only by the probe step - hence the pg_dump first."
- "N+1 rewrites kcp's own phase-0/1 objects at start while N still leads controllers for the seconds before the flip and the lease delete; harmless only for storage-compatible pairs, which is why other pairs take the planned outage."
status: done
evidence:
- id: M3-SERVE-GATEWAY-KCP
milestone: M3
title: socialweb-computer serve gateway kcp replaces container-entrypoint.sh, on the host, kcp first
depends_on: [M2-KCP-PORT0, M2-KCP-SLOTS, M2-KCP-FRONT, M2-KCP-STORE, M2-APPLY, M2-OPENBAO-SERVER, M3-FOLD, M3-CLUSTER-UNITS, M3-BAO-SYNC, M2-BOOTSTRAP-LAYER, M2-HOSTPROCESS-KIND, M2-APP-CONTROLLER]
decision: >
Third fold 2026-10-12: the kcp store, kine, front and active slot are the bootstrap layer
(M2-BOOTSTRAP-LAYER), already started by systemd from installed units (or by devsup in dev) before the
core runs; the core adopts them by name and only starts one that is missing, then writes their
HostProcess mirrors and starts the supervisor (M2-HOSTPROCESS-KIND), through which the root vault,
postgres and cluster members are placed. Runs on either backend with --host-backend required.
Order fixed 2026-10-11 and revised 2026-10-12 for the kcp slot route (decided by Claude, user may veto;
the 2026-10-07 order put the root vault, an object in kcp, BEFORE kcp): lock <state>/kcp/ -> the kcp
store (adopt or start socialweb-kcp-gw-pg, then the active slot's kine; M2-KCP-STORE) -> the kcp front on
127.0.0.1:6443 (adopt or start socialweb-kcp-gw-front; M2-KCP-FRONT) -> the active kcp slot Ready (read
kcp-slots.json, adopt the slot by name + InvocationID, or start slot a with the shared material;
/readyz on its own port; front backend set; M2-KCP-SLOTS) -> the provider, permission-inventory and
hostprocesses sets (through the front with the shard-admin context) -> the root OpenBaoServer Ready
(adopt or start the unit, unseal, apply-root) -> bootstrap-cluster when asked -> migrate -> the
gateway (in-process) -> the TLS front (Go crypto/tls proxy replacing socat, decided by Claude, user may
veto) -> the clusterprovider (in-process; adopts Postgres units, then each cluster's member units, M3-CLUSTER-UNITS)
-> the door (a deno child with today's argv and env until 0008:M3-DOOR-DENOPOD moves it into a DenoPod,
with explicit deno permissions, 0008:M2-SERVICE-TEMPLATE). kcp needs nothing from the vault and nothing
between bao and apply-root needs kcp (exploration F8), so only a fresh gateway instance's bring-up order
changes; in a running instance every unit is already up. Any in-process member failing ends the process with exit 1;
the core unit (Restart=always, StartLimit tuned, M3-HOST-INSTALL) restarts it; deno children die with
the core unit; units (kcp pg, kines, slots and front, vaults, stores, cluster member units) are adopted
on the next start rather than restarted. No container-era state dir is opened (fresh deploy, decided
by the user 2026-10-12).
Fifth fold 2026-10-12 (multi-server-to-kcp.md 4.5; decided by Claude, user may veto, within the user's
decisions of 2026-10-12): the gateway is a Gateway root object (M2-APP-CONTROLLER) with profiles prod
and market network. serve gateway kcp applies it after the sets; its app controller renders the door DenoPod (0008
M3-DOOR-DENOPOD), the root OpenBaoServer and one MarketIdentity (mode oauthDelegated) per door account,
whose sessions and the door's requester key live in the root vault (0008 M3-DOOR-STATE-VAULT). The
door's TEST_* seams (TEST_OWNER_SESSIONS_FILE, TEST_NODE_GUEST_HOST_ALIASES,
TEST_NODE_USER_DATA_BASE_FILE; door hono-socialweb-computer-k8s/mod.ts:53-67) become values of the market network
profile instead of env variables. In a market network the gateway is a child application of the MarketNetwork with
its own kcp. The tenant cluster objects (Cluster, Node, ClusterLink and the tenant side of NodeClaims)
stay the traditional exception (principle), unchanged by the application model.
files: ["internal/stack/gatewaykcp/", "cmd/socialweb-gateway and cmd/socialweb-clusterprovider bodies moved into run(ctx, cfg)"]
test_first: >
TestServeGatewayKcpStartsKcpBeforeTheRootVaultAndDiesWithAnyInProcessMember (fake members record the
order) and, behind livegate, TestServeGatewayKcpOnAnEmptyStateDirServesTheDoorAndABootstrapCluster
followed by TestServeGatewayKcpRestartedOnItsOwnStateDirListsTheSameWorkspacesAndClaims. Red first.
done_when: "on an empty state dir: door /api/config 200, the bootstrap cluster Ready, scale passes None nodes 0; after a restart the same workspaces, bindings and claims; no shell, kubectl, yq, socat or docker in the process tree."
risks: ["A panic in the core ends the gateway and the clusterprovider (one address space) but not kcp: its slot, kine, pg and front are their own units and keep serving kcp.socialweb.computer across the core's restart (route decided 2026-10-12)."]
status: done
evidence:
- id: M3-HOST-INSTALL
milestone: M3
title: The gateway as a USER unit of the lingering socialweb account, and an upgrade that is a content-addressed swap
depends_on: [M3-SERVE-GATEWAY-KCP, M1-INSTALL, M2-HOST-CONTRACT]
decision: >
Decided by the user 2026-10-11: the gateway lives in the user manager of a lingering account.
deploy/host/socialweb-computer-gateway.service is a USER unit first written and installed by `install
gateway` (M1-INSTALL, decided by the user 2026-10-12: the one explicit root step) and finalised here
under the socialweb account: ExecStart=<state>/bin/current/socialweb-computer serve gateway kcp,
Restart=always, StartLimitIntervalSec/Burst tuned, default KillMode=control-group (the 2026-10-07
KillMode=mixed is removed: transient units live in their own cgroups so ANY KillMode leaves them
untouched, and mixed would SIGKILL deno children with no graceful stop), Environment=XDG_RUNTIME_DIR
set by the manager. The account cannot bind below 1024 and does not need to: caddy (a system service,
public TLS only, never reconfigured by a kcp swap; install_trust false, M2-KCP-FRONT) stays in front of
the kcp front on 127.0.0.1:6443 and the door 8788. `upgrade` is M1-INSTALL's flow; units outlive the restart
(M2-HOST-UNITS, M3-CLUSTER-UNITS). scripts/deploy.sh is rewritten to copy the binary to the host and run
`upgrade`; no docker.
files: ["deploy/host/socialweb-computer-gateway.service", "scripts/deploy.sh"]
test_first: >
test/host_unit_test.go TestTheGatewayUserUnitVerifies (`systemd-analyze --user verify` clean) and
TestTheGatewayUnitHasNoKillModeMixed; red first.
done_when: "unit verifies; an upgrade on hat-8 swapped the binary with every cluster apiserver answering throughout."
risks: ["deploy.sh needs the no-password key (SSH_KEY=~/.ssh/id_ed25519_no_password); kept as today."]
status: done
evidence:
- id: M3-CP-CONTINUITY
milestone: M3
kcp_use: "traditional-exception (virtual Kubernetes clusters project; principle)"
title: Control-plane continuity across upgrades and restarts of the finished system, proved end to end on the market network
depends_on: [M3-CLUSTER-UNITS, M3-APISERVER-FRONT, M3-CP-SLOTS, M3-LINK-HANDOVER, M2-POSTGRES, M3-HOST-INSTALL, M3-KCP-ROLL, "0007:M3-NODE-LOSS-CLOCK", "0007:M6-CLAIM-RESUME"]
decision: >
Decided by the user 2026-10-12: continuity is planned only for upgrades of the finished system (no
cutover, no container); tenant control-plane upgrades are blue/green behind a per-cluster front. The
user's goal (2026-10-11): the gateway durably tracks which clusters own which market-rented VMs, so
after any restart or upgrade the nodes re-join their control planes, with zero downtime for clusters.
Ownership model, decided by Claude, user may veto (control-plane-handover.md 7): the owner of a VM is
exactly one NodeClaim (claim name = tenant Node name = guest vmName; market records are evidence, the
claim is the index); the owner of a control plane is the host (member units + controlplane.json +
leases.json; the Cluster CR is desired state and status mirror); join material is the cluster vault,
keyed by claim name, link keys never regenerated; liveness is the tenant Node judged by the rebased
loss clock (0007:M3-NODE-LOSS-CLOCK); a restart adopts, never reaps; the door resumes Bidding and
Provisioning claims (0007:M6-CLAIM-RESUME, decided by the user 2026-10-12). Upgrade kinds and the item
that makes each continuous: core binary (M3-CLUSTER-UNITS adoption; ccm stop/start; front restart from
the fd store), k3s and kine (M3-CP-SLOTS), dumbpipe (M3-LINK-HANDOVER), postgres minor and major
(M2-POSTGRES), bao pin (unit restart + unseal on the new InvocationID; joined nodes need no bao, new
joins and ESO wait), ESO pin (stop/start, no election), kcp (M3-KCP-ROLL: a new slot on the same kine
-> postgres store behind our own front on 127.0.0.1:6443, lease handed over; decided by the user
2026-10-12; caddy untouched). This item builds no new mechanism; it is the end-to-end proof.
files: ["internal/marketnetwork/live_test.go (continuity tests)", "docs/ARCHITECTURE.md section 5 (stated limits rewritten to the built design)"]
test_first: >
On the market network (node joined via runComputeContract, RFP -> bid -> accept -> cloud-init):
TestEveryUpgradeKindKeepsAMarketNodeReadyAndOwned (in sequence: core upgrade, k3s patch swap, dumbpipe
replacement, postgres minor restart, bao restart, a same-asset kcp roll; /readyz through the cluster
front and kcp's /readyz through the kcp front polled every 200 ms with the failure count recorded per
step and 0 for the core upgrade, the k3s swap and the kcp roll; no Ready transition, no
claim deleted, no RFP, no vm.delete) and TestAColdStartOfEveryClusterReleasesNoNodeWhosePathReturns
(`systemctl --user stop` of every member unit, then the gateway restarted: every cluster cold-starts,
the node is kept because its path returns within the rebased clock). Red first.
done_when: "both market network tests green with readings recorded (UTC); ACC6-UPGRADE and ACC6-CP-SWAP recorded."
risks: ["A user-manager restart or reboot of the gateway host is a cold start of every cluster; the rebased clock bounds its cost, it is not zero-downtime."]
status: done
evidence:
- id: M4-DEV-PARENT-KCP
milestone: M4
title: "Later: `dev host` - a parent application instance whose root:dev:<instance> workspaces hold each child instance's HostProcess objects (kcp inside kcp; composition by nesting)"
depends_on: [M2-HOSTPROCESS-KIND, M1-DEV-INSTANCE, M2-BOOTSTRAP-LAYER]
decision: >
Principle (user 2026-10-12): this is composition by nesting workspaces. The parent is itself an
application instance (its kcp, the dev-parent manifest set, a profile) whose application state is its
children's processes; each child keeps its OWN kcp (its slot is a HostProcess in the parent), and
children still talk to each other only over the atproto market protocol, never through the parent's
kcp. Decided by the user 2026-10-12: instances nest as kcp workspaces; one kcp per application instance
now, a parent kcp for nesting is fine as a later item. Decided by Claude, user may veto (dev-run-mode.md
4.4 N3): `dev host` runs one parent instance per user (its own bootstrap layer under its own devsup); each child
instance's bootstrap units, including its kcp slot, become ordinary HostProcess objects in the parent's
workspace root:dev:<instance> placed by the parent's supervisor on its devsup, so `kubectl get
hostprocesses -A` on the parent lists every worktree's processes; a dev-instance WorkspaceType with
limitAllowedChildren keeps instance trees from being nested into each other by mistake. Production is
unchanged (no parent kcp); the same shape is open later for tenant clusters (members as objects in the
cluster's home workspace).
files: ["cmd/socialweb-computer dev host", "internal/devinstance (parent mode)", "deploy/workspacetype-dev-instance.yaml"]
test_first: "TestAChildInstancesKcpSlotIsAHostProcessInItsParent, TestDeletingAChildInstanceWorkspaceStopsItsProcesses, TestAnInstanceTreeCannotBeNestedUnderAnotherInstance; behind livegate on proc. Red first."
done_when: "two child instances under one parent on hat-8 listed and stopped through the parent's kcp; readings recorded."
risks: ["A parent outage stops reconcile for every child (their units keep running under the parent's devsup)."]
status: done
evidence:
- id: M4-BUILD
milestone: M4
title: Reproducible build of the one binary per linux platform
depends_on: [M1-ASSETS, M3-FOLD, M2-KCP-PORT0]
decision: >
CGO_ENABLED=0 for the core, -trimpath, -buildvcs=true, -ldflags "-s -w -X version", GOTOOLCHAIN=local,
GOPROXY=off from the staged module cache (the Dockerfile's recipe, :32-45, run on the build host
instead), `-tags release` with only the target platform's asset file compiled in. Assets are pinned
inputs staged and checked against versions.json, never re-downloaded per build; kine (CGO_ENABLED=1)
is a separately built asset; kcp is the pinned upstream release binary, staged and checked by sha256,
never built here (route decided 2026-10-12, no patch). Two clean builds byte-identical; a
socialweb-computer.manifest.json (every asset, version, sha256, the TypeScript closure manifest from
plan 0008) printed identically by `version --json`. The cross-OS build/vet gate lives in M0-XOS-GATE;
darwin and windows release builds are plans 0012 and 0011.
files: ["scripts/build-socialweb-computer.sh", "deploy/check.sh (--full: build twice, compare, size gate)"]
test_first: "check.sh --full step reproducible-build red with a timestamp injected into one build, then green."
done_when: "two clean builds per linux platform byte-identical; manifest and sizes recorded."
risks: []
status: done
evidence:
- id: M5-FRESH-DEPLOY
milestone: M5
title: Prod gets the finished binary as a plain fresh deploy; Claude runs it
depends_on: [M3-HOST-INSTALL, M4-BUILD, M3-CP-CONTINUITY, "0007:M6-BIDDER-UPGRADE", "0007:M6-LIVE-RECOVERY"]
decision: >
Was M5-CUTOVER (container volume copy, rollback to the container, 7-day retire), void since 2026-10-12.
Decided by the user 2026-10-12: no backwards compatibility, migration, cutover, rollback or
prod-downtime planning until the finished solution exists; the gateway runs no containers; the finished
binary is deployed fresh. Decided by the user 2026-10-12 (replaces "the user runs it" of 2026-10-07):
Claude runs the deploy, start to finish, on both prod hosts, once the cut-over fixes found by
M3-CP-CONTINUITY, M6-BIDDER-UPGRADE and M6-LIVE-RECOVERY are on main, and reports the readings.
Order: (1) build the release binary from one commit (scripts/build-socialweb-computer.sh, both builds
byte-identical) and record its sha256; (2) gateway host: record what the old stack runs (units,
containers, listening ports, state dirs), stop the old gateway services so the edge ports are free,
and leave their state dirs on disk untouched (not read, not deleted; no rollback is planned); on
socialweb.computer also stop and disable the leftovers of another experiment (decided by the user
2026-10-12): socialweb-computer-pds (pds.socialweb.computer, 8789), pds2 (pds2 and test.pds2, 8794),
the town site (8791) and socialweb-computer-relay (the relay for that pds), and remove their caddy
sites; the gateway has ONE public name, k8s.socialweb.computer (decided by the user 2026-10-12: the
gateway.socialweb.computer site is removed, its 8788 web app route lives under k8s.socialweb.computer
next to the tenant apiserver front); socialweb-computer-ssh stays until 0008 M4-RUN-GATEWAY-SSH replaces
it; check the
relay config (an explicit relay list no longer adds public Jetstreams); (3) the fresh deploy below;
(4) hat-8: stop the old /opt/socialweb-computer-bidder bidder, then `install bidder --handle <the
bobvmbuilder handle>` and `run bidder --profile prod-qr`; a lost session is signed in again from the
account credentials on the host (~/.bobvmbuilder_bsky_social_source for the bidder,
~/.aliceoa_bsky_social_source for the requester alice; mode 0600, values never printed or stored
elsewhere; AGENTS.md "Prod deployments"), and the user is asked only if that fails; check that the guest DenoPod's did:key sign-in works on xrpc.fedproxy.com; (5) the finish line:
a tenant k3s cluster on the new gateway scales to zero and gets a node as a market RFP VM from that
bidder, then scales back to zero. Prepared here:
scripts/deploy.sh (host mode, M3-HOST-INSTALL) on the prod host runs `install gateway` (M1-INSTALL's
list: account, delegation drop-in, node password, unit files, linger enabled BEFORE the first start),
places the binary through `upgrade
--binary`, and starts the unit on an EMPTY state dir; serve gateway kcp brings up kcp, the root vault
and the door (M3-SERVE-GATEWAY-KCP) and the bootstrap cluster named by the deployment's config. No
volume copy, no ownership move, no rollback-to-container script, no state preservation: no
container-era workspace, cluster, claim or key is read.
files: ["scripts/deploy.sh (fresh install path)", "deploy/host/ (prod config.json for the gateway)"]
test_first: >
test/deploy_script_test.go TestTheDeployInstallsOnAnEmptyStateDirAndRunsNoDocker and
TestTheDeployEnablesLingerBeforeTheFirstStart (fake ssh, fake loginctl, temp state dir); red first.
done_when: "Claude ran it on both hosts; readings recorded (UTC): the web app /api/config 200, the bootstrap cluster Ready, an idle cluster reads None nodes 0, a tenant cluster scaled from zero to one market RFP node from the hat-8 bidder and back to zero, ACC6-ONE-BINARY and ACC6-STATE recorded."
risks: ["Decided by the user 2026-10-12: we iterate on prod, so losing sessions, VMs, clusters, claims and receipts in a deploy is acceptable; nothing is carried over or rolled back. The old stack's state dirs stay on disk untouched and the deploy report lists what it ran.", "If the password sign-in from the credential files fails (for example the bidder's prod-qr profile needs an OAuth QR), the deploy stops at that step and asks the user for one scan."]
status: done
evidence:
- id: M5-RETIRE
milestone: M5
title: Delete the container path
depends_on: [M5-FRESH-DEPLOY]
decision: >
Decided by the user 2026-10-12: the container path is simply deleted, no waiting period. Delete
deploy/control-plane-container/, deploy/container-entrypoint.sh, deploy/run-container.sh,
deploy/container.env.example, the docker-based pieces of
scripts/deploy.sh (docker save/load, docker rm/run, the docker install), deploy/install-provider.sh,
deploy/apply-permission-inventory.sh, deploy/denokcp/install-provider.sh and the folded cmd/ shims, in
one change right after M5-FRESH-DEPLOY's readings; the old image on prod is removed with the user.
files: ["the files listed", "scripts/deploy.sh", "docs/ARCHITECTURE.md sections 5 and 12", "internal/nodefirecracker, internal/nodevm, internal/noderecord, the substrate env and refusals in internal/providermain and their check-host users (folded from 0010 GAP-5 G4-DEAD-SUBSTRATE, 2026-10-12)"]
test_first: "TestNothingInTheRepoInvokesKubectlDockerOrSocatOutsideTestAssets; red while they exist."
done_when: "files gone; check.sh --full green; ARCHITECTURE updated."
risks:
- "accept.sh keeps kubectl until 0008:M4-MARKET-NETWORK-PORT; it is a test asset."
- "M1-CLI's TestEveryEnvVariableTheEntrypointReadsIsAnOptionsEnvName parses container-entrypoint.sh and container.env.example; it is rewritten in this change to read options.json alone."
status: done
evidence:
acceptance:
where: "hat-8 and the market network for every proof (live guests only through runComputeContract); prod readings after the user's fresh deploy"
checks:
- id: ACC6-ONE-BINARY
text: "Prod's gateway runs socialweb-computer alone as a user unit of the lingering socialweb account set up by one `install gateway`, plus the user units and children it started from its extracted assets (kcp's pg, kine, slot and front units; postgres, bao, each cluster's slice of member units: front, k3s, kine, external-secrets, dumbpipe; deno); the account's process tree and unit set hold no container, shell, kubectl, yq or socat (the host's own docker package is not used by the account)."
status: done
evidence: []
- id: ACC6-STATE
text: "After the fresh deploy (no container-era state), the door answers /api/config 200, the bootstrap cluster is Ready with its ClusterLinks Listening, an idle cluster reads None nodes 0, and a restart of the core lists the same workspaces, APIBindings, ClusterLinks and NodeClaims as before it."
status: done
evidence: []
- id: ACC6-UPGRADE
text: "An upgrade of the binary on the gateway host keeps every tenant apiserver answering through its front and every market node Ready throughout; every cluster member unit, postgres and vault unit, and kcp's pg, kine, slot and front units, keep their InvocationIDs. A kcp roll (M3-KCP-ROLL) keeps every /readyz through the kcp front answering (0 failed polls at 200 ms) and the controller gap under 5 s (decided by the user 2026-10-12: kcp blue/green slots behind our own front)."
status: done
evidence: []
- id: ACC6-CP-SWAP
text: "A k3s/kine slot swap on a cluster with a market node (born via runComputeContract on the market network) fails no /readyz poll through the front, causes no Ready transition, no claim change and no vm.delete; the front port, guest tickets, node secrets and kubeconfigs are unchanged, and the two slots never both held kube-system/k3s (decided by the user 2026-10-12)."
status: done
evidence: []
- id: ACC6-BAO-SYNC
text: "After a release, after a cluster deletion and after an upgrade of the binary, every cluster OpenBao and the root vault hold exactly the auth mounts, roles, policies and owned KV entries live objects ask for; no jwt-* mount without a live node (decided by Claude, user may veto)."
status: done
evidence: []
- id: ACC6-REPRO
text: Two clean builds are byte-identical per linux platform.
status: done
evidence: []
- id: ACC6-XOS
text: "Every check.sh --fast run since M0-XOS-GATE built and vetted the tree for darwin/arm64, darwin/amd64, windows/amd64, windows/arm64 and linux/arm64, which is the import boundary (decided by the user 2026-10-11: cross-OS seams built now)."
status: done
evidence: []
- id: ACC6-DEV
text: "Two dev instances from two worktrees on hat-8 run in parallel with no root and no systemd: each has its own devsup, bootstrap layer and kcp on port 0, its HostProcess objects placed on proc, and its market network renting a guest through the RFP flow; stopping one leaves the other's InvocationIDs unchanged; the hostsvc contract suite is green on proc and systemd-user (decided by the user 2026-10-12: dev run mode is kcp-native now)."
status: done
evidence: []
execution_order:
note: >
Written 2026-10-12 (third fold, the user's request (i)); mirrored with plans 0007 and 0008 in NEXT.yaml
execution_order, which is the full cross-plan breakdown. Advisory: depends_on governs. Read through the
principle (fourth fold): the order builds one application instance's kcp (store, slot, front, port 0,
apply, bootstrap layer, HostProcess) before any application is moved onto it; the market network is
wave 0 and every workstream tests in its own market network (state rule 'market network per workstream').
critical_path: >
Longest chain (15 waves, computed from depends_on 2026-10-12): M0-SIZE-BUDGET (or M0-PG-SOURCE) ->
M1-ASSETS -> M2-KCP-STORE -> M2-KCP-SLOTS -> M2-KCP-PORT0 -> M2-APPLY and M2-BOOTSTRAP-LAYER ->
M2-HOSTPROCESS-KIND -> M2-POSTGRES -> M3-CLUSTER-UNITS -> M3-BAO-SYNC (and M3-APISERVER-FRONT ->
M3-CP-SLOTS) -> M3-SERVE-GATEWAY-KCP -> M3-HOST-INSTALL -> M3-CP-CONTINUITY -> M5-FRESH-DEPLOY ->
M5-RETIRE. Feeding it: M2-HOST-CONFORMANCE -> M2-HOST-BACKEND-PROC (wave 2) and M1-CLI +
M1-PORT-LEASE-INSTANCE -> M1-DEV-INSTANCE (wave 3) -> M2-BOOTSTRAP-LAYER; M1-CLI -> M2-MANIFESTS ->
M2-TEMPLATE -> M2-APPLY; M1-ASSETS + M1-CLI -> M1-INSTALL -> M2-BOOTSTRAP-LAYER.
startable_now_in_this_plan: [M0-SIZE-BUDGET, M0-PG-SOURCE, M1-CLI, M1-HOST-INVENTORY, M1-STORE-CLEANUP, M1-PORT-LEASE-INSTANCE, M1-NO-FIXED-PORTS, M2-HOST-CONFORMANCE]
vertical_slices:
- task: binary-base
started: 2026-10-12
approved: "user-approved 2026-10-12 (the dependency relaxation of M2-KCP-STORE and M2-KCP-SLOTS, variant_dependencies)"
covers: "M1-CLI, M2-HOST-CONFORMANCE, M2-HOST-BACKEND-PROC, and a minimal `dev up --profile minimal`: kine-sqlite + one kcp slot (the dev variants of M2-KCP-STORE and M2-KCP-SLOTS), port 0 with the /proc read-back (M2-KCP-PORT0's mechanism), kcp and kine from explicit paths in a dev build, two instances side by side on one host"
not_covered: "embedded assets (M1-ASSETS), postgres (M0-PG-SOURCE), the front and blue/green slots (M2-KCP-FRONT), systemd-user placement (M2-HOST-UNITS), the full M1-DEV-INSTANCE profiles"
status_note: "Item status and evidence are unchanged by this record; the lead updates them when the slice merges."
early_because_live_tests_collide: >
Two market networks on one machine collide on fixed ports (market network 2583-2589, deno-kcp kcp 6443 and kine 23791) and on
the one-network-per-host check until 0008 M1-MARKET-NETWORK-PARALLEL and M1-DEV-INSTANCE land; until then every market network run on
hat-8 is serialized. M1-PORT-LEASE-INSTANCE, M1-NO-FIXED-PORTS (the ratchet that keeps new fixed ports
out while they land), M2-HOST-CONFORMANCE, M2-HOST-BACKEND-PROC and M1-DEV-INSTANCE are therefore first in
this plan.
application_model_fifth_fold: >
Recomputed 2026-10-12 after the fifth fold (all depends_on across 0005-0012 resolve, no cycle); the
critical path above is unchanged (15 waves). EARLY by the user's order, startable now (wave 1): 0008
M1-SSH-GATEWAY-IROH (then 0008 M1-STRIP-LEGACY-TRANSPORTS, wave 2; the ssh gateway's Live market CI green
throughout) and 0007 M1-MARKET-IDENTITY (the identity CRD), with 0007 M3-BIDDER-ACCOUNT-PDS,
M3-OIDC-ISSUER-INSTANCE and 0008 M2-MARKET-NETWORK (no dependencies), then 0008 M2-DENOPOD-LISTENERS
(wave 2, after M2-DENOKCP-FORK) and 0007 M3-BIDDER-COMPOSER (wave 3). The STRICT market network chain - everything
a market network needs so that every application in it runs its own kcp - is M2-APP-CONTROLLER (wave 8, after
M2-HOSTPROCESS-KIND and 0008 M2-SERVICE-TEMPLATE) -> 0008 M4-RUN-REQUEST (9) -> 0008 M4-MARKET-NETWORK-PORT
(10), fed by the early items above; it is ordered before any workstream moves its live tests onto `dev
up --profile market-network`. Until it lands, workstreams test in today's market network made parallel by 0008
M1-MARKET-NETWORK-PARALLEL, a recorded transitional shape (four applications in one kcp) that adds no new
cross-application coupling.
questions_for_user:
- "systemd 253 as the floor (decided 2026-10-11) refuses Debian 12 and RHEL 9 (both ship 252, to be verified in M2-HOST-CONTRACT). Recommended: accept; support there would mean dropping GetUnitByPIDFD for our own pidfd + cgroup check."