1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
# Session output, 2026-10-07
Two messages from the working session, verbatim and in the order they were sent. The first is the
architecture summary as trees; the second is the state of the backlog and of everything still biting.
The filename is the one that was asked for.
---
## Message 1 — "Structure confirmed. Trees:"
Structure confirmed. Trees:
### 1. System — request path
```
browser ─────────────────────────────────────────────────────────────┐
│ atproto OAuth (PAR+PKCE+DPoP) │
▼ │
Caddy socialweb.computer │
├─ k8s.socialweb.computer ──► door :8788 (SPA + /api/clusters/*) │
│ └─ /v1/credential ────► gateway :8443 (tenant cert mint) │
└─ kcp.socialweb.computer ──► kcp :6443 │
flush_interval -1, versions 1.1 ◄── exec/attach needs THIS │
│ │
▼ │
[container] socialweb-k8s │
│ │
├─ kcp ─── mount spec.URL ──► tenant apiserver (k3s) │
│ │
└─ clusterprovider ── scale pass @2s ──┐ │
▼ │
NodeClaim CRD (kcp workspace) │
│ │
▼ │
door watcher ── runComputeContract ──► market │
│ │
rfp ─► bid ─► accept ─► receipt │
│ │
▼ │
guest VM (bidder wins) │
cloud-init user_data │
sshd + dumbpipe listen-tcp │
│ │
▼ │
ticket iroh://… ──► /run/guest-fqdn │
│ │
door: dumbpipe connect <ticket> ◄──────┘ (stdio = the wire) │
└─ ssh2 ─► forwardIn(advertise:port) ─► tenant apiserver ◄───┘
```
### 2. Node lifecycle — who owns which half
```
clusterprovider (Go) door (Deno) market
──────────────────── ─────────── ──────
Provision
└ write NodeClaim ──────────────────► watch claim
└ add finalizer
└ runComputeContract ──► rfp/bid/accept
└ guest boots
◄──────────── status: Bidding/Provisioning/Joining ──┘
└ ssh over iroh
└ onForwarded ─► Running
confirmHeld
└ Fatal()==" " ? ─── terminal phase ──┘
└ teardownHeld ──► release() ───────► claim deleted
│ └ vm.delete (receipt)
└ delete tenant Node └ removeFinalizer
└ forgetHeld ← count stops here
```
### 3. Scale decision — the count
```
every 2s, per cluster:
Observe ─► Intent? ──no──► idle ──► hold 2m ──► Teardown (to ZERO)
│ yes
▼
counted = tenant Nodes
+ held claims the tenant does NOT name ← MARKET-4
(market substrate only)
(every node counted until ITS cluster lists it)
Target = ceil(Demand / podsPerNode) cap 10
counted < target ──► Provision (non-blocking: write claim, return)
counted = target ──► None
counted > target ──► None (scale-in all-or-nothing)
```
### 4. Repo tree
```
socialweb-computer-kcp/
├─ cmd/ 12 binaries
│ ├─ socialweb-clusterprovider ← scale pass, claims, add-ons, migrate subcmd
│ ├─ socialweb-gateway ← tenant cert mint (Vault/client-CA)
│ ├─ socialweb-nodeimage ← builds guest image (kernel+modules+agent)
│ ├─ socialweb-nodeagent, -nodeboot, -vault, -credential, -tenantcredential
│ └─ socialweb-cloud-controller-manager
├─ internal/ 38 pkgs; load-bearing ones:
│ ├─ scale, scaledriver ← Decide/Target; DefaultMaxNodes=10
│ ├─ nodemarket ← NodeClaim host (Provision/Confirm/Fatal→Terminal)
│ ├─ clusterprovider ← StepsFor, tenant grant, vault, ESO
│ ├─ kcpclient ← workspace + NodeClaim wire
│ ├─ nodefirecracker, nodeqemu ← guest substrates (kernel discovery)
│ ├─ noderecord ← pid+argv+SUBSTRATE record
│ └─ vaultcert, eso, openbao, identitybundle
├─ socialweb-computer-k8s/ the DOOR (ABC layering, own deno workspace)
│ ├─ lib/common/{socialweb-computer-common,k8s-common}
│ ├─ lib/abc/{socialweb-computer,k8s}
│ ├─ lib/socialweb-computer-k8s-nodeclaim-kcp ← claim list, 30s walk cache, 429 park
│ ├─ lib/socialweb-computer-k8s-node-market ← fulfil: contract + session + iroh
│ ├─ lib/cloud-init-plugin-k3s-join ← guest join module
│ └─ hono-socialweb-computer-k8s ← CLI
├─ deploy/ Dockerfile, entrypoint, caddy sites, ESO + tenant-grant
├─ docs/ARCHITECTURE.md ONE doc = design + ADRs + registers + readings
├─ docs/plans/0001..0004 work lists (expected stale)
└─ NEXT.yaml live backlog -> open: (none), 6 stated limits
```
### 5. Where we are
```
GOAL full thing on our GitLab, kcp cluster from SPA, market nodes
STATUS MET
├─ pipeline 76 5 jobs at once, all success
│ five DISTINCT market nodes, ~200s each
├─ count action=None nodes=7, never > ceiling, 0 releases, 0 failed
├─ prod 50 commits, both trees clean, HEAD deployed
└─ Caddy active, valid, versions 1.1, door 200 / kcp 403 anon
BACKLOG EMPTY (open: none)
closed tonight: MARKET-1, MINT-1, CCM-2, VAULT-TLS-1, ESO-2, ESO-3,
FLAKE-1, FLAKE-3 (+ earlier: MARKET-3..6, FLAKE-2, GL-1, GL-2)
remaining 6 stated LIMITS, not work
YOURS
├─ /tmp three dirs, 18.8G, cold 6h, nothing holds them
│ BUT /tmp/live-k3s-* is LIVE -> never /tmp/*
└─ ESO demo 3 options scoped; cheapest needs no new exposure
```
**Bites recorded, not hidden:** count overshoot (15 for ceiling 10) · held-node wedge (count 11 vs 4 real) · spent node name · corpse Node taking pods · ESO store dialling `10.100.0.1` nothing binds · **my own** unguarded add-on call stalling the provider (zero CPU) · reconcile deleting by name (325 writes/5min) · Terminating ExternalSecret no controller could release.
---
## Message 2 — "Two lists."
Two lists.
### STILL BITING / OPEN
```
BACKLOG open: (none) <- no item
six LIMITS remain, and they are limits not work:
1 portless member CCM binds no port, leaves no record -> sweep can't find it
(alternatives worse; stop it by hand)
2 node-to-node no guest reaches another; multi-node = no cluster DNS
for pods off the CoreDNS node. permanent.
3 ceiling ten cost knob, not a bug
4 scale-in all-or-nothing to ZERO; no 3->2
5 instance snapshot load-bearing; absent reads as EMPTY
6 vault cert host running vault won't serve a host added later
(refused loudly now, not silently broken)
```
```
LIVE, NOT TRACKED AS ITEMS
/tmp 91% full 18.8G in 3 dirs (rfp-guests, nodeboot-check,
nodeimage-preinstall-check). cold 6h, nothing holds them.
builds die: "disk quota exceeded".
LIVE: /tmp/live-k3s-* -> never /tmp/*
workaround: TMPDIR=~/.cache/scratch-tmp
deploy mid-run restart reaps nodes -> kills in-flight jobs.
measured: pipeline 74, 4/5 jobs died on corpse Nodes
ESO demo on market store can't reach a vault the tenant can dial.
closed by NOT writing it. 3 options scoped, none taken
ESO-2 residual admission measured both ways. the SECRET itself
still can't sync on market = ESO-3's subject, guest-only
reconcile per pass 3 DELETEs every pass, always NotFound.
recorded, fix known (per-cluster flag), not shipped
two GitLab runners runner 1 = INSTANCE runner on the GitLab host,
run_untagged:true -> can eat untagged jobs.
runner 5 = ours, tagged. documented
hat-8 capacity many firecracker VMs, swap full
UNVERIFIED
scale-to-zero never seen with a truly idle cluster (always workload)
FLAKE-1 repro agent's reproduction, not mine -- mine never fired
```
### CLOSED TONIGHT AND HOW
```
ITEM BITE HOW CLOSED / READING
MARKET-4 count keyed on claim PHASE had a count = tenant Nodes + held claims
gap: claim at Running, node not the tenant does NOT name; gated to
listed -> counted NOWHERE deferredJoin (guest substrates
-> 15 nodes for ceiling 10 unchanged). prod: 6 vs 6, then 4+2
2 tests watched to fail
wedge teardownHeld left host in `extra` forgetHeld on teardown.
-> count 11 while cluster had 4 11 claims/7 released/4 live vs
-> above ceiling -> NO RENTING count 11; after: 4 vs 4 + renting
test watched to fail
MARKET-5 first node named at construction; primaryNameSpent + forgetHeld.
restart re-claimed spent node1; node1 claimed 06:18:38, released
WriteNodeClaimTo keeps status 06:18:41. test watched to fail
-> own claim read as Failed
MARKET-6 Node whose guest door reaped door writes Released when reaped;
stayed Ready, took pods provider reconciles on Released;
(74: 4/5 jobs died, 58s Pending) Fatal->Terminal (any ended phase)
+ my own 1st fix deleted by NAME then: intersect with Nodes present
-> 325 writes/5min + 429 (6 removals, then 0). 3 tests
FLAKE-2 door session test died on ssh2's fixed test keypair (generator gone).
generated key, 2 runs in 4 6 concurrent suites green.
both earlier causes REFUTED
GL-1 minting CLI never run on real built+copied into image; ran as uid
data dir -- AND NOT IN THE IMAGE kcp. kubectl listed nodes; auth
whoami = cli-close-gl-1; 2nd run
refused (invariant 20)
GL-2 runner off-host could not reach CLOSED BY MEASUREMENT: kubectl exec
tenant apiserver, exec unmeasured reached containerd through mount;
kubernetes executor ran 5 jobs
MARKET-1 record named no substrate -> Record.Substrate + per-record
sweep could only reap its own identity resolve; unknown = refuse.
firecracker record swept by MARKET
gateway. test watched to fail
MINT-1 rename ran inside clusterprovider migrate SUBCOMMAND, entrypoint runs
which started AFTER gateway it BEFORE gateway. prod: log 1915
-> mint window refused migrate, 1916 gateway. test reads
entrypoint, watched to fail
CCM-2 portless member unswept closed as STATED LIMIT (item's own
option) + corrected stale ceiling
text (said default 1; refusal is
n<1 not n<default)
VAULT-TLS-1 running vault can't serve a host stated limit + READING (served leaf
added after it started vs disk leaf; client dial fails) +
next-best: refuse to hand out a URL
the listener can't present
ESO-3 store dialled 10.100.0.1:20001, not written where unreachable +
NOTHING binds it (vault is eso.Remove for upgrades + release
loopback) -> ExternalSecret had the finalizer no controller could.
never synced on market store+ES gone, operator kept.
3 tests watched to fail
ESO-2 grant hardcoded `default`; install placeholder from one source.
elsewhere DENIED. found 2nd half + installer honouring ESO_NAMESPACE
by taking the reading: installer (bundle names default in 16 places).
IGNORED ESO_NAMESPACE entirely BEFORE: probe forbidden. AFTER:
created; `default` refused though it
has RBAC -> isolates policy
FLAKE-3 openbao gate lent a port it had retry fresh port on EITHER shape
already given back -> concurrent (something answers / vault can't
run took it -> probe hit a bind). 10 concurrent runs green
stranger's TLS listener
FLAKE-1 gate test "depends on registry" mechanism: image NAME carries agent
-- WRONG. real: name carries digest; built twice; go build stamps
agent digest, built twice, go git tree. fix -buildvcs=false +
build stamps the working tree name-both-ways check + SKIP with
both digests when input moves under
the run (watched to fire).
rejected: 24s variant
DOC DRIFT docs/gitlab said ceiling updated to 10 + why; found by
defaults to 1 -- it is 10 git log -p docs/gitlab/
EARLIER IN SESSION (same run)
MARKET-3 every provision in a pass nextOrdinal seeds from held claims
resolved to SAME ordinal + ended claims + tenant Nodes
kernel rot manifests named /boot/vmlinuz- "discover" -> host-running kernel
7.2.7 which no longer exists
door 429 walk every 2s tick, no Retry- 30s walk cache, Retry-After park on
After -> self-inflicted limit reads only, poll 2s->5s
dumbpipe key endpoint secret key logged redactTransportLine
vm-ssh saved to different file than one qrSessionKey for both halves
it restored from
```
```
TALLY open items: 0
limits: 6
commits: 50 (this repo) + 1 atproto-market
prod: HEAD, 7 nodes, action=None nodes=7, never > ceiling
acceptance: pipeline 76 -- 5 jobs, all success, 5 distinct market nodes
```
