# Session output, 2026-10-07 Two messages from the working session, verbatim and in the order they were sent. The first is the architecture summary as trees; the second is the state of the backlog and of everything still biting. The filename is the one that was asked for. --- ## Message 1 — "Structure confirmed. Trees:" Structure confirmed. Trees: ### 1. System — request path ``` browser ─────────────────────────────────────────────────────────────┐ │ atproto OAuth (PAR+PKCE+DPoP) │ ▼ │ Caddy socialweb.computer │ ├─ k8s.socialweb.computer ──► door :8788 (SPA + /api/clusters/*) │ │ └─ /v1/credential ────► gateway :8443 (tenant cert mint) │ └─ kcp.socialweb.computer ──► kcp :6443 │ flush_interval -1, versions 1.1 ◄── exec/attach needs THIS │ │ │ ▼ │ [container] socialweb-k8s │ │ │ ├─ kcp ─── mount spec.URL ──► tenant apiserver (k3s) │ │ │ └─ clusterprovider ── scale pass @2s ──┐ │ ▼ │ NodeClaim CRD (kcp workspace) │ │ │ ▼ │ door watcher ── runComputeContract ──► market │ │ │ rfp ─► bid ─► accept ─► receipt │ │ │ ▼ │ guest VM (bidder wins) │ cloud-init user_data │ sshd + dumbpipe listen-tcp │ │ │ ▼ │ ticket iroh://… ──► /run/guest-fqdn │ │ │ door: dumbpipe connect ◄──────┘ (stdio = the wire) │ └─ ssh2 ─► forwardIn(advertise:port) ─► tenant apiserver ◄───┘ ``` ### 2. Node lifecycle — who owns which half ``` clusterprovider (Go) door (Deno) market ──────────────────── ─────────── ────── Provision └ write NodeClaim ──────────────────► watch claim └ add finalizer └ runComputeContract ──► rfp/bid/accept └ guest boots ◄──────────── status: Bidding/Provisioning/Joining ──┘ └ ssh over iroh └ onForwarded ─► Running confirmHeld └ Fatal()==" " ? ─── terminal phase ──┘ └ teardownHeld ──► release() ───────► claim deleted │ └ vm.delete (receipt) └ delete tenant Node └ removeFinalizer └ forgetHeld ← count stops here ``` ### 3. Scale decision — the count ``` every 2s, per cluster: Observe ─► Intent? ──no──► idle ──► hold 2m ──► Teardown (to ZERO) │ yes ▼ counted = tenant Nodes + held claims the tenant does NOT name ← MARKET-4 (market substrate only) (every node counted until ITS cluster lists it) Target = ceil(Demand / podsPerNode) cap 10 counted < target ──► Provision (non-blocking: write claim, return) counted = target ──► None counted > target ──► None (scale-in all-or-nothing) ``` ### 4. Repo tree ``` socialweb-computer-kcp/ ├─ cmd/ 12 binaries │ ├─ socialweb-clusterprovider ← scale pass, claims, add-ons, migrate subcmd │ ├─ socialweb-gateway ← tenant cert mint (Vault/client-CA) │ ├─ socialweb-nodeimage ← builds guest image (kernel+modules+agent) │ ├─ socialweb-nodeagent, -nodeboot, -vault, -credential, -tenantcredential │ └─ socialweb-cloud-controller-manager ├─ internal/ 38 pkgs; load-bearing ones: │ ├─ scale, scaledriver ← Decide/Target; DefaultMaxNodes=10 │ ├─ nodemarket ← NodeClaim host (Provision/Confirm/Fatal→Terminal) │ ├─ clusterprovider ← StepsFor, tenant grant, vault, ESO │ ├─ kcpclient ← workspace + NodeClaim wire │ ├─ nodefirecracker, nodeqemu ← guest substrates (kernel discovery) │ ├─ noderecord ← pid+argv+SUBSTRATE record │ └─ vaultcert, eso, openbao, identitybundle ├─ socialweb-computer-k8s/ the DOOR (ABC layering, own deno workspace) │ ├─ lib/common/{socialweb-computer-common,k8s-common} │ ├─ lib/abc/{socialweb-computer,k8s} │ ├─ lib/socialweb-computer-k8s-nodeclaim-kcp ← claim list, 30s walk cache, 429 park │ ├─ lib/socialweb-computer-k8s-node-market ← fulfil: contract + session + iroh │ ├─ lib/cloud-init-plugin-k3s-join ← guest join module │ └─ hono-socialweb-computer-k8s ← CLI ├─ deploy/ Dockerfile, entrypoint, caddy sites, ESO + tenant-grant ├─ docs/ARCHITECTURE.md ONE doc = design + ADRs + registers + readings ├─ docs/plans/0001..0004 work lists (expected stale) └─ NEXT.yaml live backlog -> open: (none), 6 stated limits ``` ### 5. Where we are ``` GOAL full thing on our GitLab, kcp cluster from SPA, market nodes STATUS MET ├─ pipeline 76 5 jobs at once, all success │ five DISTINCT market nodes, ~200s each ├─ count action=None nodes=7, never > ceiling, 0 releases, 0 failed ├─ prod 50 commits, both trees clean, HEAD deployed └─ Caddy active, valid, versions 1.1, door 200 / kcp 403 anon BACKLOG EMPTY (open: none) closed tonight: MARKET-1, MINT-1, CCM-2, VAULT-TLS-1, ESO-2, ESO-3, FLAKE-1, FLAKE-3 (+ earlier: MARKET-3..6, FLAKE-2, GL-1, GL-2) remaining 6 stated LIMITS, not work YOURS ├─ /tmp three dirs, 18.8G, cold 6h, nothing holds them │ BUT /tmp/live-k3s-* is LIVE -> never /tmp/* └─ ESO demo 3 options scoped; cheapest needs no new exposure ``` **Bites recorded, not hidden:** count overshoot (15 for ceiling 10) · held-node wedge (count 11 vs 4 real) · spent node name · corpse Node taking pods · ESO store dialling `10.100.0.1` nothing binds · **my own** unguarded add-on call stalling the provider (zero CPU) · reconcile deleting by name (325 writes/5min) · Terminating ExternalSecret no controller could release. --- ## Message 2 — "Two lists." Two lists. ### STILL BITING / OPEN ``` BACKLOG open: (none) <- no item six LIMITS remain, and they are limits not work: 1 portless member CCM binds no port, leaves no record -> sweep can't find it (alternatives worse; stop it by hand) 2 node-to-node no guest reaches another; multi-node = no cluster DNS for pods off the CoreDNS node. permanent. 3 ceiling ten cost knob, not a bug 4 scale-in all-or-nothing to ZERO; no 3->2 5 instance snapshot load-bearing; absent reads as EMPTY 6 vault cert host running vault won't serve a host added later (refused loudly now, not silently broken) ``` ``` LIVE, NOT TRACKED AS ITEMS /tmp 91% full 18.8G in 3 dirs (rfp-guests, nodeboot-check, nodeimage-preinstall-check). cold 6h, nothing holds them. builds die: "disk quota exceeded". LIVE: /tmp/live-k3s-* -> never /tmp/* workaround: TMPDIR=~/.cache/scratch-tmp deploy mid-run restart reaps nodes -> kills in-flight jobs. measured: pipeline 74, 4/5 jobs died on corpse Nodes ESO demo on market store can't reach a vault the tenant can dial. closed by NOT writing it. 3 options scoped, none taken ESO-2 residual admission measured both ways. the SECRET itself still can't sync on market = ESO-3's subject, guest-only reconcile per pass 3 DELETEs every pass, always NotFound. recorded, fix known (per-cluster flag), not shipped two GitLab runners runner 1 = INSTANCE runner on the GitLab host, run_untagged:true -> can eat untagged jobs. runner 5 = ours, tagged. documented hat-8 capacity many firecracker VMs, swap full UNVERIFIED scale-to-zero never seen with a truly idle cluster (always workload) FLAKE-1 repro agent's reproduction, not mine -- mine never fired ``` ### CLOSED TONIGHT AND HOW ``` ITEM BITE HOW CLOSED / READING MARKET-4 count keyed on claim PHASE had a count = tenant Nodes + held claims gap: claim at Running, node not the tenant does NOT name; gated to listed -> counted NOWHERE deferredJoin (guest substrates -> 15 nodes for ceiling 10 unchanged). prod: 6 vs 6, then 4+2 2 tests watched to fail wedge teardownHeld left host in `extra` forgetHeld on teardown. -> count 11 while cluster had 4 11 claims/7 released/4 live vs -> above ceiling -> NO RENTING count 11; after: 4 vs 4 + renting test watched to fail MARKET-5 first node named at construction; primaryNameSpent + forgetHeld. restart re-claimed spent node1; node1 claimed 06:18:38, released WriteNodeClaimTo keeps status 06:18:41. test watched to fail -> own claim read as Failed MARKET-6 Node whose guest door reaped door writes Released when reaped; stayed Ready, took pods provider reconciles on Released; (74: 4/5 jobs died, 58s Pending) Fatal->Terminal (any ended phase) + my own 1st fix deleted by NAME then: intersect with Nodes present -> 325 writes/5min + 429 (6 removals, then 0). 3 tests FLAKE-2 door session test died on ssh2's fixed test keypair (generator gone). generated key, 2 runs in 4 6 concurrent suites green. both earlier causes REFUTED GL-1 minting CLI never run on real built+copied into image; ran as uid data dir -- AND NOT IN THE IMAGE kcp. kubectl listed nodes; auth whoami = cli-close-gl-1; 2nd run refused (invariant 20) GL-2 runner off-host could not reach CLOSED BY MEASUREMENT: kubectl exec tenant apiserver, exec unmeasured reached containerd through mount; kubernetes executor ran 5 jobs MARKET-1 record named no substrate -> Record.Substrate + per-record sweep could only reap its own identity resolve; unknown = refuse. firecracker record swept by MARKET gateway. test watched to fail MINT-1 rename ran inside clusterprovider migrate SUBCOMMAND, entrypoint runs which started AFTER gateway it BEFORE gateway. prod: log 1915 -> mint window refused migrate, 1916 gateway. test reads entrypoint, watched to fail CCM-2 portless member unswept closed as STATED LIMIT (item's own option) + corrected stale ceiling text (said default 1; refusal is n<1 not n ExternalSecret had the finalizer no controller could. never synced on market store+ES gone, operator kept. 3 tests watched to fail ESO-2 grant hardcoded `default`; install placeholder from one source. elsewhere DENIED. found 2nd half + installer honouring ESO_NAMESPACE by taking the reading: installer (bundle names default in 16 places). IGNORED ESO_NAMESPACE entirely BEFORE: probe forbidden. AFTER: created; `default` refused though it has RBAC -> isolates policy FLAKE-3 openbao gate lent a port it had retry fresh port on EITHER shape already given back -> concurrent (something answers / vault can't run took it -> probe hit a bind). 10 concurrent runs green stranger's TLS listener FLAKE-1 gate test "depends on registry" mechanism: image NAME carries agent -- WRONG. real: name carries digest; built twice; go build stamps agent digest, built twice, go git tree. fix -buildvcs=false + build stamps the working tree name-both-ways check + SKIP with both digests when input moves under the run (watched to fire). rejected: 24s variant DOC DRIFT docs/gitlab said ceiling updated to 10 + why; found by defaults to 1 -- it is 10 git log -p docs/gitlab/ EARLIER IN SESSION (same run) MARKET-3 every provision in a pass nextOrdinal seeds from held claims resolved to SAME ordinal + ended claims + tenant Nodes kernel rot manifests named /boot/vmlinuz- "discover" -> host-running kernel 7.2.7 which no longer exists door 429 walk every 2s tick, no Retry- 30s walk cache, Retry-After park on After -> self-inflicted limit reads only, poll 2s->5s dumbpipe key endpoint secret key logged redactTransportLine vm-ssh saved to different file than one qrSessionKey for both halves it restored from ``` ``` TALLY open items: 0 limits: 6 commits: 50 (this repo) + 1 atproto-market prod: HEAD, 7 nodes, action=None nodes=7, never > ceiling acceptance: pipeline 76 -- 5 jobs, all success, 5 distinct market nodes ```