Session output, 2026-10-07
Two messages from the working session, verbatim and in the order they were sent. The first is the architecture summary as trees; the second is the state of the backlog and of everything still biting. The filename is the one that was asked for.
Message 1 — "Structure confirmed. Trees:"
Structure confirmed. Trees:
1. System — request path
browser ─────────────────────────────────────────────────────────────┐
│ atproto OAuth (PAR+PKCE+DPoP) │
▼ │
Caddy socialweb.computer │
├─ k8s.socialweb.computer ──► door :8788 (SPA + /api/clusters/*) │
│ └─ /v1/credential ────► gateway :8443 (tenant cert mint) │
└─ kcp.socialweb.computer ──► kcp :6443 │
flush_interval -1, versions 1.1 ◄── exec/attach needs THIS │
│ │
▼ │
[container] socialweb-k8s │
│ │
├─ kcp ─── mount spec.URL ──► tenant apiserver (k3s) │
│ │
└─ clusterprovider ── scale pass @2s ──┐ │
▼ │
NodeClaim CRD (kcp workspace) │
│ │
▼ │
door watcher ── runComputeContract ──► market │
│ │
rfp ─► bid ─► accept ─► receipt │
│ │
▼ │
guest VM (bidder wins) │
cloud-init user_data │
sshd + dumbpipe listen-tcp │
│ │
▼ │
ticket iroh://… ──► /run/guest-fqdn │
│ │
door: dumbpipe connect <ticket> ◄──────┘ (stdio = the wire) │
└─ ssh2 ─► forwardIn(advertise:port) ─► tenant apiserver ◄───┘
2. Node lifecycle — who owns which half
clusterprovider (Go) door (Deno) market
──────────────────── ─────────── ──────
Provision
└ write NodeClaim ──────────────────► watch claim
└ add finalizer
└ runComputeContract ──► rfp/bid/accept
└ guest boots
◄──────────── status: Bidding/Provisioning/Joining ──┘
└ ssh over iroh
└ onForwarded ─► Running
confirmHeld
└ Fatal()==" " ? ─── terminal phase ──┘
└ teardownHeld ──► release() ───────► claim deleted
│ └ vm.delete (receipt)
└ delete tenant Node └ removeFinalizer
└ forgetHeld ← count stops here
3. Scale decision — the count
every 2s, per cluster:
Observe ─► Intent? ──no──► idle ──► hold 2m ──► Teardown (to ZERO)
│ yes
▼
counted = tenant Nodes
+ held claims the tenant does NOT name ← MARKET-4
(market substrate only)
(every node counted until ITS cluster lists it)
Target = ceil(Demand / podsPerNode) cap 10
counted < target ──► Provision (non-blocking: write claim, return)
counted = target ──► None
counted > target ──► None (scale-in all-or-nothing)
4. Repo tree
socialweb-computer-kcp/
├─ cmd/ 12 binaries
│ ├─ socialweb-clusterprovider ← scale pass, claims, add-ons, migrate subcmd
│ ├─ socialweb-gateway ← tenant cert mint (Vault/client-CA)
│ ├─ socialweb-nodeimage ← builds guest image (kernel+modules+agent)
│ ├─ socialweb-nodeagent, -nodeboot, -vault, -credential, -tenantcredential
│ └─ socialweb-cloud-controller-manager
├─ internal/ 38 pkgs; load-bearing ones:
│ ├─ scale, scaledriver ← Decide/Target; DefaultMaxNodes=10
│ ├─ nodemarket ← NodeClaim host (Provision/Confirm/Fatal→Terminal)
│ ├─ clusterprovider ← StepsFor, tenant grant, vault, ESO
│ ├─ kcpclient ← workspace + NodeClaim wire
│ ├─ nodefirecracker, nodeqemu ← guest substrates (kernel discovery)
│ ├─ noderecord ← pid+argv+SUBSTRATE record
│ └─ vaultcert, eso, openbao, identitybundle
├─ socialweb-computer-k8s/ the DOOR (ABC layering, own deno workspace)
│ ├─ lib/common/{socialweb-computer-common,k8s-common}
│ ├─ lib/abc/{socialweb-computer,k8s}
│ ├─ lib/socialweb-computer-k8s-nodeclaim-kcp ← claim list, 30s walk cache, 429 park
│ ├─ lib/socialweb-computer-k8s-node-market ← fulfil: contract + session + iroh
│ ├─ lib/cloud-init-plugin-k3s-join ← guest join module
│ └─ hono-socialweb-computer-k8s ← CLI
├─ deploy/ Dockerfile, entrypoint, caddy sites, ESO + tenant-grant
├─ docs/ARCHITECTURE.md ONE doc = design + ADRs + registers + readings
├─ docs/plans/0001..0004 work lists (expected stale)
└─ NEXT.yaml live backlog -> open: (none), 6 stated limits
5. Where we are
GOAL full thing on our GitLab, kcp cluster from SPA, market nodes
STATUS MET
├─ pipeline 76 5 jobs at once, all success
│ five DISTINCT market nodes, ~200s each
├─ count action=None nodes=7, never > ceiling, 0 releases, 0 failed
├─ prod 50 commits, both trees clean, HEAD deployed
└─ Caddy active, valid, versions 1.1, door 200 / kcp 403 anon
BACKLOG EMPTY (open: none)
closed tonight: MARKET-1, MINT-1, CCM-2, VAULT-TLS-1, ESO-2, ESO-3,
FLAKE-1, FLAKE-3 (+ earlier: MARKET-3..6, FLAKE-2, GL-1, GL-2)
remaining 6 stated LIMITS, not work
YOURS
├─ /tmp three dirs, 18.8G, cold 6h, nothing holds them
│ BUT /tmp/live-k3s-* is LIVE -> never /tmp/*
└─ ESO demo 3 options scoped; cheapest needs no new exposure
Bites recorded, not hidden: count overshoot (15 for ceiling 10) · held-node wedge (count 11 vs 4 real) · spent node name · corpse Node taking pods · ESO store dialling 10.100.0.1 nothing binds · my own unguarded add-on call stalling the provider (zero CPU) · reconcile deleting by name (325 writes/5min) · Terminating ExternalSecret no controller could release.
Message 2 — "Two lists."
Two lists.
STILL BITING / OPEN
BACKLOG open: (none) <- no item
six LIMITS remain, and they are limits not work:
1 portless member CCM binds no port, leaves no record -> sweep can't find it
(alternatives worse; stop it by hand)
2 node-to-node no guest reaches another; multi-node = no cluster DNS
for pods off the CoreDNS node. permanent.
3 ceiling ten cost knob, not a bug
4 scale-in all-or-nothing to ZERO; no 3->2
5 instance snapshot load-bearing; absent reads as EMPTY
6 vault cert host running vault won't serve a host added later
(refused loudly now, not silently broken)
LIVE, NOT TRACKED AS ITEMS
/tmp 91% full 18.8G in 3 dirs (rfp-guests, nodeboot-check,
nodeimage-preinstall-check). cold 6h, nothing holds them.
builds die: "disk quota exceeded".
LIVE: /tmp/live-k3s-* -> never /tmp/*
workaround: TMPDIR=~/.cache/scratch-tmp
deploy mid-run restart reaps nodes -> kills in-flight jobs.
measured: pipeline 74, 4/5 jobs died on corpse Nodes
ESO demo on market store can't reach a vault the tenant can dial.
closed by NOT writing it. 3 options scoped, none taken
ESO-2 residual admission measured both ways. the SECRET itself
still can't sync on market = ESO-3's subject, guest-only
reconcile per pass 3 DELETEs every pass, always NotFound.
recorded, fix known (per-cluster flag), not shipped
two GitLab runners runner 1 = INSTANCE runner on the GitLab host,
run_untagged:true -> can eat untagged jobs.
runner 5 = ours, tagged. documented
hat-8 capacity many firecracker VMs, swap full
UNVERIFIED
scale-to-zero never seen with a truly idle cluster (always workload)
FLAKE-1 repro agent's reproduction, not mine -- mine never fired
CLOSED TONIGHT AND HOW
ITEM BITE HOW CLOSED / READING
MARKET-4 count keyed on claim PHASE had a count = tenant Nodes + held claims
gap: claim at Running, node not the tenant does NOT name; gated to
listed -> counted NOWHERE deferredJoin (guest substrates
-> 15 nodes for ceiling 10 unchanged). prod: 6 vs 6, then 4+2
2 tests watched to fail
wedge teardownHeld left host in `extra` forgetHeld on teardown.
-> count 11 while cluster had 4 11 claims/7 released/4 live vs
-> above ceiling -> NO RENTING count 11; after: 4 vs 4 + renting
test watched to fail
MARKET-5 first node named at construction; primaryNameSpent + forgetHeld.
restart re-claimed spent node1; node1 claimed 06:18:38, released
WriteNodeClaimTo keeps status 06:18:41. test watched to fail
-> own claim read as Failed
MARKET-6 Node whose guest door reaped door writes Released when reaped;
stayed Ready, took pods provider reconciles on Released;
(74: 4/5 jobs died, 58s Pending) Fatal->Terminal (any ended phase)
+ my own 1st fix deleted by NAME then: intersect with Nodes present
-> 325 writes/5min + 429 (6 removals, then 0). 3 tests
FLAKE-2 door session test died on ssh2's fixed test keypair (generator gone).
generated key, 2 runs in 4 6 concurrent suites green.
both earlier causes REFUTED
GL-1 minting CLI never run on real built+copied into image; ran as uid
data dir -- AND NOT IN THE IMAGE kcp. kubectl listed nodes; auth
whoami = cli-close-gl-1; 2nd run
refused (invariant 20)
GL-2 runner off-host could not reach CLOSED BY MEASUREMENT: kubectl exec
tenant apiserver, exec unmeasured reached containerd through mount;
kubernetes executor ran 5 jobs
MARKET-1 record named no substrate -> Record.Substrate + per-record
sweep could only reap its own identity resolve; unknown = refuse.
firecracker record swept by MARKET
gateway. test watched to fail
MINT-1 rename ran inside clusterprovider migrate SUBCOMMAND, entrypoint runs
which started AFTER gateway it BEFORE gateway. prod: log 1915
-> mint window refused migrate, 1916 gateway. test reads
entrypoint, watched to fail
CCM-2 portless member unswept closed as STATED LIMIT (item's own
option) + corrected stale ceiling
text (said default 1; refusal is
n<1 not n<default)
VAULT-TLS-1 running vault can't serve a host stated limit + READING (served leaf
added after it started vs disk leaf; client dial fails) +
next-best: refuse to hand out a URL
the listener can't present
ESO-3 store dialled 10.100.0.1:20001, not written where unreachable +
NOTHING binds it (vault is eso.Remove for upgrades + release
loopback) -> ExternalSecret had the finalizer no controller could.
never synced on market store+ES gone, operator kept.
3 tests watched to fail
ESO-2 grant hardcoded `default`; install placeholder from one source.
elsewhere DENIED. found 2nd half + installer honouring ESO_NAMESPACE
by taking the reading: installer (bundle names default in 16 places).
IGNORED ESO_NAMESPACE entirely BEFORE: probe forbidden. AFTER:
created; `default` refused though it
has RBAC -> isolates policy
FLAKE-3 openbao gate lent a port it had retry fresh port on EITHER shape
already given back -> concurrent (something answers / vault can't
run took it -> probe hit a bind). 10 concurrent runs green
stranger's TLS listener
FLAKE-1 gate test "depends on registry" mechanism: image NAME carries agent
-- WRONG. real: name carries digest; built twice; go build stamps
agent digest, built twice, go git tree. fix -buildvcs=false +
build stamps the working tree name-both-ways check + SKIP with
both digests when input moves under
the run (watched to fire).
rejected: 24s variant
DOC DRIFT docs/gitlab said ceiling updated to 10 + why; found by
defaults to 1 -- it is 10 git log -p docs/gitlab/
EARLIER IN SESSION (same run)
MARKET-3 every provision in a pass nextOrdinal seeds from held claims
resolved to SAME ordinal + ended claims + tenant Nodes
kernel rot manifests named /boot/vmlinuz- "discover" -> host-running kernel
7.2.7 which no longer exists
door 429 walk every 2s tick, no Retry- 30s walk cache, Retry-After park on
After -> self-inflicted limit reads only, poll 2s->5s
dumbpipe key endpoint secret key logged redactTransportLine
vm-ssh saved to different file than one qrSessionKey for both halves
it restored from
TALLY open items: 0
limits: 6
commits: 50 (this repo) + 1 atproto-market
prod: HEAD, 7 nodes, action=None nodes=7, never > ceiling
acceptance: pipeline 76 -- 5 jobs, all success, 5 distinct market nodes