johnandersen777.bsky.social/socialweb-computer-kcp

Switch repository

Loading…

plan: 0005-zero-ssh-market-nodes
written: 2026-10-07
source: >
  Decided with the user on 2026-10-07 in a question-by-question session after prod was found pinned
  at six market nodes for 7.5 idle hours. goal.txt carries the same content in compressed form.
state_rules:
  - TypeScript lives in this repository (decided by the user 2026-10-12): the closure's ten repositories were vendored with their history under ts/<repository>/ and the originals are frozen; a path this plan names under ts/ is changed here, never in the original.
  - This file is the only state. Every step updates status and evidence.
  - status is one of todo, doing, done, blocked.
  - An item is done only when its evidence is recorded (commit SHAs, test names watched to fail and
    then pass, prod or GitLab readings with UTC timestamps).
  - Open items are mirrored in NEXT.yaml.
  - A decision marked "decided by Claude, user may veto" was not confirmed by the user.
ownership:
  owned: >
    All code in socialweb-computer-kcp, the prod deployment on socialweb.computer, the vendored
    deno-kcp market network, and the atproto-market requester libraries.
  not_owned: >
    The market protocol records and lexicons, and the core architecture: kcp in front of
    k3s --disable-agent clusters at zero nodes, with nodes only ever born through
    requester -> RFP -> bid -> accept -> cloud-init user_data. Changes there are asked first.
m3_contract:
  decided: "2026-10-07, decided by Claude, user may veto (names only; the mechanism is the user's)"
  api: "commit 0c1c7e5: ClusterLink (spec cluster/service/targetPort, status phase/endpointId/did/ticket/message), NodeClaim.status.grant {issuer, subject, audience, role}; NodeClaim schema revision 3"
  links:
    - "one ClusterLink per tenant cluster per service, named <clusterKey>-apiserver and <clusterKey>-openbao, in the cluster's home workspace; spec and status written by the provider only"
    - "each is a dumbpipe listen-tcp process-group component started after k3s, forwarding to the k3s apiserver port or the cluster OpenBao port on loopback"
    - "iroh secret key per link generated once and kept in the cluster OpenBao, KV v2 mount socialweb-links, path <service>; dumbpipe gets it as IROH_SECRET; the public key is published as did:key (multicodec ed25519-pub 0xed01, base58btc, z prefix)"
  node_secrets:
    - "the provider writes KV v2 mount socialweb-nodes, path <nodeName>: token, nodePassword, serverCA, apiserverTicket, advertiseAddress, apiserverPort; deleted at release"
    - "policy node-<nodeName>: read on socialweb-nodes/data/<nodeName> only"
  jwt:
    - "the door writes status.grant at accept from the winning bid's grant (issuer, subject) plus audience socialweb.computer/node/<clusterKey>/<nodeName>"
    - "the provider ensures JWT auth mount jwt-<first 12 hex of sha256(issuer)> with oidc_discovery_url = issuer, and role <nodeName> with bound_subject, bound_audiences, user_claim sub, token_ttl 5m, token_policies node-<nodeName>; role and policy deleted at release"
  guest:
    - "the --secrets capability delivers /etc/socialweb/node/openbao-ticket, /etc/socialweb/node/openbao-ca.pem and /etc/socialweb/node/claim.json {node, cluster, jwtMount, role, audience, kvMount, kvPath}"
    - "user_data units: socialweb-openbao-link (dumbpipe connect-tcp --addr 127.0.0.1:18200), socialweb-node-secrets (oneshot: exchange the workload token for the audience at the bidder issuer, POST /v1/auth/<jwtMount>/login, read the KV path, write files 0600), socialweb-apiserver-link (advertise address on lo, dumbpipe connect-tcp --addr <advertiseAddress>:<apiserverPort>), k3s-agent; all Restart=always; no sshd use"
    - "the cluster OpenBao serving certificate must verify as 127.0.0.1 with the delivered CA"
  door:
    - "requestCompute handlers: write contract state; at accept write status.grant; return at onSecretsFetched, then dispose capabilities; no ssh session; release only when the claim is deleted; startup never releases a live claim"
acceptance:
  where: >
    Our GitLab (inspect git log -p docs/gitlab/, ssh gitlab, $GITLAB_TOKEN, chrome devtools MCP for
    the OAuth flow) running against a cluster created through the SPA on prod.
  checks:
    - id: ACC-ZERO
      text: An idle cluster sits at 0 nodes; a pipeline scales it up; nodes scale in one by one and it returns to 0 with the control plane Ready.
      status: done
      evidence:
        - "GitLab pipeline 77 (project johnandersen777/kcp-ci-probe, branch acc-0005, runner 6 acc-kcp-tenant-spa on hat-10 with ONLY the SPA kubeconfig + socialweb-credential) on cluster acc-gitlab-0001 created through the SPA by aliceoa.bsky.social (headless OAuth login) on prod, 2026-10-08, SCALE_HOLD=0s, DEBUG_SOCIALWEB_NODE_SSH_DIR still set (key unused): 3 jobs, one per node"
        - "cluster idle at 0 nodes before the run (kubectl get nodes: No resources found 00:00:50Z); 00:04:56Z pipeline -> 00:04:59Z Provision nodes:3 -> jobs running 00:06:07-09Z; scale-in one by one: node2 cordoned+drained 00:07:12Z released 00:07:15Z (Release nodes:2), node3 00:08:11Z/00:08:13Z (Release nodes:1), node1 released 00:09:15Z Teardown nodes:0; control plane Ready throughout (10 Ready status lines at 00:09:26Z, 0 nodes)"
        - "owed: the same run with the debug key off (final configuration)"
        - "FINAL CONFIGURATION: GitLab pipeline 78 (project johnandersen777/kcp-ci-probe branch acc-0005, runner 6 on hat-10 holding only the SPA kubeconfig + socialweb-credential) on SPA-created cluster acc-gitlab-0001, prod 2026-10-08, DEBUG_SOCIALWEB_NODE_SSH_DIR unset (container.env), SCALE_HOLD=0s, scope refresher deployed"
        - "0 nodes before (No resources found); pipeline 00:11:46Z -> Provision nodes:3 00:12:08Z -> claims Joining 00:12:39-46Z -> 3 jobs one per node; scale-in one by one: cordon+drain 00:14:29Z Release nodes:2 00:14:32Z, cordon+drain 00:15:33Z Release nodes:1 00:15:36Z, Teardown nodes:0 00:16:27Z; tenant lists no nodes after; control plane Ready"
    - id: ACC-MESH
      text: A pod on node A reaches a pod on node B, and DNS resolves on a node that does not run CoreDNS.
      status: done
      evidence:
        - "prod 2026-10-07T23:20:54Z: Deployment mesh-probe (2 replicas, required pod anti-affinity) on test-cluster-alice-0001 scaled the cluster 0 -> 2 market nodes ('Provision nodes:2'); node96 and node97 registered 23:22:30Z, Ready 23:22:31Z; pods 10.42.2.4 on node96 and 10.42.4.3 on node97"
        - "23:23:06Z kubectl exec (apiserver->kubelet over the agent tunnel) into both pods: node97 -> node96 pod IP answered by node96's httpd (HTTP 404 before its index existed), node96 -> node97 pod IP returned node97's content"
        - "23:23:16Z from the pod on node97 (CoreDNS runs on node96): resolv.conf nameserver 169.254.20.10; kubernetes.default.svc -> 10.43.0.1, mesh-probe.default.svc -> 10.43.21.222, gitlab.com resolves; wget via the Service ClusterIP returned node96's pod content"
        - "still owed: the same reading from a GitLab pipeline on an SPA-created cluster (acceptance 'where')"
        - "GitLab pipeline 77 (project johnandersen777/kcp-ci-probe, branch acc-0005, runner 6 acc-kcp-tenant-spa on hat-10 with ONLY the SPA kubeconfig + socialweb-credential) on cluster acc-gitlab-0001 created through the SPA by aliceoa.bsky.social (headless OAuth login) on prod, 2026-10-08, SCALE_HOLD=0s, DEBUG_SOCIALWEB_NODE_SSH_DIR still set (key unused): 3 jobs, one per node"
        - "probe-b on node3 at 00:06:09Z while CoreDNS ran on node2: resolv.conf nameserver 169.254.20.10 answered kubernetes.default.svc, kube-dns.kube-system.svc and gitlab.com; nslookup via the kube-dns Service 10.43.0.10 answered (pod on node3 -> CoreDNS pod on node2, cross-node)"
        - "FINAL CONFIGURATION: GitLab pipeline 78 (project johnandersen777/kcp-ci-probe branch acc-0005, runner 6 on hat-10 holding only the SPA kubeconfig + socialweb-credential) on SPA-created cluster acc-gitlab-0001, prod 2026-10-08, DEBUG_SOCIALWEB_NODE_SSH_DIR unset (container.env), SCALE_HOLD=0s, scope refresher deployed"
        - "jobs 277/278/279 all success: each resolved via node-local 169.254.20.10, via the kube-dns Service 10.43.0.10 and externally (gitlab.com); CoreDNS ran on node2 for the whole run (evicted at node2's release 00:15:33Z), so probe-a on node1 and probe-c on node3 reached the CoreDNS pod on node2 across nodes"
    - id: ACC-NOSSH
      text: Logs show zero ssh sessions to any guest for the whole run.
      status: done
      evidence:
        - "FINAL CONFIGURATION: GitLab pipeline 78 (project johnandersen777/kcp-ci-probe branch acc-0005, runner 6 on hat-10 holding only the SPA kubeconfig + socialweb-credential) on SPA-created cluster acc-gitlab-0001, prod 2026-10-08, DEBUG_SOCIALWEB_NODE_SSH_DIR unset (container.env), SCALE_HOLD=0s, scope refresher deployed"
        - "door log 00:11:40Z-00:16:30Z: 0 lines naming debug_node_ssh, sshAuthorizedKey, ssh sessions; prod container: no ssh or dumbpipe-connect processes to guests; the three guests of the run (fc-1379be663dd9-bd5c385d, -374a2854, -12381479) carry an EMPTY /root/.ssh/authorized_keys in their user_data and their consoles show no sshd connection; join path entirely OpenBao JWT over dumbpipe links (M3)"
        - "LOCAL reading (hat-8, no prod; prod acceptance stays open): TestAMarketNodeJoinsATenantClusterThroughItsOpenBaoWithNoSsh PASS 753.57s at 2026-10-07T23:55:13Z asserts it twice per run - no line of the door's log names runSession, ssh_session, pollReady, sshReady or ssh_connect, and the guest's own sshd records no Accepted, no session opened, no Failed password and no Invalid user, over a run that covered boot, the secrets fetch, the OpenBao login, the join, a control-plane restart and the release. The only sshd line the gate accepts is the guest's shell prompt and one 'Connection closed by <guest gateway> [preauth]', which is the compute provider's own TCP readiness probe of port 22, unauthenticated and pre-auth, not a session."
        - "the same reading at the claim level: the door's request carries no sshPublicKey and no sshAuthorizedKeys (node_market_test.ts 'the door asks for no ssh key and builds no ssh session'), and the debug key option DEBUG_SOCIALWEB_NODE_SSH_DIR was unset for every run of this gate"
items:
  - id: M0-BASELINE
    milestone: M0
    title: Measured baseline delta so a cluster holding only our stuff reads as nothing deployed
    decision: >
      Record the objects our installers wrote plus a hash of their live spec read back after apply;
      pods are resolved through their controller chain to a baseline workload; a baseline workload
      whose spec changed counts as intent. Chosen by the user over hand-stamping.
    status: done
    evidence:
      - "commit 65cd9d7 An idle cluster of only our own add-ons scales to zero"
      - "installer applies under field manager socialweb-addons; provider records what that manager applied with a live spec hash in the cluster's addon-baseline.json; scale.Narrow drops baseline workloads and pods whose owner is counted"
      - "tests watched to fail then pass: TestOurInstalledOperatorAloneIsNothingDeployed (demand 6 before), TestABaselineWorkloadWhoseSpecChangedIsIntent, TestATenantWorkloadBesideTheBaselineIsIntent, TestABaselineEntryNamesOneNamespace, TestTheSpecHashIsStableAndSeesAReplicaChange, TestObserveLeavesAnIdleClusterOfOurAddOnsAtNoIntent, TestTheBaselineIsWhatOurFieldManagerApplied, TestTheBaselineSurvivesARoundTripThroughItsFile; deploy/check.sh --fast green"
      - "prod 2026-10-07T16:48:28Z baseline measured: 3 ESO Deployments in default"
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): 65cd9d7->d389636"
  - id: M0-DEMAND
    milestone: M0
    title: Demand stops counting a pod and its owning workload twice
    decision: A pod whose controller workload is counted is not counted again.
    status: done
    evidence:
      - "commit 65cd9d7 (same change as M0-BASELINE)"
      - "tests watched to fail then pass: TestAPodAndItsDeploymentAreOneUnitOfDemand (6, want 3), TestPodsOfACountedJobStatefulSetOrDaemonSetAreNotCountedTwice"
      - "re-verified 2026-10-10: 65cd9d7 is d389636 after the 2026-10-08/09 history rewrite. Parent d389636^ with d389636's baseline_test.go: FAIL (build: undefined KindDeployment). Behavioural mutation on a scratch HEAD worktree 03:01:30Z (Narrow's pod/owner dedupe removed): TestAPodAndItsDeploymentAreOneUnitOfDemand FAIL 'demanded 6, want 3', TestPodsOfACountedJobStatefulSetOrDaemonSetAreNotCountedTwice FAIL 'demanded 5, want 3'; unmutated HEAD a7932be: ok internal/scale"
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): 65cd9d7->d389636"
  - id: M0-PROD-ZERO
    milestone: M0
    title: Prod idle cluster observed at 0 nodes with control plane Ready and no new RFPs
    decision: Proof required before M1 starts.
    status: done
    evidence:
      - "2026-10-07T16:48:30Z..16:51:37Z 96 of 96 scale passes: None, nodes 0, nothing running and no node; 0 RFPs, 0 claims, 0 no_bids"
      - "2026-10-07T16:52:07Z independent check: 111 of 111 passes in 5m None nodes 0; rfps=0 no_bids=0; tenant get nodes: No resources found; /readyz ok"
      - "re-verified 2026-10-10T02:59:16Z..03:00Z on prod: provider logged node scale action Teardown nodes 0 for test-cluster-alice-0001; test-cluster-alice-0001, acc-gitlab-0001 and acc-gitlab-0002 each list 0 nodes; bob runs 0 guests"
  - id: M0-PROD-BIDDER
    milestone: M0
    title: Retire the bidder that ran on the prod host
    decision: The user decided no bidder runs on prod; both doors rely on user-run bidders.
    status: done
    evidence:
      - >
        2026-10-07 ~16:00 UTC: socialweb-computer-bidder-firecracker stopped, disabled, unit moved
        out of /etc/systemd/system; 6 fc-1379be663dd9-* containers removed; no process left under
        its guests dir; 17G of guest disks deleted; bidder-key.hex, bidder.env and the unit kept in
        /root/retired-bidder; systemctl is-active reads inactive; prod / at 26%.
      - "re-verified 2026-10-10T03:01:53Z on socialweb.computer: systemctl is-active socialweb-computer-bidder-firecracker = inactive; no bidder-firecracker unit in /etc/systemd/system; /root/retired-bidder holds bidder-key.hex, bidder.env, socialweb-computer-bidder-firecracker.service; 0 fc-* containers"
  - id: M0-TMP
    milestone: M0
    title: Delete hat-8's three dead /tmp directories, never /tmp/live-k3s-*
    decision: >
      /tmp/rfp-guests, /tmp/nodeboot-check, /tmp/nodeimage-preinstall-check only, each re-checked for
      open files and process cwds immediately before deletion.
    status: done
    evidence:
      - "2026-10-07T16:51Z: no process cmdline or cwd referenced any of the three (own shell excluded), none written in the last hour; rfp-guests 6.6G, nodeboot-check 6.6G, nodeimage-preinstall-check 5.6G removed (its root-owned stage/ via sudo after /proc/mounts showed no mount under it)"
      - "2026-10-07T16:52:03Z /tmp 24G tmpfs at 12% (2.6G used, was 92%); all 7 /tmp/live-k3s-* untouched"
      - "re-verified 2026-10-10T03:01:44Z on hat-8: /tmp/rfp-guests, /tmp/nodeboot-check and /tmp/nodeimage-preinstall-check absent (test -e false for each); /tmp tmpfs 24G, 6.9M used"
  - id: M1-MARKET-NETWORK
    milestone: M1
    title: Vendor deno-kcp CRDs and deploy/market-network (with PR 2's bidder and bob PDS) as the live market network
    decision: >
      Vendored into this repo (user chose vendoring over building in deno-kcp). The bidder runs
      --compute-provider-firecracker with our nodeimage and nodeboot. Replaces the ad-hoc local
      dispatcher plus fake PLC harness for live tests. The bidder and requester use the in-cluster
      PLC (https://127.0.0.1:2587) and the market network's own dispatcher (relay.localhost:2588, TLS front
      for guests on 2589), never plc.directory or xrpc.fedproxy.com; the bidder runs --policy open
      because every identity is minted per run (decided by Claude, user may veto).
    status: done
    evidence:
      - "commit e997fe6 vendor deno-kcp f5b7c10c (pr-2) schemas, exports, workspace types, scripts and the market example; deploy/denokcp/vendor.json records commit and sha256 per file"
      - "commit c1451df build-provider.sh builds deno-kcp at the recorded commit against kcp-libs f6911d2 plus kcp-libs-no-default-issuer.patch (OpenBao 2.6/2.7 answer 400 'no default issuer' on an empty PKI mount; kcp-libs read only 404 as no authority)"
      - "commit fb6da3a firecracker bidder, dispatcher, TLS front, market-requester/mod.ts (runComputeContract, no ssh session), accept.sh market checks, provider --pod-timeout 24h, livegate test"
      - "commit d4448f2 guest onNetwork end to end (TLS front reads the HTTP/2 authority; requester user_data sets CURL_HOME=/root so units read the bidder's .curlrc)"
      - "commit 3e1f2c6 guest work root under /var/tmp, apply.sh refuses a work root whose firecracker socket path exceeds 107 bytes"
      - "tests watched to fail then pass: TestTheVendoredDenoKcpTreeIsExactlyItsRecord, TestTheRecordMatchesTheDenoKcpCommitItNames, TestEveryVendoredManifestDecodesAgainstTheVendoredSchemas (no vendor.json, then pass; a mutated 70-bidder.yaml fails all three); TestTheMarketNetworkRentsAFirecrackerGuestThroughTheRFPFlow (FAIL 964 s at 2026-10-07T20:09:51Z, guest socket path too long; PASS 684 s at 2026-10-07T20:22:12Z)"
      - "accept.sh on hat-8 2026-10-07: run 1 17:23Z 0/17 (OpenBao 400); run 2 17:30:42Z deno-worker bidder as upstream, accept: pass 17/17 (upstream bidder registered did:plc:lm4cljto7fyrug2xgjvsefh3 in the public plc.directory); run 3 17:45Z firecracker bidder up, RFP 403 scope filter (only-me); run 4 17:53Z bid, accept, receipt verified, guest booted, every DenoPod killed at the 5m pod timeout; runs 5-8 vm.delete 200 and guest gone, guest never got its OIDC token; run 9 19:53:16Z accept: pass 23/23, onNetwork iroh ticket 335 s after the RFP, vm.delete 200, guest container gone 2 s after release"
      - "deploy/check.sh --fast green 2026-10-07T20:23:04Z on the branch merged with main"
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): 3e1f2c6->5060160 c1451df->04dece8 d4448f2->09e4f31 e997fe6->0c3dab8 fb6da3a->2d40964"
  - id: M1-MARKET-ONLY
    milestone: M1
    title: Retire clusterprovider's local firecracker and qemu substrates
    decision: >
      The market path with a local bidder covers the same market network; one node path remains. The VMM
      code survives where the bidder's nodeboot needs it. decided by deepseek executor, lead may
      veto: the subnet allocator STAYS in the composition root, because internal/materialiser
      requires it to release a cluster's range beside its ports on teardown and removing it ripples
      through materialiser, the legacy-key migrator and check-host for no behaviour a market cluster
      has; the node hosts no longer take it. decided by deepseek executor, lead may veto: the node
      record sweep keeps walking and keeps refusing to signal on a record it cannot verify, but a
      record from a retired substrate is now REPORTED and left in place rather than reaped, because
      the market path writes no record at all (a rented node is a NodeClaim) and every remaining
      record names a process whose argv this provider can no longer reconstruct.
    status: done
    evidence:
      - "commit cd12808: internal/nodeqemu deleted; internal/nodefirecracker trimmed to the image, manifest, namespace-child, fingerprint and guest-document halves the bidder's nodeboot builds and boots; SOCIALWEB_CLUSTERPROVIDER_SUBSTRATE accepts market alone and refuses firecracker/qemu by name and retirement date; the variables only the retired substrates read (FIRECRACKER, QEMU, NODE_AGENT, IMAGE_OUT_DIR, IMAGE_CONFIG, NETWORK, NODE_DISK_MIB) are refused rather than parsed; GuestDialled, GuestGateway, nodeVaultStage, the per-node network allocation, AssertGuestAdvertiseInvariant, dialFromAGuest and internal/nodevm's guest-dial helpers deleted; internal/clusterprovider's guest advertise invariant and image-config check deleted; deploy/control-plane-container's computer target and nodeimage-qemu stage, deploy/test-rig and cmd/socialweb-rig-inputs deleted; run-container.sh keeps the k8s target"
      - "tests watched to fail then pass: cmd/socialweb-clusterprovider TestTheRetiredSubstratesAreRefusedAtStartup, TestTheRetiredSubstratesAreRefusedAndNameTheirRetirement, TestNoPlacementSuppliesAProcessIdentityAnyMore, TestTheMarketSweepLeavesARetiredSubstratesRecordAlone, TestASweepLeavesARecordWhoseSubstrateItCannotRead, TestShutdownLeavesTheMarketNodesToTheNextProvider, TestTheCredentialSourceIsTheClustersOwnCA; internal/clusterstack TestAStackWithNoDialFlagValidates, TestOnlyAMarketClusterDropsFlannel; internal/portallocator dropped AssertGuestAdvertiseInvariant and AssertGuestDialAddress with no remaining caller"
      - "deploy/check.sh --fast green 2026-10-07T21:50:55Z on the branch at cd12808 (gofmt, go build, apigen no drift, go vet, staticcheck, cargo, go test ./...); go build ./... and go vet ./... clean again at b2e931a 2026-10-07T21:51:03Z"
      - "re-verified 2026-10-10: cd12808 is 9d85916, b2e931a is 5b48f5d after the history rewrite. Parent 9d85916^ with 9d85916's config_test.go/market_test.go: FAIL (build: undefined RetiredSubstratesRetiredOn). Behavioural mutation 03:01:30Z (checkPlacement's default branch returns nil): TestTheRetiredSubstratesAreRefusedAtStartup FAIL 'firecracker was accepted as a placement'; HEAD a7932be: ok cmd/socialweb-clusterprovider"
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): b2e931a->5b48f5d cd12808->9d85916"
  - id: M1-AGENT-SPLIT
    milestone: M1
    title: Split socialweb-nodeagent into a static guest binary (init and agent) and a host binary (namespace and VMM runner)
    decision: >
      User suggested the split; market guests stop carrying a host half that always fails. Mapping on
      2026-10-07 showed nodeboot already re-execs ITSELF for the namespace verb, and the agent verb
      needs an identity disk only local substrates attach, so on market guests it always exits
      KCP_AGENT fatal. With M1-MARKET-ONLY the split therefore lands as: guest binary
      socialweb-nodeinit (initramfs /init only), host binary socialweb-nodeboot (unchanged); the agent
      verb, /kcp/kcpagent and kcp-agent.service are deleted, the guest joining through cloud-init
      units (M3-UNITS). Done in one change with M1-MARKET-ONLY after M1-MARKET-NETWORK proves the market
      path, because both rewrite the same files. decided by Claude, user may veto. decided by
      deepseek executor, lead may veto: the image configuration's old key names are REFUSED with a
      message naming the new ones rather than accepted for a release, and the operator's
      configurations under /home/johnandersen777/kcp-image/ were rewritten to the new names and
      fingerprinted again (the old spellings are refused, so leaving them would have made every
      build host fail).
    status: done
    evidence:
      - "commit cd12808: cmd/socialweb-nodeagent -> cmd/socialweb-nodeinit with the init verb alone; agent.go, audit.go, agent_test.go, the guest identity-disk join, LoadNodeDocument, /kcp/kcpagent and kcp-agent.service deleted; kcp-agent.service added to rootfs staleUnits so an image built on a base that carried it retires it; image config keys init_package/init_binary/init_go_flags with RefuseRenamedKeys refusing agent_package/agent_binary/agent_go_flags; every JSON config in the repo and scripts/deploy-firecracker-bidder.sh updated"
      - "image built by the real builder from the renamed configuration 2026-10-07T21:47:45Z (logs/image-build.log): fingerprint sha256-bcdfbb6824a7743d8fd4c15d1be1ad1bda35785fe97a0f1278ebfa507b98be00, stage /kcp holds guest.yaml alone, multi-user.target.wants holds no kcp unit, and the initramfs's /init is a stripped static x86-64 Go binary whose strings carry KCP_INIT fatal and 13 nodeinit markers and no KCP_AGENT marker"
      - "internal/nodefirecracker TestAnInitPackageIsRelativeToTheModuleAndNotToWhereTheReaderStands and cmd/socialweb-nodeimage TestAnImageBuiltBeforeTheRecordCoveredTheInitIsRefused, plus the input-role table renamed; fingerprint tests updated deliberately (the input's name and content are both inputs)"
      - "the market network live test was attempted twice and did NOT reach a verdict on this change: 2026-10-07T21:48Z SKIPPED (ports 2583-2589 held by an earlier session's market network), then 2026-10-07T21:55:29Z FAIL in 120 s with the MARKET NETWORK's own bring-up broken -- apply.sh failed 3/3, no OpenBao ready, no DenoPod in any workspace, the OpenBao CA unreadable, and no guest work directory ever created -- while that other session's market network came back up on the same fixed ports (ss shows the listeners again). The failure is upstream of everything this change touches: the market network never reached an RFP, so the node image, the init binary and the renamed image keys were never read. Nothing in this change is known-failing; the live reading is owed on a host with no second market network running"
      - "REOPENED 2026-10-07T22:50Z by the lead: the image bcdfbb68 above was built WITHOUT -preinstall (image.json preinstall \"\"), so its rootfs carries no cloud-init (no /etc/cloud, no cloud-init.target.wants, no /usr/bin/cloud-init; debugfs on the bidder store). Prod guest fc-1379be663dd9-4d49fee1 booted it 22:23Z, console stops after fcnet.service, no cloud-init stage, door node_claim_ended outcome network_timeout 22:43:08Z. Done needs an image built WITH preinstall-market.json, shipped, and a market guest that runs cloud-init on it"
      - "live 2026-10-07: guests on images 0437ff0d and 52df8a09 (built from cmd/socialweb-nodeinit, no kcp-agent unit) ran cloud-init and joined; console shows no KCP_AGENT line"
      - "re-verified 2026-10-10: cd12808 is 9d85916 after the history rewrite; tree at a7932be has cmd/socialweb-nodeinit and no cmd/socialweb-nodeagent; parent 9d85916^ with 9d85916's inputs_test.go: FAIL [build failed]; HEAD: ok cmd/socialweb-nodeimage (TestAnImageBuiltBeforeTheRecordCoveredTheInitIsRefused)"
      - "live 2026-10-10T03:01:44Z: the image prod guests boot today (sha256-5b823ce0, built 02:43Z from main 1ecc684) has /kcp holding guest.yaml alone, no kcp-agent.service in /etc/systemd/system, its multi-user.target.wants or /usr/lib/systemd/system, and no /usr/local/bin/socialweb-nodeagent; guests on it joined via cloud-init at 02:44Z (np-client/np-server nodes) and 02:58:45Z (sched-probe node)"
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): cd12808->9d85916"
  - id: M1-IMAGE-SHIP
    milestone: M1
    title: The bidder installer ships exactly the image the current inputs name, never a stale one
    decision: >
      scripts/deploy-firecracker-bidder.sh asked the build host -ensure -json -reuse-stale, which answers with
      the newest whole image in the store whatever its inputs, so it shipped bcdfbb68 (no preinstall) while the
      current inputs fingerprint to sha256-144a6d33 (target ensure 2026-10-07T22:22Z reusedFrom bcdfbb68). The
      build host's ensure must name the exact image of the current inputs, building it when absent through
      sudo -n with the go.mod toolchain (the preinstall chroot needs CAP_SYS_CHROOT) and handing ownership back;
      the target keeps -reuse-stale (busybox naming). An image whose manifest lacks the preinstall the config
      asks for is refused. decided by Claude, user may veto. decided by deepseek executor, lead may veto: the
      build host's ask is a new non-building mode of the builder (socialweb-nodeimage -ensure -json -present-only)
      rather than a reuse answered by the store, because -ensure alone answers the same question by BUILDING and
      an unprivileged build only fails at the preinstall chroot, after the base image and every seeded image have
      been unpacked into the stage (option A, chosen by the lead 2026-10-07T22:50Z); the build under sudo runs
      with TMPDIR unset (env -u TMPDIR) because the preinstall chroot inherits TMPDIR and dpkg's ca-certificates
      configure step then dies in mktemp; -reuse-stale with -present-only is refused rather than ordered. The
      build host's own step (a) runs for real in --dry-run too, because it is the question and it writes nothing,
      and the store is not checksummed in a preview of a host that does not hold the image yet.
    status: done
    evidence:
      - "5409398 nodeimage: -ensure refuses any image it resolves (present, reused or built) whose manifest disagrees with the run about whether a preinstall exists, naming the image and the manifest; -present-only is the non-building presence question (prints {state:absent,...} and exits non-zero when the exact-fingerprint image is not whole, builds/reuses/writes nothing); a host that must not build asks it before deciding to build"
      - "9797f50 + ea4f664 bidder deploy (ea4f664 is the state of the installer to run): the build host answers in two steps -- (a) -ensure -json -present-only as the invoking user, (b) sudo -n env -u TMPDIR on the toolchain the go line in go.mod selects, then chown the new image dir and the stage back and ask (a) again, which must answer present; --dry-run prints every one of these steps; the target keeps -reuse-stale but its resolved dir must now be the shipped fingerprint, and remote_vars is a function so the target's script can carry FINGERPRINT"
      - "tests watched to fail then pass: TestEnsureRefusesAnImageThatRecordsNoPreinstallWhenOneIsNamed and TestEnsureRefusesAnImageThatRecordsAPreinstallWhenNoneIsNamed (before: the stale image was handed back), with controls TestEnsureUsesAnImageThatRecordsThePreinstallTheInputsName, TestPresentOnlySaysAbsentAndBuildsNothing, TestPresentOnlySaysPresentForTheImageTheInputsName; TestTheBidderDeployShipsTheImageTheInputsNameAndBuildsItUnderSudo runs the script with --dry-run and asserts the build host's ask is -present-only with no -reuse-stale, the sudo build step with TMPDIR unset and the go.mod toolchain, and the chown back (before: the preview died on '-reuse-stale asked for an image already in the store, and the store at ... holds none that this configuration could have produced')"
      - "deploy/check.sh --fast green at ea4f664 (gofmt, go build, apigen no drift, go vet, staticcheck, cargo fmt, cargo test, go test ./...) -- the last commit, and the working tree the check ran on is that commit"
      - "not proven here, and the lead runs them: a real build under the new (b) and a real install. Expect the first real install to fail at the target's check if the target's store still holds the preinstall-less bcdfbb68 newer than the shipped image: that is the check doing its job, and the fix is to reap the target's other store directories"
      - "live 2026-10-07T23:39Z: the installer resolved 'the build host holds no image of these inputs: building it', built sha256-52df8a09 under sudo, shipped it, the target resolved exactly that dir; a guest booted it 23:47:43Z"
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): 5409398->317e739 9797f50->3e5d90a ea4f664->2010161"
  - id: M1-BIDDER-UNIT
    milestone: M1
    title: System template unit socialweb-computer-bidder@<handle>.service
    decision: >
      System template unit (user chose it over a user unit because firecracker needs kvm and
      runuser); %i is the atproto handle; QR sign-in is read from the journal by the operator;
      enabled with systemctl enable --now socialweb-computer-bidder@<handle>.service.
      Decided by Claude, user may veto: the launcher holds the QR sign-in to %i by reading the handle
      the bidder logs and refusing (session deleted, exit 1, restart into a fresh QR) any other
      account; the shared dir is /var/lib/socialweb-computer-bidder/host and every instance runs as
      swc-bidder; MANAGE_DOCKER_DAEMON=no leaves daemon.json alone on a host with other containers.
    status: done
    evidence:
      - "defect found 2026-10-07 ~19:20Z: QR waits time out after 5m and the unit restarted under a NEW did:plc each time (4 PLC registrations in 20 min, did:plc:474wle.., yhwrng.., 7ambb2.., jznwst..); unit stopped"
      - "5330ff2: launcher writes the 0600 key before the first start; test TestTheBidderStartsWithTheSameKeyEveryTimeBeforeASignIn watched to fail (no key at start) then pass; reinstalled on hat-8 with deploy-firecracker-bidder.sh --local"
      - "hat-8 2026-10-07 ~19:55-20:01Z: private_key_loaded_from_path, oauth_qr_did_registered did:plc:4iobc5hpgsfb5flxisath2pt once, next cycle oauth_qr_did_resolved same did; unit running, awaiting the operator QR scan"
      - "commit 51f9000 The bidder is a template unit keyed by the account's handle (deploy/bidder/socialweb-computer-bidder@.service, deploy/bidder/socialweb-computer-bidder-run, scripts/deploy-firecracker-bidder.sh reworked with --local and --dry-run, test/bidder_unit_test.go)"
      - "tests watched to fail (files absent, script still writing bidder.env and an account secret) then pass: TestTheBidderUnitIsATemplateKeyedByTheAccountHandle, TestNoAccountSecretReachesTheBidder, TestTheBidderDeployInstallsTheTemplate, TestTheLauncherSignsInByQRIntoItsOwnStateDirectory, TestTheLauncherRefusesAnAccountOtherThanItsInstance (restored, scanned, unnamed), TestTheLauncherAcceptsTheHandleInAnyCase, TestTheLauncherRefusesAnInstanceThatIsNotAHandle, TestTheBidderUnitVerifies (systemd-analyze verify, any output fails); deploy/check.sh --fast green"
      - "hat-8 2026-10-07T17:10Z deploy-firecracker-bidder.sh --local (MANAGE_DOCKER_DAEMON=no, prod untouched): rc 0, store verified, template/launcher/common.env installed, module cache warm, target ensure -reuse-stale answered reused from sha256-6e4f4f95...; second run rc 0 with all three unchanged and nothing restarted"
      - "hat-8 2026-10-07T17:11:24Z systemctl start socialweb-computer-bidder@bobvmbuilder.bsky.social with its session seeded as a 0600 swc-bidder copy of ~/.cache/pdr-market/bob-bidder-oauth-session.json: restore refused by the PDS (invalid_grant, Refresh token replayed -- the source file's refresh token was already spent, the source is unchanged), bidder deleted the copy, journal at 17:11:26Z shows the 60-char = frame, the QR, https://qr.fedfork.com/#oauth=did%3Aplc%3Awadxdef7inpvtaen77n7cfxw&n=... and oauth_qr_awaiting_session; bidder_ready not reached without a scan"
      - "hat-8 2026-10-07T17:11:38Z systemctl disable --now: is-active inactive, is-enabled disabled, 0 swc-bidder processes; template, launcher, common.env and bobvmbuilder.bsky.social.env (BIDDER_POLICY=tangled-vouch) left installed"
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): 51f9000->e0e2c6e 5330ff2->910f622"
  - id: M2-EVENT-CORE
    milestone: M2
    title: ABC-layered event-driven requester core in atproto-market
    decision: >
      Callbacks onRfpSubmitted, onBid (accept or deny), onAccepted, onReceipt, onNetwork,
      onSecretsFetched; each decides whether the flow continues or returns. No ssh session is held
      by the core.
    status: done
    evidence:
      - "atproto-market pre-iroh cdb0f9c A compute contract is a state that can be saved, and its events can stop it"
      - "atproto-market pre-iroh 28a1be5 The requester's flow runs without an ssh session, and release stands alone"
      - "packages @publicdomainrelay/compute-request-abc (ContractState, phases requested..released, handlers onRfpSubmitted/onBid/onAccepted/onReceipt/onNetwork/onSecretsFetched, parse/serialize) and @publicdomainrelay/compute-request-xrpc (requestCompute, releaseCompute, awaitComputeNetwork); secrets capability reports onServed -> onSecretsFetched"
      - "tests fail->pass: test/compute_request_test.ts (event order, return at onRfpSubmitted, return at onNetwork submits no vm.delete, onBid deny never accepted, all-denied no contract, onSecretsFetched return, awaitComputeNetwork resume), test/compute_request_state_test.ts (7), test/secrets_capability_test.ts onServed; whole suite 109 passed 0 failed 4 ignored (was 92)"
      - "not yet proven: awaitComputeNetwork backfill path; onSecretsFetched with a real guest; a live early return -- closed by M2-CLAIM-RECEIPT and M3 live runs"
      - "re-verified 2026-10-10: cdb0f9c is a260ed0 and 28a1be5 is 1a0a83b on atproto-market pre-iroh after the history rewrite; 03:02:05Z at pre-iroh 57bbcd3 test/compute_request_test.ts + compute_request_state_test.ts + secrets_capability_test.ts: 35 passed, 0 failed"
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): 28a1be5->1a0a83b(atproto-market) cdb0f9c->a260ed0(atproto-market)"
  - id: M2-RELEASE
    milestone: M2
    title: Standalone releaseCompute(receipt) that submits vm.delete
    decision: Release no longer depends on a session returning inside the process that accepted the bid.
    status: done
    evidence:
      - "28a1be5 releaseCompute(pds, state) submits vm.delete from a saved ContractState"
      - "test: releaseCompute from a JSON round-tripped state submits vm.delete from another process; no receipt refs submits nothing"
      - "live: bidder_container_integration_test vm_delete_result 200 through the wrapper (release from the SAME owner DID; the door releases as the owner, so a foreign signer is not needed)"
      - "re-verified 2026-10-10: 28a1be5 is 1a0a83b on pre-iroh. Behavioural mutation 03:02:43Z (releaseCompute returns before vm.delete even with receipt refs; file restored by git checkout, status clean): 'releaseCompute from a JSON round-tripped state submits vm.delete from another process' FAILED (17 passed, 1 failed); unmutated 57bbcd3: 35 passed, 0 failed"
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): 28a1be5->1a0a83b(atproto-market)"
  - id: M2-WRAPPER
    milestone: M2
    title: runComputeContract reimplemented as a thin wrapper over the event core
    decision: socialweb-computer-ssh and request-vm-ssh stay unchanged (user chose to leave the ssh door on the wrapper).
    status: done
    evidence:
      - "28a1be5 runComputeContract = requestCompute + existing ssh session + releaseCompute unless keepVm; signature, options, ContractFlowResult unchanged; old flow code deleted"
      - "test/bidder_container_integration_test.ts green before (38s) and after (44s), SSH_OK_VIA_IROH"
      - "deno check green: socialweb-computer-ssh (task check, requester-inproc, live_market_test), socialweb-computer-k8s (task check, node-market, requester-inproc, node_market_test, live_market_node_test)"
      - "re-verified 2026-10-10: 28a1be5 is 1a0a83b on pre-iroh; every prod contract since runs through it: today's joins (02:44Z, 02:58Z) ran requestCompute -> releaseCompute through the door, and bob ran 0 guests after the 02:59:16Z Teardown"
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): 28a1be5->1a0a83b(atproto-market)"
  - id: M2-CLAIM-RECEIPT
    milestone: M2
    title: NodeClaim tracks receipt and contract URIs
    decision: >
      The claim is the durable record a restarted door resumes from. decided by Claude, user may
      veto: the fulfiller drives requestCompute; onRfpSubmitted/onAccepted/onReceipt/onNetwork each
      await a status write of the full ContractState (legacy vmUri/rfpUri/acceptUri/receiptUri/
      receiptCid mirrored); after the ssh session the door reads the claim back (new
      NodeClaimStore.get) and calls releaseCompute with the state parsed from the claim. A claim
      needs release iff it has receipt + submitEventRef and contract phase != released; released
      claims keep the receipt plus release refs. Startup / not-ours / deleted claims are released by
      releaseCompute from the saved state (reason requester_restarted), replacing reapContract; still
      a reap (adopting is M3-NODE-LOSS). Released only on release.ok. Unreadable claim at run end
      defers the release to the next pass. No early return from onAccepted (would lose the receipt).
      ARCHITECTURE section 6 "The claim is the durable record a door releases from".
    status: done
    evidence:
      - "merged to main as 8af4f67 (via 444755e, 4c60f9d); door deno task test 80 passed 0 failed on main"
      - "prod deploy 2026-10-07 ~18:40Z: door started, 95 historical claims (68 Released, legacy receipt fields only) produced no release and no vm.delete in the first 4 minutes (18:41:28Z reading)"
      - "pending: a live door restart between accept and release against a real bidder; it runs on the M1 market network firecracker bidder once M1-MARKET-NETWORK lands"
      - "444755e node-market: requestCompute + handlers writing status.contract, releaseFromClaim via store.get + releaseCompute; reapContract/contractFromEvent/runContract option removed; NodeClaimStore.get in abc + nodeclaim-kcp (404 -> null, 429 hold applies)"
      - "instrument: test/node_market_test.ts 'a door restarted between accept and release still releases the node with vm.delete from the receipt on its NodeClaim' -- door 1 (fake kcp store + fake requester PDS, scripted requestCompute) hangs after onReceipt and is abandoned; a fresh fulfiller with a fresh PDS submits exactly one vm.delete whose event.receipt == the claim's receipt and goes to the claim's submitEventRef; claim ends Released, contract phase released, release.ok; second pass does not release twice"
      - "red first: the new suite against the old fulfiller (stub exports only) 16 failed / 3 passed, the instrument among them"
      - "mutations (scratch copy only, reverted): release from in-memory state instead of the read-back claim -> 'reads the receipt back from the claim' FAILS; restarted door ignoring the saved ContractState -> instrument FAILS (7 fail); onReceipt not writing the claim -> instrument + milestone test FAIL"
      - "green: socialweb-computer-k8s deno task check exit 0; deno task test 80 passed 0 failed 1 ignored (live_market_node_test: SOCIALWEB_K8S_LIVE unset and SOCIALWEB_FIRECRACKER_* unset; k3s/kine/kubectl/docker present); run from a scratch org root (siblings symlinked, worktree rsynced) because the worktree's ../../ imports do not resolve; deploy/check.sh --fast green"
      - "test/live_market_node_test.ts compiles against the new API (requestCompute wrapper adds extraBidderDids + guestHostAliases; asserts contract phase released + release.ok after delete); NOT run"
      - "not proven: a live door restart against a real bidder; the live instrument end to end on the new path; status writes failing mid-flow (a failed onReceipt PATCH leaves the claim without its receipt and the VM unreleasable from the claim); deploy needed (not done, user deploys) -- door code only, no schema change"
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): 444755e->d81ff42 4c60f9d->d2a7426 8af4f67->314bc58"
  - id: M2-CODEGEN
    milestone: M2
    title: Go api/v1alpha1 generates the APIResourceSchema and a TS types plus validator package for the door
    decision: >
      Go types are the source of truth; drift check in deploy/check.sh; shared JSON fixtures both
      suites round-trip. Protobuf rejected because CRDs travel as JSON/OpenAPI through kube and kcp.
      Mapping rules (reflection over json tags, apigen:"optional" struct tag, enums from source
      constants, revision lock), decided by Claude, user may veto; recorded in ARCHITECTURE section 6
      "Generated from api/v1alpha1".
    status: done
    evidence:
      - "merged to main as 5d5ce05; check.sh --fast green (apigen: 8 generated files, no drift); door deno task test 73 passed 0 failed"
      - "prod deploy 2026-10-07 ~17:47Z: workspace b3wgagx6dzltmtsx binding clusters-3pj7y now bound to v1alpha1-2.nodeclaims.socialweb.computer (re-pointed export, no rebind); all 95 pre-existing claims still list; door logged no claim refusal, only one transient 429 storage (re)initializing at 17:47:56Z right after kcp start"
      - "d1cb357 cmd/socialweb-apigen + internal/apigen: go generate ./api/... writes deploy/*-apiresourceschema.yaml, both APIExports, deploy/apiresourceschema-revisions.json and socialweb-computer-k8s/lib/common/socialweb-computer-k8s-api/mod.ts; Cluster, Node, AtprotoIdentity schemas and both exports byte-identical to the hand-written ones"
      - "d1cb357 NodeClaim status.contract += every ContractState field (version, phase, vmName, requesterDid, marketDid, payload/rfp/policy/bid/accept/receipt strongRefs, winner, receiptOk, vmAddress, bids, release, outcome, error, timestamps), all optional; old six fields kept; schema renamed v1alpha1-2.nodeclaims.socialweb.computer, clusters export re-pointed"
      - "99e894c door: socialweb-computer-k8s-common drops its NodeClaim wire types; the claim view moves to lib/abc; nodeclaim-kcp parses every claim with parseNodeClaim and asserts a status before PATCH"
      - "ee7e07c live proof internal/apigen TestAnAlreadyBoundWorkspaceKeepsTheNewContractFieldsAfterTheSchemaUpgrade on kcp v0.33.0 + kine: already-bound tenant prunes new fields on v1alpha1-1, in-place spec change Forbidden, re-point moves the existing binding to v1alpha1-2 (identity hash unchanged), new fields persist; PASS 14.8s; mutated to re-apply the old export it FAILS (still bound to v1alpha1-1)"
      - "red first: socialweb-computer-k8s/test/k8s_api_fixtures_test.ts failed TS2307 (package missing); internal/apigen fixtures_test failed to build (no model); then both green"
      - "drift red: a Go field added to NodeClaimStatus without a Revision bump -> check.sh FAILED apigen (revision lock: spec of v1alpha1-2 changed); with the bump but not regenerated -> check.sh FAILED apigen drift (diff of lock, export, schema, mod.ts); reverted"
      - "green: deploy/check.sh --fast (ok apigen, 8 generated files); socialweb-computer-k8s deno task check exit 0; deno task test 72 passed 0 failed 1 ignored (from a real-org-root copy; the worktree's ../../ siblings do not resolve)"
      - "deploy needed (not done, user deploys): install-provider.sh applies deploy/nodeclaim-apiresourceschema.yaml (creates v1alpha1-2) then deploy/cluster-apiexport.yaml (re-point); no binding migration; the door and the schema must ship together"
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): 5d5ce05->e9eab79 99e894c->9322188 d1cb357->42b18ed ee7e07c->5364b92"
  - id: M2-BACKOFF
    milestone: M2
    title: Exponential no_bids back-off per cluster, capped at 10m
    decision: >
      decided by Claude, user may veto. A claim ending Failed with status.contract.rfpUri set and
      acceptUri empty (nodemarket.MarketDeclined: the requester's NO_CONTRACT_OUTCOMES class, no_bids,
      bids_denied, policy_rejected, which all loop identically) backs that cluster's further market
      provisions off 30s*2^(n-1), capped at 10m; one end counted per confirm pass. A node newly
      joining or intent dropping to none resets it; a provision does not. Classified from record
      URIs, not message text; no status.reason field because APIResourceSchema.spec is immutable
      and a schema change does not reach existing APIBindings (internal/versions/versions.json), so
      no schema bump and no door code change. State in memory only: a provider restart resets it.
      Failures after an accept keep today's behaviour. ARCHITECTURE section 10.
    status: done
    evidence:
      - "merged to main as f4ce9be (fast-forward); deployed to prod 2026-10-07 ~17:34Z"
      - "prod proof 2026-10-07: probe pod (pause) in tenant default ns at 17:36:15Z, no bidder on the market; RFP rounds at 17:36:19, 17:37:24, 17:39:01, 17:41:37 (gaps 65/97/156/272s = 30s bid window + back-off 30/60/120/240s); log: backing off market provisions until 17:37:21Z/17:38:57Z/17:41:33Z/17:46:09Z with 1..4 provisions in a row"
      - "pod deleted 17:45:16Z; next 30s: 15 passes None nodes 0 nothing running and no node (intent gone resets, no RFP at 17:46:09)"
      - "0807250 scale.State.Backoff + scale.BackoffDelay + Decision.BackoffUntil; scaledriver Declined/Joined; nodemarket.MarketDeclined + Host.End; confirmHeld reports declined/newly-joined nodes, wiring feeds the driver before the pass and logs backoffUntil on the node-scale line"
      - "0807250 also fixes tenantNodes.retain dropping ordinal 1 when the primary slot was empty: a first node claimed again after the first was given up was never held, confirmed, released or counted in flight (surfaced by the back-off test: one gap read 5s)"
      - "d336b86 door guard: socialweb-computer-k8s test/node_market_test.ts 'a no-bids end keeps the rfp it asked with and names no accept'; door deno task check green, deno task test 67 passed 0 failed 1 ignored (run from a copy at org-root depth, the worktree's ../../ siblings do not resolve)"
      - "tests watched to fail (against stubs: gaps read [5s 5s 5s 5s 5s 5s 5s], the prod loop) then pass: cmd/socialweb-clusterprovider TestNoBidsEndsBackTheMarketOffAtGrowingIntervalsCappedAtTenMinutes (gaps 35s 1m5s 2m5s 4m5s 8m5s 10m5s 10m5s at a 5s pass), TestAMarketNodeThatJoinsResetsTheBackoff, TestIntentGoneResetsTheMarketBackoff, TestAFirstNodeClaimedAgainAfterItWasGivenUpIsHeld; internal/scaledriver TestConsecutiveNoBidsEndsProvisionAtGrowingIntervalsCappedAtTenMinutes, TestAJoinedNodeResetsTheBackoff, TestIntentGoneResetsTheBackoff, TestADeclinedEndRecordsWhyAndUntilWhen; internal/scale TestTheBackoffDoublesFromItsBaseAndStopsAtItsCap, TestADeclinedMarketHoldsTheNextProvisionUntilTheWaitIsOver, TestTheBackoffSurvivesAProvisionAndDiesWithTheIntent; internal/nodemarket TestAnEndWithAnRFPAndNoAcceptIsTheMarketDeclining, TestTheHostReportsADeclinedEndWithTheDoorsWords"
      - "guards that passed first time: TestAFailureAfterABidWasAcceptedDoesNotBackOff, TestANodeCountAtTheTargetIsNotReportedAsBackingOff"
      - "deploy/check.sh --fast green after 0807250"
      - "not yet proven on prod: a cluster with intent and no bidder reading RFPs at 30s/1m/2m/4m/8m/10m gaps and the 'backing off' node-scale line -- needs a deploy"
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): 0807250->ee19ae2 d336b86->ec19df7 f4ce9be->5a067f9"
  - id: M3-CLUSTERLINK
    milestone: M3
    title: ClusterLink CRD and per-cluster dumbpipe listeners for the k3s apiserver and the cluster's OpenBao
    decision: >
      Listeners are procgroup components started after k3s beside postgres, openbao, kine and k3s.
      Guests dial the cluster; the door holds no per-node connection.
    status: done
    evidence:
      - "merged to main 0f24132 (links) and 3656ae9 (provider rents only once both links listen); deployed 2026-10-07 ~19:13Z"
      - "prod 2026-10-07T19:14:59Z: ClusterLink 2wjeqtgx-test-cluster-alice-0001-apiserver Listening endpointId 2374b8ee..8059 and -openbao Listening endpointId a7d94692..1a6c; processes dumbpipe listen-tcp --host 127.0.0.1:20003 --ipv4-addr 0.0.0.0:20008 and --host 127.0.0.1:20001 --ipv4-addr 0.0.0.0:20009"
      - "3656ae9: TestNothingIsRentedUntilBothClusterLinksListen (mutation: gate disabled -> FAIL); TestConfirmJoinedNeedsTheGuestToHaveFetchedItsSecrets (mutation: Joining not accepted -> FAIL); TestTheDoorFindsTheClusterOpenBaoCABesideTheTokenTheClaimNames"
      - "41349d6 portallocator purposes apiserver-link and openbao-link (width 1, Purpose.Datagram); portscan reads /proc/net/udp{,6} state 07 for datagram leases and the TCP tables for the rest"
      - "596bb86 procgroup.ComponentApiserverLink/ComponentOpenBaoLink and Step.Prepare (runs once the steps before it are ready; its spec is what starts)"
      - "7c308b4 kcpclient WriteClusterLink (create spec, then status subresource; spec and status merge-patched only when changed), WriteClusterLinkStatus (never creates; NotFound is not written), ReadClusterLink, DeleteClusterLink"
      - "8f2211a clusterprovider/links.go: StackConfig.Links appends two steps after every StackOrder member, argv dumbpipe listen-tcp --host 127.0.0.1:<k3s secure or vault port> --ipv4-addr 0.0.0.0:<link lease>; gate = ticket printed and naming the vault key; reporter writes Listening with retries, Failed (exit + output tail) on an unexpected exit, Pending when the group was reaped"
      - "90f4130 wiring: on when SOCIALWEB_CLUSTERPROVIDER_DUMBPIPE is set (entrypoint default /usr/local/bin/dumbpipe), link purposes leased with the rest, restart argv derived without unleased links, a deleted cluster's two ClusterLinks deleted after its reap"
      - "c90f9f2 ARCHITECTURE section 5 'The cluster links', component 8, CRD table, section 8 ports; permission-inventory: socialweb-cluster-provider may write clusterlinks and clusterlinks/status"
      - "tests watched to fail then pass: internal/clusterprovider TestTheLinksAreComponentsOfTheGroupStartedAfterK3sInFrontOfTheApiserverAndTheVault, TestEachLinkIsRecordedListeningWithTheStableTicketAndTheDidKeyOfItsKey, TestAProviderRestartBringsTheLinksBackAsTheSameEndpoints, TestALinkThatDiesIsRecordedFailedWithWhatItLastSaid, TestTheLinksAreReapedWithTheGroupFirstInReverseOrder (red with Prepare removed, and with the stop-vs-crash split removed); internal/portscan TestALinkLeaseTakesItsOwnersFromTheDatagramTableAndEveryOtherLeaseFromTheStreamTable, TestASurvivingLinkIsAttributedToItsClusterLikeEveryOtherComponent (red with Datagram ignored); internal/kcpclient TestWriteClusterLinkCreatesItInTheHomeWorkspaceAndThenWritesItsStatus (red with the status patch skipped); cmd/socialweb-clusterprovider TestADeletedClusterLosesBothItsClusterLinks, TestARestartOverAClusterLeasedBeforeLinksExistedDerivesItsArgvWithoutThem (red without the delete / without WithoutUnleasedLinks)"
      - "guards green first time: TestLinksWithoutAVaultToKeepTheirKeysAreRefused, TestLinksWithNoLeaseAreRefusedAndAStartupScanDerivesTheRestWithoutThem, TestALinkWhoseKeyTheFakeDoesNotKnowNeverPassesItsGate, TestAClusterSwitchedOffKeepsItsClusterLinks, TestTheRecorderWritesIntoTheHomeWorkspaceOfTheClusterKey, TestWriteClusterLinkStatusNeverRecreatesALinkTheTeardownDeleted, procgroup TestPrepareRunsAfterTheStepsBeforeItAreReadyAndItsSpecIsWhatStarts, TestAFailedPrepareStartsNothingAndReapsWhatCameBefore"
      - "live (SOCIALWEB_LOCAL=1, dumbpipe v0.39.0, 2026-10-07): TestARealDumbpipeLinkCarriesTCPThroughItsPublishedTicketWhichARestartLeavesByteIdentical PASS 23.4s -- two real listeners in front of tcp echo servers, connect-tcp with each published (id-only) ticket echoes, group reaped (Pending) and respawned, same endpoint id, did and ticket bytes, and the pre-restart ticket echoes again"
      - "deploy/check.sh --fast green at c90f9f2 (gofmt, build, apigen 9 files no drift, vet, staticcheck, go test)"
      - "stated limits (ARCHITECTURE 'The cluster links'): a link that dies after bring-up is not restarted until the group is next brought up; link ports are probed for TCP only; the IPv6 socket is unpinned"
      - "not yet proven on prod: needs a deploy (install-provider.sh re-applies the permission inventory); then a ClusterLink pair per cluster in each home workspace, phase Listening"
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): 0f24132->2d9f6b1,68abfc8 3656ae9->cc8f58c 41349d6->771f833 596bb86->5d9851b 7c308b4->d61fed1 8f2211a->10f2845 90f4130->c9ecd62 c90f9f2->dfa175a"
  - id: M3-IROH-KEYS
    milestone: M3
    title: Iroh secret keys persisted in the cluster's OpenBao, public key expressed as did:key
    decision: Stable tickets across deploys; ed25519 encodings per artob/edky.
    status: done
    evidence:
      - "prod 2026-10-07T19:14:59Z: did:key:z6MkgqfYYtADsuzQxt3cpxX4ZWFSFkvmqh8LCTrMHZsDFK8C (apiserver), did:key:z6MkqkUHpCn3SmfKqMob14YDDaCWG8BUc9CC3vasnVk4pqST (openbao), keys in the cluster OpenBao socialweb-links"
      - "3c80658 internal/didkey: did:key = did:key:z + base58btc(0xed 0x01 + pub); endpoint id = lowercase hex of the pub, the form iroh prints"
      - "017d8ee internal/clusterlink: EnsureKey reads socialweb-links/<service> or generates once and writes check-and-set 0 (a losing writer takes the winner's key); an unreadable or self-inconsistent record is refused, never regenerated over; Key.Env() = IROH_SECRET=<64 hex seed>; ParseTicket reads the endpoint id out of endpoint<base32 postcard>"
      - "8f2211a VaultLinkKeys: enables the socialweb-links KV v2 mount if absent with the root token from unseal.json, retries while the mount upgrades to versioned data, Create posts {options:{cas:0}}"
      - "measured on dumbpipe v0.39.0: IROH_SECRET is the 32-byte ed25519 seed as 64 hex (dumbpipe prints 'using secret key <hex>' when it makes one); IROH_SECRET=f855..91d8 printed tickets naming endpoint cb6686..693c = ed25519.NewKeyFromSeed(seed).Public(); two runs with one key printed different ticket bytes (relay and addresses) with the same endpoint id; with no network it still prints a ticket after 'Failed to connect to the home relay'"
      - "tests watched to fail then pass: internal/didkey TestTheDidKeySpecVectorEncodesAnEd25519PublicKey (spec vector 3b6a27bc..da29 -> did:key:z6MkiTBz1ymuepAQ4HEHYSF1H8quG5GLVVQR3djdX3mDooWp), TestTheDidKeySpecVectorDecodesBackToItsPublicKey, TestTheEndpointIDDumbpipeDerivesFromIrohSecretIsTheSeedsPublicKeyInHex (red with multicodec 0xed00 and a truncated id); internal/clusterlink TestTheKeyIsGeneratedOnceAndAProviderRestartReadsTheSameOneBack, TestALosingFirstWriteTakesTheKeyThatWon, TestAnUnreadableStoredKeyIsNeverRegeneratedOver (red with the stored key ignored)"
      - "live (SOCIALWEB_LOCAL=1, bao + embedded postgres): TestTheLinkKeyIsKeptInTheClusterVaultAndReadBackAfterTheVaultRestarts PASS 2.5s -- key written, a second check-and-set first write refused, vault stopped and restarted on the same store, same endpoint id and did read back, the openbao link has its own key"
      - "stability decision (decided by Claude, user may veto): status.ticket is clusterlink.StableTicket(key), the id-only ticket that IROH_SECRET=<seed> dumbpipe generate-ticket prints (fixture endpointadfwnbtf..rutyaa for seed f855..91d8), so it is byte-stable for as long as the key is in the vault; the printed ticket (relay + addresses, different bytes per run) is only checked to name the key. Watched to fail with the printed ticket published: TestThePublishedTicketIsWhatDumbpipeGenerateTicketPrintsForTheKeyAndNeverChanges, TestTheTicketDumbpipePrintsIsFoundAndNamesTheEndpointItsSecretDerives, TestEachLinkIsRecordedListeningWithTheStableTicketAndTheDidKeyOfItsKey"
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): 017d8ee->c02b559 3c80658->2c9206d 8f2211a->10f2845"
  - id: M3-SECRETS-TICKET
    milestone: M3
    title: OpenBao ticket delivered through the --secrets capability
    decision: >
      Only the public portion travels through the secrets flow. Door half decided by Claude, user may
      veto (ARCHITECTURE section 9 "A market node joins through its cluster's OpenBao, never through
      ssh"): the bundle is exactly openbao-ticket (ClusterLink <clusterKey>-openbao status.ticket),
      openbao-ca.pem (read from <dirname(spec.tokenFile)>/openbao/tls/ca.crt until the provider
      publishes a path) and claim.json {node, cluster = cluster KEY, jwtMount, role = node, audience,
      kvMount, kvPath = node}; claim.json joins the bundle when the capability is granted, because
      jwtMount needs the winning issuer; status.grant {issuer, subject, audience, role} is awaited
      inside the capability's onContract, before the accept record exists; the door returns at
      onSecretsFetched (claim Joining), disposes capabilities, never returns at onAccepted, and
      releases only on claim deletion.
    status: done
    evidence:
      - "0f3a771 + b489218 (socialweb-computer-k8s only): node-market fulfiller without ssh (guest.ts, node_session_test.ts, NodeJoinDocument helpers, the door's dumbpipe download and node-apiserver-host removed); grantingCapability; ClusterLinkReader in lib/abc + createKcpClusterLinkReader in nodeclaim-kcp; accountHashOf/clusterKeyOf/nodeAudience/jwtMountName/nodeClaimDocument in socialweb-computer-k8s-common"
      - "red first: the new node_market_test.ts + k3s_join_test.ts against the previous door, join module and common with stub exports only (deno test --no-check): 29 failed / 3 passed"
      - "green: node_market_test.ts 'at accept the claim carries the workload grant for its node's audience before the guest can be authorized' (grant on the claim when onAccepted runs, grant write precedes the accepted write, claim.json in the bundle before the inner capability is authorized), 'the secrets bundle carries exactly the OpenBao ticket, the OpenBao CA and claim.json' (link read as root:acct:ci/d90m41be-ci-openbao; jwtMount = jwt-<sha12(issuer)>), 'the flow returns at onSecretsFetched, the claim is Joining, capabilities are disposed and nothing is released', 'the door asks for no ssh key and builds no ssh session' (no sshPublicKey/sshAuthorizedKeys in the request; the fulfiller source names no SshSessionProvider, ssh2, runNodeSession or dumbpipe), 'a grant that cannot be written stops the flow once the receipt is known and fails the claim', 'a cluster whose OpenBao link has no ticket fails the claim without starting a contract', \"the door's cluster key is the provider's\" (d90m41be-ci and 8moqwclj-ci printed by kcpclient.ClusterKey); nodeclaim_kcp_test.ts 'a cluster link is read from the claim's workspace by name, and a missing one reads as null'"
      - "mutations on a scratch copy, each reverted, each caught: grant written after the capability is authorized (2 fail), grant not written (2 fail), onSecretsFetched returning continue (1 fail)"
      - "green: socialweb-computer-k8s deno task check exit 0; deno task test 85 passed 0 failed 1 ignored (live_market_node_test: SOCIALWEB_K8S_LIVE unset); run from a scratch org root (siblings symlinked, worktree rsynced) because the worktree's ../../ imports do not resolve; no Go changed, deploy/check.sh not run"
      - "not proven: a real guest fetching the bundle and logging in (needs the provider's ClusterLink ticket, OpenBao JWT mount/role and KV entry); not deployed"
      - "interface needs from the provider (ARCHITECTURE section 9 list): ClusterLink <clusterKey>-openbao with status.ticket before the claim exists; a published OpenBao CA path (NodeClaim spec field or ClusterLink status) or the guarantee of <dirname(tokenFile)>/openbao/tls/ca.crt; KV fields token, nodePassword, serverCA (exact server-ca.crt bytes), apiserverTicket (bare), advertiseAddress (IPv4), apiserverPort; nodemarket.ConfirmJoined accepting Joining"
      - 'prod 2026-10-07 fresh guest fc-1379be663dd9-9d1763c5 (bob, image sha256-0437ff0d, door+provider at 3919df6/eada921): claim written 23:15:58.76Z, RFP 23:15:59.79Z, winner 23:16:11.30Z, receipt 23:16:12.32Z, microVM booted 23:16:13.67Z, setup-secrets token exchange at bob /v1/oidc/issue 23:17:49Z, door secrets_served + claim Joining 23:17:52.5Z, node-secrets ''fetched the join secrets'' (OpenBao JWT login over the dumbpipe openbao link) 23:17:56Z, apiserver link + k3s agent, Node registered 23:18:03Z, Ready 23:18:04Z; no ssh session to that guest (its debug key was never used)'
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): 0f3a771->6b6a762 117ab30->333108a(atproto-market) 3919df6->4c7e5b3 657a915->41705ed 87562e8->07ac55a a1b467f->86a68c3(atproto-market) b489218->05523fc eada921->d86b93c"

      - "LOCAL gate (hat-8, no prod, rerunnable): TestAMarketNodeJoinsATenantClusterThroughItsOpenBaoWithNoSsh in internal/marketnetwork PASS 753.57s, finished 2026-10-07T23:55:13Z, on the 7th run of the day; the 6 earlier runs are the readings that closed the defects below. It brings the vendored market network up held (accept.sh ACCEPT_HOLD_DIR), runs the deployed control-plane container image socialweb-k8s:m3live built from this tree against it (SOCIALWEB_CLUSTERPROVIDER_SUBSTRATE=market, the same entrypoint prod runs), creates a cluster the way kcpclient does, and applies ONE Deployment to the tenant. Readings (UTC): 23:48:21Z market network held up (bidder did:plc:vvb2czwkqncbgb6fnyledro2); 23:49:18Z cluster 11l3u1ii-m3 Ready with ClusterLinks 11l3u1ii-m3-apiserver and 11l3u1ii-m3-openbao Listening; 23:49:18Z workload applied at zero nodes; 23:49:21Z NodeClaim 11l3u1ii-m3-node1 written (creationTimestamp 23:49:19Z) after both links listened; 23:49:24Z status.grant {issuer https://did-key-zq3shpydjewhxwsdvhzmhk2gra2vduhp3h2y4pv8yecmmrjsm.relay.localhost, audience socialweb.computer/node/11l3u1ii-m3/11l3u1ii-m3-node1, role 11l3u1ii-m3-node1} and the provider created the OpenBao JWT role 11l3u1ii-m3-node1; 23:49:27Z guest fc-7066f9bdb453-d87bccdc booted from the bidder's cloud-init; 23:49:32Z guest fetched the --secrets bundle; 23:49:37Z guest logged in to the cluster OpenBao over its openbao link and read its KV; 23:49:47Z tenant Node 11l3u1ii-m3-node1 Ready with the workload pod Running on it and the door phase Joining, 29s after the Deployment was applied; 23:50:03Z the control plane container replaced (door and provider restarted) with the node up; 23:52:35Z claim adopted (same uid), node Ready, guest running, no vm.delete, observed for 2m30s; 23:52:35Z workload deleted; 23:55:12Z claim deleted and the door released the node (vm_delete_result 200 at 23:55:11.733Z); 23:55:13Z guest gone, zero nodes with the control plane Ready. For this item: the guest's setup-secrets finished 5s after boot (23:49:32Z), writing the three files the capability served (openbao-ticket, openbao-ca.pem, claim.json), and its own node-secrets then read claim.json - neither file is a secret value, and nothing in the door log names an ssh session."
      - "LOCAL gate defects this item's runs found and closed: (1) the door derived the cluster key again from the claim instead of taking spec.cluster, so it read ClusterLink <hash>-<hash>-<cluster>-openbao and every live claim failed at 20:59:06Z with 'carries no ticket yet' (87562e8, main 657a915); (2) the requester waited 30s on its own firehose when a bidder that had already dispatched the accept answered submitAccept with no receipt, and ended receipt_invalid with a VM provisioned and no receipt to release it by - eight guests rented in three minutes for one node at 21:06:39Z (atproto-market a1b467f: the winner's repo is asked for the market.receipt whose accept.uri is this accept); (3) setup-secrets.sh appended '==' to every JWT payload, and GNU base64 -d exits 1 on surplus padding when the payload is 0 or 3 mod 4, so a subject of that length never fetched anything (atproto-market 117ab30, red-first test test/cloud_init_secrets_subject_test.ts, 3 lengths, 2 of 3 fail before the fix)."
  - id: M3-JWT
    milestone: M3
    title: Guest authenticates to the cluster OpenBao by JWT and fetches k3s token, node password, server CA and apiserver ticket
    decision: >
      One JWT auth mount per bidder OIDC issuer (oidc_discovery_url is the bidder's public issuer),
      one role per NodeClaim bound to that guest's workload subject and audience, short token TTL,
      read-only on the claim's path, role deleted on release. Mount and discovery details decided
      by Claude, user may veto.
    status: done
    evidence:
      - "provider half built 2026-10-07: ff3ca9c (vaultlayout names, openbao client auth/list/delete/LoginJWT, nodemarket secrets + SyncSecrets + revoke, clusterprovider.NodeVault), 9dff4fa (kcpclient.ReadClusterLink, nodes.go wiring, main.go Links + NodeSecrets); ARCHITECTURE section 7 'A market node's secrets live in its cluster's OpenBao, behind a JWT role'"
      - "names: KV v2 mount socialweb-nodes, secret socialweb-nodes/data/<node> {token (pinned K10<sha256 serverCA>::server:<token>), nodePassword, serverCA, apiserverTicket, advertiseAddress, apiserverPort (int)}; policy node-<node> read socialweb-nodes/data/<node>; auth mount jwt-<sha256(issuer)[:12]> type jwt, oidc_discovery_url = bound_issuer = issuer; role <node>: role_type jwt, bound_subject, bound_audiences [audience], user_claim sub, token_ttl 5m, token_policies [node-<node>]"
      - "decided by Claude, user may veto: secret held back (claim not) until ClusterLink <clusterKey>-apiserver is Listening with a ticket, filled by a later pass; policy failure fails Provision before the claim; status.grant.role never written by the provider (deterministic, logged); jwt mount kept when its last role goes (delete races a concurrent grant); revoke best-effort, 10 s bound, logged only, after the claim delete is accepted"
      - "instrument (real OpenBao v2.6.2, bao server -dev, in-test ES256 OIDC issuer over http): internal/clusterprovider TestANodeLogsInWithItsWorkloadTokenAndReadsOnlyItsOwnSecret -- own token logs in and reads its path; other subject and other audience refused; 403 on another node's path; after Revoke role/policy/secret gone and login refused, other node's policy and the mount kept; second Revoke ok"
      - "fakes: internal/nodemarket TestProvisionWritesTheNodePolicyAndSecretBeforeTheClaim (order policy, secret, claim; exact fields; link read from <workspace>/liveweb-apiserver), TestASecretWaitsForTheApiserverLinkWithoutHoldingTheClaimBack, TestAGrantOnTheClaimBecomesOneJWTRole, TestAReleaseRemovesTheNodesVaultState, TestAPolicyThatCannotBeWrittenRentsNothing, TestSecretsWithoutALinkReaderAreRefused; cmd TestTheConfirmPassWritesTheNodesSecretsAndTurnsAGrantIntoALogin; vaultlayout TestTheNodeJWTNamesAreTheOnesTheContractNames (mount hash checked against sha256sum); kcpclient TestReadClusterLinkCarriesTheTicketAndReportsAbsence"
      - "watched to fail (scratch mutations, restored): no secrets at Provision -> 3 tests FAIL; no revoke at release -> nodemarket + cmd FAIL; no grant cache -> 2 FAIL; role without bound_subject -> real-bao test FAIL; policy wildcard socialweb-nodes/data/* -> vaultlayout + real-bao FAIL; confirm pass skips SyncSecrets -> cmd FAIL; no KV mount ensure -> real-bao FAIL; ticket dropped -> 2 FAIL"
      - "green: deploy/check.sh --fast exit 0 (gofmt, build, apigen, vet, staticcheck, go test)"
      - "not done: the door (claim.json, --secrets delivery, exchange for the audience), the guest units (M3-UNITS), a live login from a market guest; the provider has not been deployed"
      - "interface for the door/guest: login POST /v1/auth/jwt-<sha256(issuer)[:12]>/login {role: <node>, jwt: <token with sub = grant.subject, aud = grant.audience>}; read GET /v1/socialweb-nodes/data/<node> with X-Vault-Token; the bidder issuer must serve <issuer>/.well-known/openid-configuration and its JWKS to the provider host"
      - 'prod 2026-10-07 fresh guest fc-1379be663dd9-9d1763c5 (bob, image sha256-0437ff0d, door+provider at 3919df6/eada921): claim written 23:15:58.76Z, RFP 23:15:59.79Z, winner 23:16:11.30Z, receipt 23:16:12.32Z, microVM booted 23:16:13.67Z, setup-secrets token exchange at bob /v1/oidc/issue 23:17:49Z, door secrets_served + claim Joining 23:17:52.5Z, node-secrets ''fetched the join secrets'' (OpenBao JWT login over the dumbpipe openbao link) 23:17:56Z, apiserver link + k3s agent, Node registered 23:18:03Z, Ready 23:18:04Z; no ssh session to that guest (its debug key was never used)'
      - 'live read 2026-10-07T22:53Z of the tenant OpenBao: mount jwt-df356681a07e bound_issuer = bob''s issuer, role <node> bound_subject actx:<bidder>:plc:<owner>:role:<node>, bound_audiences socialweb.computer/node/<clusterKey>/<node>, token_ttl 300, policy node-<node>; stale mounts from earlier issuers tracked in M3-BAO-SYNC'
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): 3919df6->4c7e5b3 9dff4fa->afc0c51 eada921->d86b93c ff3ca9c->4700d8c"

      - "LOCAL gate (hat-8, no prod, rerunnable): TestAMarketNodeJoinsATenantClusterThroughItsOpenBaoWithNoSsh in internal/marketnetwork PASS 753.57s, finished 2026-10-07T23:55:13Z, on the 7th run of the day; the 6 earlier runs are the readings that closed the defects below. It brings the vendored market network up held (accept.sh ACCEPT_HOLD_DIR), runs the deployed control-plane container image socialweb-k8s:m3live built from this tree against it (SOCIALWEB_CLUSTERPROVIDER_SUBSTRATE=market, the same entrypoint prod runs), creates a cluster the way kcpclient does, and applies ONE Deployment to the tenant. Readings (UTC): 23:48:21Z market network held up (bidder did:plc:vvb2czwkqncbgb6fnyledro2); 23:49:18Z cluster 11l3u1ii-m3 Ready with ClusterLinks 11l3u1ii-m3-apiserver and 11l3u1ii-m3-openbao Listening; 23:49:18Z workload applied at zero nodes; 23:49:21Z NodeClaim 11l3u1ii-m3-node1 written (creationTimestamp 23:49:19Z) after both links listened; 23:49:24Z status.grant {issuer https://did-key-zq3shpydjewhxwsdvhzmhk2gra2vduhp3h2y4pv8yecmmrjsm.relay.localhost, audience socialweb.computer/node/11l3u1ii-m3/11l3u1ii-m3-node1, role 11l3u1ii-m3-node1} and the provider created the OpenBao JWT role 11l3u1ii-m3-node1; 23:49:27Z guest fc-7066f9bdb453-d87bccdc booted from the bidder's cloud-init; 23:49:32Z guest fetched the --secrets bundle; 23:49:37Z guest logged in to the cluster OpenBao over its openbao link and read its KV; 23:49:47Z tenant Node 11l3u1ii-m3-node1 Ready with the workload pod Running on it and the door phase Joining, 29s after the Deployment was applied; 23:50:03Z the control plane container replaced (door and provider restarted) with the node up; 23:52:35Z claim adopted (same uid), node Ready, guest running, no vm.delete, observed for 2m30s; 23:52:35Z workload deleted; 23:55:12Z claim deleted and the door released the node (vm_delete_result 200 at 23:55:11.733Z); 23:55:13Z guest gone, zero nodes with the control plane Ready. For this item: node-secrets exchanged its workload token for the claim's audience, logged in to the cluster OpenBao at the jwt mount the provider created (logged 'the node can log in to the cluster OpenBao with its workload token', role and policy named for the node) and read its own KV entry at 23:49:37Z; a first attempt failed and the unit's Restart=on-failure retried it five seconds later, which is the retry design."
  - id: M3-UNITS
    milestone: M3
    title: user_data installs systemd units for dumbpipe connect-tcp and the k3s agent; no ssh to guests
    decision: >
      Binaries come from our nodeimage when present, otherwise from the door hash-pinned (static
      musl amd64 and arm64, sha256 written into user_data). Invariant 13 changes from ssh stdin to
      OpenBao JWT over the cluster link. Measured for the market path: the iroh transport module
      (setup-iroh.sh, /usr/local/bin/dumbpipe) and the k3s module (setup-k3s.sh, /usr/local/bin/k3s)
      the door already composes install both binaries in the same runcmd before the units start, so
      door-served binaries are not needed there. Decided by Claude, user may veto: the agent unit is
      socialweb-k3s-agent (the k3s installer owns k3s/k3s-agent); socialweb-node-secrets is a oneshot
      with Restart=on-failure (systemd refuses Restart=always on oneshot) and every unit has
      StartLimitIntervalSec=0; node-secrets refuses a non-IPv4 advertiseAddress, a non-digit port and a
      ticket outside [a-z0-9]+ because they reach a unit command line; the token is pinned
      K10<sha256(serverCA)>::server:<token> as the ssh join pinned it.
    status: done
    evidence:
      - "prod probe 2026-10-07T20:54Z (first bidder online, bob on hat-8): the door asked for ClusterLink 2wjeqtgx-2wjeqtgx-...-openbao -- it re-derived the cluster key from spec.cluster, which the provider already writes as the key; every claim failed before its RFP. Fixed in 657a915 (door test now pins spec.cluster as the key; 4 tests failed first), deployed ~20:57Z"
      - "prod probe 2026-10-07T20:57:40Z: RFP -> bob won -> status.grant written -> accept -> receipt verified -> guest booted on hat-8 (fc-1379be663dd9-a58a9c15), cloud-init wrote all four socialweb units; then provisioning-token.service FAILED: bob's provider answered /v1/oidc/prove with OIDC token failed validation: signature verification failed (token kid uCOvMc.., the issuer now serves kid EQCTHB..); setup-secrets, guest-onnetwork, node-secrets, apiserver-link and k3s-agent failed after it; door ended network_timeout at 21:17:57Z and released the VM. BLOCKED on a bidder-side OIDC signing-key defect in hono-compute-provider (not ours); root cause being investigated"
      - "0f3a771: cloud-init-plugin-k3s-join writes socialweb-openbao-link, socialweb-node-secrets, socialweb-apiserver-link, socialweb-k3s-agent (+ three 0700 scripts); the socialweb-k3s-join stdin script, the sshd GatewayPorts drop-in and its sshd reload are gone; the iroh transport module is untouched"
      - "ARCHITECTURE invariant 13 rewritten (secrets cross only OpenBao JWT over the cluster link; no ssh to guests), invariant 29 rewritten, invariant 32 and the market row of section 9 corrected"
      - "green: k3s_join_test.ts 'the node's user_data installs four units that join it through OpenBao and the apiserver link, in order' (composed with iroh, k3s, the join module and secrets: After chains, Restart, StartLimitIntervalSec=0, agent argv, setup-iroh.sh and setup-k3s.sh before the start), 'the node's user_data carries no secret value and no ssh join path', 'node-secrets exchanges the workload token for the claim's audience, logs in to OpenBao and writes the join files 0600' (real bash + jq, fake curl asserting aud, sub, --cacert, role, jwt and X-Vault-Token), 'node-secrets refuses a KV entry that would put an unsafe value on a unit's command line', 'node-secrets fails, and is restarted by systemd, when the KV entry lacks a field', 'the OpenBao link dials the delivered ticket and listens where node-secrets looks', 'the apiserver link puts the advertise address on lo and dials the apiserver ticket there'; node_market_test.ts 'the user_data the door asks for renders without any secret value'"
      - "mutations caught: agent unit not ordered after the apiserver link (1 fail); token not pinned to the CA (1 fail)"
      - "not proven: the units on a real guest (systemd ordering, dumbpipe connect-tcp against a provider listen-tcp, k3s agent Ready); needs M3-CLUSTERLINK and M3-JWT; not deployed"
      - 'prod 2026-10-07 fresh guest fc-1379be663dd9-9d1763c5 (bob, image sha256-0437ff0d, door+provider at 3919df6/eada921): claim written 23:15:58.76Z, RFP 23:15:59.79Z, winner 23:16:11.30Z, receipt 23:16:12.32Z, microVM booted 23:16:13.67Z, setup-secrets token exchange at bob /v1/oidc/issue 23:17:49Z, door secrets_served + claim Joining 23:17:52.5Z, node-secrets ''fetched the join secrets'' (OpenBao JWT login over the dumbpipe openbao link) 23:17:56Z, apiserver link + k3s agent, Node registered 23:18:03Z, Ready 23:18:04Z; no ssh session to that guest (its debug key was never used)'
      - commits 3919df6 (CNI under --flannel-backend=none), bde602f (units log to the console), 2501ea9 (node-diagnose)
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): 0f3a771->6b6a762 2501ea9->15c8d9f 3919df6->4c7e5b3 657a915->41705ed bde602f->8636fa2 eada921->d86b93c"

      - "LOCAL gate (hat-8, no prod, rerunnable): TestAMarketNodeJoinsATenantClusterThroughItsOpenBaoWithNoSsh in internal/marketnetwork PASS 753.57s, finished 2026-10-07T23:55:13Z, on the 7th run of the day; the 6 earlier runs are the readings that closed the defects below. It brings the vendored market network up held (accept.sh ACCEPT_HOLD_DIR), runs the deployed control-plane container image socialweb-k8s:m3live built from this tree against it (SOCIALWEB_CLUSTERPROVIDER_SUBSTRATE=market, the same entrypoint prod runs), creates a cluster the way kcpclient does, and applies ONE Deployment to the tenant. Readings (UTC): 23:48:21Z market network held up (bidder did:plc:vvb2czwkqncbgb6fnyledro2); 23:49:18Z cluster 11l3u1ii-m3 Ready with ClusterLinks 11l3u1ii-m3-apiserver and 11l3u1ii-m3-openbao Listening; 23:49:18Z workload applied at zero nodes; 23:49:21Z NodeClaim 11l3u1ii-m3-node1 written (creationTimestamp 23:49:19Z) after both links listened; 23:49:24Z status.grant {issuer https://did-key-zq3shpydjewhxwsdvhzmhk2gra2vduhp3h2y4pv8yecmmrjsm.relay.localhost, audience socialweb.computer/node/11l3u1ii-m3/11l3u1ii-m3-node1, role 11l3u1ii-m3-node1} and the provider created the OpenBao JWT role 11l3u1ii-m3-node1; 23:49:27Z guest fc-7066f9bdb453-d87bccdc booted from the bidder's cloud-init; 23:49:32Z guest fetched the --secrets bundle; 23:49:37Z guest logged in to the cluster OpenBao over its openbao link and read its KV; 23:49:47Z tenant Node 11l3u1ii-m3-node1 Ready with the workload pod Running on it and the door phase Joining, 29s after the Deployment was applied; 23:50:03Z the control plane container replaced (door and provider restarted) with the node up; 23:52:35Z claim adopted (same uid), node Ready, guest running, no vm.delete, observed for 2m30s; 23:52:35Z workload deleted; 23:55:12Z claim deleted and the door released the node (vm_delete_result 200 at 23:55:11.733Z); 23:55:13Z guest gone, zero nodes with the control plane Ready. For this item: the guest is born from the RFP's cloud-init user_data - the four join units plus main's pod network units are enabled and started after setup-secrets, node-secrets is the oneshot that fetches, and the apiserver link then the k3s agent follow, which is what registers the Node 8s before it reports Ready."
  - id: M3-NODE-LOSS
    milestone: M3
    title: A Node NotReady for 1 minute is released and replaced; door or provider restarts never reap
    decision: >
      User chose 90s; 76c4c864 (2026-10-13) shortened NodeLossGrace to 1 minute (internal/providermain/nodes.go). Door half decided by Claude, user may veto: a restarted door adopts Joining and
      Running claims with no write and no release; it marks Bidding and Provisioning claims Failed and
      does not release them, because their guest's user_data names the old door's per-process secrets
      server, so the guest can never fetch, and awaitComputeNetwork would only find an address the door
      no longer needs; the release comes with the claim's deletion from the receipt it carries. stop()
      releases nothing and writes nothing.
    status: done
    evidence:
      - "deployed 2026-10-07 ~19:44Z with main 6af8994 (adoption, shutdown no longer reaps, per-node scale-in 932b139): 19:45:36Z passes None nothing running and no node, ClusterLinks re-recorded after restart; a restart with live market nodes awaits a bidder"
      - "provider half built 2026-10-07 in 9dff4fa: tenantNodes.confirmHeld reads each held market node's tenant Node; Ready False/Unknown with now - lastTransitionTime >= NodeLossGrace (90s) -> teardownHeld (claim deleted, vault state revoked, door releases with vm.delete, tenant Node deleted), reported in confirmOutcome.lost; the scale pass in the same tick claims a replacement if demand remains. ARCHITECTURE section 10 'A market node NotReady for 90 s is released'"
      - "decided by Claude, user may veto: clock is the Node's lastTransitionTime (survives provider restarts), zero transition time falls back to first-seen in memory; no Node object or no Ready condition is not covered; held nodes only"
      - "instrument: cmd/socialweb-clusterprovider/nodeloss_test.go TestAMarketNodeNotReadyFor89sIsKeptAndAt90sIsReleased (False and Unknown: 89s keeps claim + Node, 90s deletes both, revokes, reports lost), TestANodeTheClusterNeverReportedIsNotTouchedByTheLossRule (1h on, no Node, claim kept), TestAReadyNodeIsNotLostHoweverOldItsTransition"
      - "watched to fail (scratch mutations, restored): grace '>' for '>=' -> 90s case FAIL (both subtests); ignore lastTransitionTime -> FAIL; unreported node judged -> never-reported test FAIL"
      - "superseded by the provider restart half below: a PROVIDER restart re-holding market nodes it did not provision in this process (was: neither confirmed nor lost)"
      - "door half, 0f3a771 + b489218: node_market_test.ts 'a door restarted with live claims adopts them and submits no vm.delete' (Joining, Running and a legacy Running claim: no records, no vm.delete, no status write), 'a door restarted mid-flight marks the orphaned claim Failed, releases nothing, and releases it once deleted' (exactly one vm.delete naming the claim's receipt, to its submitEventRef, then Released, finalizer gone, not released twice), 'a deleted Joining claim is released from the receipt on the claim, not the door's memory', 'a claim deleted while the door waits for the guest's fetch is released from its claim at once', 'stopping the door leaves in-flight claims for the next door and releases nothing', 'an unreadable claim at deletion defers the release to the next pass, which reads it again'"
      - "mutations caught: a restart reaping live claims (2 fail); a release at the end of a successful flow (3 fail)"
      - "provider restart half, 9ca4c04: decided by Claude, user may veto. Shutdown no longer releases market nodes (nodeHosts.reap skips the market substrate; it deleted every claim, so every deploy made the door destroy every machine). A restarted provider ADOPTS every live NodeClaim of a cluster (spec.cluster = key, <cluster>-node<N> name, phase not Failed/Released, no deletionTimestamp): nodemarket.Host.Adopt marks it provisioned with no claim, policy or secret write (secret marked written only for Joining/Running; earlier phases rewrite it on SyncSecrets; JWT role ensured once from status.grant), retained at its ordinal, live, counted in started, snapshot rewritten. Runs at startup right after the empty snapshot (before the cluster's CCM starts), on every pass from the claim list reconcile already reads (no extra kcp read), and once before an OFF cluster's reapCluster. ARCHITECTURE section 9 'A restarted provider never reaps; it adopts', sections 10 and 11 updated"
      - "tests watched to fail against the stub (adopt a no-op) then pass: cmd/socialweb-clusterprovider TestARestartedProviderAdoptsItsJoiningAndRunningNodesOnceEach (held [] -> both; counted 2; no claim/policy/secret write; snapshot names both), TestAnAdoptedNodeNotReadyFor90sIsReleased, TestAnAdoptedIdleNodeIsScaledIn (decided None -> Release node3), TestTeardownToZeroReleasesAdoptedNodes (deletes 1 -> 2), TestAProvisionAfterARestartSkipsAdoptedNames (node3 never claimed -> claimed, adopted claims unchanged), TestAProviderShutdownLeavesItsMarketNodesForTheNextOne (2 claims deleted + 2 revokes -> none); TestAnEndedClaimIsNotAdopted passed first (guard)"
      - "mutations (in place, restored with git checkout): reconcile not adopting -> TestTheFirstScalePassAfterARestartAdoptsBeforeItCounts FAIL; reapOne without adoptOnce -> TestAClusterSwitchedOffAfterARestartReleasesItsAdoptedNodes FAIL; nodemarket.Adoptable always true -> TestAnEndedClaimIsNotAdopted + TestAnEndedOrDeletingClaimIsNotAdopted FAIL; secret never marked written -> TestARestartedProvider... + TestAnAdoptedJoinedHostRewritesNoSecretAndLogsInFromTheGrant FAIL; secret always marked written -> TestAnAdoptedClaimTheGuestHasNotFetchedYetGetsItsSecretWritten FAIL; adopted not counted in started -> TestAProvisionAfterARestartSkipsAdoptedNames FAIL"
      - "green: deploy/check.sh --fast exit 0 (gofmt, build, apigen, vet, staticcheck, go test)"
      - "with the door half (0f3a771 + b489218, merged), prod's door and provider can now both restart without reaping a live market node -- in code and tests; not deployed, not yet read on prod"
      - "not done: deploy; a prod provider restart with market nodes up re-registering within the 1 minute loss grace after the group comes back; a live NotReady release against a real bidder"
      - 'live 2026-10-07T22:52:55Z: a node NotReady since 22:51:24Z was released by the 90s rule and a replacement rented; 2f0ad94+eada921: a never-Ready node gets a join grace (2m default, SOCIALWEB_CLUSTERPROVIDER_NODE_JOIN_GRACE) instead; restarts adopt: 23:01:4xZ deploy -> ''adopted a live market node a previous provider process rented''; 3539ad8: a held node whose claim was deleted is released and replaced (live 22:22:53Z)'
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): 0f3a771->6b6a762 2f0ad94->d872ca7 3539ad8->90740fd 6af8994->0b02637 932b139->a6072ee 9ca4c04->acd218f 9dff4fa->afc0c51 b2624c6->3f0d73b b489218->05523fc eada921->d86b93c"

      - "LOCAL gate (hat-8, no prod, rerunnable): TestAMarketNodeJoinsATenantClusterThroughItsOpenBaoWithNoSsh in internal/marketnetwork PASS 753.57s, finished 2026-10-07T23:55:13Z, on the 7th run of the day; the 6 earlier runs are the readings that closed the defects below. It brings the vendored market network up held (accept.sh ACCEPT_HOLD_DIR), runs the deployed control-plane container image socialweb-k8s:m3live built from this tree against it (SOCIALWEB_CLUSTERPROVIDER_SUBSTRATE=market, the same entrypoint prod runs), creates a cluster the way kcpclient does, and applies ONE Deployment to the tenant. Readings (UTC): 23:48:21Z market network held up (bidder did:plc:vvb2czwkqncbgb6fnyledro2); 23:49:18Z cluster 11l3u1ii-m3 Ready with ClusterLinks 11l3u1ii-m3-apiserver and 11l3u1ii-m3-openbao Listening; 23:49:18Z workload applied at zero nodes; 23:49:21Z NodeClaim 11l3u1ii-m3-node1 written (creationTimestamp 23:49:19Z) after both links listened; 23:49:24Z status.grant {issuer https://did-key-zq3shpydjewhxwsdvhzmhk2gra2vduhp3h2y4pv8yecmmrjsm.relay.localhost, audience socialweb.computer/node/11l3u1ii-m3/11l3u1ii-m3-node1, role 11l3u1ii-m3-node1} and the provider created the OpenBao JWT role 11l3u1ii-m3-node1; 23:49:27Z guest fc-7066f9bdb453-d87bccdc booted from the bidder's cloud-init; 23:49:32Z guest fetched the --secrets bundle; 23:49:37Z guest logged in to the cluster OpenBao over its openbao link and read its KV; 23:49:47Z tenant Node 11l3u1ii-m3-node1 Ready with the workload pod Running on it and the door phase Joining, 29s after the Deployment was applied; 23:50:03Z the control plane container replaced (door and provider restarted) with the node up; 23:52:35Z claim adopted (same uid), node Ready, guest running, no vm.delete, observed for 2m30s; 23:52:35Z workload deleted; 23:55:12Z claim deleted and the door released the node (vm_delete_result 200 at 23:55:11.733Z); 23:55:13Z guest gone, zero nodes with the control plane Ready. For this item, the restart half: the container was replaced with a Joined node up at 23:50:03Z and for 2m30s the claim kept its uid, the node stayed Ready, the guest stayed up and no vm.delete was submitted (the door adopted it and the provider re-held it); the release came only from the claim's deletion at 23:55:12Z, with vm.delete accepted and the guest gone 1s later."
      - "LOCAL gate defect this item's runs found and closed: a provider restarted while a claim was Bidding/Provisioning read it as already Failed, never adopted it, and nothing ever deleted it, so its machine ran on (5fzvzwkw-m3-node1 left Failed after a container restart at 21:36:41Z). b2624c6 makes reconcile delete a Failed, undeleted claim no host of this process holds, and TestAnEndedClaimIsNotAdopted - which pinned the leak - now wants it deleted; both it and the new TestAFailedClaimNoProcessHoldsIsDeletedSoTheDoorReleasesItsMachine FAIL with the call removed."
  - id: M3-BAO-SYNC
    milestone: M3
    title: The cluster OpenBao's JWT mounts, roles, policies and KV entries match the live claims, always
    decision: >
      User asked 2026-10-07 that the plans keep OpenBao backends and roles in sync with what is active.
      Read live 22:53Z: the tenant OpenBao holds 5 jwt-* mounts, one per issuer bob has had (its issuer
      is a fresh did:key relay keypair every restart), and only jwt-df356681a07e serves a live node. Rule:
      per cluster, desired = for each live (Adoptable) claim with a grant: mount jwt-<sha12(issuer)>,
      role <node>, policy node-<node>, KV socialweb-nodes/<node>. The provider reconciles actual to desired
      every confirm pass and at adoption: a role, policy or KV entry with no live claim is deleted; a
      jwt-* mount with no role left is disabled. Release keeps revoking eagerly; the reconcile is the
      backstop for restarts and crashes. Unit test against a fake OpenBao seeded with stale mounts/roles;
      live reading on prod: after a release and after a provider restart, `bao auth list` and
      `bao list auth/<m>/role` name exactly the live nodes. Plan 0007's stable bidder keys shrink the
      churn; this keeps the store honest regardless. decided by Claude, user may veto.
    status: done
    evidence:
      - 'commit 0ff704b: NodeVault.Reconcile (internal/clusterprovider/nodesecrets.go) declared as nodemarket.NodeSecretsReconciler, a sibling of NodeSecrets rather than a fourth method on it; deletes every role under a jwt-<12 hex> mount whose name is not a live node or whose mount is not that node''s grant mount, every policy named node-<node> that names no live node, every socialweb-nodes/metadata/<node> entry naming no live node, and disables a jwt-<12 hex> mount left with no role that no live grant names; creates nothing, writes nothing else, never touches a mount, policy or path outside those naming patterns, and deletes nothing when any of its three listings fails (each reports its own error out of Reconcile)'
      - 'commit 0ff704b: the provider calls it (cmd/socialweb-clusterprovider/vaultsync.go) with live = the claims it holds plus the Adoptable claims in the home workspace (liveNodeGrants), after adoptClaims at adoption (adopt.go) and on every pass (nodes.go reconcile), rate-limited to once per cluster per VaultSyncInterval (30 s); every deletion is logged at INFO with its reason (NodeVault.WithLogger, wired in main.go)'
      - 'green: internal/clusterprovider/nodesecrets_sync_test.go against a real bao server -dev (v2.6.2) and an in-test ES256 issuer, seeded with 4 stale jwt mounts (one with a stale role, one holding a live node''s role under a stale issuer, two empty), a stale policy, a stale KV entry, and one live node''s full set: TestReconcileLeavesOnlyTheLiveNodesMaterialInTheClusterVault (only the live mount/role/policy/KV remain, an approle mount, a jwt-<non-hex> mount and postgres/default/root survive, a second pass changes nothing), TestReconcileWithNoLiveNodeRemovesEveryNode, TestReconcileKeepsTheMountALiveNodesGrantNamesBeforeItsRoleIsWritten, TestReconcileDeletesNothingWhenAListFails (an httputil proxy in front of the dev vault answers every LIST with 500: Reconcile errors, names each of the three failed reads, and every role, policy and KV entry survives)'
      - 'green: cmd/socialweb-clusterprovider/vaultsync_test.go: TestAdoptionReconcilesTheClusterOpenBaoWithTheLiveClaims (adoption reconciles once with the live node and its grant; a second pass inside 30 s does not read the vault again; 30 s later it does), TestReconcileNamesEveryLiveNodeAndOnlyThose (a Released claim is not live, a claim with no grant is live with none), TestReconcileKeepsANodeTheProviderStillHoldsWithNoClaim'
      - 'mutations caught (9, each reverted): a LIST failure read as an empty list for roles, for policies and for KV; a role kept under a mount that is not its node''s grant mount; the policy naming filter dropped; the jwt-<12 hex> mount filter dropped; a mount named by a live grant disabled anyway; the rate limit removed; a Released claim counted live; the held nodes dropped from live'
      - 'docs/ARCHITECTURE.md: new section "The cluster OpenBao is reconciled against the live claims (plan 0005 M3-BAO-SYNC)"; the M3-JWT mount row now reads "disabled by the reconcile" instead of "never", and the "the mount outlives its last role" decision is superseded there'
      - 'not done: the live reading on prod (after a release and after a provider restart, bao auth list and bao list auth/<m>/role name exactly the live nodes) -- the lead deploys and reads; nothing here was deployed'
      - "live prod 2026-10-07T23:51:42Z after the merged reconcile was deployed (23:41:44Z), a provider restart and two node releases: the tenant OpenBao lists 0 jwt-* auth mounts (5 at 22:53Z, one per bob issuer ever seen), 0 node-* policies and no socialweb-nodes KV entries, with 0 live nodes"
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): 0ff704b->034afff"
  - id: M3-DEBUG-SSH
    milestone: M3
    title: DEBUG_SOCIALWEB_NODE_SSH_DIR gives the operator an ssh key into market nodes, off by default
    decision: >
      User asked 2026-10-07 for a debug ssh path into live nodes, gated by a DEBUG_ env var. The door
      generates an ed25519 key once in that directory, the requester adds the public half to the guest's
      user_data authorized_keys (RFP/cloud-init path), and the door logs debug_node_ssh with the private
      key path and an ssh command over the guest's iroh ticket (the iroh module's dumbpipe listener in
      front of sshd). Unset = no key, so ACC-NOSSH runs with it unset and the door's no-ssh test still
      holds for the default. decided by Claude, user may veto.
    status: done
    evidence:
      - "commit a6a67e2: node-market debugSsh option, CLI option debug-node-ssh-dir (env DEBUG_SOCIALWEB_NODE_SSH_DIR), scripts/deploy.sh passes it into the container env file"
      - "test watched to fail then pass: socialweb-computer-k8s test/node_market_test.ts 'only a door given a debug key puts it in the guest's authorized_keys, and it is told the guest's address' (FAILED: sshAuthorizedKeys undefined); suite 106 passed"
      - "used 2026-10-07T23:11Z to find the CNI root cause (M5-CNI-READY); acceptance run 78 with it unset showed empty authorized_keys"
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): a6a67e2->4a47aa5"
  - id: M5-CNI-READY
    milestone: M5
    title: A market node's kubelet reports NetworkReady with the bridge CNI
    decision: >
      Prod 2026-10-07 22:51-23:00Z: three market nodes joined over the no-ssh path (secrets bundle,
      OpenBao JWT login, dumbpipe apiserver link, Node registered ~75-80 s after boot) and every one stayed
      NotReady: "container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady
      message:Network plugin returns error: cni plugin not initialized", although socialweb-pod-cni wrote
      <data-dir>/agent/etc/cni/net.d/10-socialweb-bridge.conflist ~76 s after boot. The bridge CNI had never
      run under a real kubelet (M5 evidence is zero-node and unit level). Hypotheses: H4 containerd's conf
      watch was never armed because net.d did not exist when it started (with flannel off nothing creates
      it before pod-cni); H1 a plugin named by the conflist (bridge/host-local/portmap, loopback) is not in
      containerd's bin_dirs; H2 containerd's conf_dir is not the dir we write; H3 the conflist is rejected.
      Isolation: socialweb-node-diagnose (2501ea9) prints crictl info's lastCNILoadStatus, confDir, binDirs
      and their contents to the console; DEBUG ssh (M3-DEBUG-SSH) for containerd's own log; a host-side
      netns test of the rendered conflist against k3s's bundled plugins (cnitool, no guest) for H1/H3; a
      unit that creates net.d before the agent for H4. The join grace (2f0ad94, 10m) is a placeholder to be
      replaced by 2x the measured register->Ready time once a node reaches Ready. decided by Claude, user
      may veto.
    status: done
    evidence:
      - "2026-10-07T22:52:55Z provider: node96 NotReady since 22:51:24Z released by the 90s loss rule; tenant Node condition read 22:58Z: KubeletNotReady cni plugin not initialized"
      - 'root cause read over the debug ssh on guest fc-1379be663dd9-48148a7e 23:11Z: k3s with --flannel-backend=none writes no cni section into containerd''s config, so crictl info shows confDir=/etc/cni/net.d binDirs=[/opt/cni/bin], lastCNILoadStatus ''no network config found in /etc/cni/net.d''; linking k3s''s bundled cni into /opt/cni/bin and the conflist into /etc/cni/net.d -> NetworkReady true and Node Ready at 23:12:04Z within seconds'
      - 'fix 3919df6 (tests watched to fail: pod-cni symlink test, path test, diagnose false test; suite 108 passed); deployed 23:15:28Z; fresh node registered 23:18:03Z Ready 23:18:04Z with no hand change'
      - 'claim->Ready 125 s on prod: apt-get update 55 s, apt install 15 s, dumbpipe 7 s, k3s install 9.5 s, bid window 6.6 s; targets in M3-BOOT-TIME'
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): 3919df6->4c7e5b3"

  - id: M3-BOOT-TIME
    milestone: M3
    title: Claim to Ready in about 30 s by baking the guest's software into the node image
    decision: >
      Measured 125 s claim->Ready on prod 2026-10-07 (M5-CNI-READY evidence); 85 s of it is the guest
      installing software at boot. (1) preinstall-market.json carries openssh-server, jq, curl, iptables and
      everything else the user_data packages lists, and the user_data skips package_update when nothing is
      missing (~70 s); (2) k3s and dumbpipe come from the node image (k3s is already a nodeimage input) or
      door-served sha256-pinned, never get.k3s.io/GitHub at boot (~16 s); (3) setup-secrets starts once the
      network is up instead of at the end of cloud-init's runcmd (~10 s). Bid window (6.6 s, policy
      bidWindowSec 8) is market policy: ask the user before changing it. decided by Claude, user may veto.
    status: done
    evidence:
      - "code: socialweb-computer-kcp bcccf22 (deploy/bidder/preinstall-market.json in the repo; scripts/deploy-firecracker-bidder.sh LOCAL_PREINSTALL defaults to it; socialweb-computer-k8s/test/preinstall_manifest_test.ts; node_network_test.ts reads the guard), 8cd8e12 (the dumbpipe run unpacks into a directory of its own, so tar never has to set the mode of /tmp); atproto-market d20612d (lib/common/cloud-init-common/mod.ts: packagesGuardCommand + guardPackages in buildUserData, package_update in UserDataPatch, the k3s and dumbpipe skip paths, the secrets module runcmd -> runcmdPrepend; test/cloud_init_packages_test.ts, test/cloud_init_baked_binaries_test.ts; cloud-init fixtures regenerated)"
      - "the manifest carries cloud-init, jq, unzip, ca-certificates (what the file already held) plus curl, iptables, openssh-server and tar, and a run that installs sha256-pinned dumbpipe (x86_64 9ac5e71983eba4cf4f47e92e4694dad0f0133abba4f14dd2a2a8428cece88fef, aarch64 8cd099afe80c69ac58bfbc975b28e29d41caea141f3dd396457345a739836a0e) at /usr/local/bin/dumbpipe with its release written to /usr/local/bin/dumbpipe.version. Every name in the door's user_data (iroh, k3s, k3s-join, secrets) is covered: openssh-server, jq, curl, iptables"
      - "the composed document no longer has a packages list at all: cloud-init's own module is what runs apt-get update on every boot whatever the image holds. The packages become one runcmd step at the head of the document -- dpkg -s per package, apt-get update and install only for the missing ones -- so a node image that carries them does no apt at all and a container or droplet guest nothing was baked for still installs what it needs (test/cloud_init_packages_test.ts: 6 tests, including the ordering against the steps that need jq and curl and a name that is not a package name refused rather than put in a shell command)"
      - "k3s: setup-k3s.sh skips get.k3s.io when /usr/local/bin/k3s --version reports v1.36.4+k3s1. The node image stages the pinned release at exactly that path (k3s_binary in nodeimage-market.json, staged by cmd/socialweb-nodeimage rootfs.go:214), and the build host's /usr/local/bin/k3s reports v1.36.4+k3s1. dumbpipe: dumbpipe has no --version flag (checked against the real v0.39.0 archive: 'error: unexpected argument --version found'), so the skip reads /usr/local/bin/dumbpipe.version, which the preinstall run is what writes; a caller that pins another release through ctx.dumbpipeVersion gets a guest that installs that one. Decision: dumbpipe rides the preinstall manifest's runs rather than a new nodeimage config key, because the builder refuses a config whose required keys are unset and the market config lives outside every checkout (~/kcp-image/nodeimage-market.json), so a new required key would refuse the next build"
      - "secrets: the unit is enabled and started --no-block from runcmdPrepend instead of runcmd, so it starts at the head of runcmd, behind the package guard and ahead of setup-iroh.sh (7 s) and setup-k3s.sh; it is a oneshot with ConditionPathExists on its stamp, After=network-online.target provisioning-token.service, and it waits for accept.json itself"
      - "watched to fail (scratch mutations, restored): no guardPackages in buildUserData -> 3 FAIL in test/cloud_init_packages_test.ts and 1 in test/preinstall_manifest_test.ts; openssh-server dropped from the manifest -> 1 FAIL; the iroh skip removed -> 2 FAIL; the k3s skip removed -> 1 FAIL; the secrets start moved back to the end of runcmd -> 1 FAIL; the manifest's dumbpipe release changed to v0.40.0 -> 1 FAIL"
      - "green: atproto-market deno test -A test/ 124 passed 4 ignored; socialweb-computer-k8s deno test -A test/ 112 passed 1 ignored (live_market_node_test.ts skipped: a worktree has no sibling org root for its ../../../atproto-market/test/fetch-interceptor.ts import; run with the sibling repos present); hono-compute-provider deno test -A test/ 51 passed; deploy/check.sh --fast exit 0"
      - "the preinstall run was executed for real outside a chroot (busybox sh, target dirs redirected): sha256sum -c - OK against the pinned digest, the binary installs executable, dumbpipe.version holds the release, and the guest's own skip check takes the skip when that file matches the pin and downloads when it does not"
      - "not done: the image is not rebuilt and nothing is measured on prod -- the lead ships it and measures claim->Ready (the target is about 30 s against the 125 s of 2026-10-07). tmux (the wootty module) is deliberately not in the manifest: no market user_data asks for it, and the manifest is the market image's"
      - "live prod 2026-10-07T23:47:24.8Z (image sha256-52df8a09 built by the installer under sudo with the repo preinstall manifest; door/provider 0a7e9b0): pod created -> Running 54.0 s (was ~130 s); boot -> cloud-init finished 5.9 s (was 92 s), no apt fetch at all; secrets served 23:47:54.8Z, Node registered 23:48:09Z, Ready 23:48:11Z"
      - "remaining: policy eval 5.3 s (M2-BID-LATENCY), OpenBao link+login 6.4 s, apiserver link+agent registration 8 s, bob bid 3.6 s, claim->RFP 2.9 s, receipt->boot 3.4 s"
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): 0a7e9b0->cd3c96b 8cd8e12->7393baf bcccf22->78641f5 d20612d->6f51515(atproto-market)"
  - id: M3-JOIN-PATH
    milestone: M3
    title: >
      A node dials both links at once, and the guest stops unpacking k3s on its way to joining
    decision: >
      Measured on prod 2026-10-07T23:47Z (M3-BOOT-TIME evidence): secrets served at uptime ~11 s,
      "fetched the join secrets" at 17.4 s, Node registered about 8 s later. The chain is serial for
      one reason -- the apiserver link's dumbpipe ticket arrives only in the KV entry the OpenBao JWT
      login reads -- and the guest's first k3s command unpacks k3s's own runtime (about 254 MiB) into
      <data-dir>/data/<sha256 of its data tarball> on a filesystem it is already booting. Three
      changes, none of them to the market protocol or the cluster design: (1) every join script prints
      its own step times off the guest's /proc/uptime, so the split reads off the console instead of
      being inferred from two timestamps 8 s apart; (2) the door delivers the apiserver link's ticket
      with the claim's advertise address and port in the secrets bundle, and
      socialweb-apiserver-link waits on setup-secrets instead of socialweb-node-secrets, so both
      dumbpipe dials are up before the agent needs either. An apiserver ticket is a capability, not a
      credential: the apiserver still refuses a connection that does not carry the k3s token, and that
      token, the node password and the server CA stay in OpenBao. The KV copy of the ticket is still
      read when the bundle does not carry one -- an older door, or a link that is not listening yet --
      so those nodes still join, serially. (3) the image build stages k3s's runtime assets into the
      rootfs, checked against the .sha256sums and .links k3s itself checks after it unpacks, with
      data/current and the data/cni plugin links written to name the guest's path rather than the
      build host's. decided by Claude, user may veto.
    status: done
    evidence:
      - "the image reading the third change rests on: debugfs on the newest market image in ~/kcp-image/out-market (sha256-52df8a09..., the one M3-BOOT-TIME measured) shows /var/lib/rancher/k3s holding agent and nothing else, so the guest unpacks the runtime itself at its first k3s command"
      - "code: 69f91cf (STEP_PROBE and per-step lines in the six join scripts), e6b5d55 (NODE_LINK_ENV_PATH, renderNodeLinkEnv and nodeLinkEnv in k8s-common; the door reads <clusterKey>-apiserver beside <clusterKey>-openbao; node-secrets keeps a bundled apiserver.env and refuses one that names another node; apiserver-link waits on setup-secrets; ARCHITECTURE invariants 13 and 29 and section 9), 460521f (a link the door cannot read is withheld like an absent one, never fatal to the claim), 4a3f516 (cmd/socialweb-nodeimage/k3sdata.go: k3sAssetDir, parseK3sAssetDir, stageK3sAssets, verifyHashes, verifyLinks, linkK3sCurrent, linkK3sCNI, called from buildRootfs after stageK3s)"
      - "tests watched to fail then pass: k3s_join_test.ts 'node-secrets keeps the apiserver link the bundle already wrote, and needs no ticket in the KV entry' (mutation: the bundle branch disabled -> 3 FAILED), 'every join script times its own steps' and 'node-secrets prints its start and done times' (mutation: the two done lines dropped -> 2 FAILED); node_market_test.ts 'the secrets bundle hands the guest the apiserver link's ticket with the OpenBao ticket' FAILED first with the four-path diff, and 'an apiserver link that is not listening', 'an apiserver ticket that would ride a unit's command line' and 'an apiserver link the door cannot read' FAILED as undefined bundle entries; k3sdata_test.go TestAnAssetDirectoryWhoseSumsDoNotMatchIsRefused and TestAnAssetDirectoryWithNoSumsAtAllIsRefused (mutation: the sums check skipped -> 2 FAILED)"
      - "k3s's own trigger, measured on the build host: k3s --debug --data-dir <stage> check-config <kernel config> prints msg=\"Asset dir /var/lib/rancher/k3s/data/5a9973ddf4c7ec074f657c06287e0e6a07a24ecafd6d326827f70ef1e95bdd2d\". k3s falls back to its default data dir before it extracts, so on a host that already has this k3s the line names a directory outside the stage; the build copies that directory in and checks it either way. A probe run of stageK3sAssets against the real directory staged it in 3.5 s, with data/current -> /var/lib/rancher/k3s/data/5a9973dd... and data/cni/{bandwidth,bridge,cni,firewall,flannel,host-local,loopback,portmap} each pointing at the guest's bin/cni, the same shape as the host's own data/cni. The probe file was scratch and is deleted"
      - "green at 4a3f516 and 460521f: socialweb-computer-k8s deno test -A test/ 125 passed (live_market_node_test.ts excluded: a worktree has no sibling org root for its ../../../atproto-market/test/fetch-interceptor.ts import), deno check hono-socialweb-computer-k8s/mod.ts exit 0, cmd/socialweb-nodeimage go test -count=1 exit 0, gofmt/go vet/staticcheck clean, deploy/check.sh --fast exit 0"
      - "not done: the lead deploys, rebuilds the image and measures. No image was built here and nothing was measured on prod. The numbers this is aimed at are the 6.4 s OpenBao link+login and the 8 s apiserver link+agent registration M3-BOOT-TIME lists as remaining; the staged assets add about 254 MiB to the image, and a rebuild that -reuse-stale answers with an older image would measure none of it"
      - "step 4 of the handoff (start the OpenBao dial before the token exchange finishes) needed no change: socialweb-openbao-link already waits only on setup-secrets, so it dials while socialweb-node-secrets, which is where the token exchange lives, is still running"
      - "the image's name does not cover the builder's own behaviour, and that is now load-bearing: internal/nodefirecracker's BuildConfig fingerprint digests the config's inputs (kernel, modules, base image, k3s binary, busybox, init binary and package, preinstall, seeded images, boot args) and not the builder's code, so an image built before this change and one built after it share a name and -present-only answers present for either. Measured while checking the deploy path (scripts/deploy-firecracker-bidder.sh --dry-run, ORG_ROOT and LOCAL_NODEIMAGE_BIN redirected): the current inputs name sha256-d1a844929f46a01f5ea06a777bbf02c340d1725b7f6ddaaa1ba5a4052a5d0d07, the store in ~/kcp-image/out-market does not hold it, and the deploy's own guard (the answered dir's basename against the shipped FINGERPRINT) refuses any other directory -reuse-stale would answer with. So the lead's rebuild does include this change, but the next builder change under the same inputs would not be visible in the name, and fixing that (an input covering the builder, or the built binary's digest) changes how every image is named and is left to the lead"
      - "live prod 2026-10-08T00:23Z (image 52df8a09 rebuilt with the k3s runtime assets after moving the stale same-named build aside; door/provider 1abe746): bundle at uptime 11.03 s, apiserver-link and openbao-link both start 11.03 s (parallel), node-secrets done 17.66 s, Node registered ~22 s, CNI 23.9 s: secrets->registered ~11 s (was ~14 s); pod->Node registered 33.6 s (was 44.2 s). Follow-ups split into M3-JOIN-FOLLOWUPS"
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): 1abe746->35bb7d2 460521f->2c3bec3 4a3f516->eb85648 69f91cf->e596fb5 e6b5d55->c1178bf"
  - id: M3-JOIN-FOLLOWUPS
    milestone: M3
    title: The join's remaining waits, and an image name that changes when the builder does
    decision: >
      From the 2026-10-08T00:23Z prod reading: (1) socialweb-node-secrets starts with the bundle, fails
      because the OpenBao link has not connected, and systemd retries it 5 s later (RestartSec) -- poll the
      link inside the script at ~0.5 s for a bounded time instead; (2) Ready 00:23:46Z but the busybox pull
      began 00:24:03Z: ~16 s of pod sandbox creation (5 s in earlier runs), cause unknown -- read kubelet and
      containerd on a debug-ssh node; (3) internal/nodefirecracker's image fingerprint covers inputs, not
      the builder, so a builder-only change keeps the name and the installer ships the stale image (hit
      2026-10-08: the operator moved the old build aside to force a rebuild) -- add the nodeimage binary's
      digest (or a builder version) to the fingerprint. decided by Claude, user may veto.
    status: done
    evidence:
      - "(2) done in ee75a03. Market clusters now name --kube-scheduler-arg=config=<DataDir>/kube-scheduler.yaml, rendered pure by clusterstack.KubeSchedulerConfig/KubeSchedulerConfigPath/SchedulerKubeconfig and written before the steps are built by clusterprovider.WriteKubeSchedulerConfig (the OpenBao config's shape), with podInitialBackoffSeconds and podMaxBackoffSeconds at 1 and a clientConnection block (a --config file makes kube-scheduler IGNORE the deprecated --kubeconfig flag k3s still passes, so the block is what carries the apiserver)"
      - "(2) the flags the plan preferred do not exist here: MEASURED against k3s v1.36.4+k3s1, kube-scheduler's own --help (printed when it is handed a flag it does not know) lists --config and the deprecated --pod-max-in-unschedulable-pods-duration, and --kube-scheduler-arg=pod-initial-backoff-seconds=abcde is not rejected, so --pod-initial-backoff-seconds and --pod-max-backoff-seconds are not flags this binary defines. The file form IS read: a config of garbage bytes makes kube-scheduler exit (k3s logs `scheduler exited`) while the server stays up, so the failure is a cluster whose pods are never placed rather than one that fails to start"
      - "(2) tests, against the real binary under SOCIALWEB_LOCAL=1: TestK3sPlacesAZeroNodePodWithTheMarketSchedulerConfig creates a pod at zero nodes and requires it placed on the node that arrives (PASS 7.73 s, pod on probe-node); TestK3sPlacesNoPodWhenTheSchedulerConfigIsUnreadable is the same probe with a garbage config and requires the pod NOT placed (PASS 62.19 s), so a green acceptance result is about the file and not about a line k3s prints either way. Unit tests pin the market argv and the absence of the arg for a plain cluster; internal/clusterprovider pins where the file is written (mode 0600, same inode on a second write, nothing written for a plain cluster)"
      - "(2) what is NOT claimed: on this host a pod waiting on a node was requeued ~50 ms after the node became schedulable with the DEFAULT backoff in every shape tried (node Ready, a taint removed, a label added), so the 16 s prod reading was not reproduced here and this change bounds the worst case (a pod requeued into backoffQ waits at most 1 s) rather than an observed wait. The next reader should not treat the backoff as the measured cause of the 16 s"
      - "green at ee75a03: deploy/check.sh --fast exit 0 (gofmt, go build, apigen no drift, go vet, staticcheck, cargo fmt, cargo test, go test ./...)"
      - "(1) done in 4e5d9cf. renderNodeSecretsScript's login and KV read now go through a bounded wait on the link rather than one attempt: 120 attempts 0.5 s apart (NODE_SECRETS_LINK_ATTEMPTS, 60 s) around the curl to https://<OPENBAO_LINK_LOCAL_ADDR>/v1/..., each failed attempt printed on the console as `the OpenBao link on 127.0.0.1:18200 has not answered yet (attempt N of 120, curl exit S)`, and exit 1 after the last so systemd still owns the retry. The token exchange at the door is NOT retried: the door issued the workload identity token in accept.json before the guest existed, while the link started in the same `systemctl start --no-block` as node-secrets and its listener was not bound yet -- which is the failure the prod console recorded at 11.0 s uptime, followed by RestartSec=5 at 16.6 s"
      - "(1) tests, written first and watched fail at the red revision (deno test test/k3s_join_test.ts: 17 passed, 4 failed, every failure the prod symptom `OpenBao login at auth/jwt-0123456789ab as ci-node1 failed`): node-secrets waits for the OpenBao link the bundle started beside it, and retries the login itself (shimmed curl refuses the first 3 logins -> exit 0, 4 login calls, >=1500 ms); node-secrets gives the link a bounded wait and then says how long it waited before failing to systemd (linkWaitAttempts 3 -> exit 1, exactly 3 login calls, the message names the bound, no done stamp); node-secrets retries the read of the KV entry as well as the login (2 refused reads -> exit 0, 3 read calls); the wait node-secrets puts on the link is 60 s of half-second attempts, and names the listener it is waiting for. No existing node-secrets test changed: on the happy path the first attempt succeeds and the call list is what it was"
      - "(1) the other join script that dials a just-started link needed no change: the k3s agent dials the apiserver link, and a k3s agent that cannot complete the bootstrap against the address it dials LOOPS on `failed to get CA certs` rather than exiting (internal/versions/versions.json, the k3s v1.36.4+k3s1 invariant, MEASURED -- the same measurement that records a node dialing the wrong port looping), with Restart=always/RestartSec=5 and StartLimitIntervalSec=0 behind it on every join unit; socialweb-pod-cni already waits for the Node to register with its own bounded retry (CNI_WAIT_ATTEMPTS x 2 s), and meshd is a daemon k3s's own API serves rather than a dialer of the link. dumbpipe's listener ordering is not observable on this host (dumbpipe exists only in the guest), so what is claimed here is the agent's tolerance, not the link's bind order"
      - "(3) done in 1a5cd4d. nodefirecracker.BuildInputs gains builderDigest, the sha256 of the executable that reads the configuration (os.Executable + fileDigest), recorded as a read input role beside configPath and this host's busybox, so a builder-only change moves the name an image is given. RecordInputs takes the digest from a function it is handed and Fingerprint is unchanged, so every existing caller (ensure, main, gc, build, imagestore) keeps its signature; cmd/socialweb-nodeimage/inputs_test.go's role map gained the builderExecutableField declaration, which is the test that says the record and the reads are one set"
      - "(3) tests, written first and watched fail with builderDigest held at its pre-fix value (empty, i.e. no builder input at all): TestABuilderThatChangedNamesAnotherImageOverOneConfiguration named one configuration sha256-e777d066ae6075a9ceea43be64f425699f440cfab7b3a89cfbab5be94aecac8a under both builders; TestTheBuilderTheRecordNamesIsTheBinaryThatReadTheConfiguration recorded an empty digest where the running binary digests to sha256:0256433b164854ac7e2705b1acae5bc9f70346184b48d5e484bcfff3f1976dfb; TestEveryFieldTheBuildReadsIsOneTheRecordCovers reported that the record carries builderDigest and nothing is declared to read it. All three pass after the change; no other test in internal/nodefirecracker or cmd/socialweb-nodeimage changed"
      - "(3) the deploy still resolves an image end to end, measured with scripts/deploy-firecracker-bidder.sh --dry-run (ORG_ROOT set, LOCAL_NODEIMAGE_BIN redirected to scratch so nothing under ~/kcp-image is written): the fresh builder answers ABSENT for the same ~/kcp-image/nodeimage-market.json that M3-GUEST-JOIN's dry-run named sha256-d1a84492..., and this one names sha256-2216954253f95069c2ea1058f4c184514df8121ee516b6e348f6e1e9ca8673cf, so the build host builds and ships a new image instead of answering with the one the earlier builder wrote. EVERY IMAGE IS RENAMED ONCE BY THIS and the build host rebuilds it (the record grew a field, so nothing built before hashes to a name this builder resolves): expected, and it is the rebuild the defect was hiding"
      - "(3) what the change costs elsewhere, checked rather than assumed: socialweb-computer-kcp-check-host/checks.go:490 derives the name in ITS OWN process, so with the builder digest in the name its host.image-store check reports MISSING on a host whose store holds an image the nodeimage binary built -- a false MISSING for an operator; no Result in imageStoreResult sets Build:true, so -build exit statuses and deploy/control-plane-container/validate.sh's summary line are unaffected, and no Go test calls LoadImage. Nothing on the market path is affected, because nothing else resolves that name: the target asks `-ensure -json -reuse-stale` and is held to the shipped basename (scripts/deploy-firecracker-bidder.sh:929), the bidder's provider asks with --compute-provider-firecracker-reuse-image (deploy/bidder/socialweb-computer-bidder-run:108, deploy/market-network/70-bidder.yaml:24), and a host that builds for itself builds with the binary doing the asking. Two follow-ups for the lead, not done here: check-host could be handed the builder binary as a fact, and the deploy's builder build could pass -buildvcs=false, because a binary built inside the repo carries vcs.revision, so an unrelated commit also renames the image and buys a 2.3 GB rebuild"
      - "(2) not touched: the ~16 s pod sandbox gap still needs kubelet and containerd read on a debug-ssh node, left to the lead as the handoff asks"
      - "green at 1a5cd4d: deploy/check.sh --fast exit 0 (gofmt, go build, apigen 9 files no drift, go vet, staticcheck, cargo fmt, cargo test, go test ./...); socialweb-computer-k8s deno test -A test/ 130 passed with live_market_node_test.ts excluded (a worktree has no sibling org root for its ../../../atproto-market/test/fetch-interceptor.ts import, so it needs the import-map override deno.worktree.json this worktree carries)"
      - "(1),(3) cherry-picked onto rewritten main as 3957880, 63b2b5c, 831a370; deployed 2026-10-10T01:54Z; the installer built a NEW name sha256-51e20605 for unchanged inputs because the builder digest is now an input (3 proven: the old name 52df8a09 is no longer reused); guest console 02:01Z: node-secrets start 10.26 s -> done 11.71 s with no failed attempt (1) (was 11.04 -> fail -> 16.66 -> 17.66)"
      - "(2) root cause read 2026-10-10T02:02Z over debug ssh: Node Ready 02:02:13Z, kubelet idle 02:02:14.5-02:02:29.0, pod PodScheduled 02:02:29Z (CoreDNS and metrics-server also started 02:02:30Z): the 16 s is the k3s SERVER's kube-scheduler backoff for pods that were unschedulable while the cluster had no node, not the guest. Fix in flight: market clusters run kube-scheduler with ~1 s pod backoff (--kube-scheduler-arg)"
      - "(2) merged ds-sched-backoff (ee75a03, 11e23e8; market kube-scheduler config file with 1 s backoffs) as b223f8a and deployed 2026-10-10T02:51Z: the first pod STILL scheduled 15 s after its Node was Ready (Ready 02:53:18Z, PodScheduled 02:53:33Z). k3s server log: kube-scheduler starts when a node registers (02:53:17.22Z) and waited out its previous incarnation's leader lease ('Attempting to acquire leader lease' 02:53:17.35Z, 'Successfully acquired lease' 02:53:33.08Z)"
      - "fix 22d1841: the config sets leaderElection.leaderElect false (one k3s server per cluster); TestTheSchedulerConfigTurnsLeaderElectionOff watched to fail then pass; TestK3sPlacesAZeroNodePodWithTheMarketSchedulerConfig PASS 8.82 s against the real k3s v1.36.4 (SOCIALWEB_LOCAL=1); deployed 02:57:14Z (final configuration: debug key unset, SCALE_HOLD=0s)"
      - "live prod 2026-10-10: pod created 02:57:58.3Z, Node registered 02:58:43Z, Ready 02:58:45Z, PodScheduled 02:58:45Z (0 s after Ready, was 15-16 s), Running 02:58:52.9Z (Ready->Running 6 s)"
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): 11e23e8=ok 1a5cd4d->63b2b5c 22d1841=ok 3957880=ok 4e5d9cf->3957880 63b2b5c=ok 831a370=ok b223f8a=ok ee75a03=ok"
  - id: M5-NETPOL
    milestone: M5
    title: NetworkPolicies are enforced on market nodes
    decision: >
      Read 2026-10-10T02:02Z over debug ssh on a market node: k3s logs "Failed to load kernel module
      nft-expr-counter with modprobe", kube-router's network policy controller "Aborting sync. Failed to run
      iptables-restore ... RULE_APPEND failed (No such file or directory): rule in chain KUBE-POD-FW-*", and
      `iptables -S` answers "iptables: Incompatible with this kernel" (iptables-nft 1.8.10; nft_compat,
      nf_tables, xt_comment, xt_mark, xt_set loaded). Pod traffic works (M5 proven) but tenant NetworkPolicies
      are not enforced. Fix: ship the missing nft expression modules (nft_counter and whatever else the rules
      need) in the node image's module set or kernel config, prove a deny-all NetworkPolicy blocks a cross-node
      connection on prod. decided by Claude, user may veto.
    status: done
    evidence:
      - "root cause (2026-10-10, local market network + offline image reading): not a missing module FILE and not an iptables backend mix. The image copies the build host's whole module tree (debugfs on ~/kcp-image/out-market/sha256-51e20605.../rootfs.ext4 lists nft_limit, nft_log, nft_reject*, nfnetlink_log under /lib/modules/7.2.8-200.fc44.x86_64), but the Fedora kernel is built with CONFIG_MODPROBE_PATH=\"/usr/bin/modprobe\" and the Ubuntu 24.04 rootfs has modprobe only at /usr/sbin/modprobe (-> ../bin/kmod; /sbin -> usr/sbin; no /usr/bin/modprobe). So every request_module() in the guest exec'd a missing path and no module autoloads; only systemd-modules-load's list and userspace modprobe-on-PATH calls load anything. kube-router's failing restore line (61 on the market network) is `-m mark ! --mark 0x10000/0x10000 -j NFLOG --nflog-group 100 -m limit --limit 10/minute --limit-burst 10`; Ubuntu's iptables-nft 1.8.10 writes NFLOG and limit as NATIVE nft expressions (`nft --debug=netlink` on that ruleset replayed with the image's own xtables-nft-multi in a user+net namespace on the build host: `[ log group`, `[ limit rate`, `[ counter`, compat `match name` for comment/conntrack/addrtype/set/physdev), so it needs nft_log and nft_limit, which the module list never named (it names xt_NFLOG/xt_limit, which only the xtables path uses) -> ENOENT -> the whole sync aborts. The same replay on the build host kernel, where autoload works, applies cleanly and `iptables -S` lists it, so the ruleset and the iptables binary are fine"
      - "nft-expr-counter: harmless. The host config has no CONFIG_NFT_COUNTER and modules.alias no nft-expr-counter; the counter expression is built into nf_tables.ko, and the replayed rules carry `counter` expressions that load. k3s's own modprobe of that alias still warns on every boot"
      - "not explained: prod's `iptables -S` -> `iptables: Incompatible with this kernel` (iptables-nft maps errno 0 to that text, i.e. a userspace parse failure, not a kernel errno); not reproduced on the market network (no shell there) -- re-read on prod after the rebuilt image ships"
      - "fix b363cb8: cmd/socialweb-nodeimage/modprobe.go linkKernelModprobe, called in buildRootfs after applyPreinstall: reads CONFIG_MODPROBE_PATH from the staged kernel config (kernel default /sbin/modprobe when unset; an empty or relative value refused), resolves it inside the stage (symlinks followed within the stage, never on the build host, 40-hop bound), and if it does not resolve links it to the rootfs's own modprobe (first of /usr/sbin, /sbin, /usr/bin, /bin that resolves), refusing a rootfs with none. The build prints `modprobe /usr/bin/modprobe -> /usr/sbin/modprobe (...)`. The builder digest is an image input (M3-JOIN-FOLLOWUPS (3)), so the next deploy names and builds a new image"
      - "unit tests (modprobe_test.go), watched to fail with linkKernelModprobe a no-op stub: TestAKernelThatExecsAModprobeTheRootfsDoesNotHoldGetsALinkToTheRootfsOwn (readlink usr/bin/modprobe: no such file) and TestARootfsWithNoModprobeAtAllIsRefused (accepted) -> 2 FAIL; pass after. Also TestTheModprobeTheKernelExecsIsReadFromItsConfig, TestAModprobePathThatIsNotAbsoluteOrIsEmptyIsRefused, TestAKernelWhoseModprobeTheRootfsAlreadyHoldsIsLeftAlone, TestResolvingInsideTheStageNeverLeavesIt (absolute links stay in the stage, a self-link does not resolve)"
      - "live gate 58f61e1: internal/marketnetwork live_m3_netpol_test.go assertNetworkPolicyEnforced, called by TestAMarketNodeJoinsATenantClusterThroughItsOpenBaoWithNoSsh once the node is Ready: busybox httpd server + client pinned to the market node; the client must fetch with no policy, be refused 3 fetches in a row within 2 min of a deny-all ingress NetworkPolicy on the server, and fetch again once it is deleted; on failure it reads kernel.modprobe and /proc/modules from the client pod and the console's policy-sync errors"
      - "RED on the market network 2026-10-10T02:22Z (image sha256-51e20605, the one prod runs; control plane socialweb-k8s:m3live): node joined, no-policy fetch ok, then `a deny-all ingress NetworkPolicy on netpol-server did not stop the client within 2m0s: the last fetch answered \"netpol-ok\"`; guest kernel.modprobe=/usr/bin/modprobe; /proc/modules had nf_tables nfnetlink_log nft_compat nft_ct nft_masq nft_nat and xt_* but NO nft_limit, NO nft_log; console: `Aborting sync. Failed to run iptables-restore ... line 61: RULE_APPEND failed (No such file or directory): rule in chain KUBE-POD-FW-H4TFJTPHAUNIO5IH` repeating -- the prod symptom"
      - "GREEN on the market network 2026-10-10T02:29-02:35Z, image sha256-9b6879e3 built by the fixed builder from ~/kcp-image/nodeimage-market.json + deploy/bidder/preinstall-market.json into a scratch store (debugfs: /usr/bin/modprobe -> /usr/sbin/modprobe): TestAMarketNodeJoinsATenantClusterThroughItsOpenBaoWithNoSsh PASS 501.44s; nft_limit and nft_log loaded on demand; zero `Aborting sync`/`RULE_APPEND` lines on the guest console; deny-all ingress enforced 9 s after it was applied (`wget: can't connect to remote host (10.42.0.6): Connection refused`), fetch restored when deleted; the rest of the gate (restart adoption, release, zero nodes, no ssh) passed unchanged"
      - "green at b363cb8+58f61e1: deploy/check.sh --fast exit 0 (gofmt, build, apigen, vet, staticcheck, cargo, go test ./... incl. test/nocomments_test.go)"
      - "single-node only: the gate's cluster is capped at one node (SCALE_MAX_NODES=1), so this proves same-node enforcement (kube-router's physdev/FORWARD path); a cross-node deny on prod is the lead's proof, after a deploy rebuilds the image"
      - "live prod 2026-10-10 (bidder reinstalled from main 1ecc684: image sha256-5b823ce0, /usr/bin/modprobe -> /usr/sbin/modprobe; bob restarted 02:44:12Z): np-client on node1 (10.42.4.2), np-server on node2 (10.42.2.2), CROSS-NODE: 02:45:37Z baseline reachable; deny-all ingress NetworkPolicy on role=server applied 02:45:37Z -> refused 02:45:38Z, 02:45:40Z, 02:45:42Z; policy deleted 02:45:42Z -> reachable 02:45:43Z"
      - "node2 over debug ssh 2026-10-10T02:46Z: /proc/sys/kernel/modprobe=/usr/bin/modprobe (link to /usr/sbin/modprobe); nft_log nft_limit nft_compat xt_NFLOG xt_limit loaded; 0 'Aborting sync. Failed to run iptables-restore'; iptables -S lists 110 rules (64 KUBE-POD-FW) -- the earlier 'Incompatible with this kernel' was the same missing-module cause"
  - id: M6-CLONE
    milestone: M6
    title: A CI job on a market node can fetch its repository
    decision: >
      The acceptance run used GIT_STRATEGY none: market nodes resolve public names, but the GitLab host is a
      tailnet name (gitlab.dorper-matrix.ts.net), so the runner helper cannot clone. Options: a public
      clone_url for the runner (a reverse proxy in front of GitLab's HTTPS), or reaching the tailnet from
      the cluster. User 2026-10-08: "It's a funnel" -- GitLab :8443 is a Tailscale Funnel. Measured 2026-10-10 from socialweb.computer (no tailscale): gitlab.dorper-matrix.ts.net resolves publicly (208.111.34.11) and https://gitlab.dorper-matrix.ts.net:8443/users/sign_in answers 200, so the GIT_STRATEGY none was a wrong assumption; prove a market-node job clones. decided by Claude, user may veto.
    status: done
    evidence:
      - "GitLab pipeline 79 (project kcp-ci-probe, branch acc-clone, runner 6, SPA cluster acc-gitlab-0001, prod 2026-10-10): job 280 with the default GIT_STRATEGY ran 02:04:19-02:05:27Z on a market node: 'Fetching changes with git depth set to 20... Created fresh repository. Checking out 8cc8fd76 as detached HEAD', test -f .gitlab-ci.yml passed, Job succeeded; docs/gitlab/04 corrected (the clone limit was an untested assumption)"
  - id: M2-BID-LATENCY
    milestone: M2
    title: A first-free winner is accepted within 0.5 s of its bid, not after 6 s of policy evaluation
    decision: >
      User asked 2026-10-07 why first-free waits. Measured on prod (door log, claim node96 23:16Z):
      firstFree true, bidWindowSec 30; bid present 23:16:04.97Z, first_free_winner 23:16:11.30Z. The window
      is not the cost: BidCollector waits on allow() = decideBid + evaluateCandidate, which spent ~3 s
      silent (policy worker start, record/DID resolution) then ~3.3 s scanning the same delegated-trust
      vouch chains (badgeBlueKeys, operator vouches) three times, uncached. User 2026-10-07: caching must
      keep the policy engine generic from the execution of it. So the requester learns nothing about what
      a policy does: it mirrors the bidder's host wiring (market-bidder/mod.ts: one createScopeCache() per
      process passed to createPolicyEvaluator, firehose trust events -> scopeCache.applyEvent); the door
      creates the one ScopeCache and shares it across contracts; candidates are pre-warmed through the
      generic evaluator.scope at bidder_discovery; verdict memoization stays inside policy execution
      (runPolicy's opaque cache). Timing logs per phase. decided by Claude, user may veto.
    status: done
    evidence:
      - "merged atproto-market 87e9ebf (502a1bf timing logs, 2b4bfd1 requester scopeCache + prewarm + trust invalidation), door 0a7e9b0 (one createScopeCache per process shared by every contract)"
      - "live 2026-10-07T23:47Z: bid_decide 0 ms; bid 23:47:34.18Z -> first_free_winner 23:47:39.46Z = 5.3 s (was 6.3 s): the previous contract was 5.5 min earlier, past the 5 min positive TTL, and scope_prewarm logged at 23:47:35.7Z, after the bid, though bidder_discovery was 23:47:29.4Z. Still doing."
      - "merged 2026-10-10 onto the rewritten histories: policy-engine fast-forward to b049d54 (47fb02f one in-flight scope run per (policy, counterparty, args); b049d54 gha-lite trust lookups concurrent, once per DID); atproto-market cherry-picks cfb9a73 (refresh in place), 754658c (delegated trust concurrent), 02bb5c6 (first-free wins when the shared run lands); tests: policy-engine 52 passed, atproto-market check + 130 unit passed, bidder lock --frozen ok; door main 831a370 130 passed"
      - "live prod COLD (first contract after the 01:54Z door restart) 2026-10-10: bidder_discovery 02:01:47.648Z, scope_prewarm_done 02:01:49.242Z (1.6 s, was ~6 s), waiting_for_bids 02:01:51.120Z, first_free_winner 02:01:51.375Z: bid->winner 0.255 s cold (was 5.3-5.7 s); warm reading 2026-10-08T00:23Z 0.25 s"
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): 02bb5c6->57bbcd3(atproto-market) 0a7e9b0->cd3c96b 2b4bfd1->d030da3(atproto-market) 502a1bf->47013be(atproto-market) 754658c->e6f0529(atproto-market) 831a370=ok 87e9ebf->MISSING(atproto-market) cfb9a73->a28e714(atproto-market)"
  - id: M4-SCALE-IN
    milestone: M4
    title: Per-node scale-in
    decision: >
      A node holding nothing beyond baseline for the 2m hold is cordoned, drained and released.
      Supersedes plan 0002 section 4.4 (all-or-nothing). Decided by Claude, user may veto: "nothing
      beyond baseline" means no live pod bound to the node other than DaemonSet pods, mirror pods,
      k3s add-on pods and pods of measured-baseline workloads; one node per hold (a release restarts
      the hold of the nodes left), emptiest (fewest baseline pods) then newest; never below the
      target while intent exists; no release while a Pending pod with no node is younger than the
      hold, and never of a node a Pending pod is nominated or hostname-pinned to; per-node idle
      clocks in memory (a provider restart restarts the hold); only nodes this provider process
      holds; cordon, re-observe, uncordon and keep a node that took work or whose eviction is
      refused, evict through the Eviction API, then the existing teardownHeld. A DaemonSet is never
      intent or demand (its desired count follows the nodes).
    status: done
    evidence:
      - "deployed 2026-10-07 ~19:44Z (main 932b139 + 6af8994); idle cluster unchanged at 0 nodes; a live release awaits a bidder"
      - "fb4ca07 internal/scale: scale.Loads (per-node Busy/Evictable/Incoming, Unplaced), Decide's Release branch (State.NodeIdleSince, Decision.Release, scale.Releaser), DaemonSets and DaemonSet/mirror pods out of Intent and Demand"
      - "fc16e9b provider: NodeObserver computes Placement; scaledriver performs Release through scale.Releaser (ErrNoReleaser otherwise); cmd/socialweb-clusterprovider/scalein.go tenantNodes.Release (cordon, re-observe, drain, teardownHeld) and nodeHosts.observeHeld (marks held nodes, adds in-flight)"
      - "ARCHITECTURE section 10 'Scale-in is per node' and 'A DaemonSet is never intent'; plan 0002 section 4.4 marked superseded"
      - "tests: internal/scale TestThreeIdleNodesWithDemandForOneReleaseTwoOverTwoHoldsAndKeepOne (releases c-node3 at 2m, c-node2 at 4m, c-node1 kept), TestANodeWithARunningNonBaselinePodIsNeverReleased, TestANodeThatBecameIdleLessThanTheHoldAgoIsKept, TestTheHoldIsExactForOneIdleNode, TestScaleInNeverGoesBelowTheTarget, TestAPodAboutToBeScheduledHoldsItsNode, TestAnUnplacedPendingPodHoldsEveryReleaseUntilItIsOlderThanTheHold, TestANodeThisProviderDoesNotHoldIsNeverReleased, TestLoadsCountsOnlyWhatHoldsANode, TestAReleaseRestartsTheHoldOfTheNodesLeft, TestADaemonSetIsNeverIntentAtZeroNodesOrOnceUp, TestADaemonSetOnlyClusterScalesToZero; internal/scaledriver TestAReleaseDecisionReleasesThatOneNodeAfterTheHold, TestAFailedReleaseLeavesTheStateWhereItWas, TestAnActuatorThatCannotReleaseOneNodeIsAnError; internal/clusterprovider TestObservePlacesEachPodOnItsNode; cmd TestIdleMarketNodesAreCordonedDrainedAndReleasedOnePerHold (3 held market nodes, real driver + observer over a fake tenant: DaemonSet-only node released at 2m without evicting its pod, node with a k3s add-on pod at 4m with that pod evicted, claims deleted, vault revoked, Nodes deleted, working node kept to 20m), TestANodeThatTakesWorkBeforeTheCordonIsUncordonedAndKept, TestARefusedEvictionKeepsTheNode"
      - "superseded tests rewritten: TestADaemonSetIsInertAtZeroNodesAndHoldsOnceUp (asserted a DaemonSet with a node is intent) -> TestADaemonSetIsNeverIntentAtZeroNodesOrOnceUp; DaemonSet rows of TestTheNarrowTriggerCannotScaleToZero and TestDemandAgreesWithIntentOnEveryIntentFixture; TestPodsOfACountedJobStatefulSetOrDaemonSetAreNotCountedTwice now wants 3; TestTheObservationRangesOverExactlyThePredicatesKinds admits Placement and UnreconciledCustomResources"
      - "watched to fail (scratch mutations, restored): decision with no per-node hold -> 4 FAIL; busy ignored -> 2 FAIL; incoming ignored -> 1; no hold restart after a release -> 3; held ignored -> 1; emptiest ignored -> 1; oldest first -> 2; unplaced pod ignored -> 1; DaemonSet pods as intent -> 2; DaemonSet desiredNumberScheduled as intent (the old rule) -> 3; the old None branch for count > target -> 7; release down to the target -> 3. Provider: no cordon -> 2; no re-observe -> 1; no uncordon -> 2; no eviction -> 2; DaemonSet pods evicted -> 1; refused eviction ignored -> 1; release without teardownHeld -> 1; held never marked -> 3"
      - "green: deploy/check.sh --fast exit 0 (gofmt, build, apigen, vet, staticcheck, go test)"
      - "not done: deployed; a live market cluster releasing a node one hold after its work moved off (ACC-ZERO)"
      - "'only nodes this provider process holds' now includes market nodes adopted after a provider restart (M3-NODE-LOSS provider restart half, 9ca4c04): TestAnAdoptedIdleNodeIsScaledIn"
      - 'live 2026-10-07T23:18:56Z: the cluster''s last node torn down ''nothing running for 2m0.03s, past the 2m0s hold''; tenant then lists no nodes, control plane up. Per-node scale-in with >1 node still owed (ACC-ZERO)'
      - "live 2026-10-07T23:26:15Z with 2 market nodes and the workload cut to 1 replica at 23:23:43Z: node96 'held nothing beyond the baseline for the hold, so it is cordoned, drained and released' (evicted kube-system/coredns, metrics-server), its OpenBao role/policy/secret removed, VM released confirmed, Node removed 23:26:16.7Z; decision 'emptiest and newest of 1 such nodes'; node97 kept the workload"
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): 6af8994->0b02637 932b139->a6072ee 9ca4c04->acd218f fb4ca07->9c90581 fc16e9b->7d0f1c3"
  - id: M4-ESO-BESIDE
    milestone: M4
    title: ESO runs beside the control plane per cluster and is removed from the tenant
    decision: >
      A process in our container against the tenant apiserver, webhook by loopback URL, store
      reaching the cluster OpenBao on loopback; works at zero nodes and closes the ESO-3 market gap.
    status: done
    evidence:
      - "merged to main 933a7c9; deployed 2026-10-07 ~19:57Z: external-secrets controller (--enable-leader-election=false, live 127.0.0.1:20012) and webhook (:20010, healthz 127.0.0.1:20011) run in the container; tenant has no ESO Deployment; ClusterSecretStore openbao Valid/ReadWrite/Ready True; ExternalSecret users/openbao-demo SecretSynced Ready True at ZERO nodes on the market substrate (ESO-3 closed on prod)"
      - "defect seen in that deploy: at 19:57:39Z one pass read intent and wrote 3 claims; the migration deleted the old Deployments with BACKGROUND propagation, so their ReplicaSets' Pending pods outlived them, named no baseline workload, and read as demand 3. The claims failed and were released within seconds with no RFP published (no bidder online). Fixed: Retire deletes Foreground and waits until each object is gone; TestRetiringWaitsUntilTheOldWorkloadAndItsPodsAreGone and TestRetiringRefusesWhenTheOldWorkloadNeverGoes watched to fail (build) then pass"
      - "commits 411678e (internal/eso beside install, migration split, webhook TLS), afe36e8 (process-group members, leases, grant), 209d724 (cmd wiring, add-on pass), fa35e32 (image pins, install.sh retired); docs commit follows"
      - "two members external-secrets-webhook then external-secrets after the CCM in procgroup.StackOrder; leases external-secrets-health, external-secrets-webhook, external-secrets-webhook-health; metrics off, leader election off; identity system:socialweb:external-secrets bound to upstream ClusterRole external-secrets-controller; webhook by url https://127.0.0.1:<lease> under a minted CA, failurePolicy Ignore"
      - "image: /usr/bin/external-secrets from ghcr.io/external-secrets/external-secrets:v2.11.0@sha256:66fb710878cbf3eba4a927e35c5d75e29a202c9bab238aad64bfb23b786b962b to /usr/local/bin/external-secrets (sha256 9ea572f5e1d6..., COPY probe built and compared); bundle /opt/socialweb/eso/external-secrets.yaml sha256 c427124642886d240af8f28d294714a168d920ef8e5d55db14600e00ed0cfb80 (esobundle stage built, checksum passed); versions.json pins all three and TestTheImageCarriesTheExternalSecretsThisManifestPins holds the Dockerfile to them"
      - "live 2026-10-07, SOCIALWEB_LOCAL=1: TestAZeroNodeMarketClusterSyncsAnExternalSecretFromItsOwnVaultWithNoESOPodInTheTenant PASS 25.4 s; Ready in 22.68 s; zero nodes; webhook denied a data-less ExternalSecret on its loopback URL; store Ready=True Valid on https://127.0.0.1:<vault>; users/openbao-demo synced 500 ms after apply; no ESO pod or Deployment in the tenant"
      - "watched to fail first: 7 internal/eso tests against a stub; the live test before the grant change (SecretSyncedError, VAP socialweb-mint-identity denied the beside identity); a CRD-schema-refused probe replaced by a webhook-only one"
      - "unit: TestTheTenantKeepsOnlyTheCRDsAndTheRolesAndEverythingThatRanThereRetires, TestTheMigrationDeletesEveryRetiredObjectBeforeAnythingIsApplied, TestAWebhookRegistrationStillAddressedByServiceIsReplacedNotMerged, TestTheWebhooksAreRegisteredByLoopbackURLUnderTheMintedCA, TestTheExternalSecretsControllerBindsOnlyItsLeasedLoopbackHealthPortAndElectsNoLeader, TestTheExternalSecretsWebhookServesItsLeasedPortWithTheMintedCertDir, TestAClusterLeasedBeforeThePairExistedRestartsWithoutItUntilItsNextSpawnLeasesIt, TestTheBaselineIsReMeasuredEmptyOnceTheOperatorLeavesTheTenant, TestAJoinWithoutTheExternalSecretsBinaryOrBundleIsRefused; deploy/check.sh --fast green"
      - "not deployed: prod's next deploy migrates its one cluster (ARCHITECTURE section 15)"
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): 209d724->2611347 411678e->ff1e515 933a7c9->47181ca afe36e8->ea807e8 fa35e32->6b7840d"
  - id: M4-PENDING-CRS
    milestone: M4
    title: Intent counts workloads beyond baseline plus our operators' CRs not yet reconciled
    decision: >
      User chose workloads plus pending CRs; leftover namespaces, RBAC and ConfigMaps never hold nodes.
      Decided by Claude, user may veto: a CR counts as intent only if its operator runs IN the tenant
      cluster and the CR is not reconciled, because an operator beside the control plane (ESO after
      M4-ESO-BESIDE) reconciles at zero nodes. Registry scale.Operators of {Group, Kind,
      ReadyCondition, InCluster}; not reconciled = ReadyCondition not True, or that condition's or
      status.observedGeneration older than metadata.generation, or deleting behind a finalizer; each
      such CR is one unit of demand. scale.OurOperators ships empty: no operator of ours runs in a
      tenant cluster once ESO moves. NodeObserver refuses to observe when the registry names an
      in-cluster kind and no lister is supplied.
    status: done
    evidence:
      - "fb4ca07 internal/scale/operators.go (Operators, OperatorKind.Reconciled, Unreconciled, CustomResourceOf, OurOperators), Observation.UnreconciledCustomResources in Intent and Demand and carried through Narrow"
      - "fc16e9b internal/clusterprovider NodeObserver.Operators + CustomResources lister and the no-lister refusal; main.go wires scale.OurOperators"
      - "ARCHITECTURE section 10 'A custom resource is intent only while an operator of ours that runs IN the tenant cluster has not reconciled it'"
      - "tests: internal/scale TestACustomResourceCountsOnlyWhileItsInClusterOperatorHasNotReconciledIt (no status, Ready False, stale condition generation, lagging status.observedGeneration and finalizer-held deletion count; Ready True, deleted without finalizer, beside-the-control-plane kind and unregistered kind do not), TestAnUnreconciledCustomResourceAsksForOneNodeAndAReconciledOneForNone, TestNoOperatorOfOursRunsInTheTenantClusterToday, TestCustomResourceOfReadsTheFieldsTheRuleNeeds, the CR row of TestDemandAgreesWithIntentOnEveryIntentFixture; internal/clusterprovider TestAnInClusterOperatorKindNeedsALister"
      - "watched to fail (scratch mutations, restored): InCluster flag ignored -> 1 FAIL; stale generation ignored -> 1; CRs not intent -> 2"
      - "green: deploy/check.sh --fast exit 0"
      - "registry empty by design; correct only once M4-ESO-BESIDE lands -- until then ESO in the tenant is baseline and its CRs are not intent, which is the behaviour before this item"
      - "re-verified 2026-10-10: fb4ca07 is 9c90581 and fc16e9b is 7d0f1c3 after the history rewrite. Parent 9c90581^ with 9c90581's tests: FAIL (build: undefined Operators); HEAD a7932be 03:00Z: ok internal/scale (TestACustomResourceCountsOnlyWhileItsInClusterOperatorHasNotReconciledIt)"
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): fb4ca07->9c90581 fc16e9b->7d0f1c3"
  - id: M5-DNS
    milestone: M5
    title: Per-node CoreDNS on a link-local IP set as kubelet clusterDNS
    decision: >
      Removes the docs/gitlab DNS wait hack. Decided by Claude, user may veto (ARCHITECTURE section 9
      "Every market node resolves names on its own CoreDNS"): 169.254.20.10 on a dummy interface
      swdns0 made by a guest unit (socialweb-node-dns-link), not an init container; the agent passes
      --kubelet-arg=cluster-dns=169.254.20.10 and --node-label=socialweb.computer/node-local-dns=true
      (not the server's --cluster-dns, which is also kube-dns's ClusterIP); resolv-conf left at k3s's
      choice; a host-network CoreDNS DaemonSet kube-system/socialweb-node-dns (k3s's seeded CoreDNS
      image, Corefile bound to the address, kubernetes plugin over the in-cluster Service the apiserver
      link carries, forward . /etc/resolv.conf) applied in EVERY tenant by the add-on pass under
      socialweb-addons and selected by that label, so a node without the address never runs it;
      k3s's own CoreDNS is kept.
    status: done
    evidence:
      - "commits 2125360 (internal/tenantnet: manifests, Apply, wiring into tenantAddOns before the vault step), 58dfd82 (guest units, agent flags), 4e72a73 (live zero-node proof); docs commit follows"
      - "tests watched to fail then pass: internal/tenantnet TestApplyServerSideAppliesTheNodeReaderAndThePerNodeDNSUnderTheAddOnManager, TestThePerNodeDNSServesOnTheLinkLocalAddressOnlyOnNodesThatCarryIt, TestTheDNSImageIsOneTheNodeImageSeeds (3 failed against a stub Objects/Apply); cmd/socialweb-clusterprovider TestTheTenantAddOnsApplyTheNodeReaderAndThePerNodeDNSBeforeAnythingElse (applied [] before the wiring); socialweb-computer-k8s test/node_network_test.ts 'the node's user_data adds the pod network units, and the k3s agent never waits on meshd' (agent flags and After), 'the DNS link puts the per-node DNS address on a dummy interface, once', 'the guest, the tenant DaemonSet and the k3s server agree on the DNS address, its label and the cluster CIDR' (12 of 12 in that file failed against the previous module with its new exports stubbed, deno test --no-check)"
      - "guard that passed first: internal/clusterprovider TestThePodNetworkAddOnsAreBaselineAndNeverIntent (DaemonSet applied by socialweb-addons is baseline; a node running only its pod has Busy 0 and no intent)"
      - "live 2026-10-07 hat-8, SOCIALWEB_LOCAL=1: internal/clusterprovider TestAZeroNodeMarketClusterAllocatesPodCIDRsAndTakesTheNodeNetworkAddOnsAsBaseline PASS 10.3 s -- market cluster Ready in 7.7 s at zero nodes; the real apiserver admitted every tenantnet object twice; the DaemonSet kept its selector and host network; measured baseline holds kube-system/socialweb-node-dns; Observe reads no intent; zero nodes"
      - "docs/gitlab: pre_get_sources_script removed from 02-gitlab-runner.md and 04-spa-cluster-runner.md; README 'Why the DNS wait is gone' explains it; 03 rewritten for market clusters; the local-substrate live GitLab test keeps its wait (flannel, one CoreDNS)"
      - "stated limits: a node that just became Ready starts its DNS pod beside its first workload, unordered (the removed wait covered that window; a startup taint lifted once DNS answers is the fix, open); no live multi-node reading"
      - "green: deploy/check.sh --fast exit 0; door deno task check exit 0, deno task test 103 passed 0 failed 1 ignored (scratch org root, siblings symlinked)"
      - "not done: deploy; ACC-MESH's 'DNS resolves on a node that does not run the cluster CoreDNS' on a live market cluster (needs guests through the RFP flow; not hand-provisioned)"
      - "23:23:16Z from the pod on node97 (CoreDNS runs on node96): resolv.conf nameserver 169.254.20.10; kubernetes.default.svc -> 10.43.0.1, mesh-probe.default.svc -> 10.43.21.222, gitlab.com resolves; wget via the Service ClusterIP returned node96's pod content"
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): 2125360->69166e1 4e72a73->7470210 58dfd82->d89a123"
  - id: M5-MESHD
    milestone: M5
    title: Rust iroh pod mesh in rust/socialweb-meshd
    decision: >
      QUIC datagrams over iroh, a tun device, each peer Node's podCIDR routed to its iroh endpoint id;
      k3s --flannel-backend=none with bridge CNI. Lives in this repo (user chose).
    status: done
    evidence:
      - "status moved back to doing on 2026-10-07: the daemon is done (below); the integration half is built and unit-proved but not deployed and not read on a live multi-node market cluster"
      - "integration commits c8ba259 (k3s server --flannel-backend=none --cluster-cidr=10.42.0.0/16 for MarketDialled only), 2125360 (node-reader RBAC in every tenant via internal/tenantnet, byte-identical to rust/socialweb-meshd/deploy), 58dfd82 (guest units: socialweb-pod-cni, socialweb-pod-masquerade, socialweb-meshd-install, socialweb-meshd + drop-in, meshd.env from node-secrets), 4cbebf1 (door /downloads/<sha256>/<name>, loadPinnedDownloads, meshdPinOf, fulfiller meshd option), 4e72a73 (image ships both builds; live proof); docs commit follows"
      - "decisions (decided by Claude, user may veto; ARCHITECTURE section 9 'The integration'): podCIDR fed by a oneshot polling the Node through the kubelet's own kubeconfig with k3s kubectl (host-local has no range-from-Node mode), written once; cluster CIDR named explicitly at k3s's default; kube-router network policy kept; masquerade -s 10.42.0.0/16 ! -d 10.42.0.0/16 by iptables -w with a comment; meshd binary kept from a node image when its sha256 matches else downloaded from the door by hash; binaries built on the host by rust/build.sh inside build.sh k8s (or KCP_MESHD_DIST) and passed as a named context, not a Docker builder stage; copied after the /opt/org-root chown; door refuses to start on a sidecar mismatch and offers no meshd (meshd_not_offered) without the pair or public-origin"
      - "tests watched to fail then pass: internal/clusterstack TestAMarketClusterRunsNoFlannelAndNamesItsPodNetwork (argv lacked the flag), TestOnlyAMarketClusterDropsFlannel, TestK3sAcceptsTheMarketPodNetworkFlags (real k3s v1.36.4 probe); socialweb-computer-k8s test/node_network_test.ts (12/12 red against the previous module): pod-cni polls until the podCIDR exists and writes bridge isGateway ipMasq false mtu 1160 host-local + portmap, fails without a CIDR and refuses a non-IPv4 one; masquerade adds the rule once; meshd-install keeps a matching node-image binary, downloads <door>/downloads/<pin>/socialweb-meshd-<arch> and installs it 0755, refuses tampered bytes leaving nothing, asks for aarch64 on arm64; the module refuses a non-sha256 pin and a non-origin door; k3s_join_test.ts node-secrets writes meshd.env 0600 (1 red); test/pinned_downloads_test.ts + node_market_test.ts (5 red against stubs): exact bytes and sha256 at /downloads/<sha256>/<name>, 404 for wrong hash/name/uppercase/unsidecarred file, start refused on a mismatched or misnamed sidecar, pin needs both arches and an origin, the fulfiller's user_data carries the pin and installs no meshd without one; test/meshd_image_test.go (red against the previous Dockerfile and build.sh)"
      - "mutation caught (scratch copy): socialweb-meshd.service added to the k3s agent's After -> 'the k3s agent is After socialweb-meshd.service: deadlock'"
      - "live 2026-10-07 hat-8: TestAZeroNodeMarketClusterAllocatesPodCIDRsAndTakesTheNodeNetworkAddOnsAsBaseline PASS 10.3 s -- k3s ran kube-controller-manager --allocate-node-cidrs=true --cluster-cidr=10.42.0.0/16; a system:node identity was denied list nodes before the node reader and allowed after (RBAC: allowed by ClusterRoleBinding socialweb-meshd-node-reader)"
      - "measured 2026-10-07 hat-8, k3s v1.36.4+k3s1 --disable-agent: started with the default backend, stopped, restarted on the same datastore with --flannel-backend=none --cluster-cidr=10.42.0.0/16 -> /readyz ok in 2.1 s, 0 'critical configuration' lines, 0 fatal lines, a ConfigMap written before the switch still read back; zero nodes before and after"
      - "image: build.sh k8s with KCP_MESHD_DIST built socialweb-k8s:m5-meshd (local only, not deployed) at 1,711,865,393 bytes; the meshd layer is 17.2 MB (socialweb-meshd-x86_64 9358064 bytes sha256 7c563cd6..ad64, socialweb-meshd-aarch64 7875400 bytes sha256 47df1302..212a, rebuilt identical to a56b60d's); sha256sum -c OK in the image; the first placement (before the chown) cost a second 17.2 MB copy-up, fixed"
      - "smoke: the door's loadPinnedDownloads + route over those two real binaries returned 200 with bytes hashing to each pin"
      - "green: deploy/check.sh --fast exit 0 (gofmt, build, apigen, vet, staticcheck, cargo fmt, cargo test, go test); door deno task check exit 0, deno task test 103 passed 0 failed 1 ignored"
      - "not done: deploy (provider + door together); ACC-MESH pod A on node 1 reaching pod B on node 2 -- a real two-node run needs two guests born through requester -> RFP -> bid -> accept -> cloud-init on the M1 market network; no guest was hand-provisioned, so this stops at the unit, zero-node live and image level"
      - "commits c67b7b6 (crate, unit tests, netns proof), ee9285e (kube mode against k3s, node-reader RBAC), 11c9a29 (static musl builds, check.sh hook), a56b60d (15 s idle timeout, relist per watch); ARCHITECTURE section 9 'Pods reach pods on other market nodes through socialweb-meshd'"
      - "scope: the daemon only. Built, NOT deployed, NOT installed by any user_data. The integration (k3s --flannel-backend=none, bridge CNI conflist mtu 1160 on spec.podCIDR, cluster-cidr-only SNAT, node-reader RBAC in every tenant, user_data module installing the sha256-pinned binary, systemd unit) is listed in that section and is not done; ACC-MESH stays todo"
      - "decisions (decided by Claude, user may veto): hand-rolled kube client over iroh's reqwest/rustls/ring instead of kube-rs; tun MTU 1160 = 1200-byte QUIC path - 38 bytes noq overhead - 2; tun address = own podCIDR network address /32 with routes src it; one packet per datagram, no header; lazy dial, 512-packet per-peer queue; connection idle timeout 15 s; blake3 pure-Rust backend for the aarch64 musl build"
      - "found: k3s v1.36.4+k3s1 node authorizer refuses list nodes to system:node:<name> ('cannot read all nodes, only its own Node object'); rust/socialweb-meshd/deploy/node-reader-rbac.yaml (group system:nodes get/list/watch nodes) is required; patching another Node is 403 and meshd exits 4"
      - "tests watched to fail then pass: 18 of 30 unit tests failed against mutated route computation, packet dispatch and did:key encoding (did:key spec vector did:key:z6MkiTBz1ymuepAQ4HEHYSF1H8quG5GLVVQR3djdX3mDooWp and the dumbpipe IROH_SECRET vector shared with internal/didkey), then 41 of 41 pass; two_meshd_in_two_netns_carry_ping_between_two_pod_cidrs failed with route install disabled and with emptied datagrams, then passed; with_a_kubelet_identity_meshd_annotates_its_own_node_and_follows_the_node_list failed on the 403 list before the RBAC existed, then passed"
      - "measured: max_datagram_size 1162 on a 1228-byte underlay, 1414 on 1500; 1132-byte DF ping crosses, 1400-byte DF refused by the node, 1400-byte fragmentable ping crosses; one-way recovery after a peer SIGKILL+restart 30.8 s at iroh's 30 s idle, 15.4 s at 15 s"
      - "binaries at a56b60d, rustc 1.95.0: socialweb-meshd-x86_64 9358064 bytes sha256 7c563cd6cd10e4113aab6a8dc3fc45e041728376499e6a215b1d51f0dfa2ad64; socialweb-meshd-aarch64 7875400 bytes sha256 47df130296100de6b08e9008421a6351fa5848264ebb71ec586245daf7f1212a; identical from a second target dir on hat-8; both static binaries pass the netns and k3s proofs (aarch64 under qemu binfmt)"
      - "green: deploy/check.sh --fast exit 0 (cargo fmt, cargo test); SOCIALWEB_MESHD_NETNS=1 SOCIALWEB_MESHD_K3S=1 cargo test on hat-8 with sudo -n"
      - "prod 2026-10-07T23:20:54Z: Deployment mesh-probe (2 replicas, required pod anti-affinity) on test-cluster-alice-0001 scaled the cluster 0 -> 2 market nodes ('Provision nodes:2'); node96 and node97 registered 23:22:30Z, Ready 23:22:31Z; pods 10.42.2.4 on node96 and 10.42.4.3 on node97"
      - "23:23:06Z kubectl exec (apiserver->kubelet over the agent tunnel) into both pods: node97 -> node96 pod IP answered by node96's httpd (HTTP 404 before its index existed), node96 -> node97 pod IP returned node97's content"
      - "23:23:16Z from the pod on node97 (CoreDNS runs on node96): resolv.conf nameserver 169.254.20.10; kubernetes.default.svc -> 10.43.0.1, mesh-probe.default.svc -> 10.43.21.222, gitlab.com resolves; wget via the Service ClusterIP returned node96's pod content"
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): 11c9a29->f5a8319 2125360->69166e1 4cbebf1->c1f3494 4e72a73->7470210 58dfd82->d89a123 a56b60d->8981e07 c67b7b6->c392aa6 c8ba259->38f279a ee9285e->debf915"
  - id: M6-RUNNER1
    milestone: M6
    title: GitLab runner 1 run_untagged false
    decision: User confirmed we own that GitLab config. Set through the API (PUT /runners/1), not gitlab-rails.
    status: done
    evidence:
      - "2026-10-07T22:11:42Z GET /api/v4/runners/1 before: run_untagged true, tag_list [socialwebcomputer], online"
      - "2026-10-07T22:11:42Z PUT run_untagged=false; read back run_untagged false, tag_list [socialwebcomputer], paused false, online"
      - "commit 9adc4d8 'M6-RUNNER1: runner 1 takes only tagged jobs'"
  - id: M6-LIMITS
    milestone: M6
    title: Fix the old stated limits
    decision: >
      CCM child gets Pdeathsig; an absent instance snapshot makes CCM refuse Node deletes; OpenBao
      SIGHUP TLS reload for a host added later; reconcile's NotFound deletes behind a per-cluster
      flag. decided by Claude, user may veto.
    status: done
    evidence:
      - "merged to main 01f9b2e; deployed with 5cde08e (Retire foreground) 2026-10-07 ~20:42Z: 20:43:39Z 13/13 passes None nothing running and no node, 0 claims written across the restart, ExternalSecret users/openbao-demo still SecretSynced"
      - "commit 65e3297 procgroup: every member dies with the provider (Pdeathsig SIGKILL from a goroutine holding runtime.LockOSThread for the child's life); TestAGroupMemberDiesWithTheProviderAndNotWithTheThreadThatSpawnedIt watched to fail twice: no Pdeathsig, member outlived the SIGKILLed provider by 5 s; bare Pdeathsig, member died with its spawning thread under a live provider. After: alive 500 ms on, gone 0-20 ms after the provider SIGKILL"
      - "commit 3cfc6ae cloudprovider: absent nodes.json is ErrNoSnapshot and the controller deletes nothing; [] still means empty. TestTheControllerDeletesNoNodeOverAnAbsentSnapshotAndDeletesOverAnEmptyOne, TestAMissingInstanceFileIsAnErrorNotAnAbsence, TestAnAbsentSnapshotIsRefusedAndAnEmptyOneIsEmpty watched to fail"
      - "commit 273c109 vaultcert.EnsureServedVaultTLS: re-sign, read served leaf, SIGHUP the one port holder whose argv is this cluster's bao server -config, read back up to 10 s, refuse if not served. OpenBao v2.6.2 on hat-8: [198.18.255.254 127.0.0.1] -> [198.18.255.254 127.0.0.1 10.100.0.1] served 114 ms after the call; with a no-op reload TestARunningVaultServesAHostItsLeafGainedOnceItIsReloaded failed with ErrVaultReloadDidNotTake. Refusal kept: TestAReloadThatDoesNotTakeKeepsTheRefusal, TestAListenerThatIsNotTheVaultIsNeverSignalled"
      - "commit 2665f00 nodeHosts memo of released claims (name/UID) whose Node is gone, per cluster: TestTheReleasedNodeReconcileCostsTheTenantNothingOnceItsNodesAreGone, 5 passes before 1 DELETE + 6 calls (5 lists), after 1 DELETE + 2 calls. The ESO per-pass eso.Remove deletes had no caller since M4-ESO-BESIDE"
      - "NEXT.yaml limits: three removed (portless member not swept, absent snapshot reads empty, vault host added later); ARCHITECTURE section 15 entry 'Four stated limits, fixed'; deploy/check.sh --fast green; 21 internal/clusterprovider vault tests pass under SOCIALWEB_LOCAL=1"
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): 01f9b2e->185a12c 2665f00->8b60d64 273c109->edd7c3a 3cfc6ae->4376da6 5cde08e->04059a4 65e3297->54bf63b"
  - id: M6-SPA-CREATE
    milestone: M6
    title: A cluster created in the SPA is reachable through its kubeconfig without operator help
    decision: >
      Found by the acceptance run 2026-10-08: (1) the door's createCluster call times out at 30 s while the
      gateway's Ensure waits for a Cluster status.URL that cannot appear, because CreateCluster sets running
      only AFTER Ensure returns; the request's cancellation then also drops ControlPlaneUp, leaving the
      cluster running=false; (2) the kcp mount child (workspace "clusters" with spec.URL + mount ref) is only
      ever created inside Ensure and only if status.URL is already set, so nothing creates it later; (3) the
      gateway talks to kcp as kcp-admin (admin.kubeconfig context root), and kcp refuses spec.URL from a
      non-privileged user ("spec.URL can only be set by system privileged users"), so even a later Ensure
      cannot create or repair the child; the operator had to create it as system:admin; (4) errForDeny maps
      every Ensure error to "the identity store could not be read" (503), hiding the real cause. Fix: set
      running before waiting; never wait for the URL in the request path (return Initializing); a reconciler
      (provider, which owns the Cluster status) creates/repairs the mount child once status.URL appears,
      with a privileged identity scoped to that write; Ensure errors keep their own message/code and are
      logged. decided by Claude, user may veto.
    status: done
    evidence:
      - "commit 9a56762: CreateCluster records spec.running FIRST on context.WithoutCancel and answers the phase it finds (Initializing is a 200, not a refusal); errForDeny maps only revocation.ErrDenyUnreadable to the identity-store message and everything else to ClusterUnavailable 'the cluster could not be prepared: <cause>', with the full error logged"
      - "commit df51a4b: Ensure stops at the Cluster CR and never reads it back or waits for status.URL; kcpclient.MountChild (one method, EnsureMountChild) creates the child or repairs an empty spec.URL, built from SOCIALWEB_CLUSTERPROVIDER_MOUNT_CONTEXT (default system:admin) in cmd/socialweb-clusterprovider; Provider.addressMount runs it on every pass right after the status write and fails the pass on a refusal; Build refuses a join without the writer (ErrNoMountChild)"
      - "tests watched to fail then pass: gateway TestCreateClusterRecordsRunningBeforeItReadsAnythingBack, TestACreateTheCallerAbandonedStillLeavesTheClusterRunning, TestCreateReportsThePhaseItFoundRatherThanRefusing, TestAnEnsureFailureKeepsItsOwnCauseAndIsLogged (FAILED: 'the identity store could not be read'); kcpclient TestEnsureStopsAtTheClusterCRAndNeverWaitsForAURL and TestEnsureCreatesTheTree (FAILED against the old wait: 'Ensure POSTed the mount child 1 time(s)', 'read the Cluster CR back 10 time(s)'), TestTheMountChildIsWrittenByItsOwnWriter (4 subtests), internal/kcpclient/context_test.go; clusterprovider TestAPassAddressesTheMountChildOnceTheCRReportsAURL, TestEveryPassRepairsTheMountChildItFinds, TestAMountChildThatCannotBeWrittenFailsThePass (FAILED against a stubbed addressMount)"
      - "deploy/check.sh --fast green at df51a4b (gofmt, build, apigen 9 files no drift, vet, staticcheck, cargo fmt/test, go test)"
      - "door unchanged: it already answers a create with the phase as data (test/cluster_api_test.ts 'the account is the proved one, never one named in the body' asserts 200 {name, phase: Initializing}) and its 30 s budget now covers a create that no longer waits"
      - "owed: a create through the SPA on prod with the mount child written by the system:admin context, and the live-gated mount proofs (internal/kcpclient/mount_child_live_test.go, internal/clusterprovider/live_mount_test.go, internal/kcpclient/live_apiserver_test.go) run against a live kcp -- they were updated but NOT RUN here"
      - "live prod 2026-10-08 after merge e27c6f6 deployed: the SPA (headless OAuth as aliceoa.bsky.social) created acc-gitlab-0002 and returned its kubeconfig 00:27:14Z->00:27:31Z with no clusterUp workaround; the provider wrote the mount child 'clusters' (spec.URL https://127.0.0.1:20029) itself, Ready 32 s after creation; from hat-10 the SPA kubeconfig + socialweb-credential reached the cluster through kcp.socialweb.computer at 00:28:45Z (~74 s after create) with no operator action (acc-gitlab-0001's child had needed a manual system:admin create)"
      - "SHAs rewritten by the user's 2026-10-08/09 history rewrite (old->current, matched by subject): 9a56762->2015b46 df51a4b->e83444b e27c6f6->361b083"
  - id: M6-DOCS
    milestone: M6
    title: Rewrite ARCHITECTURE invariant 13 and update docs/gitlab
    decision: >
      Invariant 13 read that a market node's secrets cross only OpenBao JWT over the cluster link; it
      now reads what the guest is handed before it has authenticated to anything and what stays in
      OpenBao, and invariant 29 and section 9 moved with it. That half was done by M3-JOIN-PATH on
      2026-10-08, because the join path could not be shortened without the invariant saying where the
      apiserver ticket travels. What remains here is docs/gitlab.
    status: done
    evidence:
      - "docs/gitlab/04-spa-cluster-runner.md: 'The hop this document cannot prove' now opens with the 2026-10-08 settlement (pipelines 77/78 exec'd through Caddy->kcp->mount); new section 'What was measured on 2026-10-08' with the SPA create, the runner config and run command, the pipeline, the pipeline-78 timeline and the limits found (no clone from market nodes to the tailnet GitLab, the SCALE_HOLD knob, one job = one rented node); invariant 13 rewritten by M3-JOIN-PATH"
      - "commits 5589f4e (docs/gitlab: the SPA runner recipe ran end to end on 2026-10-08) and 453c0a8 (clone-through-funnel correction, 2026-10-10)"
queued_after_this_plan:
  - docs/plans/0006-socialweb-computer-unified-binary.yaml
  - docs/plans/0007-compute-provider-kcp.yaml
  - docs/plans/0008-embedded-deno-and-typescript-denopods.yaml