# socialweb-computer-kcp - architecture Service which serves virtual Kubernetes clusters. A tenant cluster's k3s **nodes** scale to zero while its control plane stays live. A single ATProto account may have multiple clusters. Module `github.com/johnandersen777/socialweb-computer-kcp`, Go. CRDs in API group `socialweb.computer`. --- ## 0. The organizing principle: kcp is part of each application kcp (the Kubernetes control plane) is a core part of each application, not infrastructure shared between applications. The one exception is the virtual Kubernetes clusters project, which uses kcp in the traditional way. This section states the principle and the rules that follow from it. - **An application instance = its own kcp + a manifest set + a profile.** A bidder handle, a requester `run request`, the gateway, a market network, a dev instance. kcp belongs to the application instance, never to a machine; a host is only where instances run, and any number run on one host. - **Application state = kcp objects** in that instance's kcp: ComputeVM, ComputeContract, NodeClaim, OpenBaoServer, Postgres, HostProcess, the component kinds (Pds, Plc, Relay, Bidder, GuestControlPlane, Requester, GatewayAPI, SshServer, IngressProxy, DispatcherTLS, PolicyEngine) and their Service and Exposure objects. - **Application components** (the servers that would otherwise sit inside one process) = component instances that run in process in the core, in the component runtime of `internal/component`, and HostProcesses for third-party binaries; both are reconciled from those objects. - **Modes = profiles over one manifest set.** A bidder in OAuth QR mode against a real account has no Pds component; a bidder in the market network has a Pds component of its own. - **Composition = applying manifest sets, or nesting instances as workspaces** (a parent's workspace holds a child instance's HostProcess objects; in a market network the child application itself is a workspace of the parent's kcp). - **Applications talk to each other only over the atproto market protocol.** A shared kcp (the market network profile) is composition and isolation, never a data channel. - **The market network** (`deploy/market-network`) is the seed of this model and the vehicle for parallel workstreams: each workstream's live tests run in its own market network, brought up in its own git worktree of this repository; the shared real bidder (bobvmbuilder) and alice are a final soak only. - **The one traditional use** is the virtual Kubernetes clusters project this document mostly describes (sections 1-4, 9-11): tenant k3s `--disable-agent` control planes fronted by kcp (clusterprovider, socialweb-computer-k8s, NodeClaims). Even there, the gateway's kcp is one application instance's kcp. **Port 0 rule.** Every listener behind the edge binds port 0 and reports its bound address into its own kcp object's status (a HostProcess, kcp slot, kine or front is read back from its single listening socket). An in-process component does not listen itself: it registers handlers, a Service names the component and a handler, and an Exposure publishes the Service (`internal/exposure`). A local Exposure listens on a port the core keeps in its port registry (`/ports/ports.json`, a 64-port block of 20000-29999 chosen by the state dir, `internal/stack/componentcore`), so the number survives a core restart, and records it in `status.address`. Peers resolve each other by service name through kcp: an address is application state in kcp, not config. Why: N instances per machine; no allocator and no bind-close-rebind race; nesting; blue/green slots; isolation (an application learns addresses only from its own kcp, so it can never reach another instance's components by accident). Exception: only stable public edge entry points keep known addresses - caddy on 443 and an application's front that caddy proxies to (the gateway's kcp front on 127.0.0.1:6443, the GatewayAPI web app front on 127.0.0.1:8788, `internal/stack/gatewaykcp` `DefaultWebAppFront`). A tenant cluster's members are third-party binaries that take a port number on their command line; for them the cluster's apiserver front binds port 0 and holds the socket for the life of the cluster, and the member binds that number beside the hold (section 8), so the number is stable across member restarts and no other process on the host can take it. What stays about the machine: `/dev/kvm`, the systemd user manager and its units (production placement only), `install`, the host contract, and the bootstrap layer that runs an instance's kcp itself. **How a multi-server process decomposes into kcp:** - **One root object per application** in its own kcp - MarketBidder, MarketRequester, Gateway, MarketNetwork. Its spec carries the application's options and profile; the core renders the profile into owned children, and a profile switch deletes what the new profile omits. - **Which kcp, by profile (user decision 2026-10-17, amending the strict rule of 2026-10-12).** The prod profiles keep a strict kcp per application: the gateway on its host, a bidder handle on its host, a `run request` without `--network`. The market network profile (dev, CI, soak) runs every application in ONE kcp, each application in its own workspace: the network's own services in root:global (PLC), root:relay (relay, dispatchers), root:alice and root:bob (PDSes), and each child application (a bidder, the gateway, a requester invocation, an ssh gateway) in root:, its components in root::app. One manifest set serves both shapes: in prod it is applied into the application's own kcp root; in the network the applier rebases every phase workspace under root: (kcpapply.Options.Base; root:*-provider and the sets of scope kcp stay shared) and service names come from the phase workspace (`.default.{{.ServiceSuffix}}`). The child keeps its own core (the network's dev host places it, as a HostProcess in root:dev:), scoped to its workspace, with no kcp or kine of its own; its components run in process in that core. Children learn the network from values the composer copies onto their root objects and from valueFrom references that name another workspace of the same kcp (plan 0019). - **Every CLI option is kcp-native.** Each option is a root-object field or a profile (the TypeScript hono-bidder and request-vm-ssh CLIs that came before plan 0026 are gone, and only legacy guest transports were dropped). Sign-in is an object, MarketIdentity: the URL to visit, the association to approve and an expired session are status that the foreground CLI renders (`internal/appcli`). A bidder or requester without a real account keeps its repo in a Pds component of its own (a LocalRepo, `internal/pds/local`). - **kcp only.** No component runs as a standalone process; every run goes through `socialweb-computer` and kcp. - **The requester is a fresh kcp per invocation** whose state dir (kcp root + vault) persists; `run request release|resume ` restarts that kcp to finish the contract. - **Secrets only in the application's own OpenBao** - OAuth sessions, repo key, relay keypairs, the OIDC signing key (the identity engine of the bidder's own OpenBao), requester key, the gateway's keys; kcp objects hold references. - **Every listener reports into status** (a component's `status.endpoints`, a Service's `status.endpoints`, an Exposure's `status.address`, a HostProcess's `status.listeners`), so no peer scrapes stdout for a DID, a port or readiness. - **iroh is the default and required guest transport.** The ssh gateway moves to iroh; the tunnel, fedproxy-ssh, fedproxy-web and old browser-terminal modules, websocat ProxyCommand paths, the relay-FQDN guest addresses and the guest package registry are removed. Two other transports are opt-in per entry of `spec.tunnels`: transport `opentunnel` (an https URL to a guest port) and transport `webrtc` (a datachannel to a guest port through socialweb-guest-agent); each comes back as a tagged `vm.onNetwork` with its kind (plans 0032 and 0034, invariant 36). The browser shell is ssh over webrtc first, with wootty behind an opentunnel tag (`spec.tunnels[].shell`) as the fallback. - **Components:** the bidder's guest-facing endpoints (an allow-list proxy to the bidder's OpenBao for the guest's bind, login and identity token, `POST /v1/events`, `GET /v1/downloads` and the `compute.guest.subscribeEvents` subscription) are the GuestControlPlane component of the bidder; the NodeClaim fulfiller is the NodeClaim controller (`internal/controllers/nodeclaim`) bound by the gateway's GatewayAPI component. ## 1. The objective, stated once This objective is the virtual Kubernetes clusters project, the one traditional use of kcp (section 0). ``` kcp + openbao + kine + embedded-postgres + k3s, as gateway-side processes. A tenant cluster's k3s NODES scale to zero while the kcp + k3s CONTROL PLANE stays live. ``` Per virtual cluster: ``` kine over an external store <- the datastore k3s server --disable-agent <- the apiserver. NOT a node. Always up. k3s agent(s) <- the ONLY nodes. 0 or 1. Provisioned on apply. ``` `--disable-agent` is the piece that makes "zero nodes" a real state rather than a contradiction: a single-node k3s where the server IS the node can never produce zero nodes. **One apiserver per store, no exceptions.** kine's revision is a monotonic sequence with dummy fill records specifically so no second client can create a lagging event, and two apiservers over one store is unsafe. This constraint forces the shape. The user and the nodes talk to the SAME API. There is no handover and no second store: the mount's `status.URL` points at our apiserver at zero nodes and at N nodes alike. **Zero nodes is not a cluster state.** The cluster stays `Ready` while nodes go to zero. ## 2. Scope boundary ``` IN the virtual cluster: workspaces, the zero state, the store, the gateway, the trigger, identity, reachability, the CRDs, the ports, the lifecycle OUT how a guest COMES TO EXIST in production: the requester, the RFP record, the bid, the accept, the bidder, the market's money SEAM ComputeProvider - provision(vm{user_data}, requesterDid, spec?) / destroy(id) ``` The seam is "given this cloud-init, make me a guest and give me back an id". A provider supplies a CLOUD; it never supplies the CONNECTION SEMANTICS. `kcp + k3s` in this document are **gateway-side processes**, not market VMs. Where text elsewhere says "market VM", read it as the substrate a provider made, wherever it came from. The market is reached through a NodeClaim. A tenant cluster's nodes are rented from the market as the CLUSTER'S OWNER, through that owner's atproto OAuth session, which the gateway holds. The cluster provider reaches past the `ComputeProvider` seam: `internal/nodemarket` writes a `NodeClaim` into the cluster's home workspace, and the NodeClaim controller (`internal/controllers/nodeclaim`, bound by the GatewayAPI component with the mapper of `internal/gateway/nodeclaim`) turns it into one contract entry of the cluster's Infra (plan 0031), whose controller applies the ComputeContract that the gateway's Requester runs through the RFP flow with the owner's session. The winning bidder boots the guest from cloud-init, the guest joins over the cluster's apiserver ClusterLink, and the controller writes the claim's status. The `internal/scale` decision that says WHEN a node is needed is separate. This does not change: - The cloud-init is the real composer's (`BuildUserData` plus the module registry in `internal/atproto/cloudinit`). The substitution is who won the bid, never what the guest is told to install. - A guest is born from cloud-init `user_data`, never hand-assembled. Provisioning goes through the declared path, never a shell. The market substrate adds no provisioning path of its own: it writes a record and waits. - The claim carries PATHS, never secrets. The node host reads the join token, the per-node password and the server CA out of the cluster's data dir (files k3s writes), stores them in the cluster's OpenBao at `socialweb-nodes/`, and the guest fetches them itself with a JWT login (invariant 29). ## 3. The zero state - the mechanism ``` zero nodes -> server up. No kubelet, so it never registers a Node. kubectl get pods -> objects, Pending, Unschedulable apply -> provision an agent -> it joins the SAME server -> the pod runs idle -> agent torn down -> back to zero nodes, server still up ``` The honest failures at zero, which are the correct failures: ``` exec -> unable to upgrade connection: pod does not have a host assigned port-forward -> unable to forward port because pod is not running. Current status=Pending ``` **Because the store is external, the control plane is cattle.** A destroyed server rebuilds against the same store: no `bootstrap data ... different token`, identical CA fingerprint, objects intact, and a kubeconfig issued before the rebuild still valid. The token is the only local-disk secret that matters. **There is no syncer.** There is one apiserver, not two, so there is nothing to propagate between them: intent goes in, kubelet sees it. What remains is the binder: ``` podStrategy.PrepareForCreate -> Phase: Pending, free our binder writes -> PodScheduled=False, FailedScheduling reason ...and later spec.nodeName + PodScheduled=True ``` Same reconcile loop in both states. **Two requirements `--disable-agent` adds**, both from k3s source: - `--egress-selector-mode`: use `cluster`. It is a static authorizer (node IPs plus cluster/pod CIDR) with no per-pod watch dependency. `agent` suffices for apiserver->kubelet but fails for service webhooks and aggregated APIs with a 10 s hang rather than a fast refusal. All modes produce byte-identical config files - the mode changes the TUNNEL, not the config. - The identity bundle must be pre-seeded. See section 8. ## 4. The store **Each cluster's store is `kine` over that cluster's own postgres.** k3s is started with `--datastore-endpoint http://127.0.0.1:` (`clusterprovider.DaemonURL`) and dials kine as an etcd endpoint; kine's `--endpoint` is the cluster's postgres DSN (`internal/providermain/main.go` `storeEndpoint`: the per-cluster postgres registry, unless `SOCIALWEB_CLUSTERPROVIDER_STORE_ENDPOINT` names an operator-supplied store for every cluster). The same postgres also holds the cluster vault's storage (`internal/clusterstack/openbao.go`). k3s is never pointed at a bucket: measured against k3s v1.36.4+k3s1, the S3 form is refused at startup with `unknown driver`, and a `t4` endpoint makes k3s start its OWN kine against the bucket instead of dialing ours - two writers on one store (`internal/clusterprovider/stack.go:39-43`, `ErrNoStore`). **A central store is not used.** `kine` plus `t4` over a central `rustfs` (bucket per tenant, prefix per cluster) is not built. The store is the per-cluster postgres above, and k3s is never pointed at a bucket. No `rustfs` name is in the Go code or the pins; the only `t4` is in `stack.go`'s `ErrNoStore` measurement. **Several kine over one database are the k3s HA shape.** Measured on stand-ins (`docs/research/2026-10-11-vm-lifecycle/control-plane-handover.md` 2-3): two k3s servers, each with its own kine on one postgres and the same token, share CAs and service-account key and both serve; kine's unique `(name, prev_revision)` is the compare, so a racing second write fails like an etcd compare. Planned, not built: a k3s/kine upgrade starts a second slot beside the first on the same postgres for the seconds of a swap behind a per-cluster front. Section 1's "one apiserver per store" is the rule outside that overlap. atproto is the durable **mirror**, not the datastore: it cannot be one. No compare-and-swap, no durable watch (the sequencer resets `seq` to 0 on restart), no compaction, and every write is a signed commit. What it carries is **intent and identity** - the `Cluster` CR mirror and account records. Not data, and not portability: the export is deferred. An empty `--datastore-endpoint` is refused. k3s would otherwise keep its objects in its own sqlite, so a cluster that scales to zero and comes back comes back EMPTY. The join cannot reach that state: `clusterstack.Spec.DatastoreEndpoint` is REQUIRED and `StepsFor` refuses an empty value, because empty is the one value k3s resolves SILENTLY into a brand-new cluster. **The store is durable against the launcher.** Durability rests on the data directory under `SOCIALWEB_CLUSTERPROVIDER_STATE_DIR`. "The cluster is gone" means someone removed the data directory - a caller decision, not an automatic consequence of a process exiting. ## 5. Components Everything is either a controller-runtime reconcile loop or a deploy config. No new language. This section lists the components. The table below is the virtual-clusters project, section 0's traditional exception. It and the component subsections that follow are organized by section 0: each application instance has its own kcp, manifest set and profile, its components run in process in the core's component runtime (`internal/component`) and its third-party binaries are HostProcesses, and listeners behind the edge bind port 0. | # | Component | Layer | Why it exists | |---|---|---|---| | 1 | `Cluster` CRD - the mount object | Go | Carries `status.URL` and `status.phase`, which are the real contract. Owns `Draining`, `conditions[TenantPathShut]`, `conditions[Reachable]`, `status.nodeExpiry` | | 2 | **Materialiser** - controller watching kcp with a host selector | Go | Places and stops each cluster's member units through HostProcess objects. This IS the gateway's supervisor | | 3 | Cluster provider | Go | Owns the `Cluster` CR lifecycle: mints the instance token, fetches the identity bundle, sets `status.URL` once the apiserver answers. ONE process serves EVERY cluster in the kcp tree, and a cluster is named inside it by its home workspace, not by the bare name the owner chose | | 4 | Node provider | Go | Watches the home workspace for intent, actuates one node, writes `Node` state | | 5 | kcp provider workspace | YAML | `APIResourceSchema` plus `APIExport` for our CRDs; tenants `APIBinding` | | 6 | k3s cloud-init module | Go `internal/atproto/cloudinit`, `UserDataModule` | Writes the k3s, secrets, iroh and node modules cloud-init runs in a market guest. The only sanctioned way a guest-side component arrives | | 7 | OpenBao plus ESO | deploy config | The vault, at two levels: a root vault one per deployment and a cluster vault one per ON cluster, both in the production stack. OpenBao holds the per-cluster 13-entry identity bundle and is its AUTHORITATIVE store - the wake reads it out of the vault and materialises the file tree at `/server/tls`+`cred/`, because k3s has no vault plugin and must read files at runtime (sections 9, 14 and 15). ESO materialises vault-backed Secrets into the tenant cluster (row 9); the bundle is custody and wake-time restore, not the runtime location | | 8 | Cluster links | Go, `internal/clusterlink` + `internal/clusterprovider/links.go` | Two `dumbpipe listen-tcp` members of every cluster's process group, one in front of the k3s apiserver and one in front of the cluster vault, each an iroh endpoint whose key lives in that cluster's vault. A node dials the cluster through them; the gateway holds no per-node connection. Recorded as `ClusterLink` objects | | 9 | External Secrets pair | pinned upstream binary, `internal/eso` + `internal/clusterprovider/externalsecrets.go` | The External Secrets Operator v2.11.0 run as TWO members of every cluster's process group, beside the control plane: the controller (`external-secrets`) and its admission webhook (`external-secrets webhook`), one digest-pinned binary the release embeds as its `external-secrets` asset. They dial the tenant apiserver with a minted credential and reach the cluster vault on loopback, so an ExternalSecret syncs at ZERO nodes. The tenant keeps only the bundle's CRDs, ClusterRoles and a URL webhook registration; no ESO pod runs in it | | 10 | Node network | Rust `rust/socialweb-meshd`, Go `internal/tenantnet` + `internal/atproto/cloudinit` + `internal/nodeagent` | Market clusters run `--flannel-backend=none`; each market node gets, from its `user_data`, a bridge CNI conflist on its Node's pod CIDR, cluster-CIDR-external SNAT, the sha256-pinned `socialweb-meshd` (downloaded from the gateway's web app by hash, `internal/nodeagent/host.go`) carrying pod traffic to other nodes over iroh, and `169.254.20.10` for its own CoreDNS; every tenant gets the meshd node reader RBAC and the per-node DNS DaemonSet as baseline (section 9) | **Not building**, each rejected on evidence: a CNI plugin (the market path writes a conflist for k3s's bundled `bridge`/`host-local`/`portmap`; the cross-node half is component 10), a CAPI provider, an OIDC issuer, an ATProto bridge, a syncer, a virtual workspace, a second reverse proxy, a reachability substrate, an underlay, a warm node pool, an admission plugin (RBAC already enforces the identity classes exactly). A registry plus volume-commit machinery is not built either and is not rejected on evidence: its only reason was S3-backed PVs on the central rustfs that is not used (section 4). ### The per-cluster slice of member units Each cluster is a slice of member units. The cluster provider never forks a member: it writes one HostProcess object per member into the gateway's own kcp (workspace `root` unless `SOCIALWEB_CLUSTERPROVIDER_HOSTPROCESS_WORKSPACE` says otherwise), and the HostProcess supervisor in the core places it on systemd-user (production) or under devsup (proc, dev). The reconcile loop is still the only supervisor; there is no cluster-group verb and no Pdeathsig. A core or provider restart or upgrade restarts no member. ``` pg-.service the cluster's Postgres object, unix socket only cluster-.slice one slice per cluster c--bao.service After/Wants the pg unit c--front.service no ordering, Restart=always, fd store (section 8) c--kine@a.service After/Wants the pg unit c--k3s@a.service After/Wants kine@a c--ccm.service After/Wants front c--esowh.service After/Wants front c--eso.service After/Wants front c--link-apiserver.service After/Wants front, Restart=always c--link-openbao.service After/Wants bao, Restart=always ``` - **Names.** `` is `SOCIALWEB_CLUSTERPROVIDER_UNIT_PREFIX`: `socialweb-` in production, the dev instance's HostProcess prefix in dev (`dev env` prints it). The key's dashes become `_` in the slice name, so a dash in a key adds no slice level. The HostProcess object of a member is `c--` (`@` becomes `-`), labelled `socialweb.computer/cluster` and `socialweb.computer/member`, with the finalizer `socialweb.computer/cluster-member`. - **Wants, not Requires.** kine and bao want the pg unit and are ordered after it; a postgres crash or restart does not stop them. They also gate on the Postgres object being Ready before they are placed (`procgroup.ErrPostgresUnavailable`). - **Bring-up.** `internal/clusterprovider/units.go` (`UnitsHost`) walks the steps in `procgroup.StackOrder` order and then the two links. For each member: run its Prepare, write its HostProcess, wait for Running, run its readiness gate against the member's log file, then record the gated invocation in the label `socialweb.computer/verified-invocation`. A member whose object is Running with that invocation is adopted and not touched. `/controlplane.json` records the active slot, the slice, the store unit and each member's unit and invocation. - **Readiness follows member health.** A pass first asks the host which members are unhealthy: a missing object, a phase other than Running, or a running invocation that differs from the verified one (the member was restarted by its unit's restart policy). A Ready cluster with an unhealthy member is written `Initializing` with Ready=False, reason `MembersUnhealthy`, naming the members. The next pass re-places the missing or lost members and re-runs the gate of every unverified one (a restarted bao is unsealed there), then the cluster is Ready again once the apiserver answers. A killed link is therefore never hidden behind a Ready cluster. - **Adoption.** On start the provider lists each cluster's member objects and logs them as adopted; it reaps nothing and spawns nothing over them. Leases are kept as long as the Cluster CR is. - **Teardown.** Spec.running false, or a deleted Cluster: every member object is deleted, the supervisor stops each unit, and the provider waits until every member is Stopped before it drops the finalizers. A deleted cluster then loses its Postgres object (its data directory is kept); an OFF cluster keeps it. - **Blue/green control-plane slots.** A cluster has slots a and b, each a `kine@` + `k3s@` pair on the cluster's one Postgres object, with its own leases (section 8) and its own k3s `--data-dir` (slot a: the cluster data dir; slot b: `/slots/b`); only the active slot runs. `controlplane.json` records `activeSlot`, the slots (k3s and kine executables, version, ports, data dir, hostname, units) and the front; `StepsFor` builds the active slot from it. `units roll cluster --cluster-state-dir [--k3s-path] [--kine-path]` writes `/roll-request.json`; the provider's roller (`clusterprovider.Roller`) takes it when the cluster is Ready, under the cluster's lock (Spawn and Reap take it too), and writes `/rolls/.json`, which the CLI prints. A roll: k3s minor in {active, active+1} and within three minors above every joined guest; the kubernetes EndpointSlice is only the front pair; idle first (waits while a non-kube-system pod is Running on a node, up to `--idle-bound`, default 30m; zero nodes swap at once); a minor bump takes a `pg_dump` of k3s and openbao to `/backups/`; the green slot's ports are re-leased if another process holds them (`Allocator.Relet`); kine@green is gated on its listener and "tables and indexes are up to date", k3s@green on its own port and the first kube-system Deployment and on the same server, client and request-header CAs as blue; `controlplane.json` is written with the green slot active BEFORE the front switches; then blue's k3s and kine objects are deleted, blue's front connections drained, and for 60 s the EndpointSlice, the kube-system lease holders and every Node's Ready transition are recorded in the report. Blue's data dir and its record stay for a same-minor rollback, which is the next `units roll cluster`. During a roll the cluster stays Ready: `Unhealthy` judges only the members `controlplane.json` records. Each k3s slot runs under its own hostname `swc--` through `internal ns-exec` (util-linux `unshare --user --map-current-user --keep-caps --uts --mount`, then sethostname, a bind of `/rancher` over `/etc/rancher`, ambient capabilities cleared, exec k3s; no resident code), on both backends, so wrangler's `kube-system/k3s` lease never has two holders under one name. The market kube-scheduler elects with a 4 s lease (renew 3 s, retry 1 s). Measured on a dev instance (k3s v1.36.4+k3s1, zero nodes): green ready 16-18 s after the request, authenticated `/readyz` through the front every 200 ms: 0 failures over both swaps; kube-controller-manager and k3s-cloud-controller-manager leases moved to green 22-26 s after blue stopped, wrangler's `k3s` lease 52 s (45 s lease, not released on stop). - **Secrets.** A link's key reaches dumbpipe through `spec.envFiles` of its HostProcess: a 0600 file under the cluster's data dir. The key is never written into a kcp object. - **The cluster's OpenBao.** The `bao` member is an OpenBaoServer object `c--bao`: unit `socialweb-c--bao.service` in the cluster's slice, After/Wants its postgres unit, postgres storage, the cluster's own vault TLS with client certificates optional. Its `spec.declared` is the cluster's whole vault state; the core syncs the vault to it. The cluster provider reaches the vault only as client `clusterprovider` (policy `clusterprovider`); `/openbao/access.json` names the address and the client certificate files, never a token. - **The core binary as member code.** The ccm and the cluster front run the core's resolved executable, `/bin//socialweb-computer` on an installed host, so a core upgrade changes no code inside a running member and the upgrade's collection keeps that directory while the member runs. - **Not yet.** Port attribution by cgroup (`internal/portscan`) is not needed while nothing reaps by port. ### The External Secrets controller runs beside the control plane The External Secrets Operator runs beside the control plane, not inside the tenant cluster. Installed into the tenant cluster as three Deployments (controller, cert-controller, webhook), it does not work: at zero nodes, the resting state, their pods are Pending so nothing syncs, and on the market substrate the store cannot reach the cluster vault at all. Running the operator in the process group avoids both. The decisions: - **Two members, one binary.** `procgroup.ComponentExternalSecretsWebhook` then `procgroup.ComponentExternalSecrets`, after the cloud controller manager in `StackOrder`. The binary is the upstream `ghcr.io/external-secrets/external-secrets:v2.11.0` image's own `/usr/bin/external-secrets`, embedded by digest as the release's `external-secrets` asset (`internal/versions/versions.json`); the webhook is its `webhook` subcommand. The cert-controller is not run: its only job was minting the webhook's serving certificate and injecting caBundles, which the provider now does itself. - **The webhook step installs the tenant half, before either process starts.** Its `Prepare` (`internal/clusterprovider/externalsecrets.go`) mints the pair's tenant credential, mints the webhook's serving pair, and calls `eso.InstallBeside` with the cluster's admin client against the pinned bundle (the `external-secrets-bundle` asset, sha256-checked at release build). `eso.SplitBundle` sorts every bundle object three ways and refuses a kind it does not place: KEPT (the 25 CRDs and the ClusterRoles `external-secrets-controller`, `-view`, `-edit`, `-servicebindings`), RE-REGISTERED (the two ValidatingWebhookConfigurations), RETIRED (everything else). Retired objects are deleted first, then the kept ones are server-side applied under `socialweb-addons`, then the CRDs are awaited `Established`, because the controller starts an informer per kind and an informer for an unserved kind fails its start. The step runs at every spawn, so the install is also the migration. - **Identity and RBAC.** The pair authenticates as `system:socialweb:external-secrets` (group of the same name), a client certificate minted from the cluster's CA exactly as the cloud controller manager's is, written to `/external-secrets.kubeconfig` (0600) and handed to both processes as `KUBECONFIG` and nothing else in their environment. A ClusterRoleBinding `socialweb-external-secrets` binds that GROUP to the upstream `external-secrets-controller` ClusterRole, so least privilege is upstream's own rule set rather than a fork of it. The upstream ClusterRoleBindings to the ServiceAccount `external-secrets` are retired, not kept: a binding to a ServiceAccount nothing runs as is one a tenant who creates that ServiceAccount inherits. The tenant grant's `socialweb-mint-identity` policy admits the beside username (its `__SOCIALWEB_ESO_USER__` placeholder) where it used to admit the in-tenant ServiceAccount. - **The webhook is registered by URL on loopback.** Each webhook's `clientConfig` becomes `url: https://127.0.0.1:` with `caBundle` the CA the provider mints per cluster (`eso.EnsureWebhookTLS`: CA 10 years, key under `/external-secrets/webhook-ca.key`, outside the cert dir the webhook reads; leaf 2 years, SANs `localhost` and `127.0.0.1`, renewed at a spawn inside its last year). A registration still naming a Service is deleted before the URL apply, because the field another manager owns survives a server-side apply and a webhook carrying both is refused. MEASURED: the k3s apiserver (`--egress-selector-mode cluster`) dials the loopback URL directly at zero nodes. The registrations stay `failurePolicy: Ignore`: the pair and the cloud controller manager are member units with `Restart=on-failure`, so a dead pair comes back, and a dead webhook under `Fail` would still refuse the provider's own store writes and deletes until then. - **Ports and singletons.** The upstream defaults (`:8080` metrics, `:8082` controller health, `:8081` webhook health, `10250` webhook) are one set for the host, so each cluster leases three (section 8): the controller's `--live-addr` and the webhook's `--healthz-addr` bind `127.0.0.1`; metrics are off (`--metrics-addr=0`: nothing scrapes them, and a leased port nobody reads is a lease held for nothing); leader election is off, because the process group is already the singleton per cluster. The webhook's `--port` has no host flag and binds every interface (measured: `Serving webhook server` with an empty host); no public edge entry point forwards to it, and the listener answers only admission reviews under a CA no client but the tenant apiserver trusts. - **Readiness.** The webhook's gate waits for its `/readyz` (which itself checks the serving pair) and then dials the serving port verifying the minted CA for `127.0.0.1`, which is what the apiserver does. The controller's `/readyz` is a ping that answers before any cache syncs, so its gate also waits for the `"Starting workers"` line of the `externalsecret` controller. - **The store dials the vault on loopback, on every substrate.** `esovault.ApplyVaultStore` writes `server: https://127.0.0.1:` and the vault CA; no substrate branch keeps the store off the market substrate. - **Clusters leased before the pair.** `StackConfig.WithoutUnleased` drops the pair from the restart reap's argv derivation when a cluster's lease-file entry holds none of its ports; the bring-up that follows leases them, because the materialiser allocates every purpose at spawn. Embedded-postgres is a special case: it is a Go **library** that runs the server as a child of whoever links it, so there is no argv for a step and `processrunner.Start` refuses the empty argv such a step would have. `procgroup.Options.PostgresUp func() bool` exists for this: the component holding the library handle answers for it, and a **nil predicate is a refusal, not a skip**. ### Control-plane continuity across upgrades and restarts The gateway's own state is kcp objects plus host records; the processes that serve a tenant are units that outlive the core. Each kind of change is continuous by one mechanism, and none of them restarts a VM or rents one: - **Core binary** (`upgrade --binary --sha256`, `rollback`, a crash or `kill -9`): only the core unit restarts. kcp's store, kine, slots and front are bootstrap units, every cluster member is a unit of its slice, so they keep running; the new core adopts them by name and invocation. The cluster provider adopts every live claim and claims again a joined Node whose claim was lost; the web app resumes in-flight claims from the root vault and re-reads the joined ones. Nothing is written to a joined claim, no RFP is posted and no vm.delete is sent. - **kcp** (`units roll kcp`): a new slot on the same store behind the kcp front, the controller lease handed over. - **k3s and kine** (`units roll cluster`): a green slot behind the cluster's apiserver front, idle first. - **The cluster links** (`units roll link`): a second listener with the same iroh key, then the old one stops. - **Postgres, OpenBao, ESO, the CCM**: unit restarts; a restarted OpenBao is unsealed on its new invocation and synced to its declaration. The members (the gateway XRPC and the cluster provider, in the core) reach kcp as kcp-admin through a token file (`/members.token`), because every kcp start issues a new kcp-admin token. The roll rewrites the file from the new slot's `admin.kubeconfig` right after the front flips and before the old slot stops; a member's client re-reads it on its first 401. The core checks the file against the active slot every 5 s as a fallback and warns when it had to rewrite it. Readings on hat-8, 2026-10-12 UTC: a staged release gateway (`install gateway --root`, systemd-user, the core's PATH `/usr/bin:/bin`) with tenant cluster `gc` and one market node rented through the RFP flow from a market network's `spec.kcp` bidder (guest a VM unit of test-0001), `/readyz` polled through the cluster's front every second. `upgrade` 15:44:18: core members up 15:44:33, the VM unit's InvocationID, the claim's uid and the node's Ready transition (15:25:17) unchanged. `units roll kcp` a to b 15:45:09-15:45:22: front flipped 15:45:15.86, lease handed over 15:45:21.6, probe Ready 0.53 s after; it left the members Unauthorized (their kcp-admin token was the old slot's), fixed by the token file; after the fix, roll b to a 16:00:51-16:01:02 left the cluster provider Unauthorized for 10 s (6 passes) until the client re-read the file. `units roll cluster` a to b 16:02:24-16:04:56: the swap waited the 60 s idle bound (the workload pod), green Ready 80 s after start, front flipped 16:03:48.905, blue stopped 0.79 s later with 303 open connections drained, the EndpointSlice never lost the front pair, 1 failed `/readyz` poll (16:03:49.288), no Ready transition. `units roll link` apiserver a to b 16:05:14-16:05:29: green Ready 10 s after start, the old listener stopped 0.2 s later, the node's lease renewed through the new listener. A `rollback` to a build without the token file restarted the core into a loop (bootstrap Unauthorized) while every member unit kept serving and the node stayed Ready. Between two builds that both carry it: `upgrade` 16:22:45 (members up 16:24:10), `rollback` 16:24:10 (members up 16:24:14 on the previous binary), `kill -9` of the core 16:24:22.1 (systemd restarted it 16:24:27, members up 16:24:29.2). Over the whole run (15:25:22-16:25, 3392 polls): 1 failed `/readyz` (the cluster roll above), no Ready transition of the market node, the same claim uid and receipt, the guest's VM unit kept its InvocationID and pid, no RFP and no vm.delete after the first rental. Stated limits: a user-manager restart, linger removal or a reboot removes every transient unit, so every cluster cold-starts; the NotReady clock counts from when the path is back, so a node whose path returns is kept, but it is not zero-downtime. The core's own listeners (the gateway XRPC, its TLS front, the web app) are down while the core restarts. ### The cluster links Each cluster has two links, one to the k3s apiserver and one to the cluster vault. - **What runs.** Per cluster, `dumbpipe listen-tcp --host 127.0.0.1: --ipv4-addr 0.0.0.0:0` twice (`clusterlink.Argv`): target = the cluster's apiserver front port for `apiserver-link`, the vault's held port for `openbao-link`. dumbpipe v0.39.0, the same release the guests install (the `dumbpipe` asset in `internal/versions/versions.json`). - **On or off.** On when `SOCIALWEB_CLUSTERPROVIDER_DUMBPIPE` names the binary; the gateway sets it to the extracted `dumbpipe` asset (`internal/stack/gatewaykcp`). Unset leaves every cluster without links, and the provider logs `cluster links on=false` once at start. Links need the vault in the Stack (`ErrNoLinkSeam`) and a recorder (`ErrNoLinkRecorder`). - **The key.** One ed25519 key per link, generated once and kept in the cluster's OWN vault at KV v2 mount `socialweb-links`, path `` (`apiserver`, `openbao`), fields `secret` (64 hex, the 32-byte seed -- exactly the form dumbpipe reads from `IROH_SECRET`; measured: dumbpipe prints `using secret key ` when it generates one, and the ticket it prints for a given seed names `ed25519.NewKeyFromSeed(seed).Public()`), `endpointId`, `did`. The mount and the provider's policy on it are part of the cluster's OpenBaoServer declaration (`internal/clusterprovider/vaultdeclared.go`). The first write is check-and-set 0, so two writers can never both mint a key; a stored record that does not parse, or whose `endpointId` disagrees with its secret, is an error and is NEVER regenerated over (a new key is a new endpoint id and strands every node holding the old ticket). The key is read in the step's `procgroup.Step.Prepare`, which runs once the steps before it passed their gates, so the vault is up; the secret reaches dumbpipe only through a 0600 env file named in the HostProcess `spec.envFiles`. - **Identity encodings.** `internal/didkey`: endpoint id = lowercase hex of the 32-byte public key (what iroh prints); did:key = `did:key:z` + base58btc(0xed 0x01 + key). Both matched the did:key spec vector and a dumbpipe-printed ticket when they were built. - **The ticket.** dumbpipe prints `dumbpipe connect-tcp `: `endpoint` + lowercase unpadded base32 of a postcard-encoded `EndpointTicket` (byte 0 the variant 0x00, bytes 1..32 the endpoint id, then the relay URL and direct addresses). `clusterlink.ParseTicket` reads the id back, and the gate opens only when the printed ticket names the vault-held key. The printed ticket's BYTES change per bring-up (the relay chosen and the addresses are in it), so it is NOT what is published: `status.ticket` is `clusterlink.StableTicket(key)`, the id-only ticket (variant 0x00, the id, zero addresses) -- byte for byte what `IROH_SECRET= dumbpipe generate-ticket` prints -- so it is a function of the key alone and never changes while the key lives in the vault. A node secret may hold it written once. dumbpipe resolves the id's current relay and addresses by discovery; measured live against a real dumbpipe (connect-tcp with the published ticket echoes before and after a restart, and the ticket is byte-identical). Decided by Claude, user may veto: the cost is that a dialer with no discovery (no DNS to the n0 discovery service) cannot use the ticket; guests reach the internet anyway. - **The link port.** Not held: dumbpipe binds UDP port 0, and a node dials the link by its ticket, so the port number is in no ticket that is published (section 8). The `apiserver-link` and `openbao-link` port purposes are retired (`portallocator` `retiredPurposes`). - **Status.** The link's readiness gate writes the `ClusterLink` `Listening` (spec cluster/service/targetPort, status endpointId/did/ticket) with retries, so a kcp outage never reaps a cluster. A dead link is a member unit with Restart=always; while it is down or not yet re-verified the cluster is not Ready (section "The per-cluster slice of member units"). - **Teardown.** A deleted cluster's two ClusterLinks are deleted after its group is reaped (both the all-clusters path and a pass that removes the finalizer). An OFF cluster keeps them: it comes back as the same endpoints. The keys stay in the vault's data and go with it. - **Replacement without a gap.** Each link has two slots; `controlplane.json` `links.` names the slot (`a` or `b`) and the dumbpipe it runs, and `StepsFor` builds the link from it (member `link-` for slot a, `link-@b` for slot b). `units roll link --service apiserver|openbao [--dumbpipe-path]` is carried out by the provider's roller (`Roller.RollLink`): the new listener starts in the other slot with the same `IROH_SECRET` env file, its gate waits for the ticket it prints naming the vault's key (and records the ClusterLink, whose `status.ticket` is `StableTicket(key)` and so byte-identical), the record is switched, then the old listener's object is deleted. A link is never re-pointed for a k3s swap: its target is the front. Measured with the pinned dumbpipe v0.39.0 (iroh 1.0.0, sha256 81010d0a... of the binary in the pinned archive; n0's default relays; client and listeners on one host, so the path was direct): a connect-tcp client dialling the endpoint-id-only ticket keeps its dials on the OLDER listener while both run (the newer process is not promoted for an established client), and its next dial after the older one is stopped reaches the newer one with no dial lost (200 ms polls); a client that first dials while both run reaches the newer one only (the address record is last writer wins; the newer listener's first answer came 5.2 s after it started). So the replacement keeps start-then-stop; no fallback to stop-then-start was needed. On a dev instance a replacement (old object deleted 0.17 s after the new one was gated, 8 s after the request) and a SIGKILL of the link (Restart=always, a new process 0.19 s later, cluster Initializing for 4.2 s while the restarted invocation was re-verified) each lost 0 of ~260 kubectl polls through a connect-tcp client on the published ticket. - **Stated limits.** A ticket carries no direct address, so a dialer reaches the link only through discovery and the relay; the IPv4 and IPv6 sockets both take a random port. ### The vault's config is rendered, not shipped `internal/openbaoctl` renders the config from the cluster's OpenBaoServer spec (`rootvault.ServerConfigWith`: the same listener and postgres storage, client certificates optional) into its own state dir, mode 0600, rewritten only when it changed, before it ensures the HostProcess that runs `bao server -config=...`. `clusterstack.Spec.OpenBaoServerConfig` states the same rules for a cluster spec. Each line is measured on v2.6.2: ```hcl ui = false disable_clustering = true api_addr = "https://127.0.0.1:" listener "tcp" { address = "127.0.0.1:" tls_cert_file = "/server/tls/openbao.crt" tls_key_file = "/server/tls/openbao.key" } storage "postgresql" { connection_url = "postgres://@localhost/openbao?host=&sslmode=disable" } ``` The listener is loopback on the cluster's leased port, and the port is NEVER defaulted: bao's own 8200 is one port for the whole host, which is the store's byte-identical-DSN defect class. The listener is TLS against the cluster's own certificate and key, and a spec with either absent is refused: a listener with no certificate is an ACME/plaintext listener whose `GET /v1/sys/health` never answers the readiness gate. `disable_clustering = true` removes the cluster listener (default host-global 8201) and makes "one vault per cluster, no HA" a property of the file. The file carries **no `disable_mlock`**: 2.6.2 logs `unknown or unsupported field disable_mlock` and starts anyway, because it does not mlock at all. The config file must be inside the cluster's data dir, and the certificate and key paths are per-cluster: a shared config is written by whichever cluster reconciles last, so its port becomes that cluster's. Storage is the **postgres backend**, and each word of that stanza was read off the bundled `bao` rather than remembered: `storage "postgres"` answers `unknown storage type postgres`, `storage "postgresql"` with the cluster's DSN answers, and the health body's own `Storage Type` field then reads `postgresql`. The key is `connection_url`. The URL is ALWAYS this cluster's own DSN -- the unix socket directory of the cluster's Postgres object (`pgunit.DSN`) -- and NEVER a defaulted one, because two clusters' DSNs are otherwise byte-identical. It names the vault's OWN database, `openbao` (`clusterstack.OpenBaoDatabase`); kine's objects go to the database `k3s`, so the vault's tables and the objects kine writes never share one database. `OpenBaoServerConfig` refuses a DSN that is empty, names no unix socket directory, carries no user, is not a postgres URL, or names any database but `openbao`; `openbaoctl` refuses a `spec.storage.postgres` without an absolute socketDir, a user and a database. The database itself is created empty (`internal/pgunit`) before the vault starts (bao creates its own tables); measured on 2.6.2, the first vault write creates `openbao_kv_store` in `openbao` and nothing in `k3s`. The file backend is not used: 2.6.2 logs it as deprecated (`storage.file: the file physical backend is deprecated; use bao operator migrate ... by v2.7.0`) and section 6 decides against it. ### A child's output is drained, never left in a pipe Every cluster member is a HostProcess unit. Its stdout and stderr go to one log file, `procgroup.Spec.Log`, set as the HostProcess `spec.output` (`internal/clusterprovider/units.go`): systemd-user appends to it (`StandardOutput=append:`), devsup opens it itself. No member writes into a pipe, so a member may write without bound and still reach its exit. The rule dates from the in-process child runner that came before the HostProcess units, where a pipe nobody read made the child block inside `write(2)` the moment the kernel's 64 KiB buffer filled. `k3s server` and `k3s agent` both emit far more than that during startup, so the block lands mid-boot and the readiness gate reports `k3s server at https://...: no answer` - a readiness fault over a child that was alive, healthy, and had nowhere to put its log. The failure is also load-dependent and therefore intermittent, which is the worst property a startup fault can have. Measured: a child writing 500 KB never finished under a pipe; with a writer it finishes immediately. The log is bounded: the HostProcess output cap is `MemberOutputCap` (64 MiB, enforced by `internal/hostprocess` through `internal/logcap`), and a new HostProcess object for the member moves the old log to `.prev`. **The bound is the point** - a cluster's k3s logs for the life of the cluster, and a log that grows with it is a leak on the gateway host. A readiness gate reads only the last 1 MiB of the file (`unitMember.Output`, `memberOutputTail`). The tail is also what a positive assertion about a node reads: `portallocator.AssertAgentDialledOurPort` needs the agent's own dial line, and a tail that no longer holds it fails loudly rather than passing on an absence. ### A market bidder is an application instance of one installed account A bidder host is set up by the one root step `socialweb-computer install bidder`, which creates the lingering `swc-bidder` account, its subuid/subgid range for per-VM user namespaces and the kvm group membership, and ends by running `check-host -role bidder` as the account (`docs/host-contract.md`). Each bidder is then `socialweb-computer run bidder`: it applies its MarketBidder root object with `--profile prod-qr|market-network|dev-local` into the handle's own kcp, so every handle is its own application instance (section 0). Its guests are VM units of `internal/vmruntime` under the account's user manager, born only from the cloud-init `user_data` of the RFP flow. There is no systemd system template unit per handle, no docker group account and no runner containers. A bidder's policy (MarketBidder `spec.policy`, `run bidder --bidder-policy`) defaults to `bidder-tangled-vouch` on every profile (user, 2026-10-30): the bidder bids only for a requester its operator vouches (`sh.tangled.graph.vouch`). `open` is insecure and runs only when a spec or flag names it; the market network profile names it for its own dev bidders (`dev up` default values). `install bidder` writes `bidder-policy` to config.json, so every prod bidder runs `bidder-tangled-vouch` across `upgrade` and restarts. ### Host processes and the VM lifecycle This subsection describes the design for host processes and the VM lifecycle. It is built (`internal/hostprocess`, `internal/hostsvc`, `internal/devsup`, `internal/vmruntime`, `internal/install`), except where a line says it is not. Lines marked "user" are the user's decisions; the rest are Claude's, open to the user's veto in the plan files. **Invariant: no socialweb-computer code in a running VM's process tree (user).** The service manager launches the VMM; our code only does one-shot prep before the unit starts, and the tiny `socialweb-vmexec` helper creates the tap and `execve`s firecracker, so after start a VM unit's cgroup holds exactly `pasta` and `firecracker`. There is no `socialweb-nodeboot` and no resident namespace verb; the firecracker provider's only path is the VM unit of `internal/vmruntime`. The instrument that proves it (every process in a VM unit's cgroup must hash to a pinned asset) exists as `socialweb-computer units audit` (`internal/hostsvc/treeaudit`), which exits 1 naming the unit and the executable that is not pinned. Running it after every boot, restart and upgrade is not built: only the verb runs it today. **Scope and platforms (user).** Headless servers; VMs survive a crash or upgrade of the core binary and a logout; a reboot is Lost (transient units live on tmpfs). This runs on Linux only; the cross-OS seams are built now: Go interfaces (`hostsvc`, `vmruntime`, `guestnet`), GOOS-tagged files, per-OS embedded assets compiled in only for the target OS, and a check.sh gate that builds and vets darwin and windows on every run. Windows (Hyper-V HCS, SCM) and macOS (vfkit, launchd) backends follow. The live proof of this design ran on hat-8 with systemd 259.9 and firecracker v1.13.1. **Where processes live (user).** The gateway and the bidders each run in the USER manager of a dedicated lingering account (`socialweb`; one shared `swc-bidder` account for every bidder handle, units namespaced by handle). Long-lived children are transient SERVICES (never scopes) started over `$XDG_RUNTIME_DIR/systemd/private`: a VM per `socialweb-vm--.service`, the vaults (`socialweb-bao-root`, `socialweb-bao-`), each cluster's postgres (`socialweb-pg-`) and its member units in `socialweb-cluster-.slice` (kine and bao `After=` and `Wants=` the postgres unit, not `Requires=`), and kcp: on the gateway as alternating blue/green slot units (`socialweb-kcp-gw@a|b`) of the unpatched kcp asset built here from kcp-dev/kcp main (commit 74e9fb9c, versions.json assets.kcp, fetch kind gosrc; the user decided 2026-10-22, plan 0026 W0-RUNTIME, to run kcp off main), each on its own kine over a dedicated postgres unit on a unix socket (`socialweb-kcp-gw-pg`, not a CRD object), behind our own L4 front (`socialweb-kcp-gw-front`, 127.0.0.1:6443); on a bidder as ONE slot per handle (`socialweb-kcp-@a`) on one kine over sqlite behind a port-0 front, upgraded by a short restart that VMs never notice, after which the controller re-adopts them. Postgres and OpenBaoServer stay CRDs (user). Components run inside the core process and stop with it. The core unit uses the default `KillMode=control-group`; transient units are separate cgroups, so no KillMode reaches them (proved: a unit survives restart, kill -9 and stop of its starter). Only cpu, memory and pids delegation is relied on (user); `AllowedCPUs` and `IOWeight` are not enforced in a user manager and are not used; VM units set `MemorySwapMax=0` with `MemoryMax` (otherwise the guest swaps) and live in their own slice, outside the `app.slice` Fedora's systemd-oomd watches. Minimum systemd 253 with each property probed by `check-host` (user). **One explicit root step (user).** Production setup is `socialweb-computer install bidder|gateway`, idempotent, with `uninstall`: it creates the account (never created on demand anywhere else), for bidders one subuid/subgid range and kvm membership, the `user@` cpu delegation drop-in, the AppArmor profile for the embedded pasta where AppArmor restricts user namespaces, the gateway's `/etc/rancher/node/password`, the content-addressed binary, `socialweb.slice`, the core unit file and the bootstrap-layer unit files (kcp's store, kine, slot(s) and front), then linger (after the group and the drop-in, so the user manager starts with them) and `host-backend: systemd-user` in the account's config. Every other unit (the vaults, cluster slices and members, VM units) is a transient unit placed at runtime from a HostProcess object and adopted by name. **The gateway's install and upgrade.** `install gateway` writes the user unit `socialweb-computer-gateway.service` (`ExecStart=/bin/current/socialweb-computer serve gateway kcp`, `Restart=always`, default `KillMode=control-group`, optional `EnvironmentFile=-/gateway.env`) and a `config.json` that `serve gateway kcp` reads with nothing added by hand: the state dir, `host-backend: systemd-user`, the asset cache, and the two front addresses caddy proxies to (kcp front 127.0.0.1:6443, TLS front 127.0.0.1:8443; the web app front's 127.0.0.1:8788 is the prod Gateway spec's default), plus `unit-prefix` and `gateway-spec-file` when install is given them. A release gateway takes the clusterprovider's external-secrets binary, its bundle (the embedded asset `external-secrets-bundle`, the release's `external-secrets.yaml`), dumbpipe, and every tenant cluster's k3s, kine, bao and pg_dump from its own assets, each by the absolute path under `runtime//`; a release clusterprovider never looks in PATH (a dev build names k3s with `k3s-path`). `upgrade --binary --sha256` checks the digest, copies the binary to `bin//`, runs its self-check, retargets `bin/current` (the old target becomes `bin/previous`) and restarts the core unit; every installed and transient unit is adopted, only the core gets a new InvocationID. `rollback` swaps `current` and `previous` and restarts. After either, `bin//` directories are collected except `current`, `previous`, any directory a running process executes from (`/proc//exe`) and any a unit file names (the account's unit dir and the user manager's runtime and transient unit dirs). Member code that is the core binary (the ccm and the cluster front) runs the core's resolved executable, `bin//socialweb-computer`, never `bin/current`, so a member keeps its code across an upgrade and its directory is kept while it runs. `--root ` stages an install for a proof run: the host files (`/etc`, the linger file, the install record) go under ``, the state dir is used as given, the account is the invoking user and the units run in that user's own manager; `--unit-prefix` names every unit of the installed application, the core included (`computer-gateway.service`). **Each application instance's kcp as its service manager; dev run mode (user).** Long-lived host processes are `HostProcess` objects in their application instance's own kcp (section 0) reconciled by a supervisor inside the core (`internal/hostprocess`) onto a placement backend: `systemd-user` (transient units in the account's user manager, production installs only) or `proc` (dev: a plain supervised process tree under a per-instance `devsup` that is every unit's parent and child subreaper). Postgres, OpenBaoServer, each cluster member and each ComputeVM own HostProcess children. Below kcp sits the BOOTSTRAP LAYER - kcp's store, kine, slot(s), front and the core - which kcp cannot supervise: installed systemd units in production, devsup units in dev, mirrored into kcp read-only once it answers (the static-pod/mirror-pod pattern). `host-backend` is required with no default. `socialweb-computer dev up|down|status` runs one instance per worktree with no root, no install and no systemd: state under `$XDG_STATE_HOME/socialweb-computer/dev//`, its own kcp on port 0 (gateway store sqlite for quick runs, postgres for the market network and prod-like profiles), leases keyed by instance, the k3s node password bind-mounted in a per-instance user namespace, AppArmor reported not fixed, one uid for every role. Instances nest as kcp workspaces (one kcp per application instance; a parent instance that holds its children's processes later; applications never share a kcp, section 0). Guests in dev still come up only through the RFP flow. **Adoption is a tri-state verdict (user).** A restarted core adopts units by name, never re-creates them. Identity is read from systemd and the kernel (unit, InvocationID, VMM pid and start ticks through a pidfd, `GetUnitByPIDFD`, firecracker's own `--id` from `GET /`), never from a file the child wrote. Only bus dial, auth or timeout errors and an unreachable API socket are Unknown (retried, never Lost). A changed boot id is Lost. A definite mismatch on an active unit is Lost and the unit is stopped at once. **Guest side (user).** Every guest gets the identical address plan inside its own pasta namespace and dials a constant host address, so there is no network-index allocator and user_data is final at create. Guest images (kernel + rootfs) are built per hypervisor and arch by a build-host script that runs on any Linux host (CI shells out to the same script), pinned by sha256 and fetched; bidders need no root, chroot, apt or Go toolchain. The seed is a separate pure-Go NoCloud cidata drive, the rootfs is copied as-is and the guest grows its own root, so no host runs debugfs, e2fsck or resize2fs. Guest SSH host keys are generated in the guest and reported by it, with proof of possession, in a bidder-signed `vm.hostKeys` record replicating `vm.onNetwork`; every ssh to a guest verifies them with `StrictHostKeyChecking=yes`, and no host scans a guest. The guest console goes to a capped file in the VM's work dir and is deleted with it; tenant data never enters the host journal. Stop is graceful first: `SendCtrlAltDel` over the API socket, a grace window measured live through the RFP flow, then `KillSignal=SIGKILL` (firecracker, PID 1 of pasta's PID namespace, ignores SIGTERM); arm64 goes straight to SIGKILL. The Bidder component runs inside the bidder's core, which shares a uid with the user manager it places units in; the VM units are what is isolated (below). **VM isolation without on-demand accounts (user).** The bidder account is shared by every handle, but each VM unit runs pasta and firecracker in its own user namespace mapped to a distinct 65536-id sub-range of the account's reserved subuid/subgid range (the rootless-podman pattern through `newuidmap`), with access to exactly its own work files by ACL, so a VMM escape lands in an unprivileged uid that owns no bidder file. **Stated limits.** Handles on one bidder host share a uid for their cores, and nothing but that uid separates a bidder's core from its user manager (the deno permission sandbox of the TypeScript bidder before plan 0026 is gone with it). Restarting `user@`, removing linger or `loginctl terminate-user` stops every unit and is treated as a reboot. **Control-plane continuity across upgrades (user).** There is no cutover and no container path: the binary is deployed fresh. No backwards compatibility, migration, rollback or prod-downtime handling is planned before the finished solution exists, and the gateway runs no containers. Continuity is for upgrades of the finished system: a per-cluster TCP front owns the existing apiserver port (guest tickets, node secrets, kubeconfigs and `status.URL` never change); a k3s/kine upgrade starts a new slot beside the old on the same postgres, the front swaps, the old stops, idle clusters first; the market kube-scheduler elects a leader with a short lease and each slot has its own hostname so k3s's own leader lease never has two holders; a cluster link keeps its iroh key across restarts; a restarted gateway (the NodeClaim controller, `internal/controllers/nodeclaim`) resumes Bidding and Provisioning claims from a per-claim secrets-server key in the cluster OpenBao; and NODE-LOSS counts only time when our own path was up. Design and measurements: `docs/research/2026-10-11-vm-lifecycle/control-plane-handover.md`. The gateway's kcp upgrades follow the same pattern behind our own front, not caddy (a bidder's single kcp slot is restarted instead): slots run `--enable-leader-election` and `--shard-base-url` = the front, a new slot gets the five shared root files and starts beside the old on the same kine -> postgres store, the front flips after `/readyz` on the new slot's own port, the old slot exits, and the Lease `kube-system/kcp-controllers` is deleted so the new slot leads at once (61 s gap without it, measured); a pg_dump precedes every roll, and kcp release pairs whose storage versions or bootstrap schemas differ take a planned short outage instead (`kcp-blue-green.md`). caddy stays public TLS only, is never reconfigured by a swap, and any caddy config we render pins `install_trust false`. ### compute-provider-kcp: a ComputeVM per guest, reconciled by a Go controller This subsection describes the compute-provider-kcp design. It is built in Go: the Bidder component (`internal/services/bidder`) writes a ComputeVM per guest (`computevm.ComputeVMFor`), the ComputeVM controller (`internal/controllers/computevm`) and the FirecrackerVM controller (`internal/controllers/firecrackervm`) reconcile it over `internal/computevm`, `internal/compute/firecracker` and `internal/vmruntime`. The host mechanics it relies on are the subsection above. The TypeScript provider that came before plan 0026 is gone. - **The bidder is an application instance with its own kcp (user).** A bidder handle = one kcp (one slot on kine-sqlite behind a port-0 front) + the bidder manifest set + a profile, rooted at a MarketBidder object. Its guests are ComputeVM objects in that kcp; two handles on one host never share a kcp. Requesters and the gateway never read a ComputeVM: what crosses applications is a market record (RFP, bid, accept, receipt, onNetwork, vm.hostKeys, vm.terminated). A bidder kcp upgrade is a short restart of its one slot that VMs never notice; the controller re-adopts every VM afterwards. - **ComputeVM plus a Go controller (user).** The Bidder component writes a ComputeVM (`computevm.ComputeVMFor`, from `internal/market/bidder` through `internal/services/bidder`); the ComputeVM controller prepares, places, adopts and releases it through its backend, a FirecrackerVM named in `spec.backendRef`. Spec: backendRef, vm, owner.handle, userData (a SecretRef into the bidder's vault at `guests//user-data`), requester, contract, accept, image, stopGraceSeconds, guestLogin, release; no network index. Status is OS-neutral (phase, lostReason, runtime {backend as a free string, ref, instance, pid, startTime, hostBootID, vmmID}, network, image, conditions, market) so the Windows and macOS backends need no schema revision. - **What the bidder reads back.** The bidder keeps no guest state of its own: it reads its ComputeVMs and Bid objects from its kcp, and a restarted bidder carries on from them. A release is the ComputeVM's `spec.release`, written by the bidder from the requester's `vm.delete`. A ComputeVM that ends is reported by the ComputeVM controller as a `vm.terminated` StateSync (`computevm.EnsureTerminationSync`, reason `computevm.GoneReason`: HostRebooted -> host_reboot, OOMKilled -> oom, IdentityMismatch -> identity_mismatch, GuestPoweroff -> guest_poweroff, every other -> vmm_exited; `deleted_on_request` for a release). The guest-facing endpoints (the allow-list proxy to the bidder's OpenBao, `POST /v1/events`, `GET /v1/downloads` and the guest event subscription) are the GuestControlPlane component (`internal/services/guestcontrolplane`). - **A process per VM under a generic runner was rejected, on evidence.** Before plan 0026 a runner of the deno runtime SIGKILLed a deleted pod's process group with no release step, and its 5-minute pod timeout killed the market network's bidder mid-contract (run 4), while a VM lives as long as its contract and needs typed guest fields and `/dev/kvm`. A VM is a ComputeVM with its own controller. - **ComputeVM is the source of truth.** A bidder that held guests and the network index in memory lost its receipts and leaked guests on a restart (the TypeScript firecracker provider before plan 0026). Now the contract is ComputeVM `spec.contract`, and a restarted bidder re-populates from its ComputeVMs plus live VMM state. - **No socialweb-computer code in a running VM's process tree (user).** Prep is one-shot in the controller; each ComputeVM owns exactly one HostProcess (`computevm.ChildName`, `vm-`) placed as a transient unit `socialweb-vm--` whose tree after start is pasta -> firecracker only: on `systemd-user` in production, in the user manager of the one shared lingering `swc-bidder` account, surviving core crash, upgrade and logout; on `proc` in dev, as a unit of the per-instance `devsup`, the parent and child subreaper of every unit (`internal/devsup/server_linux.go`). devsup in a dev tree is the dev backend by design, never production. Both backends satisfy `hostsvc.Manager` (`internal/hostsvc/hostsvc.go`, the `internal/contract/host` Units interface). Reboot = Lost. - **Tri-state verdict: Running | Lost | Unknown (user).** `hostsvc.Verdict` and `LostReason` (`internal/hostsvc/hostsvc.go`, from `internal/contract/host`). Bus dial, auth or timeout and an unreachable API socket are Unknown: a condition, phase unchanged, retried, never Lost. A changed boot id is Lost(HostRebooted); a missing unit Lost(UnitGone); a failed or inactive unit Lost(UnitFailed | OOMKilled | VMMExited); an identity mismatch on an active unit (InvocationID, pidfd + start ticks, `GetUnitByPIDFD`, firecracker `GET /` id) is Lost(IdentityMismatch) and the unit is stopped at once; a unit placed by another backend is Lost, never adopted. Why: a false Lost costs a tenant a VM. A Lost guest is never booted again; the requester replaces it through a new RFP (user). - **Stop is CtrlAltDel, then SIGKILL (user).** firecracker is PID 1 of pasta's PID namespace and ignores SIGTERM, so a plain stop hangs 45 s. Releasing sends `PUT /actions SendCtrlAltDel`, waits `stopGraceSeconds`, then the unit stops with `KillSignal=SIGKILL`; arm64 goes straight to SIGKILL. Measured live through the RFP flow: 0.21 s for an idle guest. The finalizer is dropped only after the work dir and API socket are gone (32 dirs and 63 GB leaked on hat-8 without it). - **Per-VM subuid user namespace (user).** `install bidder` reserves one subuid/subgid range for the shared account; each VM unit runs pasta and firecracker in its own user namespace mapped to a distinct 65536-id sub-range through `newuidmap`, so a VMM escape lands in a uid that owns nothing. No Unix account is created on demand. - **cidata seed (user).** The seed is a separate pure-Go NoCloud cidata drive; the rootfs is reflink-cloned as-is and truncated to the contract's disk; the guest grows its own root. No host runs debugfs, e2fsck or resize2fs. - **Identical guest network per pasta namespace (user).** Every guest gets the same address plan inside its own pasta user+net namespace and dials a constant host address (pasta's gateway), so the network-index allocator of the TypeScript firecracker provider before plan 0026 is gone and user_data is final at create. - **Guest images from a build-host script (user).** Kernel + rootfs per hypervisor x arch, built by a script on any Linux host (CI shells out to it), pinned by sha256 and fetched; a bidder needs no root, chroot, apt or Go toolchain. - **iroh is the default and required guest transport (user; relaxed 2026-10-24).** The bidder installs no guest package registry, websocat path, tunnel or fedproxy module. The requester may ask, per contract, for the opt-in opentunnel module (`spec.tunnels`, plan 0032); its user_data installs it, and iroh stays on every guest. Guest SSH host keys are guest-reported in `vm.hostKeys` and verified with `StrictHostKeyChecking=yes`. - **vm.terminated (user).** The bidder reports why and when a guest ended in a signed `com.publicdomainrelay.temp.compute.events.vm.terminated` inside the existing market.event envelope (5.6 s from guest poweroff to the requester's session end, measured live). The ComputeVM controller emits it from every Lost verdict and every release through a termination StateSync (`internal/computevm/termination.go`, reasons as above); an ended ComputeVM answers a late `vm.delete` 200. vm.terminated and vm.hostKeys are the only protocol additions, with the optional `tag` of vm.onNetwork (plan 0032). ## 6. CRDs and the API surface Every CRD here is INTERNAL. No CRD is user-facing: the user's surface is the mounted Kubernetes API (standard kinds), and cluster lifecycle is an **XRPC call**. None of these schemas carries a compatibility promise. | CRD | For | Writer | Tenant-visible | |---|---|---|---| | **Cluster** | the mount object - `is-mount` annotation, `status.URL`/`phase`, `Draining`, conditions, `status.nodeExpiry` | cluster provider, from an XRPC lifecycle call - never a kubectl apply | no | | **Vault** | OpenBao, per ON cluster over its own embedded-postgres | system | no | | **Postgres** | embedded-postgres - the vault's store and the one per-cluster local state | system | no | | **Kine** | not a CRD today: kine is a member of each cluster's process group over that cluster's own postgres (section 4; the "kine plus t4 to central S3" row was the unbuilt rustfs design) | system | no | | **Node** | one node. The only bound on leaks | node provider | no | | **NodeClaim** | one node RENTED FROM THE COMPUTE MARKET for a cluster, as that cluster's owner. Carries paths (`tokenFile`, `nodePasswordFile`, `serverCAFile`) and never a secret; the NodeClaim controller is the only writer of its `status` | cluster provider, spec and metadata only | no | | **ClusterLink** | one iroh listener in front of a cluster's apiserver or vault, named `-apiserver` / `-openbao` in the cluster's home workspace. `spec` {cluster (the key), service, targetPort}; `status` {phase `Pending`/`Listening`/`Failed`, endpointId (64 hex), did (did:key), ticket (id-only, a function of the key: byte-stable), message}. Never carries the secret | cluster provider, spec and status | no | | **AtprotoIdentity** | the DID to identity mapping, in the home workspace | us | no | | **Account** | the DID to shard map. Not a CRD - a real kcp `Workspace` of an account `WorkspaceType` in root | us | no | Do **not** use kcp's shipped home (`~`): it is a bare LogicalCluster with no owner and no `directlyDeletable`, and `LogicalCluster` delete is Forbidden without that field, so the shipped home can never be deleted. `ClusterPhase`: `Initializing`, `Ready`, `Unavailable`, `Draining`. `NodePhase`: `Pending`, `Running`, `Draining`. ### Generated from api/v1alpha1 The Go types in `api/v1alpha1` are the ONLY source of every CRD's shape. `go generate ./api/...` (`cmd/socialweb-apigen -root .`, logic in `internal/apigen`) writes, and nothing else may write: | Output | What | |---|---| | `deploy/-apiresourceschema.yaml` | one kcp `APIResourceSchema` per entry of `v1alpha1.APIResources` | | `deploy/-apiexport.yaml` | one `APIExport` per distinct `Export` (cluster, atprotoidentity, computecontract, computevm, hostprocess, marketidentity, ...), resources in registry order | | `deploy/apiresourceschema-revisions.txt` | the revision lock: every schema name ever generated and the sha256 of its `spec`, one ` sha256:` line each, in the order they were added (the generator keeps the file order and appends new names); `.gitattributes` merges it with `merge=union`, so two branches that each add a revision never conflict and the merged order needs no regeneration, and a name recorded twice with two hashes is a generator error | | `api//v1alpha1/zz_generated.register.go`, `zz_generated.conditions.go` | the scheme registration and the condition helpers of each API group | How the Go type maps, decided by Claude (user may veto): - Struct layout comes from reflection over `encoding/json` tags, so the schema is what Go actually writes. A field is `required` exactly when its json tag lacks `omitempty`, unless it carries the struct tag `apigen:"optional"` (only `ClusterSpec.running`, which is written as `false` and was never required). Struct tags, because the no-comments rule forbids kubebuilder markers and so rules out controller-gen. - Enums are read from SOURCE (`go/parser`): a named string type's enum is its typed string constants in declaration order. A named string type with no constants is a generator error, so an enum cannot silently become a free string. - `metav1.ObjectMeta` is `type: object`; `[]metav1.Condition` items are `type: object` with `x-kubernetes-preserve-unknown-fields`; `int` is `integer`, `int64` adds `format: int64`. A named type outside the group's own package is a generator error. - `apigen.Model.Validate` checks a value by the same rules as the schema plus one kube does not apply: an UNKNOWN field is an error, not pruned. `metadata` and conditions stay open. The first generation reproduced the hand-written Cluster, Node and AtprotoIdentity schemas and both exports BYTE FOR BYTE; NodeClaim differs only by its name and the added `status.contract` fields. `deploy/check.sh` (every mode) regenerates into a temp dir and `diff -u`s every output against the tree: any drift fails with `run: go generate ./api/...`. Protobuf was rejected: a CRD travels through kube and kcp as JSON validated by OpenAPI, so a second wire format is a second source of truth. **Schema revisions.** kcp holds `APIResourceSchema.spec` IMMUTABLE (re-measured: an update of a changed spec under the same name is Forbidden). So a schema is named `v1alpha1-..` with `Revision` set per kind in `api/v1alpha1/resources.go`, and the revision lock refuses to generate a spec that differs from what it recorded under that name: changing a Go type without bumping its `Revision` fails generation, and so `check.sh`, before any host sees a schema it cannot apply. Old names stay in the lock forever; old `APIResourceSchema` objects stay in kcp and are harmless. **What the re-point does to tenants, MEASURED** (kcp v0.33.0 over kine, by an `internal/apigen` test that plan 0026 deleted with every Go test): a workspace that bound the export while it named `v1alpha1-1` PRUNES the new fields (a write returns `unknown field` warnings and drops them). Creating `v1alpha1-2` and re-pointing the export moves that ALREADY-BOUND `APIBinding`'s `boundResources` to `v1alpha1-2` within seconds, with the identity hash unchanged (so the etcd prefix and every stored claim are untouched), the pre-upgrade claim intact, and the new fields then persisting. No binding is deleted or recreated: the migration IS the manifest set (`deploy/sets.yaml`, `socialweb-computer apply`) applying the new schema and then the re-pointed export, in that order. Without the re-point the binding stays on `v1alpha1-1` and the proof fails. **NodeClaim `status.contract`** carries, beside the original `vmUri|rfpUri|acceptUri|receiptUri|receiptCid|submitEventRef` (kept: prod claims hold them and the no_bids back-off keys on `Failed` + `rfpUri` without `acceptUri`), every field of the contract state (`v1alpha1.NodeClaimContract`): `version`, `phase` (`requested|rfp_submitted|winner_selected|accepted|receipt|network|released`), `vmName`, `requesterDid`, `marketDid`, strongRefs `{uri, cid}` `payload|rfp|policy|bid|accept|receipt`, `winner {did, submitAccept?}`, `acceptPending`, `receiptOk`, `vmAddress` (relay FQDN or `iroh://` ticket), `bids`, `release`, `termination`, `hostKeys`, `outcome`, `error` and `timestamps` keyed by contract phase. All optional, so every existing claim stays valid. None of it is secret: records are public repo records, `submitAccept`/`submitEventRef` are bidder endpoints. **The claim is the durable record a node is released from; the contract carries the flow.** The Requester component runs a ComputeContract through the RFP flow (`internal/market/requester`, the contract runner of `runner.go`) and writes the evolving contract state into the ComputeContract's `status.contract` at each stage (RFP submitted, accepted, receipt, network); each write lands before the flow moves on, so the receipt is on the contract before the guest boots, and a restarted runner resumes from the recorded stage. The NodeClaim controller copies `status.contract` onto the claim's `status.contract` on every transition (`internal/gateway/nodeclaim` `Mapper.Status`), with the legacy `receiptUri|receiptCid` filled from the receipt strongRef, so the provider's no_bids back-off is unchanged. A node is released when its claim is DELETED: the NodeClaim controller sets the claim's Infra entry to Released, the Infra controller sets `spec.release` on the contract, and the Requester releases it with the state read FROM THE CONTRACT, never from its memory (section 9, "A market node joins through its cluster's OpenBao, never through ssh"). Decisions, made by Claude, user may veto. The two marked SUPERSEDED are kept for the record: - A claim needs releasing iff its contract has receipt refs (`receipt` + `submitEventRef`) and its phase is not `released`. A released claim keeps its receipt and gains `release {event, payload, ok}`; the legacy receipt fields are no longer cleared, the contract phase is what says "done". - SUPERSEDED (M3): a claim with no live session in this process (startup, or a live claim this process did not start) was released with `releaseCompute` from its saved state, reason `requester_restarted`. A restarted gateway now adopts a `Joining` or `Running` claim without a write or a release, and marks a `Bidding` or `Provisioning` claim `Failed`; the release waits for deletion. A pre-upgrade claim carrying only the legacy fields is released from a `ContractState` synthesised from them. - `Released` only when `release.ok`; an unconfirmed release is `Failed`. A release that threw leaves the phase unreleased, so the delete path retries it. - SUPERSEDED (M3): an unreadable claim at the end of a run deferred the release to the next pass. A run no longer releases at its end; an unreadable claim at DELETION defers the release, the finalizer stays, and the next pass reads the deleted claim again. - A deleted claim, or a contract whose `spec.release` is set, asks its run to release (`contractRunner.Update`), and the run releases from the state recorded on the contract. The decision behind it stays: a flow must not end between the accept and the receipt, because that would leave a VM nothing can release. **Claim layering.** The claim's shape is the Go type `v1alpha1.NodeClaim`, the source of its schema; the NodeClaim controller (`internal/controllers/nodeclaim`) reads and writes it through the typed kcp client, and the mapper (`internal/gateway/nodeclaim`) turns a claim and its ComputeContract into the values of `internal/contract/nodeclaim`. A status write is always of the schema's shape. The TypeScript claim store that came before plan 0026 is gone. ### Abstract kinds and their contracts Plan 0026 W0-ABSTRACT. A concept with two or more implementations is an abstract kind plus concrete kinds (the Cluster API infrastructureRef pattern). The rules below hold for every abstract kind; the tables after them are the contract of each one. - A consumer MUST ref only the abstract kind, never a concrete kind. - An abstract kind MUST have `spec.backendRef`, a `common/v1alpha1.ObjectRef` to exactly one concrete kind of its contract, and a contract status with the conditions `Ready` and `BackendBound` (`commonv1.ConditionReady`, `commonv1.ConditionBackendBound`). - Every concrete kind MUST carry the label `contracts.socialweb.computer/: v1alpha1` on its APIResourceSchema. `APIResource.Contracts` in the group's `resources.go` names the abstract resources; apigen writes the label and refuses a contract that names no resource with `spec.backendRef`. The keys are `commonv1.ContractLabelAtprotoRepos`, `ContractLabelVaults`, `ContractLabelComputeVMs` and `ContractLabelEventSources`; `APIResource.ContractLabels()` returns the map a controller sets or verifies. - The composer (the root for children, the creating component for intent objects) MUST write the abstract kind's spec and the concrete kind's spec, and nothing else writes either. - The concrete kind's controller MUST be the only writer of the concrete kind's status. - The abstract kind's controller MUST be the only writer of the abstract kind's status. It MUST mirror every contract field from the bound backend's status, and MUST NOT compute a contract field the backend did not report. - `BackendBound` MUST be True exactly when `spec.backendRef` resolves to an existing object of a kind that carries the contract label; otherwise it is False and `Ready` is False. - `Ready` on the abstract kind MUST equal `Ready` on the backend, observed at the backend's current generation, and MUST be False while `BackendBound` is False. - A field that only one backend can report MUST live on that concrete kind, never on the abstract kind. **AtprotoRepo** (`api/v1alpha1`, export `applications`), over **LocalRepo** and **RemoteRepo** (label `contracts.socialweb.computer/atprotorepos`). | Field | Meaning | Writer | | --- | --- | --- | | `spec.backendRef` | a LocalRepo or a RemoteRepo | composer | | `status.did` | the DID of the repo | AtprotoRepo controller, from the backend | | `status.head` | the CID of the repo's latest commit | AtprotoRepo controller, from the backend | | `status.rev` | the rev (TID) of that commit | AtprotoRepo controller, from the backend | | `status.writable` | true when a RecordWriter of the repo can write now | AtprotoRepo controller, from the backend | | `status.address` | the PDS base URL the repo is served from | AtprotoRepo controller, from the backend | | `status.conditions` | `Ready`, `BackendBound` | AtprotoRepo controller | - LocalRepo `spec.pdsRef` MUST ref the Pds component; it writes through the Pds signer handle and holds no token. RemoteRepo `spec.pdsRef` MUST ref a RemotePDS and `spec.credentialsRef` the credentials provider (a MarketIdentity) whose lease authenticates the writes. - Both carry `spec.did`; `spec.authRef` is the session the plan 0026 W2-REPO controllers replace. - Both report the contract fields plus `status.seq` and `status.keyFingerprint` in `AtprotoRepoBackendStatus`, written only by their own controller. **Vault** (`api/v1alpha1`, export `hostprocesses`), over **OpenBaoServer** (local) and **RemoteOpenBao** (label `contracts.socialweb.computer/vaults`). | Field | Meaning | Writer | | --- | --- | --- | | `spec.backendRef` | an OpenBaoServer or a RemoteOpenBao | composer | | `status.address` | the vault API base URL a client dials | Vault controller, from the backend | | `status.ready` | the vault answers, is initialized and unsealed | Vault controller, from the backend | | `status.initialized` | the vault is initialized | Vault controller, from the backend | | `status.sealed` | the vault is sealed | Vault controller, from the backend | | `status.caBundle` | the PEM CA bundle that verifies `status.address` | Vault controller, from the backend | | `status.version` | the server version | Vault controller, from the backend | | `status.conditions` | `Ready`, `BackendBound` | Vault controller | - `status.caBundle` MUST hold only public CA certificates; a client MUST reload it when it changes. - No token, unseal key or credential is ever a Vault field; the callee holds them. **ComputeVM** (`api/v1alpha1`, export `computevms`), over **FirecrackerVM** (label `contracts.socialweb.computer/computevms`); HcsVM and VzVM join when plans 0011 and 0012 land. | Field | Meaning | Writer | | --- | --- | --- | | `spec.backendRef` | a FirecrackerVM | creator (the bidder) | | `spec.vm` | the size and role asked for: cpus, mem, disk, role | creator | | `spec.owner`, `spec.requester` | the account the VM runs under and the requester DID | creator | | `spec.userData` | the SecretRef of the guest user data | creator | | `spec.contract` | the market contract refs (receipt, accept, submit endpoint) | creator | | `spec.image`, `spec.stopGraceSeconds`, `spec.guestLogin` | the image policy, the stop grace, the guest login | creator | | `spec.accept` | the market accept the VM serves | creator | | `spec.release` | `requestedAt` and the optional `request` StrongRef: the creator asks for the release | creator (field manager `socialweb.computer/bidder`) | | `status.phase`, `status.lostReason` | `Pending` to `Running` to `Releasing`, or `Lost` or `Failed` and why | ComputeVM controller, from the backend | | `status.addresses` | `guest` and `host` addresses of the VM | ComputeVM controller, from the backend | | `status.bootedAt`, `status.message` | when it booted; the last transition | ComputeVM controller, from the backend | | `status.conditions` | `Ready`, `BackendBound` | ComputeVM controller (field manager `socialweb.computer/computevm`) | | `status.market.guest` | the guest's own report: address, reportedAt, host keys | guestcontrolplane (field manager `socialweb.computer/guestcontrolplane`) | | `status.market.published` | the URIs of the market records published for the guest: onNetworkUri, hostKeysUri | bidder (field manager `socialweb.computer/bidder`) | - Release MUST be `spec.release` written by the creator; an annotation MUST NOT carry it. The ComputeVM controller answers `spec.release` by creating the termination StateSync and deleting the ComputeVM; its finalizer deletes the FirecrackerVM, waits for it to go, then drops the guest login and the guest identity grants. - The accept a VM serves MUST be `spec.accept`; an annotation MUST NOT carry it. - The ComputeVM controller creates the FirecrackerVM named by `spec.backendRef` from the ComputeVM spec, owned by the ComputeVM (controller ownerReference), and mirrors its status on each backend event. Each field manager above is a const beside the type in `api/v1alpha1/types_computevm.go`; each writer applies only its own subfield. - FirecrackerVM `spec` (owner, vcpu, mem, disk, image, userData, stopGraceSeconds) and `status` (network, image, runtime unit identity and VMM, work dir) are the FirecrackerVM controller's (`internal/controllers/firecrackervm`); a ComputeVM consumer MUST NOT read them. The backend `internal/compute/firecracker` implements `compute.Provider` and takes `host.Units`, a user data reader and a Placement as dependencies; it imports no kcp client. Unit state follows systemd D-Bus signals (`internal/hostsvc` Watch); no ticker drives it. - Bid status has one writer, the bidder, by server-side apply under `socialweb.computer/bidder` (`v1alpha1.FieldManagerBid`); it binds a Won Bid to its ComputeVM and sets Complete or Failed. **EventSource** (`api/events/v1alpha1`, export `events`), over **PdsEventSource**, **RelayEventSource** and **JetstreamEventSource** (label `contracts.socialweb.computer/eventsources`). | Field | Meaning | Writer | | --- | --- | --- | | `spec.backendRef` | one of the three backends | composer | | `status.cursor` | the last sequence number the source delivered | EventSource controller, from the backend | | `status.lastEventAt` | when that event arrived | EventSource controller, from the backend | | `status.lag` | how far the cursor is behind the upstream head | EventSource controller, from the backend | | `status.address` | the upstream the source is subscribed to | EventSource controller, from the backend | | `status.conditions` | `Ready`, `BackendBound` | EventSource controller | - PdsEventSource `spec.repoRef` MUST ref an AtprotoRepo and subscribes through its Subscriber handle. RelayEventSource refs a Relay component (`spec.relayRef`) or a user endpoint (`spec.url`). JetstreamEventSource has a user endpoint (`spec.url`). - An EventStream MUST ref its source through `spec.sourceRef` (an EventSource) and carries only its `filter` (collections, DIDs), cursor and sink; no URL of a source is an EventStream field. ### The XRPC surface How a user gets a kubeconfig. Seven procedures under `com.publicdomainrelay.temp.cluster.`: ``` createCluster create the named cluster and record the intent to run it, in one call. Idempotent. The create-on-read path getKubeconfig walks, then spec.running = true -- set FIRST, on a context the caller's hang-up cannot cancel. Answers the phase it FINDS, Initializing included: it never waits for the control plane and never answers an error for a cluster that is coming up. controlPlaneUp bring the named cluster's control plane up. Idempotent. 429 + Retry-After while Initializing. Refused once Draining. DOES NOT provision nodes. controlPlaneDown the inverse getKubeconfig the credential the user runs kubectl with deleteCluster the lifecycle end listClusters this account's clusters revokeIdentity revoke an identity: write the deny record first, then fan out ``` Auth is a service-auth token whose `aud` is the service and whose `lxm` is the method NSID. A verifier on this path MUST be passed a real NSID: it compares against the token's `lxm`, and a nil there skips the check. **A caller that wants a usable cluster gets it in ONE call.** `createCluster` exists because the only other way to a live cluster was two round trips through two different verbs - `getKubeconfig`, whose kubeconfig a caller that is provisioning has no use for, and then `controlPlaneUp` - and because a caller had to know that order. It is not a new path: it calls the same `revocation.CreateOnRead` around `Store.Ensure` that `getKubeconfig` does, and then records `spec.running = true` itself, so the deny is consulted before anything is created. THE ORDERING IS THE CONTRACT: the running write comes first and reads nothing back, and it is made on a context the caller's cancellation cannot reach (`context.WithoutCancel`, `internal/gateway/gateway.go`), because the gateway's web app abandons a call at 30 s (`internal/gatewayhttp/cluster.go`) and an abort on that call must not land before this write, which would leave a cluster whose tree exists and whose `spec.running` is false, which no pass ever brings up. The phase is the ANSWER, not a refusal: a cluster whose control plane is still coming up is a 200 carrying `Initializing` (a market join is minutes away by construction), and only a Draining cluster is refused, from the view the write returns. `controlPlaneUp` keeps the stricter reading - 429 + Retry-After while Initializing - because a caller that asks the control plane up by name is asking about the control plane. `getKubeconfig` keeps its create-on-read, which is what makes a first login work at all. **`getKubeconfig` is served by a granter the composition root must be handed.** The granter (`kcpclient.KubeconfigGranter`) is the client certificate issuer `clientcert.Issuer`, which `internal/gatewaymain` hands the kcp client. A caller may also ask for a chosen credential lifetime; the settings and the reasoning behind each are in section 15. **A routed verb is not a working verb.** A verb can be on this surface, routed and documented while the composition root hands its client no granter, so it answers `ErrNoKubeconfigGranter` to every caller that reaches it. It fails CLOSED, which reads as a gap rather than a hole, and only a live call that reads the ANSWER, not the route, shows it. **The tenant path is gated on `status.phase == "Ready"`, and our front owns that gate.** kcp's mount controller reads the referenced object's PHASE, not its conditions, before it will serve (measured: `phase: Initializing` gave `WorkspaceMountReady=False(MountObjectNotReady)` and a 503). Tenant requests no longer cross the kcp mount (below), so the same gate is now ours and these are MUST rules: - The client certificate front MUST admit a tenant request only while the `Cluster`'s `status.phase` is `Ready` and it has no `deletionTimestamp` (`kcpclient.ClusterGate`), answering 503 otherwise and 404 when the `Cluster` is absent or its `spec.account` hashes to another account. - The front MUST follow that gate by a watch on the `Cluster` (`kcpclient.FollowClusterGate`), MUST fail closed on a read error, and MUST close every open request (watches included) for a cluster whose gate it observes closed. - The tenant path is SHUT when, and only when, the `Cluster`'s own state has closed the gate for at least `materialiser.PathGateGrace`: `deletionTimestamp` set, or `phase` not `Ready` with the `Ready` condition non-True since its `lastTransitionTime`. `materialiser.shut` MUST NOT read an HTTP status code; a 502, 503 or 404 from anything is never "shut". **The gateway acts in a tenant cluster as its owner.** `kubectlApplyServerSide` and `getJob` need a credential for the cluster's owner. The gateway signs one per call from the cluster's own client CA under `//server/tls` (CN the owner's DID, O `socialweb.computer:cluster-owners`, 10 minutes), dials the apiserver front whose address `front.json` reports, and the `socialweb-owner` ClusterRoleBinding of `deploy/tenant-grant.yaml` gives that group `cluster-admin`. No ServiceAccount token and no stored credential is involved, and the certificate is never written to disk. **No tenant request crosses the kcp mount.** kcp main (kcp-dev/kcp #4427, `pkg/server/mountproxy.go`) verifies the mount target's certificate against the system roots plus `--mount-proxy-server-ca-file`, authenticates the caller itself, strips `Authorization` and forwards `X-Remote-*` over its own mount proxy client certificate. A tenant k3s presents its own server CA and authenticates its own ServiceAccount tokens, so through the mount every tenant request is a 502 (target CA) or a 401 (the k3s token at kcp). The rules: - Every request for `/clusters/root:::clusters[/...]` MUST be served by `internal/clientcertfront`, in dev (the front is `SOCIALWEB_GATEWAY_KCP_PUBLIC_HOST`) and in prod (Caddy's `@clientcert` and `@tenant` matchers send it there over the edge client certificate; every other path goes to kcp). The external URL is unchanged, so a kubeconfig's `server` is still `/clusters/:clusters`. - The front MUST forward to the address the cluster's `front.json` names, verifying the cluster's own `server/tls/server-ca.crt`, with the path below `:clusters` and the query unchanged, and MUST strip every inbound `X-Remote-*` and `X-Socialweb-Client-Certificate` header. - A request with a client certificate MUST pass the certificate checks of section 7 and goes by request-header auth over `client-auth-proxy`, with `Authorization` removed. - A request without one (or from the edge without the certificate header) MUST carry `Authorization: Bearer ...`, MUST pass the deny lookup of the `Cluster`'s `spec.account`, and is forwarded with `Authorization` UNCHANGED over a transport that presents no client certificate, so k3s authenticates the bearer itself. - The tenant liveness probe (`Registry.TenantPath`, `clusterprovider.ProbeTenantFront`) MUST dial the `front.json` address directly, verifying the cluster's server CA; its result is reported, never counted as shut. - The revoker (`kcpclient` `FanOut`, `DeleteServiceAccount`) MUST reach the tenant directly at its front with a 10 minute owner credential signed from the cluster's client CA (`Options.Tenant`); a tenant it cannot reach is `Cold`, never a completed delete. - The kcp mount (`Workspace` `clusters` with `spec.mount.ref` to the `Cluster`, `spec.URL` from `status.URL`) MAY stay for kcp-side listing and discovery only; nothing may depend on kcp forwarding through it. Why kcp changed: GHSA-c8w2-fgvx-vhv4 (CVE-2026-61682) let a client-supplied `X-Remote-Group` through the front proxy; the fix strips it, #4238 added the mount proxy client certificate, and #4427 makes kcp authenticate every caller before it forwards to a mount (v0.33.1 fell back to `InsecureSkipVerify` and forwarded an unauthenticated caller's `Authorization`). The supported contract is a target that trusts kcp's identity headers; it cannot serve tenant credentials that only the tenant k3s can verify, so tenant traffic stays off the mount. Plan 0015's reason ("kcp's local proxy runs before authentication and forwards every client header") described v0.33.1 and is superseded by this section. **The negative rule, which is a design invariant:** never configure a cluster's apiserver to trust request-header identities from the kcp mount proxy. The only request-header client a tenant trusts is its own `client-auth-proxy`, held by the front. ## 7. Identity and auth - **Bearers and certificates both go through the client certificate front, never through the kcp mount.** Caddy (prod) or the client certificate front itself (dev, market network) ends TLS for every `/clusters/:clusters/...` request. A client certificate MUST be checked by the front (`internal/clientcertfront`) against the cluster's client CA, the PKI CRL, the deny record and the `ClusterClientCertificate` of its serial (it must exist and not be revoked) and is forwarded to k3s by request-header authentication (client-auth-proxy, X-Remote-User = the CN, X-Remote-Group = `socialweb.computer:cluster-owners`). A bearer MUST be forwarded unchanged to the cluster's own front and is authenticated by k3s alone; the front adds only the cluster gate and the deny lookup. Both MUST pass the cluster gate of section 6. No JWKS, no `--authentication-config`. The front follows a home's `ClusterClientCertificate` records and its `Cluster` with watches by workspace path. A Cluster deleted and created again under the same name gets a new home logical cluster that those watches never see (read on dev: every certificate of the new cluster was 401 "no ClusterClientCertificate for this serial" until the watch went idle). So a serial the record watch does not hold, and a Cluster the gate watch does not hold, is read once directly; when the read finds it, the answer comes from that read and the watch starts again. - **ServiceAccount tokens via TokenRequest** for the machine class. - **The DID to ServiceAccount mapping** is the tenancy boundary, and it has exactly ONE spelling: a DID's ServiceAccount is `did-` plus the lowercased unpadded base32 of the DID (`kcpclient.ServiceAccountName`), which is a valid DNS-1123 label because base32 emits only `[a-z2-7]` and never a hyphen or a digit outside that range. The deployed grant carries it as the subject of the `socialweb-owner` ClusterRoleBinding, so the spelling cannot be changed later without a migration. - **The grant is rendered for the cluster's OWNER, and the owner can act in it.** The subject's name is not a literal in the deployed file: `deploy/tenant-grant.yaml` carries `__SOCIALWEB_OWNER_SERVICE_ACCOUNT__` and the bootstrap substitutes `ServiceAccountName` of the `Cluster`'s `spec.account` - the same DID the mount path is derived from - before it parses the document. `RenderTenantGrant` REFUSES a document with no placeholder rather than applying it, so a grant scoped to a name nobody owns cannot be applied by accident. The file also carries a `ClusterRoleBinding` `socialweb-owner` binding that ServiceAccount, and the group `socialweb.computer:cluster-owners`, to `cluster-admin`: it is the owner's own cluster, and until this existed a real user's minted token authenticated and was authorized for NOTHING, because no `did-*` ServiceAccount had any RBAC. - **A revocation that only deletes objects does not revoke.** Deleting a ServiceAccount is a ~10 s interruption, not a revocation: a stale, name-keyed RoleBinding resumes grants the moment the object is recreated. Revocation needs a consultable deny record, and it is per-DID. - **The token is per cluster INSTANCE**, not per cluster. A new agent cannot join an orphaned server, because it does not hold that instance's token. This is what keeps an orphan harmless. **The tenant credential is minted from the ClientCA.** `cred/admin.kubeconfig` is derived, not stored, so the provider derives it: it reads `/server/tls/client-ca.{crt,key}` and signs a fresh ECDSA P-256 client certificate with CN `system:admin`, O `system:masters`. That is the identity k3s's own admin kubeconfig carries, and it is the arm of the grant's carve-out that writes Secrets - `deploy/tenant-grant.yaml`'s `socialweb-mint-identity` policy admits a Secret create in the tenant namespace only for `request.userInfo.username == 'system:admin' && 'system:masters' in request.userInfo.groups`, or the External Secrets controller `system:socialweb:external-secrets` (the identity of the pair beside the control plane, section 5; the pair is not the in-tenant ServiceAccount `system:serviceaccount:default:external-secrets`, which a tenant who creates that ServiceAccount would inherit). Writing a Secret in the tenant namespace is the provider's act, through its own admin credential for the grant and through the External Secrets controller for the vault-backed Secrets it materialises; no TENANT subject is admitted. `system:masters` is the apiserver's built-in superuser group for client certificates, so the credential is cluster-admin-equivalent on the tenant apiserver, and it is the **provider's**: the provider applies the tenant grant, reads the tenant cluster's Nodes and Pods for the scale observation, and deletes the tenant-side Node object on teardown. **Nothing k3s generated is read.** The rejected route was the k3s-written admin kubeconfig under the data dir: its server URL and SANs are unverified against the address a node dials, and `/etc/rancher/k3s/k3s.yaml` is shared state on a host that runs several clusters. **Which source mints it follows the SUBSTRATE, and not the mere presence of a seal.** There are two sources: the cluster's own `ClientCA` on disk, and the cluster's vault, whose PKI engine issues from a copy of that same CA imported into it. The choice is an explicit input to the seam (`clusterprovider.CredentialSource`), derived by the caller from where the cluster's nodes are placed. For `market` the credential is always signed from `client-ca.{crt,key}` on disk and the vault is never consulted: a market node's machine is not this host, the cluster vault in that path exists for the guest's custody and wake material rather than for the provider's own credential, and minting from it on every pass puts a second TLS dependency between the provider and the tenant apiserver for no gain. For `firecracker` and `qemu` nothing changes: the vault issues when the cluster has left unseal material beside its data dir, and the ClientCA signs when it has not. A caller that names no source is refused (`ErrCredentialSourceUnknown`) rather than defaulted, because the default is the thing this seam exists to remove. **Minting refuses rather than downgrades.** No ClientCA on disk is a refusal, never a fallback to a wider credential. A CA key that is not its certificate's key is refused *before* anything is signed, because that pair still signs and the failure would otherwise surface as an authentication error at the first apply. An empty CN or O is refused, because it authenticates as the empty username or as an unprivileged user. A non-https URL is refused, because the URL's host is the name the apiserver's certificate must carry. The rule holds ACROSS the two sources as well as within each: a cluster whose ClientCA cannot sign is refused with the reason, and the vault is not tried instead. **The provider's credential never reaches a user.** It is a certificate credential signed by the cluster's client CA with `system:masters`. The user path is the kubeconfig of `getKubeconfig`: a client key signed from the cluster's own `client-ca.{crt,key}` on disk when they exist (`internal/clientcert`), or by the cluster's OpenBao PKI as the fallback (role `kubeconfig-user`, at most 24 h), CN the user's DID, O `socialweb.computer:cluster-owners`, which the client certificate front admits only while its `ClusterClientCertificate` exists and is not revoked. **A workspace ServiceAccount does not leave its workspace.** Its token carries `authentication.kcp.io/scopes: [cluster:]` and kcp refuses it in every other logical cluster before RBAC is consulted - measured both directions, with a real TokenRequest token: a provider token is denied in a tenant workspace even with a matching ClusterRoleBinding there, and a tenant token is denied in the provider workspace. `--as=` impersonation does NOT reproduce it (it sets the username and not the scope extra), so an impersonation probe cannot answer the question and an RBAC probe that uses one is not evidence. And the refusal is not RBAC at all: a token is refused by every workspace it was not minted in, which no subject string can spell. The consequence is load-bearing: `internal/kcpclient` addresses root, the account workspace, the home workspace and the mount from ONE `rest.Config`, so **the runtime cannot be any of the four provider ServiceAccounts**. The cluster provider holds a credential kcp does not confine to a workspace - the admin kubeconfig the operator already installs - and the four ServiceAccounts exist for a future split into four identities per workspace, not for today's single client. That is a deliberate privileged gateway-side credential: it is never handed to a tenant, and the tenant's own path is the client certificate front with a short-lived client certificate. RBAC cannot enforce the identity classes for a runtime that spans workspaces; the SCOPE does, and it enforces them by refusing the client rather than by shaping it. **The identity bundle is exactly 13 entries:** ``` tls/server-ca.crt tls/server-ca.key tls/client-ca.crt tls/client-ca.key tls/request-header-ca.crt tls/request-header-ca.key tls/etcd/server-ca.crt tls/etcd/server-ca.key tls/etcd/peer-ca.crt tls/etcd/peer-ca.key tls/service.key cred/passwd cred/ipsec.psk ``` `cred/admin.kubeconfig` is NOT one of them: it is DERIVED from the ClientCA, deterministic. Nor is the instance token, which is per-instance state. **Pre-seeding a list without request-header-ca and all four etcd CAs makes k3s REGENERATE those and INVALIDATE every kubeconfig and SA token.** The count is load-bearing: `internal/identitybundle` refuses a bundle whose size is not `len(identitybundle.Paths)` (`ErrIncomplete`). ### A market node's secrets live in its cluster's OpenBao, behind a JWT role This subsection describes how a market node's secrets are stored and read. Names are in `internal/vaultlayout/nodejwt.go`, the vault writes in `clusterprovider.NodeVault` (`internal/clusterprovider/nodesecrets.go`: the policy, the JWT mount and the role go into the cluster's OpenBaoServer declaration, `internal/clusterprovider/vaultdeclared.go`, and the core's sync writes them; the secret itself is written as the vault client `clusterprovider`), the order in `internal/nodemarket/secrets.go`. Only the market substrate wires it (`nodeHostsOptions.NodeSecrets`, set in `internal/providermain/main.go`); a firecracker or qemu guest keeps its cert-auth path (`node-secret-`, `secret/data/k3s/node/`), which is a different policy namespace and a different mount. | what | name | written | removed | |---|---|---|---| | KV v2 mount | `socialweb-nodes` | enabled on first use if `sys/mounts` lacks it | never | | node secret | `socialweb-nodes/data/` | before the claim, once `ClusterLink -apiserver` reads `Listening` with a ticket; else by the first later pass that sees one | release (`DELETE socialweb-nodes/metadata/`, every version) | | policy | `node-`: `path "socialweb-nodes/data/" { capabilities = ["read"] }` | before the claim, always | release | | JWT auth mount | `jwt-`, type `jwt`, `oidc_discovery_url` = `bound_issuer` = issuer | when a claim first carries `status.grant` with that issuer | disabled by the reconcile once no live claim's grant names it and it has no role left | | JWT role | `` under that mount: `role_type jwt`, `bound_subject` = grant subject, `bound_audiences` = [grant audience], `user_claim sub`, `token_ttl 5m`, `token_policies [node-]` | the first pass after the NodeClaim controller writes `status.grant` | release, under every `jwt-<12 hex>` mount | The secret's fields are `token` (the pinned join token `K10::server:`, composed by `nodemarket.PinnedJoinToken`, so `k3s agent --token` takes it verbatim), `nodePassword` (the per-node password `nodemarket.MintOrReadNodePassword` minted at Provision, trimmed), `serverCA` (the PEM of `/server/tls/server-ca.crt`, verbatim), `apiserverTicket` (the link's `status.ticket`), `advertiseAddress` and `apiserverPort` (an integer; the claim's own `spec` values). A secret exists only when it is whole: the guest may read "present" as "complete". Decisions, made by Claude, user may veto: - **The secret waits for the link; the claim does not.** A link that is not `Listening` holds back the secret, never the claim, and never the pass: Provision writes the policy and the claim and returns, and `confirmHeld` calls `SyncSecrets` on every held market node each pass, which writes the secret once a ticket exists (logged once per distinct reason, not per pass). A policy that cannot be written fails the Provision before the claim, so nothing is rented that could never read its join material. - **The grant is the NodeClaim controller's; the role name is not recorded in status.** The provider never writes `status.grant.role`; the role name is deterministic (the node name). `EnsureLogin` declares the grant on the cluster's OpenBaoServer, and a pass over an unchanged grant changes nothing in the declaration. - **A mount goes with its last live grant.** The declaration names a `jwt-<12 hex>` mount while a live node's grant names its issuer, so the sync removes a mount only once no live grant names it. - **Revocation is best-effort and bounded** (`nodemarket.RevokeTimeout`, 10 s), runs right after the claim delete is accepted and before the wait for the NodeClaim controller, and only logs a failure: the machine is what costs, and a leftover role binds one workload subject the released machine no longer runs. - **The issuer must answer from this host.** OpenBao fetches `/.well-known/openid-configuration` when the config is written and the JWKS at login; a bidder issuer the cluster vault cannot reach is an `EnsureLogin` error in the pass log, retried every pass. How a guest logs in, which is the requester's and the guest's half (`claim.json`, written by the guest's cloud-init from `internal/atproto/cloudinit/k3sjoin.go`, carries `jwtMount`, `role`, `audience`, `kvMount`, `kvPath`): ask `guest-login.sh token ` for a token of the bidder's OpenBao identity/oidc whose `aud` is the grant's audience (`api://socialweb-node?actx=/`) and whose `wid_sub` claim is the grant's subject, then `POST /v1/auth/jwt-<12 hex>/login` with `{"role": "", "jwt": ""}`, then `GET /v1/socialweb-nodes/data/` with the returned `client_token` in `X-Vault-Token` (`internal/nodeagent`). Measured against a real OpenBao v2.6.2 (`bao server -dev`) and an ES256 OIDC issuer when it was built: the node's own token logs in and reads its path; a token for another subject or audience is refused; the node's token gets 403 on another node's path; after `Revoke` the role, policy and secret are gone, the login is refused, the other node's policy and the mount remain. ### The cluster OpenBao is reconciled against the live claims The node entries below are part of the cluster's OpenBaoServer declaration: `NodeVault.Reconcile` sets that declaration to the live claims, and the core's sync removes whatever no declaration names. The table above is what is written; this is what is removed. `Revoke` alone is eager and therefore skippable: a provider restart or crash between a claim going away and its `Revoke` leaves a role, a policy and a secret behind, a bidder's issuer is a fresh `did:key` per restart, so every issuer a cluster has ever seen would otherwise leave its `jwt-*` mount forever (read live: 5 `jwt-*` mounts, 1 live node). `(*clusterprovider.NodeVault).Reconcile` (`internal/clusterprovider/nodesecrets.go`) takes `live map[string]*v1alpha1.NodeClaimGrant` for ONE cluster's OpenBao and rewrites the node part of that cluster's OpenBaoServer declaration to it (`setVaultNodes` in `internal/clusterprovider/vaultdeclared.go`): for every live node the policy `node-`, the KV entry `socialweb-nodes/` and, when the claim carries a usable grant, the mount `jwt-<12 hex of sha256(grant.Issuer)>` with the role `` under it. It touches only the declaration's entries of those naming patterns (`vaultlayout.NodeFromJWTPolicy`, `vaultlayout.IsNodeJWTMount`, the `socialweb-nodes` KV mount); every other policy, mount and entry of the declaration stays. The core's sync of the OpenBaoServer then makes the vault equal the declaration. The method is declared as `nodemarket.NodeSecretsReconciler`, a sibling of `NodeSecrets` rather than a fourth method on it, so a substrate's secrets implementation still has one method per verb and the provider type-asserts it the way `confirmHeld` asserts `secretSyncer` (`internal/providermain/vaultsync.go`); the provider calls it with `live` = the claims it holds plus the `Adoptable` claims in the cluster's home workspace (`liveNodeGrants`), after `adoptClaims` at adoption and on every pass, rate-limited to at most once per cluster per `VaultSyncInterval` (30 s). ### The browser refreshes OAuth tokens in a JS worker (2026-10-25) atproto refresh tokens are single-use, and every tab of the browser page runs its own Go wasm. So the browser build hands the token request to one plain-JS Worker (no wasm), which runs it single-flight across tabs, writes the rotated set to IndexedDB before it answers, and hands it back. The seam is `oauthclient.TokenRefresher` (`internal/atproto/oauthclient/refresher.go`): `RefreshTokens(ctx, TokenRefreshRequest) (TokenRefreshResponse, error)`. - Registration. `Options.Refresher` on one Session, or `oauthclient.SetTokenRefresher(r)` (js build only, `refresher_js.go`) as the default for every Session without one, read at each refresh. Unset (every native build) is the Go POST of `refresh.go`, unchanged. - The Go side keeps its part. The refresh stays one flight per refresh token in the process; the flight re-reads the session store first and adopts a newer set another tab wrote; only then does it call the refresher. It accepts a set only when `token_type` is DPoP, `sub` is the session DID (else `ErrRefreshWrongSubject`, nothing saved, the old set stays) and `scope` contains `atproto` (else `ErrScopeRejected` under `ErrSessionExpired`). It then saves the set to its own store and uses it. On `invalid_grant` it re-reads the store once and adopts or refreshes the newer set, else the session is `ErrSessionExpired`, as on the Go path. - The request (JSON on the bridge, snake_case): `session_id`, `did`, `issuer`, `token_endpoint`, `client_id`, `refresh_token`, `dpop_jwk {kty: EC, crv: P-256, x, y, d}`, `dpop_nonce`. The client is the atproto public client: `token_endpoint_auth_method` none, the form is `grant_type`, `refresh_token`, `client_id`, DPoP-bound. - DPoP: the worker signs. The request carries the session's private JWK; the worker imports it as a non-extractable ES256 CryptoKey and signs each proof with WebCrypto (`htm` POST, `htu` the token endpoint, `jti`, `iat`, `nonce`), retrying once on `use_dpop_nonce`. A sign callback into Go would need a round trip per proof to the tab that asked, and a flight shared across tabs must finish after that tab is gone. The key is in the stored session already; it never goes into a log. WebCrypto signatures are high-S as often as not; our authorization server verifies DPoP leniently (`HashAndVerifyLenient`), and other atproto servers take plain ES256. - The response: `access_token`, `refresh_token`, `token_type`, `scope`, `sub`, `expires_in`, `expires_at` (unix seconds), `dpop_nonce` (the newest the authorization server sent). The Go side reads the access token expiry from its `exp` claim, as on the Go path. A set the worker rotated is never dropped: when the IndexedDB write fails the worker still answers it. - Errors: a `*TokenRefreshError {error, error_description, status}`. `invalid_grant` is `ErrSessionExpired`. `pass_through` (`ErrRefreshPassThrough`) means the worker does not own that `client_id`, and the Go POST runs instead. Any other code (a network error, a timeout, another OAuth error) is a plain refresh error and the session stays. - Confidential clients keep the Go path. The gateway web app session is the OpenBao atproto plugin's (private_key_jwt) and never reaches this seam; no native build registers a refresher; a browser worker answers `pass_through` for any `client_id` other than the page's own public client metadata. ## 8. Ports Every per-cluster port is chosen by the kernel and held by the cluster's apiserver front (member `front`) for the life of the cluster. Nothing checks a port and binds it later. The front binds its own listener on `BindAddress:0`, keeps it in its fd store and records it in `front.json`; the HostProcess supervisor reports it in the front's `status.listeners`, and the provider reads the `k3s-secure` port from there. Every other purpose below is a HOLD: the provider asks the front (control op `hold`), the front binds `[::]:0` dual-stack with `SO_REUSEADDR` off, sets `SO_REUSEADDR` after the bind, never listens, and stores the socket in its fd store (`FDNAME=hold.`, `FDPOLL=0`). The member (k3s, kine, openbao, external-secrets: third-party binaries that take a port flag and accept no socket) binds that number beside the hold. The holds are reported in the Cluster `status.front.heldPorts`; every member's own listeners are in its HostProcess `status.listeners`. `internal/portallocator` only records which port the front holds for which purpose (`leases.json`); there is no port range and no probe. | purpose | for | |---|---| | `postgres` | the cluster's embedded-postgres - if embedded bostgres doesn't work, use real postgres | | `openbao` | the vault | | `kine` | kine's listener | | `k3s-secure` | the cluster's apiserver front (`internal apiserver-front`, member `front`): the port a node, the kcp mount, the provider, the CCM and ESO dial, and `--advertise-port` of every slot | | `k3s-slot-a`, `k3s-slot-b` | the `--https-listen-port` (supervisor) of k3s slot a and slot b behind the front; never advertised | | `k3s-apiserver-a`, `k3s-apiserver-b` | the `--apiserver-port` (a hidden k3s flag) of slot a and slot b: the apiserver listener on `127.0.0.1`, which k3s otherwise puts on `https-listen-port + 1` | | `kine`, `kine-b` | kine's listener of slot a and slot b | | `k3s-controller-manager`, `k3s-scheduler`, `k3s-cloud-controller-manager` (slot a), the same names with `-b` (slot b) | the k3s control-plane components' secure ports | | `external-secrets-health` | the External Secrets controller's `--live-addr`, on `127.0.0.1` | | `external-secrets-webhook` | the External Secrets webhook's TLS `--port`, the one its `ValidatingWebhookConfiguration` names as `https://127.0.0.1:`; it binds every interface, because the upstream flag has no host | | `external-secrets-webhook-health` | the External Secrets webhook's `--healthz-addr`, on `127.0.0.1`. Both members' metrics are off (`--metrics-addr=0`), so no metrics port is leased | **The apiserver front owns the `k3s-secure` port.** Each cluster runs member `front`: `internal apiserver-front`, the same L4 front kcp uses (`internal/apiserverfront`), bound on `BindAddress:`, its control socket `/front.sock` (or `/fronts/.sock` when that path is over 107 bytes) and its state `/front.json`. k3s listens on its slot lease and advertises the front pair (`--advertise-address` unchanged, `--advertise-port` = the front port), so `status.URL`, every node secret's `apiserverPort`, the link's `--host`, every kubeconfig and the kubernetes EndpointSlice keep their number across a k3s swap. The backend is chosen per new connection; open connections stay on their backend. The front never fails over by itself: the k3s gate points it at its slot after `/readyz` and the first kube-system Deployment answer on the slot port, then checks `/readyz` through the front and that the kubernetes EndpointSlice names only the advertise address and the front port. Its listening socket is kept in the backend's fd store (systemd `FDSTORE=1`; devsup on proc), so a front restart refuses no connection and resets only open ones. `AssertMarketAdvertiseInvariant` takes a `portallocator.Advertisement`: advertise port == front lease, link target == front lease, no slot listen port inside the front lease. **The vault's lease is ONE port.** Every known purpose is one port wide (`Purpose.Width`), `openbao` included. The config the vault is given (`clusterstack.OpenBaoServerConfig`, section 5) sets `disable_clustering = true`, which removes the cluster listener and makes "one vault per cluster, no HA" a property of the file. Both the listener stanza and `api_addr` take the same `Lease.Port`, and no code in the tree reads `Lease.Port + 1`. The k3s pair is two holds, `k3s-slot-` and `k3s-apiserver-`, because the apiserver binds `*:` AND `127.0.0.1:`, a kernel-picked port has no free neighbour to count on, and `--apiserver-port` moves the second listener (measured: `127.0.0.1:41000`, `[::1]:41000` and `127.0.0.1:41500` with `--https-listen-port 41000 --apiserver-port 41500`). **`advertise` and `kubelet` are retired purposes, and neither is a second fact.** The advertised port IS the bound port: `K3sAPIPorts` answers both questions from the `k3s-secure` lease, and the invariant refuses an `--advertise-port` that is not the port the apiserver holds. `kubelet` is not leased: a node is not a process in THIS host's kernel and network namespace, so the port is the guest's own. Both are in `retiredPurposes` (with the ssh and link purposes): leases for them are dropped from an existing lease file on open, and `Allocate` refuses them as unknown purposes. Section 15 holds both arguments in full, including what the kubelet lease was for. **The invariant that matters:** an unleased or colliding advertise **port** routes a node's tunnel to ANOTHER CLUSTER'S APISERVER, and that surfaces as an **x509 name error**. The certificate is not the fault; the lease is. The advertised port is not chosen and is not separately leased: it IS the port the `k3s-secure` lease holds, so the two cannot disagree. **A node binds no port of this host's, so a node owns no lease.** Every port its guest binds, the kubelet's included, is bound inside the guest's own network namespace. MEASURED on this host with a guest running, nothing binds 10250, 10248, 10249, 10251 or 10256, the host's k3s runs `--disable-agent` and so has no kubelet at all, and pasta is started without `-t`/`--tcp-ports`, so no guest port is published and the host cannot initiate a connection to a guest. **Only ONE port serves the API, and it is the one a node must dial.** Measured: `k3s server --disable-agent` binds `*:` AND `127.0.0.1:` as a PAIR - which is why each slot holds two ports - and in steady state it binds NOTHING on `--advertise-port`. `/cacerts` answers 200 on the listen port and 401 on the advertise port for every credential form, so an agent dialed at the advertise port loops on `failed to get CA certs: Unauthorized`. The node's `--server`, `status.URL`, the tenant credential, the readiness gate and the hardening seam all name the listen port; hardening through the advertise port is what kept the process group from ever becoming ready. **N control planes per host: proven. N nodes per host: refuted.** Two `--disable-agent` servers ran concurrently, each with its own kine and store, both serving - `kubectl get ns` against each - with the port pair above and disjoint control-plane secure ports. A node is the opposite: `127.0.0.1:6444` collides FATALLY and the loser is served the other cluster's CA; there is one `/run/k3s/containerd/containerd.sock` (hardcoded - `--data-dir` moves only `root`; measured cross-contamination, cluster A's containerd served `RunPodSandbox` for cluster B's pods), one `flannel.1` whose NAME is not redirectable and whose reconfiguration panicked the other agent (`vxlan_network.go:217`), one `8472/udp` socket, one `/run/flannel/subnet.env`, one IPAM pool, one set of `KUBE-*` chains and one `/var/lib/kubelet/pod-resources`. All four candidate fixes were refuted. That list is the measured case for a VM per node. **The hold, measured** (Linux 7.2, in a private network namespace with a 1000-port ephemeral range): a second exclusive hold of a held port is refused; a member binds `127.0.0.1:P`, `[::1]:P` or `:P` beside the hold and is dialled; a second listener is refused while the member listens; the member binds again after it restarts; 998 port-0 binds exhausted the range and none got a held port; 200 outgoing connects took none. A cluster link's dumbpipe binds UDP port 0 (`--ipv4-addr 0.0.0.0:0`); a node dials it by its ticket, so its port is not held. **The bind probe (before the holds).** Before the front held every port, the allocator chose a port from a range and probed it; the holds replaced the probe. Its lesson is why a hold binds the WILDCARD and sets `SO_REUSEADDR` only after the bind (`internal/apiserverfront/fdstore_linux.go`; the one exception re-binds a known port over a TIME_WAIT socket). A probe that relies on Go's `SO_REUSEADDR` does not coexist with a wildcard binder: a LISTEN socket conflicts with a wildcard or a specific bind whatever the prober's `SO_REUSEADDR` is. The fault is the ADDRESS, not the flag: ``` holder bound 127.0.0.2:P, LISTENing -> old probe said FREE (a wildcard bind is blocked) holder bound 0.0.0.0:P with SO_REUSEADDR, not yet LISTENing -> old probe said FREE (a wildcard bind is blocked) holder bound [::]:P, ipv6-only, LISTENing -> old probe said FREE (a wildcard bind is blocked) ``` The later probe bound the WILDCARD with `SO_REUSEADDR` cleared; both halves were load-bearing, and the old probe leased `35167` while it was already bound. `/proc`'s LISTEN table was not a substitute: it has no row at all for a bound-not-listening holder. **Unleased per-node ports.** `127.0.0.1:6444` (k3s agent supervisor), `:10248` (healthz) and `:10010` (containerd) are unleased, and `6444` is host-fixed, so two agents on one host collide on it. ## 9. A node **The contract**, which is durable and substrate-independent: - a node is one long-lived command; - closing it is the teardown, and it is the ONLY teardown - one code path, no leak surface beyond "did we close it"; - nothing is ever read back out of a node. We generate the token and set the node name ourselves. - `--node-name` is set from the `(cluster, node)` pair, never learned by a round trip. This is not hygiene: IP reuse lets a dead node shadow a live kubelet and break `logs`/`exec` with a 502. **The mechanism**, which is substrate-specific. The design has had three; exactly one is still reachable, and the other two are recorded because the evidence that removed them is the evidence that keeps them removed: | | | |---|---| | **a market guest** | **THE node path.** The node is a VM rented from the COMPUTE MARKET through a `NodeClaim` the clusterprovider writes and the NodeClaim controller (`internal/controllers/nodeclaim`, bound by the GatewayAPI component) fulfils under the cluster owner's own atproto OAuth session. **`internal/nodemarket`**, `SOCIALWEB_CLUSTERPROVIDER_SUBSTRATE=market`. NOT a guest of this host: no subnet lease, no pasta, no VMM, no node image and no `/dev/kvm` is involved. The clusterprovider's whole part in the node is the claim; the gateway's Requester runs the RFP flow, the winning bidder boots the guest from its cloud-init, and the NodeClaim controller writes the phase back; the guest itself dials the cluster's OpenBao and apiserver through the cluster's ClusterLinks and binds the advertised address (`SOCIALWEB_CLUSTERPROVIDER_MARKET_ADVERTISE_ADDRESS`, default `198.18.255.254`) on its own loopback (section 9, "A market node joins through its cluster's OpenBao, never through ssh"). Confirming is the tenant Node appearing for a claim the NodeClaim controller left `Joining`, not a local process being alive | | **a guest of this host** | RETIRED. A VM this host started itself -- `firecracker` or `qemu` behind the same `scale.Nodes` seam, each a VMM in its own network namespace, booting a node image built here and joining through an agent the image carried. Both values were refused and deleted: the market path with a local bidder covers the same market network, so a second way to place a node was a second way to be wrong about where a node is. The guest the MARKET rents is booted on the bidder's host by the bidder's own path (`internal/compute/firecracker`, `internal/vmruntime`, `internal/guestimage`, `internal/guestrootfs`, `internal/nodeimagemain`), not by any code of the retired substrate | | **local process** | DELETED. A `k3s agent` child process, the deleted `internal/nodelocal`, behind the same `scale.Nodes` seam. REFUTED as the substrate, three ways, all measured: `k3s agent` is fatal as a non-root user (`agent requires additional privilege if not run with --rootless`); it needs root for `/run/k3s/containerd`, `/var/lib/kubelet/device-plugins`, `127.0.0.1:6444` and the host `KUBE-*` chains, none of which `--data-dir` moves; and postgres is fatal AS root (`initdb: cannot be run as root`), so a host that runs the store cannot run a node | **`SOCIALWEB_CLUSTERPROVIDER_SUBSTRATE` accepts exactly one value, `market`, and every other name is refused at startup.** `checkPlacement` names `market` as the only placement implemented; the two retired names get a refusal that names what replaced them, because a name from before the retirement would otherwise read as a typo. `market` is the one value that was never a guest: it takes no subnet range, mounts no guest gateway, and its `checkPlacement` demands no VMM, no agent, no image and no pasta. The variables only the retired substrates read -- `FIRECRACKER`, `QEMU`, `NODE_AGENT`, `IMAGE_OUT_DIR`, `IMAGE_CONFIG`, `NETWORK`, `NODE_DISK_MIB` -- are not parsed into the Config, and setting one is refused rather than ignored, since an operator who set one and saw no complaint would believe it was carrying it. The local-process mechanism is **DELETED, not merely refuted**: `internal/nodelocal` and the `process` value of `SOCIALWEB_CLUSTERPROVIDER_SUBSTRATE` are gone, so a host can no longer be configured into it. The `scale.Nodes` seam stays, with one implementation behind it, because the seam is what made a second substrate cheap and a third possible. The measurements that refuted the local-process mechanism are in the git history: `git log` on the deleted `experiments/process-substrate/` carries them, with the commit messages that say what each one established. **History.** The paragraphs from here to "`k3s server` is deliberately NOT a node" record what was measured on the retired guest-of-this-host substrate: its node images (the host's own kernel and modules), the image builder, pasta namespaces and snapshots. That code is gone, and "now" in them means at the time. The market guest's image is built from `deploy/guest-image` with a kernel from `deploy/guest-kernel` that has no loadable modules (`CONFIG_MODULES=n`, section 12). **A guest kernel is obtainable with no privileges at all, and this was measured.** Firecracker boots a kernel YOU supply, and the CI kernels cannot run a k3s node: all three fail `k3s check-config` with `STATUS: 4 (fail)` - `NETFILTER_XT_MATCH_IPVS`, `XT_MATCH_COMMENT`, `XT_MATCH_MULTIPORT`, `XT_MATCH_STATISTIC`, `IP_VS`, `IP_SET`, `NF_TABLES_INET` and `NFT_MASQ` are all absent, so BOTH kube-proxy modes are unusable. Building a kernel needs `flex`, `bison` and `libelf`, which are not installed and need root. The way through is that **the host's own kernel already has every one of those symbols as modules**: extract the ELF from `/boot/vmlinuz-$(uname -r)` (zstd; the payload starts at byte offset 21196) and carry the host's own `.ko.xz` files in an initramfs, with `boot_args` that omit firecracker's default `nomodule`. Inside such a guest, `k3s check-config` returns **`STATUS: pass`**. No build, no root, no packages - and this is the single biggest de-risking of the whole substrate. **A guest kernel's module set is a MEASURED set, and `k3s check-config` cannot make that measurement.** The image's module list is what the guest agent loads before k3s starts, and it is a measurement rather than a preference. Measured on this host: with `xt_conntrack` absent from the list, kubelet inside the guest failed to create the KUBE-FIREWALL chain - the one that drops packets to 127.0.0.0/8 arriving from a non-loopback interface: ``` "Failed to ensure rule to drop invalid localhost packets in filter table KUBE-FIREWALL chain" err="error appending rule: exit status 1: Warning: Extension conntrack revision 0 not supported, missing kernel module? iptables: No chain/target/match by that name." "Failed to initialize iptables rules; some functionality may be missing." protocol="IPv4" ``` and the same for IPv6 with `ip6table_mangle` absent. **`k3s check-config` returned RC=0 both before and after.** It reads the kernel CONFIG file staged beside the kernel and reports `CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)` without ever asking whether that module is LOADED. It audits the config the kernel was built with, not the kernel the guest is running - so a node can be Ready, and a green check, with a firewall rule missing. iptables 1.8.11 (legacy) no longer autoloads xtables extensions, which is why a module has to be NAMED rather than left to the first rule that needs one. The instrument that names this class of defect is the guest console, and nothing else. The list is also bounded by what the kernel was built with, and that bound is not negotiable from here: k3s tries `modprobe nft-expr-counter` and fails because `CONFIG_NFT_COUNTER` is not set at all, so there is no module to add; it is harmless only because kube-proxy runs in iptables mode. Three further check-config complaints (`CONFIG_RT_GROUP_SCHED`, `CONFIG_CRYPTO_GHASH`, `CONFIG_INET_XFRM_MODE_TRANSPORT`) are compile-time gaps of the same kind. **The same instrument found three more, and they were not cosmetic.** With `ipt_REJECT`, `ip6t_REJECT` and `xt_nat` absent, kube-proxy's `iptables-restore` failed on the rules that target them, so the in-cluster `10.43.0.1:443` had no ClusterIP, and every system pod that talks to the apiserver - the provisioner and the metrics server - crash-looped on `dial tcp 10.43.0.1:443: connect: connection refused`. The pods were scheduled and started, so the symptom is two services down rather than a cluster that will not come up. iptables 1.8.11 (legacy) does not autoload these either. **The list is complete because it is DERIVED, not because it grew - and the weaker instrument was hiding the rest.** `iptables` inside this guest does not load an extension on demand, and `iptables-restore` stops at the FIRST extension it cannot find, so discovering names by running and reading the console is a treadmill: one name per boot, dozens of boots, and a guest whose firewall is incomplete for as long as nobody looks. Both sides are enumerable at build time. The supply side is the kernel's own tree: `modules.dep`, `modules.builtin` and `modules.alias`, which is the kernel's ` -> module` table. The demand side is the guest's own console, because a failed restore dumps its ENTIRE payload - 401 rule lines in one run - and the extensions in it are the demand. That derivation found two more, and the way it found them is the important part: `xt_NFLOG` and `xt_physdev`, absent from the list while `-j NFLOG --nflog-group 100` and `-m physdev --physdev-is-bridged` sat in kube-router's network policy payload. **And with them missing, the three system pods reached `1/1` Running, with zero restarts, and stayed there for 1m43s** - while kube-router's network policy controller aborted its whole sync. So pod readiness is the WEAKER instrument for this class of defect, and a longer module list is not evidence either. The acceptance is: pods Ready AND held for a stability window AND the guest's own console free of a failed `iptables-restore` or an aborted policy sync. The failure is racy, too - one of two boots hit it with the old list - which is what explains two agents measuring the same three pods as `1/1` and as `0/1` with restarts and both being right. The names that cannot be left to a same-name module are the ones to watch, and they are few: the target `REJECT` is `ipt_REJECT` and `ip6t_REJECT` (there is no `xt_REJECT`); `DNAT` and `SNAT` are `xt_nat` (there is no `xt_DNAT`); `NFLOG` and `physdev` are `xt_NFLOG` and `xt_physdev`. The builder now refuses a config that names no provider for an extension the cluster's rules use, and refuses a name the kernel does not carry, so this is caught at build time rather than one boot at a time. What that cannot do is prove a name NECESSARY: five names have been proven by a measured failure - `ipt_REJECT`, `ip6t_REJECT`, `xt_nat`, `xt_NFLOG`, `xt_physdev` - and the remaining ~43 are carried for modes and matches these boots did not exercise. **Why nothing loaded on demand: the kernel execs a modprobe the rootfs did not have.** The guest boots the build host's Fedora kernel, built with `CONFIG_MODPROBE_PATH="/usr/bin/modprobe"` (Fedora merged sbin into bin), over an Ubuntu 24.04 rootfs whose modprobe is `/usr/sbin/modprobe` only. Every `request_module()` the kernel makes -- the autoload of an nf_tables expression, an xt match, a logger -- exec'd a path that did not exist, so the only modules a guest ever had were the ones `systemd-modules-load` or a userspace `modprobe` on PATH loaded. Measured on the local market network (the plan 0005 M3 live gate): `/proc/sys/kernel/modprobe` read `/usr/bin/modprobe` from a pod, `nft_limit` and `nft_log` were not in `/proc/modules`, and kube-router's network policy controller aborted every sync with `iptables-restore v1.8.10 (nf_tables): line 61: RULE_APPEND failed (No such file or directory): rule in chain KUBE-POD-FW-*`; line 61 of the dumped payload is `-m mark ! --mark 0x10000/0x10000 -j NFLOG --nflog-group 100 -m limit --limit 10/minute`, which iptables-nft 1.8.10 writes as the NATIVE expressions `log group` and `limit` (`nft --debug=netlink` on the same ruleset in a user+net namespace on the build host: `[ log group`, `[ limit rate`), and those need `nft_log` and `nft_limit` -- not `xt_NFLOG` and `xt_limit`, which the module list carried and which iptables-nft never asks for. So the derivation above was complete for the xtables path and blind to the nf_tables one, and a deny-all NetworkPolicy left a pod reachable. The builder now reads `CONFIG_MODPROBE_PATH` from the kernel config it stages and, when that path does not resolve inside the stage (symlinks resolved within the stage, never on the build host), links it to the rootfs's own modprobe (`/usr/bin/modprobe -> /usr/sbin/modprobe`), and refuses a rootfs with no modprobe at all (`linkKernelModprobe` of the retired `cmd/socialweb-nodeimage`). With it, the same gate on a rebuilt image: `nft_limit` and `nft_log` loaded on demand, no aborted sync on the console, and a deny-all ingress policy refused the client 9 s after it was applied. The module list stays as a preload, not as the only way a module gets in. `nft-expr-counter` still fails k3s's own modprobe and is harmless: the counter expression is built into `nf_tables.ko` on this kernel (no `CONFIG_NFT_COUNTER`, no `nft-expr-counter` alias, and the rules above carry `counter` expressions that load). **And the same missing module is why a node's own cluster services die, which is the failure that looked like a network problem.** `iptables-restore` is ALL-OR-NOTHING: when kube-proxy's ruleset names the first extension the guest has no module for, the entire `KUBE-SERVICES` chain is dropped - including the DNAT rule for `default/kubernetes`. A packet to `10.43.0.1:443` then leaves by the default route and is refused, and every kube-system component that talks to the apiserver dies on it: ``` metrics-server panic: unable to load configmap based request-header-client-ca-file: Get "https://10.43.0.1:443/api/v1/namespaces/kube-system/configmaps/ extension-apiserver-authentication": connect: connection refused local-path-provisioner level=fatal msg="Error starting daemon: invalid empty flag helper-pod-file ... Get ".../configmaps/local-path-config": refused" coredns 0/1, never Ready, same address ``` So a missing module presents as three unrelated services failing on one address, and the cause is visible only in the guest's console 7.6 s in, before any pod needs it. The node itself is unaffected: it registers, reaches `Ready`, and runs a pod - which is why this hid behind a green four-step proof. **Two hypotheses were wrong here and are recorded so they are not re-run.** The first: that the kubelet's LEASED port breaks these pods, because the ecosystem assumes 10250. Refuted three ways - every failing and passing boot used the lease (`10251`); two boots identical but for the port both reach `1/1` and hold; and structurally the apiserver takes the kubelet's address AND port from the Node's own status (`status.daemonEndpoints.kubeletEndpoint`), with k3s starting its metrics-server `--kubelet-use-node-status-port`, so 10250 is not assumed. The endpoint that looked like evidence, `https://10.42.0.2:10250/readyz`, is the kubelet's readiness probe OF THE METRICS-SERVER POD, because k3s gives that pod `--secure-port=10250` - it points at the victim, not the cause. The second: that the failure was intermittent, because two agents measured the same three pods as `1/1` and as `0/1`. The reconciliation is that the console reads differ by how much was read - the captured consoles are bounded copies (~90-123 KB) and a failing run prints only the last 60-80 lines, so a tail read is indistinguishable from a clean console. Neither agent's measurement was wrong; one of them was looking at a truncated file. **The kubelet port lease is retired.** Its recorded justification is that two k3s AGENTS as local processes on one host contend for host-global ports (`listen tcp 0.0.0.0:10250: bind: address already in use`), and the fourteen host-global resources measured alongside it are all properties of a node that is a process in the host's network namespace - which is the substrate this design deleted. A node is now a guest with its own kernel and its own netns: measured, nothing on the host binds 10250, 10248, 10249, 10251 or 10256 while a guest runs, the host's k3s runs `--disable-agent` and has no kubelet at all, and pasta is started without `-t`/`--tcp-ports` so no guest port is published and the host cannot initiate a connection to a guest. What kept the lease alive after that was that `perNodeNetwork` derived each guest's address FROM the leased port, making it the only per-node-varying input the composition root had. The leased CIDR replaced exactly that, which is what made the lease removable, and it has been removed: `PurposeKubelet` is deleted, `kubelet` is in `retiredPurposes`, and the guest's agent passes no `--kubelet-arg` at all, so the kubelet binds its own default inside its own namespace. Nothing is lost by that, because no part of the ecosystem assumes a port: the apiserver takes the kubelet's address AND port from the Node's own status (`status.daemonEndpoints.kubeletEndpoint`), and k3s starts its metrics-server with `--kubelet-use-node-status-port`. MEASURED: two boots identical but for the port (10250 vs 10251) both reached `1/1` system pods and held. MEASURED ONCE MORE, with the flag removed entirely - one uninterrupted run against a real `k3s server --disable-agent`: k3s passes NO `--port` argument to the kubelet at all, the guest registered as a Ready node **12.9 s** after `InstanceStart`, its cluster's three system pods (`coredns`, `local-path-provisioner`, `metrics-server`) reached **`1/1` with zero restarts** 44.5 s after instance start and held for **1 m 47 s**, the node reports its own kubelet endpoint as `{"Port":10250}`, `kubectl top nodes` reads it through that port (`215m` CPU, `169Mi`), and the guest console is clean on every firewall signature. What that run also needed, and what nothing enforces: the image had to be REBUILT first. The guest's kubelet argv is decided by the agent inside the image, and the image's recorded inputs did not include that agent at the time, so a boot on the previous image passed `--kubelet-arg port=0` - the removed field read back as Go's zero value - and the kubelet refused to start with `port (--port) 0 must be between 1 and 65535`. The rule underneath the lease survives it, restated as invariant 8. **The guest never reaches a registry, which is stronger than giving it a resolver.** The image builder pulls the images the cluster's own system components use - with `go-containerregistry`, as Go, no daemon, no credential - and stages them as docker-save tarballs under `/agent/images/`, where k3s's agent imports every tarball BEFORE it starts the kubelet. Measured: the import lines are in the guest's own log at 27-150 ms each, and the whole console contains zero occurrences of `ErrImagePull`, `ImagePullBackOff`, `no such host`, `dial tcp` or `failed to resolve reference`. The system pods reach `Running` in a guest whose `/etc/resolv.conf` is still one byte, which is what makes the property provable rather than merely likely - they cannot have fetched anything. The list is explicit in the operator's config rather than derived at build time, and it WAS derived from the shipped binary before being written down: read out of the manifests `k3s server` extracts, with `pause` taken from a string inside the binary because no manifest names it. Five images, 61.5 MiB staged and 62.0 MiB of rootfs, against a 1500 MiB headroom. Three of the resolved digests appear verbatim as blob names inside upstream's own air-gap tarball for this version, which is independent confirmation that the same images were resolved. One trap the verification caught: containerd normalizes seeded names through docker's `ParseDockerRef`, so emitting `index.docker.io/rancher/...` imports under a name no pod asks for - a silent `ImagePullBackOff`. Two things this does not cover, both named rather than implied: the `svclb` image for a LoadBalancer Service is created by the service controller on demand and cannot be found by reading manifests, so the first tenant LoadBalancer Service will hang until it is added; and a k3s bump moves both the list and the digests. **pasta is the rootless network path, and it does hand out privileges.** Measured: under `pasta --config-net` the child runs as uid 0 in a fresh user and network namespace with a full capability set, and `ip tuntap add ... mode tap` SUCCEEDS there - while the same command in the host's own namespace is denied. So a guest gets a tap, a bridge and an address without root, and it reaches a HOST-LOOPBACK-ONLY listener through the gateway address. That is what lets the apiserver stay off the network: bind loopback, advertise a host address the apiserver will accept for the in-cluster `kubernetes` Service endpoint, and add the gateway address as a `--tls-san`. **A guest joins, and this is now the measured destination rather than the direction of travel.** A k3s agent inside a firecracker guest registers against the host's real `k3s server --disable-agent`: ``` liveweb-node1 Ready 0s v1.36.4+k3s1 INSTANCE_START_TO_READY_S 7.4 10.6 9.5 ``` Three runs. The time is measured ON THE HOST from the firecracker `InstanceStart` timestamp to the moment `kubectl` first reports `Ready`, because a guest's own account of its own elapsed time is not evidence: an earlier round of this work spent real effort chasing a reported 332-second boot that was a literal `echo` in an init script, printed 0.29 s after the VM started. The spread is k3s's startup, not the VM. Inside the guest, kubelet logs `Starting flannel with backend vxlan`, registers the node, sets the pod CIDR, and the kubelet it started binds the port it is given - a leased `10251` in this measurement, and its own default since the lease was retired (the kubelet lease paragraph above). Supporting numbers on the way there: a `k3s server --disable-agent` on the host is ready in **1.1-1.9 s**; the guest's namespace reaches the apiserver. And **a kernel is validated by restoring it, never by comparing versions**: 6.18.48 cannot be restored at all on this AMD host (`Bad FPU state ... restore_fpregs_from_fpstate` -> panic). That is how 6.1.186 became the pin for the CI-kernel route discussed below, a route this design then abandoned in favour of the host's own kernel - which is what actually ships, which is why the guest that joined runs the host's kernel, and which turns out to matter for snapshot restore, below. **Boot and control-plane costs, recovered from the retired experiment tree before it went**, and worth keeping because two of them closed lines of work rather than opening them: ``` kernel boot medians 0.270 / 0.341 / 0.336 s 6.1.141 / 6.1.186 / 6.18.48 kernel sizes 41.9 / 45.0 / 27.8 MB spread 0.07 s -> KERNEL CHOICE BOUGHT NOTHING. The 27.8 MB kernel is not faster than the 45 MB one. root device initramfs 0.303 s ext4 0.763 s squashfs 0.763 s (delta +0.460 s) uncompressed 1.44 MB cpio ties 0.86 MB gzip cpio -> the cost is the virtio-blk DEVICE plus the mount path, NOT decompression, so compressing the initramfs harder buys nothing snapshot create 168.24 ms, state 31 KB, mem 512 MB; loads 5.81 / 6.16 ms control plane cold 26.79 s PRE-SEEDED 1.28 s restart 1.23 s SIGTERM -> exit 0.50 s, first listener ~0.11 s, 503 -> 401 gap 26.68 s -> the 26.79 s was attributed to k3s GENERATING ITS CAs, and that attribution is REFUTED by a control experiment on the process substrate. A rebuild that must author its own CAs pays within +/-0.24 s of one that is handed them -- sometimes FASTER -- so there is no ~25 s of CA authoring to collect, and k3s's own CA generation is ~0.2 s here. What the cold figure actually contains is a fresh DATASTORE: 3.7-5.6 s of it disappears when only the store is warm and the bundle is absent. The 1.28 s pre-seeded figure reproduces (1.135-1.139 s); the 26.79 s cold one does NOT (4.97-6.75 s), so the two were measured under different store conditions and comparing them was wrong. The identity bundle's measured value is the IDENTITY it preserves -- a CA fingerprint byte-identical across a teardown and rebuild, where a control's changed every time -- not a time saving. Without it every rebuild is a new cluster under the same name, and every prior kubeconfig and ServiceAccount token stops validating silently. flannel recovery killing agent B leaves A's pods Running again 75 s later; 6 veths on cni0 shared by both clusters' pods ``` **Restoring a snapshot instead of booting is REFUTED as the provisioning primitive, and the numbers are not close.** Measured on this host against the same image: ``` cold boot InstanceStart -> Ready 9.079 9.158 10.768 s snapshot freeze with the agent running 0.698 0.719 0.731 s snapshot load firecracker's own count 3.96 ms (0.247 s for the whole path, pasta included) LOAD -> Ready 9.974 s ``` The load is **2.6 % of the operation**. Everything else is k3s reconnecting and re-reporting itself, so a 4 ms restore buys a 250 ms head start on a 9.5 s job. Counted honestly from the other end it is worse still: a node whose VM dies is not noticed by the cluster for **53.9 s**, so a cold boot from nothing beats waiting for the monitor and then restoring. **And it cannot make a NEW node at all, which is the part that kills it.** Restore does not re-run boot-time units, so the guest's own identity adoption never happens: `kcp-agent.service` does not re-run, `mountNodeDisk` and `adoptIdentity` never run, and no agent start marker appears in a restored console. A snapshot frozen as `liveweb-node1` - with its recorded node-disk path deliberately re-pointed at another node's disk before the load - comes back as **`liveweb-node1`**. So restore can thaw a node the cluster still knows and nothing else. Two consequences follow directly: a control plane can only ever scale a restored node back up, never out, and **a deleted Node object is never re-created**, because the kubelet only retries `updateNodeStatus` against a node that is not there and makes no registration attempt. Since a snapshot is also valid only for the control-plane INSTANCE it was frozen against (the join token is per instance), the mechanism's scope is one node, one name, one cluster incarnation. Two facts worth keeping even though the mechanism is gone. **A per-node disk CAN be attached to a restored guest**: a drive `PUT` whose backing file exists is accepted and the load that follows succeeds, so the constraint is not that the drive slot must exist in the snapshot. What is true is that a drive `PUT` which FAILS leaves the VMM unable to load *any* snapshot (`Loading a microVM snapshot not allowed after configuring boot-specific resources`), which is a nasty failure shape: one rejected device poisons every later attempt. And **restorable FPU state is a property of the kernel, not of the guest**: restoring the host's 7.2.7 kernel panics on the guest's first syscall return (`exc_general_protection` in `restore_fpregs_from_fpstate`, reached from `switch_fpu_return`), and `noxsave` removes the panic on this kernel - an earlier note that `noxsave` merely converts the panic into a hang was measured on 6.18.48 and does not hold on 7.2.7. That leaves two requirements in direct conflict: k3s needs the host kernel's netfilter symbols, and a restorable guest wants a kernel whose FPU state this host can actually restore. `noxsave` is an escape hatch, not a resolution, and its effect on a k3s workload is unmeasured. Firecracker v1.13.1 also carries the pre-1.15.1 kvm-clock bug: a restored guest's monotonic clock jumps forward by the pause duration. **Two failures that presented as plumbing, and were one cause: the `pasta` child inherits pasta's own SELinux domain.** `/usr/bin/pasta` is `pasta_exec_t`, so a child of it runs in `pasta_t` unless the program it execs is a type whose own policy transitions it away. In `pasta_t`, `ip tuntap add` is denied `CAP_NET_ADMIN` and `open` on a console file in `/tmp` is denied - and, the detail that made this expensive, `ifconfig_t` cannot write to the pipe `pasta_t` owns, so the error TEXT of both failures was silently discarded and arrived as an empty string. `pasta link in namespace: ""` and a permission error with no message were the same bug. The child is now `/bin/sh -c 'exec ...'`, and **the reason that works is not the one this document first gave.** Measured: on this host `/bin/sh` is `system_u:object_r:bin_t:s0`, NOT `shell_exec_t` - `/usr/bin/bash` is the `shell_exec_t` one. What actually holds is that the gateway itself runs in `unconfined_t`, so exec'ing a `bin_t` file does not transition at all. The fix is incidental rather than principled: a gateway deployed under a systemd service domain would get the original failures back, including the silent one. What survives the domain changing is a POSITIVE PROBE rather than a label. The namespace is treated as configured only once the child has been observed to find pasta's own upstream link, and nothing is mutated before that, so a child without the privileges the work needs fails at the observation instead of half-configuring a VMM. Its power is bounded and worth stating: it refuses a child that was unshared WITHOUT pasta's configuration, but a child that was never unshared at all sees the host's own interfaces and would pass - so it establishes "unshared, with pasta's config" rather than "provably pasta's namespace". A failure that writes nothing on either stream is also now a failure that names its own emptiness, because the original bug was exactly that: a denial whose account was denied too. **pasta also hands its child a PID namespace.** The guest-side code reported its VMM as pid 17 while the host saw 2188474, so a record naming that pid named nothing. The record now resolves the HOST pid from its own identity, so the record, the startup sweep and the liveness lookup agree by construction rather than by three callers happening to do the same thing. **One writable ext4 serving as both template and instance was a correctness bug, not hygiene.** The guest's writes - k3s's extracted data dir, `/etc/rancher/node/password`, generated keys - persisted into the file the next boot read, and the next boot failed `k3s check-config` with `sha256sum: does not match`. Two VMs on one ext4 would also have corrupted each other. The image is now a TEMPLATE and every node gets its own clone at `/rootfs.ext4` (`cp --reflink=auto`, free on this filesystem), removed on teardown. Overlayfs is the alternative; reflink needs no guest-side mount work and no extra drive. **A host that died at its first step and a host that is merely slow look identical until the clock runs out.** The guest's console is read for named fatal markers (`Kernel panic`, `KCP_INIT fatal`, `KCP_PREFLIGHT fatal`, `KCP_AGENT fatal`, `KCP_AGENT_EXIT rc=`), and confirming a join returns immediately with the guest's own words instead of waiting out the timeout. This is what turned a missing k3s binary from a four-minute mystery into an instant, named error. The guest agent writes the same account to `/dev/console` directly as well as to a file, so a dead or rate-limiting journald cannot turn a guest's only report of its own failure into silence. **A host prerequisite that cost real time:** the repository directory carried the SELinux type `user_tmp_t` instead of `user_home_t`. Under pasta's namespace that makes binaries built in the tree unexecutable and repo paths unreadable, and it presents as `permission denied` on unrelated-looking operations. `restorecon -R` on the tree and its parent is the fix, and a host that runs this substrate should be checked for it. **What the VM substrate requires, and what it costs.** Firecracker's CI kernels are under a DATE-based artifact prefix; a version-numbered prefix such as `firecracker-ci/v1.13/` is STALE and still served, and a year-old kernel from it BOOTS PERFECTLY, which is exactly why the mistake is invisible. Pin by the newest date prefix. The current kernels carry `VXLAN=y`, `NF_TABLES=y`, `VETH`/`BRIDGE`/`TUN=y` and `OVERLAY_FS=y`, so k3s's networking is reachable - with ONE hard exception: `CONFIG_NETFILTER_XT_MATCH_COMMENT` is UNSET, and kube-proxy's iptables mode appends `-m comment --comment` to essentially every rule it writes. It is on kubeadm's required-kernel-config list for that reason. That left two routes - build a kernel, or build `xt_comment.ko` against the 6.18 CI kernel (the only one with `CONFIG_MODULES=y`) and ship it in the rootfs - and BOTH were abandoned for the cheaper route stated above: boot the host's own kernel with the host's own modules. What survives from this finding is the rule rather than the route: **a guest kernel's module set is a MEASURED set**, and the measurement is `k3s check-config` run inside the guest, plus reading the guest's own console for the iptables failures that check-config does not catch. `CONFIG_IP_VS` being unset is irrelevant: ipvs mode is deprecated and being removed. Firecracker's DEFAULT cmdline passes `nomodule`, so a distro kernel whose virtio drivers are modules needs both a `boot_args` override and an initramfs - which makes building a kernel the better route, not the worse one. `VIRTIO_FS` is unset in both kernels, so a virtiofs root is not available. The guest needs a REAL init: k3s runs the kubelet with `--cgroup-driver=systemd`, so systemd must be managing cgroups inside the guest. A busybox-init guest proves rootless boot and networking and is not the shape a node needs. Networking is rootless either way; what changes is whether the host's apiserver has to leave loopback. With a userspace gateway such as `pasta` the guest reaches the host's loopback services through the gateway address, so the listener stays on loopback and only the SAN grows: `--tls-san` carries the gateway address the guest dials. That is why bind and advertise are now separate values. The point of the VM is not isolation for its own sake: it is that every one of those host-global resources is PER GUEST inside it, so N nodes fit on one host where a local process allowed one. The node contract above is unchanged by the substrate - one long-lived command (the VMM), closing it is the teardown, nothing is read back out, `--node-name` is set from the `(cluster, node)` pair and the token still travels as a `0600` file. **The seam must not give the dial lifecycle ownership.** If `Close` also destroys, then `Wait` and `Close` become indistinguishable in effect, and the contract above collapses: the node is supposed to die BECAUSE THE COMMAND EXITED, not because we asked. The refusal was therefore explicit, and it lived in a control that nothing reached: the package that carried it, `internal/nodetransport`, is DELETED, because a control nothing reaches is indistinguishable from a control that is not there. The rule it enforced is kept here as a rule, which is where it always belonged: **the substrate must not destroy the node it dials.** "The substrate must not destroy this" has to be a value a caller is handed, never a convention a caller has to remember. A lifecycle-owning dial puts a destroy inside the transport, which is precisely where nothing asserts it is never called. Note the control's reach, which is why it is gone: nothing in the tree ever imported it, so that refusal is recorded in section 15 alongside the package that carried it. A control nothing reaches is indistinguishable from a control that is not there, which is why it is stated here as a rule rather than trusted as a guard. The token never rides argv: it goes in a `0600` file named by `--token-file`. An argv value is readable from `/proc` by every local user, and k3s derives its bootstrap storage key from the join token, so a leaked token is a leaked cluster. **For a guest the token must not ride the kernel command line either, and the reason is not obvious.** The command line reaches a console TWICE: firecracker logs the body of its own `PUT /boot-source` request, and the kernel prints `Kernel command line:`. A `kcp.token=...` argument is therefore written to `guest.console` in plaintext, twice, every boot. The per-node secret travels instead on a per-node read-only virtio-blk disk, written `0600` on the host at provision time and destroyed at teardown, carrying the join token and the `(cluster, node)` identity. Nothing secret reaches a console, k3s still takes it from `--token-file`, and the disk is what makes a snapshot CLONE-able: the image stays generic because the per-node half is injected. **The join token is also the node's k3s password, written to three paths, and that reuse is recorded rather than latent.** `adoptIdentity` writes the same `node.Token` to the file k3s is handed by `--token-file`, to `/etc/rancher/node/password` and to `/agent/node-password`: k3s reads the second as the node password it presents when it registers, so one value is both the cluster join secret and the credential the kubelet authenticates with. The v1 ceiling is stated rather than hidden: every node of a cluster shares one node password, which is harmless at 0-or-1 nodes and is inherited by the first multi-node change, where a per-node password must be minted instead. **The image is a directory, not a file.** It holds `vmlinux`, `initramfs.cpio`, `rootfs.ext4` and `image.json`, and the boot line travels WITH the kernel inside `image.json`, so a kernel and the command line that suits it cannot be separated by an operator choosing two paths. The builder refuses to write a 2 GB image that is missing k3s, the agent, the guest document, the units or the kernel config, which fails in seconds instead of after a boot. **The record's identity is supplied by the substrate, and it is not a local-process idea.** The retired `internal/noderecord` matched a token SUBSET against `/proc//cmdline` after confirming the pid's start time, and a record whose identity does not match is NOT OURS and is never signalled - a pid reuse must stay a refusal, not a kill. A local agent is `{"agent", nodeName}`; a VMM is `{"firecracker", "--api-sock", }`, and the socket path is per node, so one node's record can never match another's VMM. An empty identity matches nothing rather than everything, which is the only safe reading of "every token present". **The guest runs with SELinux off**, `selinux=0` on its command line and `SELINUX=disabled` in the root filesystem where the base image carries a config. A node's guest boots the host's OWN kernel, which is built with SELinux, while its minimal root filesystem has no policy to enforce, so leaving it on is at best noise and at worst an unlabelled denial that surfaces as a permission error somewhere unrelated to its cause - which is exactly the shape of the two `pasta_t` failures above. The security boundary for a node is the microVM, not SELinux inside a guest that has no policy. This is guest-only; the host's policy is untouched. `k3s server` is deliberately NOT a node: it runs gateway-side with `--disable-agent`. Only the worker scales to zero. **A Node's cloud identity is the cloud controller manager's, and it runs on the host.** A guest joins as a Kubernetes Node with no cloud identity: an empty `spec.providerID` and no labels for an autoscaler to group by. The controller stamps each Node it observes with `spec.providerID = socialweb:///` and the label `node.socialweb.computer/cluster=`, and deletes a Node whose instance record is absent - the guest is gone. It reads the guest set from `/nodes.json`, which the provider writes (atomic temp-plus-rename) on every provision, teardown and reap, and once at startup as `[]`. Two things about that snapshot are load-bearing and neither is obvious. An UNREADABLE snapshot is an error and patches nothing, because invariant 14 applies and deleting a Node out from under its own running work is the cost of getting it wrong. But an ABSENT snapshot and an empty array are the same answer to the reader - `(known=false, err=nil)`, the delete path - so writing `[]` protects nothing on its own; what protects a live Node is that the snapshot LISTS it. The file is infrastructure and not a cache, and the ordinary path is what keeps the absent case out of reach: the provider writes it before the controller can be asked anything. The identity is SCOPED, never the superuser, and the provider grants it `get`, `list`, `watch`, `patch`, `update` and `delete` on `nodes` once k3s is up. That grant is not decoration: a scoped identity with no binding is authenticated-and-denied, so the informer never syncs, the readiness gate times out, and the failure reads as a controller that will not start rather than as a missing binding. ### A market node joins through its cluster's OpenBao, never through ssh Plan 0005 M3-SECRETS-TICKET, M3-UNITS and the gateway half of M3-NODE-LOSS, built to the plan's `m3_contract`. The gateway holds no connection to a market guest and opens no ssh session to one; the guest dials its cluster through the cluster's ClusterLinks and reads its join secrets from the cluster's OpenBao. Decisions not in the contract are marked "decided by Claude, user may veto". **What the NodeClaim controller and the Requester do per claim** (`internal/controllers/nodeclaim` with the mapper of `internal/gateway/nodeclaim`, bound by the GatewayAPI component; the Requester component, `internal/services/requester` over `internal/market/requester`): The NodeClaim controller maps. It runs no market flow, holds no PDS session, no claim key and no secrets key, and since plan 0031 it writes no ComputeContract. Each Cluster with a NodeClaim has one Infra `nodes-/)>` in the Gateway application's own workspace (`spec.clusterRef` names the Cluster, label `infra.socialweb.computer/cluster`; no ownerReference, because kcp garbage collection is per logical cluster). The NodeClaim controller is the only writer of its spec: one entry `node-` per claim, Staged while the join bundle is not written, Requested once it is, Released when the claim is deleting or ended, with the node fields (`ownerDid`, `adopt`, `nodeJoin`, `nodeClaimRef`) in `spec.nodes` and the bundle in `spec.secrets`, both kcp-only. The Infra controller applies the entry's ComputeContract `infra--` and runs its lifecycle; the NodeClaim controller reads that contract by name, maps its status back onto the claim and names the entry in `status.infra`. A claim that is gone drops its entry; an Infra with no entry left, or whose Cluster is deleting, is deleted, and its finalizer releases every node. The Requester component of the same application carries the flow. - *NodeClaim controller, per claim.* It names the entry's contract `infra-nodes-<20 hex>-node-<12 hex>`, puts the finalizer on the claim, reads the ClusterLink `-openbao` from the claim's own workspace (a link with no `status.ticket` fails the claim before any market round) and the cluster OpenBao CA (`/openbao/tls/ca.crt`, or the path the provider publishes), and writes the join bundle as one KV entry `contracts//bundle` in its own vault mount. The bundle holds the files `/etc/socialweb/node/openbao-ticket`, `openbao-ca.pem` and, when the `-apiserver` link is listening, `apiserver.env` (`NODE_NAME`, `ADVERTISE_ADDRESS`, `APISERVER_PORT`, `APISERVER_TICKET`). None is a credential: the tickets are the links' public halves and the CA is public. The entry, and so the contract, carries `spec.vmName` (the node name), `spec.ownerDid`, `spec.secretsRef` (the bundle), `spec.nodeJoin` (the cluster key and the pinned socialweb-meshd build), the bidders to admit and deny, the policy and `spec.vmReadyTimeoutSec`. - *Requester, per contract.* The Requester watches ComputeContracts, admits at most `maxConcurrent` of them and `perAccount` of one account (`RequesterSettings`), and runs the RFP flow for each under the owner account's delegated session; it reaches the owner's repo and the vault only through bus handles (`internal/services/requester/wiring.go`). The node-join contract adds the `k3s` and k3s-join user-data modules and wraps the secrets capability: on the grant it writes `status.grant` = `{issuer, subject, audience: api://socialweb-node?actx=/, role: }` and appends `claim.json` = `{node, cluster, jwtMount, role, audience, kvMount, kvPath}` to the bundle (`jwtMount` = `jwt-` needs the winning bid's issuer). The wrapped capability's RBAC record also names a role `-node` whose `aud` is that audience, so the bidder keeps a GuestIdentityGrant (api `socialweb-node`, actx `/`) and the guest can get the token; an audience that is not `api://?actx=` gets no grant. The cluster vault's JWT role binds `wid_sub` (the user claim) with `bound_claims`, because the `sub` of an OpenBao identity token is the entity id. A grant write that fails leaves the inner capability unauthorized and the contract `Failed`, releasable from its receipt. The Requester waits up to `vmReadyTimeoutSec` (default 120) from the accept for the guest to fetch its bundle (`status.secretsFetchedAt`), then writes phase `Held`; a guest that never fetches ends `Failed` with the reason and keeps its receipt. - *Mapping.* `Requesting` is claim phase `Bidding` until the accept, then `Provisioning`; `Held` with `secretsFetchedAt` is `Joining`; a contract `Released` with `status.contract.termination` is claim `Lost`; `Failed` is `Failed` with the contract's message. `status.grant` and `status.contract` are copied on every transition; a claim in `Joining` or `Running` changes only on `Lost`, `Failed` or `Released`, so the provider's `Running` is never overwritten. - *Release.* When a claim is deleted the NodeClaim controller sets `spec.release` on its contract (`Mapper.Settle`), waits until the contract is `Released` or `Failed`, then deletes the contract and the bundle and removes the claim's finalizer (claim phase `Released` when the VM was released or none was held, else `Failed`). A contract that is deleted while it holds a VM releases the VM first. A contract whose claim is gone is released or deleted as an orphan (`Mapper.Orphan`). - *Adoption.* A claim the provider recreates for a node that already joined (`Adopting`) becomes a contract with `spec.adopt`: no RFP; the Requester finds the receipt and the bidder-signed `vm.hostKeys` in the owner's records and holds it `Held` (`contractRunner.runAdopt`), which the NodeClaim controller maps to `Running`. **A restarted gateway loses nothing.** Every step the Requester takes is in `status.contract` of its contract; a restarted Requester resumes each contract from it (no second RFP) and a contract that already fetched its bundle is only watched for the bidder's `vm.terminated`. A restarted NodeClaim controller finds each claim's contract by name and maps it again. **A restarted provider never reaps; it adopts** (M3-NODE-LOSS, provider half). Decided by Claude, user may veto. Before this, the provider's shutdown reap (`nodeHosts.reap`) called the market host's `Reap`, which DELETES the claim, so every provider stop (every deploy) made the gateway destroy every market machine; and a provider that started held nothing, so a node that survived anyway was never confirmed, lost, scaled in, torn down or counted in flight, and its name could be claimed again. Now: - *Shutdown leaves market nodes.* `nodeHosts.reap` returns without touching a market host (logged `leaving the market nodes running for the next provider process`). The guest needs no session of ours; its agent and links are `Restart=always` and reconnect when the next process brings the cluster's group back. A cluster switched OFF or deleted still releases every node (`reapCluster`), and the guest substrates still reap on shutdown, because their guest is a child of this host. - *Adopt at startup, before the first pass.* `join`/`joinAll` call `nodeHosts.adopt` for every cluster right after the empty instance snapshot is written, so the snapshot names the adopted nodes before the cluster's cloud controller manager starts (it deletes every Node the snapshot does not name). A failed adoption is a warning and the next pass retries. - *Adopt on every pass, from the list it already reads.* `nodeHosts.reconcile` lists the claims of the cluster's home workspace each pass for the `Released` rule; the same list goes through `adoptClaims`, so any live claim of the cluster that nothing holds is held from that pass on. No extra kcp read per pass. An OFF cluster, which gets no scale pass, adopts once (`adoptOnce`, one list until a reap that acted resets it) before `reapCluster`, so a cluster switched off while the provider was down still releases its machines. - *What is adopted.* A claim whose `spec.cluster` is the cluster key, whose name carries the `-node` ordinal, whose phase has not ended (not `Failed`, not `Released`) and which is not being deleted. Pending, `Bidding` and `Provisioning` claims are adopted too: the NodeClaim controller owns their end, and the given-up rule tears them down when it writes `Failed`. An ended claim is never adopted; the `Released` rule still deletes its Node. - *How.* A `nodemarket.Host` is built for that name exactly as `Provision` would build it, then `Host.Adopt(claim)` marks it provisioned WITHOUT writing the claim, the policy or the secret: the claim exists, so its policy was written before it (Provision writes the policy first); a `Joining` or `Running` claim's guest already fetched its secret, so the secret is marked written; an earlier phase's secret may have been held back for a link, so it is written on the next `SyncSecrets` (an identical rewrite at worst). The JWT role is ensured once from `status.grant` on the next pass, as for any node whose grant this process has not yet seen. The host is retained at its ordinal, marked live, counted in `started` (so `Provision` never takes the primary path over an adopted `node1`), and the snapshot is rewritten. A never-provisioned primary host is dropped when adoption creates or meets the cluster's entry, so it cannot be counted in flight. - *Then every rule applies unchanged.* confirm (`ConfirmJoined`, `markJoined`), the node loss rule, per-node scale-in (`Held` is set from the held names), teardown to zero, `inFlight` (an adopted `Joining` claim with no Node is in flight, a reported one is not, so each adopted node counts once) and `nextOrdinal` (held names are present, so the next claim is named past them). - *Not proven live:* a prod deploy restarting the provider with market nodes up, the guests re-registering within the minute the loss rule allows after the cluster's group comes back (the node lifecycle controller starts the NotReady clock when the group restarts, not when it stopped). With the gateway half above, both the gateway and the provider can restart without reaping a live market node (`internal/providermain/adopt.go`, `internal/nodemarket`). **What the guest runs** (`internal/atproto/cloudinit`, `K3sJoinModule`, built in Go). The module writes the resolver, the sysctl file, `/etc/socialweb/node/join.json`, the unit files and one pinned-download bootstrap shell. The bootstrap fetches `/downloads//socialweb-node-` from the gateway's web app, checks the sha256, installs it 0755 and execs `socialweb-node units start`; everything after that is the one Go binary `cmd/socialweb-node` (`internal/nodeagent`), built on the shared `internal/systemdunit` library that HostProcess/hostsvc also use. The pin is the contract's `spec.nodeJoin.nodeAgent` (`ComputeContractNodeJoin.NodeAgent`), filled from the `socialweb-node-x86_64` and `socialweb-node-aarch64` downloads the web app serves. Every unit carries `StartLimitIntervalSec=0`. | unit | after | does | |---|---|---| | `socialweb-node-bootstrap` | `network-online` | fetch and verify the pinned `socialweb-node`, then `units start`; oneshot | | `socialweb-node-prep` | `network-online`, `setup-secrets` | `socialweb-node join`: workload-token exchange, OpenBao JWT login, KV read, the `0600` files and `meshd.env`; then supervise the OpenBao and apiserver dumbpipe links | | `socialweb-node-dns-link` | prep | `169.254.20.10/32` on `swdns0` | | `socialweb-pod-masquerade` | prep | `net.ipv4.ip_forward=1` and the cluster-CIDR-only SNAT rule | | `socialweb-pod-cni` | prep | poll the Node's podCIDR, link the CNI plugins, write the bridge conflist with `encoding/json` through a rename | | `socialweb-node-diagnose` | prep | print containerd's CNI status to the console | | `socialweb-meshd-install` | prep | fetch the pinned meshd from the gateway's web app, verify and install it 0755 | | `socialweb-meshd` | install, prep | run meshd with the kubelet kubeconfig and secret | | `socialweb-k3s-agent` | prep, DNS link, masquerade | `k3s agent` as before | The join's credentials are fetched by the Go agent itself: it runs `guest-login.sh token ` for the claim's audience, logs in at `/v1/auth//login` with `{role, jwt}` and `--cacert openbao-ca.pem`, reads `/v1//data/`, and requires `token`, `nodePassword` and `serverCA`. It writes `server-ca.pem`, `token`, `node-password` and `apiserver.env` under `/etc/socialweb/node/` and `/etc/rancher/node/password`, all `0600`; the token is the `K10` form unless the stored token already starts with `K10`. The advertise address, port and ticket are validated before they reach a unit's command line. The k3s agent is ordered after prep, the DNS link and masquerade, and never after meshd, its install or the CNI: those wait for the Node the agent registers. Gone: the `lib/cloud-init-plugin-k3s-join` TypeScript module, its three `0700` scripts and four units. The separate `setup-secrets`, k3s-install and iroh modules remain cloud-init shell modules; the join path's own logic is the Go binary. **MEASURED locally end to end, 2026-10-07** (this reading predates the Go node agent; section 15 records the rewrite's reading; the plan 0005 M3 live gate, a Go test that plan 0026 deleted, PASS 753.57s, finished 23:55:13Z; the vendored market network held up by `accept.sh`'s `ACCEPT_HOLD_DIR`, and the deployed control-plane container `socialweb-k8s:m3live` built from the tree running against it, so prod's own entrypoint and provider are what was measured). One Deployment on a tenant cluster took the whole path: both ClusterLinks Listening and the cluster Ready 23:49:18Z, the claim written 23:49:21Z after them, `status.grant` 23:49:24Z, the OpenBao JWT role the same second, a firecracker guest booted from the RFP's cloud-init 23:49:27Z, the `--secrets` bundle fetched 23:49:32Z, the OpenBao JWT login over the openbao link and the KV read 23:49:37Z, the tenant Node Ready with the workload pod Running on it and the claim `Joining` 23:49:47Z - 29s after the Deployment was applied - then a control-plane restart with the node up, which adopted the claim (same uid, node Ready, guest up, no `vm.delete`) for 2m30s, and then the workload's deletion, the claim's deletion, `vm.delete` accepted at 23:55:11.733Z, the guest gone 1s later and the cluster back to zero with the control plane Ready. Zero ssh sessions: no line of the web app's log names one, and the guest's sshd records only the compute provider's own unauthenticated pre-auth TCP probe of port 22. Three defects, all found by this gate and all closed with tests watched to fail first: the web app of the time re-derived a claim's cluster key instead of taking `spec.cluster` from the provider (87562e8, main 657a915); the requester waited out its firehose when a bidder that had already dispatched the accept answered `submitAccept` with no receipt, ending `receipt_invalid` with a VM nobody could release (atproto-market a1b467e: the winner's repo is asked for the `market.receipt` naming this accept); and `setup-secrets.sh` appended `==` to every JWT payload, which GNU `base64 -d` rejects when the payload is 0 or 3 mod 4 (atproto-market 117ab30). Two things the gate had to name rather than assume: the image store's newest image had been rebuilt with no preinstall manifest and so carried no cloud-init, while the bidder reuses a stale image when it has none for its configuration (the gate now reuses only images that carry cloud-init and refuses to run when none does); and a node's bridge conflist has to land in `/etc/cni/net.d` with the plugins linked into `/opt/cni/bin`, because under `--flannel-backend=none` containerd reads the stock directories rather than k3s's agent directory (3919df6). **What the NodeClaim controller needs from the provider** (the interface, as built): 1. The ClusterLink `-openbao` in the claim's home workspace with `status.ticket` set (a bare dumbpipe ticket) BEFORE the claim is written; otherwise the claim fails at once. The ClusterLink `-apiserver` is read the same way and delivered to the guest in the same bundle when it holds a ticket, which it does on every cluster the provider rents nodes from; when it does not, the guest reads that ticket from the KV entry below instead, and joins serially. 2. A published path for the cluster OpenBao CA (a `NodeClaim` spec field such as `openBaoCAFile`, or a ClusterLink status field), or the guarantee that `/openbao/tls/ca.crt` is it and is readable by the gateway's core. 3. KV `socialweb-nodes/` with `token` (bare, or the full `K10` form), `nodePassword`, `serverCA` (the exact bytes of `server-ca.crt`, since k3s pins its hash), `apiserverTicket` (bare), `advertiseAddress` (IPv4) and `apiserverPort` (number or digit string). 4. JWT mount `jwt-` and role `` bound to `status.grant.subject` and `status.grant.audience`, as the contract says, and an OpenBao serving certificate valid for `127.0.0.1`. 5. `nodemarket.ConfirmJoined` accepts a claim in `Joining` or `Running`; the NodeClaim controller writes `Joining`, and the provider confirms the join from the tenant Node. ### Pods reach pods on other market nodes through socialweb-meshd Market guests cannot reach each other: each sits behind its own NAT or pasta, so k3s's flannel overlay has nothing to run over (the limit "A node cannot reach another node" in NEXT.yaml). The user's decision: QUIC datagrams over iroh, a tun device, each peer Node's podCIDR routed to that Node's iroh endpoint id, k3s with `--flannel-backend=none` and a bridge CNI for the node-local half. `rust/socialweb-meshd` (a cargo workspace at `rust/`, the first Rust in this repo) is that daemon. It is built and proved on hat-8, and the Go cloud-init module installs it on every market guest (the integration, below). A mesh probe scaled `test-cluster-alice-0001` from 0 to 2 market nodes, pods reached pods across the nodes both ways, and the node without CoreDNS resolved through 169.254.20.10; a NetworkPolicy denied cross-node traffic. The binary ships the meshd downloads (item 6 below), and the first two-node proof on the binary path is in section 15. **Identity.** One ed25519 secret per node, read from `--secret-file` as 64 hex characters (the `IROH_SECRET` encoding dumbpipe and the cluster links use), generated 0600 through a rename if absent. The endpoint id is the public key in lowercase hex; the did:key is multicodec `0xed01` + key, base58btc, `z` prefix - the same spelling as `internal/didkey`, and the Rust tests carry the same two vectors (the did:key spec vector `did:key:z6MkiTBz1ymuepAQ4HEHYSF1H8quG5GLVVQR3djdX3mDooWp` and the dumbpipe v0.39.0 `IROH_SECRET` -> endpoint id vector). The secret is node-local and never leaves the guest: a node's mesh identity lives and dies with the guest, so nothing about it goes through OpenBao. `socialweb-meshd identity --secret-file F` prints `{endpoint, did}`. **Peers are Nodes.** With `--kubeconfig` (the kubelet's own, `/var/lib/rancher/k3s/agent/kubelet.kubeconfig`) meshd merge-patches its own Node with `socialweb.computer/mesh-endpoint=` and `socialweb.computer/mesh-did=`, lists and watches all Nodes, and relists (rebuilding the client, so a rotated kubelet certificate is picked up) after every watch. The route table, computed by `RouteTable::compute` from a Node list and unit-tested on Node lists: - every OTHER Node with `spec.podCIDRs` (else `spec.podCIDR`) and a valid `mesh-endpoint` is one route per pod CIDR to that endpoint; a Node missing either is skipped silently (it has not joined yet); - this node's own pod CIDRs are never routed, and come from its own Node object; - rejected and reported in `/status`: a malformed endpoint, an endpoint equal to this node's own, a `mesh-did` naming a different key than `mesh-endpoint`, a pod CIDR overlapping this node's, and BOTH of two peers whose pod CIDRs overlap; - `socialweb.computer/mesh-addrs` (comma-separated `ip:port`) is a read-only dialing hint; meshd never writes it. It is how the tests dial directly with discovery off. `--peers-file` replaces the kube source with a NodeList JSON file (`kubectl get nodes -o json` shape), re-read when its mtime changes, and writes nothing. It exists for the network-namespace proof. **The node authorizer does NOT let a node list Nodes.** Measured against k3s v1.36.4+k3s1 with a `system:node:node-a` client certificate minted from the cluster client CA: `list nodes` answers 403 `node 'node-a' cannot read all nodes, only its own Node object` (AuthorizeNodeWithSelectors). Every tenant cluster therefore needs `rust/socialweb-meshd/deploy/node-reader-rbac.yaml` - ClusterRole `socialweb-meshd-node-reader` (`nodes` get/list/watch) bound to group `system:nodes`. With it the same identity lists, watches and patches its own Node's annotations (node authorizer plus NodeRestriction allow annotations on one's own Node), and patching ANOTHER Node is 403, which meshd turns into exit 4. Until the binding exists meshd logs `list_nodes_forbidden` with the manifest's path, stays not ready, and retries. The binding is ours and installed by us, so it belongs in the addon baseline, not intent. **The Kubernetes client is hand-rolled, not kube-rs.** meshd makes three calls (list, watch, merge patch). iroh already brings reqwest, rustls and ring, so a client over those plus serde_yaml for the kubeconfig adds almost nothing to a static musl binary; kube-rs would add k8s-openapi and a second TLS configuration. The kubeconfig reader follows `current-context`, resolves file paths against the kubeconfig's directory, takes `*-data`, `token` and `tokenFile`, and REFUSES `insecure-skip-tls-verify` and a cluster with no CA, because the Node list decides where pod traffic goes. `tls-server-name` is refused as unsupported. **Data plane.** A tun device (`--tun-name`, default `swmesh0`, `IFF_TUN|IFF_NO_PI`, so one read is one IP packet). The tun takes the network address of each own pod CIDR as a host address (`10.42.1.0/32` for `10.42.1.0/24`, the flannel.1 convention; host-local IPAM never hands out `.0`), and each peer pod CIDR is routed `dev swmesh0 src `, so traffic the node itself originates leaves with a source inside its own pod CIDR and the reply comes back through the mesh. One IP packet is one QUIC datagram, with no header: the connection already authenticates the sender's endpoint id. - outbound: longest-prefix match of the destination over peer routes; a destination with no route, or a source outside this node's own pod CIDRs, is dropped and counted by reason; - inbound: the sending endpoint must have a route, the source must be inside THAT endpoint's pod CIDRs and the destination inside ours, else dropped and counted; a connection from an endpoint no Node routes to is closed at accept (`connections_refused`); - connections are dialed lazily on the first packet for a peer, kept, and reused in both directions (an accepted connection is used for sending too); a per-peer queue holds 512 packets, a dial has 15 s, a failed dial backs off 1 s doubling to 30 s and drops packets meanwhile; a peer leaving the Node list has its connection closed and its routes removed. ALPN `socialweb.computer/meshd/0`. - The node-local half is untouched: own pod CIDR traffic never reaches the tun. **The tun MTU is 1160.** noq's largest datagram is `current_mtu - (1 flag + 8 CID + 4 PN + 16 AEAD tag) - 9 (datagram frame bound)`, and `current_mtu` is the minimum over a connection's live paths, starting at QUIC's 1200. A relay path stays at 1200, so with relays on the floor is 1200 - 38 = 1162; 1160 leaves two bytes. Measured: `max_datagram_size` 1162 on a 1228-byte underlay (a 1200-byte UDP payload path) and 1414 on a 1500-byte one after MTU discovery. The bridge CNI's `mtu` must be 1160: a don't-fragment packet above it is refused by the node with frag-needed (proved), and a fragmentable 1400-byte ping crosses as fragments (proved). **A crashed peer is reachable again in about 15 s.** A peer that is SIGKILLed and restarts with the same key leaves the other side holding a connection nobody answers; one-way traffic waits for that connection's idle timeout. Measured 30.8 s at iroh's default 30 s idle timeout and 15.4 s after meshd set it to 15 s (three of iroh's 5 s heartbeats). A clean stop sends CONNECTION_CLOSE and the peer re-dials on its next packet. **Relay and discovery** are iroh's n0 preset by default (pkarr publish, DNS lookup, n0 production relays), since guests have outbound internet: `--relay default|disabled|`, `--discovery n0|none`, `--bind ip:port[,...]`. No inbound port is needed. The proofs run with relays and discovery off in network namespaces; the n0 path itself is iroh's and was not exercised here. **Health and lifecycle.** JSON log lines on stderr (tracing, `--log-filter`, default `info,iroh=warn,...`); `GET /status` (JSON: identity, own pod CIDRs, routes with connected flag and max datagram size, installed kernel routes, rejections, counters), `/metrics` (Prometheus text) and `/healthz` on `--metrics-listen`, default `127.0.0.1:9469` (a non-loopback address is warned about). Ready - and `READY=1` to `$NOTIFY_SOCKET` - means the tun is up, the own Node is seen with a pod CIDR, its annotations are visible through the watch (kube mode), and the routes synced. Exit codes: 0 SIGTERM/SIGINT, 1 a task ended, 2 usage, 3 secret or peers file, 4 kubeconfig refused or own-Node patch forbidden, 5 tun create/up/read, 6 iroh bind, 7 status bind. A missing kubeconfig, an unreachable apiserver, a forbidden list and an unregistered own Node are retried, not fatal. **Build.** `rust/build.sh [x86_64] [aarch64]` writes `rust/dist/socialweb-meshd-` and `socialweb-meshd-.sha256` (sha256sum format): static musl, ring's C compiled by `clang --target`, linked by the toolchain's `rust-lld`, release profile `opt-level=s`, fat LTO, one codegen unit, `panic=abort`, stripped, with the checkout, registry and sysroot paths remapped. Two different target directories on hat-8 produced identical bytes; identity across hosts is not measured, so the pin a user_data carries is the `.sha256` written next to the binary it ships. blake3 is forced to its pure-Rust backend because its NEON C includes libc headers and there is no aarch64 musl sysroot. `rust/dist/` is not committed. `deploy/check.sh` runs `cargo fmt --check` and `cargo test` in `--local` and `--full` (skipped with a reason without cargo; `--full` fails instead), and adds the network-namespace and k3s proofs in `--local`/`--full` when `sudo -n` works. **The integration.** Every guest piece is written by the Go cloud-init module (`internal/atproto/cloudinit`, `K3sJoinModule`) into the `user_data` the RFP flow carries; nothing reaches a guest any other way. Decisions not in the meshd author's list are marked "decided by Claude, user may veto". 1. *Server.* `clusterstack.Spec.ServerArgv` adds `--flannel-backend=none --cluster-cidr=10.42.0.0/16` when `MarketDialled` (a market cluster, the only kind since the guest substrates were retired). Decided by Claude, user may veto: the cluster CIDR is named explicitly, at k3s's own default (so a running cluster's CIDR does not change), because the guest's masquerade rule names it and must not depend on a k3s default; kube-router's network policy controller is KEPT (no `--disable-network-policy`), since it works on pod IPs over any bridge and the meshd inbound check keeps pod source addresses. The real k3s v1.36.4 binary accepts the flags, and a live zero-node market cluster read `--allocate-node-cidrs=true --cluster-cidr=10.42.0.0/16` on the controller manager k3s runs, so a Node gets the `spec.podCIDR` the bridge waits for. 2. *Bridge CNI on the Node's pod CIDR.* host-local has no "range from the Node" mode (that is Calico's IPAM; flannel feeds host-local through its own `subnet.env`), so decided by Claude, user may veto: `socialweb-pod-cni` (after `socialweb-node-prep`, whose `join` writes `NODE_NAME`) polls `k3s kubectl --kubeconfig /var/lib/rancher/k3s/agent/kubelet.kubeconfig get node $NODE_NAME -o jsonpath={.spec.podCIDR}` every 2 s for 3 minutes (the node authorizer lets a kubelet read its own Node), refuses anything that is not an IPv4 CIDR, and writes `/var/lib/rancher/k3s/agent/etc/cni/net.d/10-socialweb-bridge.conflist` through a rename: `bridge` `cni0` (`isGateway`, `ipMasq: false`, `hairpinMode`, `mtu: 1160`) with `host-local` `ranges [[{subnet: }]]` and a default route, then `portmap`; cniVersion 1.0.0, the plugins k3s bundles. Until it exists the kubelet reports the network not ready and nothing is scheduled there; `Restart=on-failure` retries a Node that has no CIDR yet. Stated limit: the conflist is written once; a Node deleted and re-registered with a different CIDR is not followed (the CCM deletes a Node only when its claim is gone, which also releases the machine). 3. *Masquerade only what leaves the cluster.* `socialweb-pod-masquerade` (oneshot) sets `net.ipv4.ip_forward=1` (also persisted in `/etc/sysctl.d/90-socialweb-node.conf`) and adds, once (`-C` before `-A`), `iptables -w -t nat -A POSTROUTING -s 10.42.0.0/16 ! -d 10.42.0.0/16 -m comment --comment socialweb-pod-egress -j MASQUERADE`; `iptables` joins the module's packages. 4. *Node reader RBAC in every tenant.* `internal/tenantnet` embeds a byte-identical copy of `rust/socialweb-meshd/deploy/node-reader-rbac.yaml`, and the provider's tenant add-on pass (`internal/providermain/eso.go` `tenantAddOns`, run by the bootstrap on every provider process's first Ready of each cluster) server-side applies it under the `socialweb-addons` field manager BEFORE it needs the cluster vault, then measures the baseline. Live: a `system:node:` in `system:nodes` is denied `list nodes` before the apply and allowed after it (`RBAC: allowed by ClusterRoleBinding "socialweb-meshd-node-reader"`). RBAC objects are not workloads, so they never enter intent. 5. *meshd itself.* With a pin, the module writes `socialweb-meshd-install` (oneshot, after `network-online`, `Restart=on-failure`): it picks the pin for `uname -m` (x86_64/amd64 or aarch64/arm64), keeps `/usr/local/bin/socialweb-meshd` if a node image already put a binary with that sha256 there, and otherwise downloads `/downloads//socialweb-meshd-` into a temporary file beside the target, refuses bytes whose sha256 is not the pin, and renames it in 0755. The unit is `rust/socialweb-meshd/deploy/socialweb-meshd.service` byte for byte, plus the drop-in `socialweb-meshd.service.d/10-socialweb-node.conf`: `After=`/`Wants=` the install and `socialweb-node-prep`, `StartLimitIntervalSec=0`. The `join` process in `socialweb-node-prep` writes `/etc/socialweb/node/meshd.env` (`SOCIALWEB_MESHD_NODE_NAME=`, `SOCIALWEB_MESHD_KUBECONFIG=`), since it is the process that learns the node name. `socialweb-k3s-agent` is ordered after `socialweb-node-dns-link` and `socialweb-pod-masquerade` and NEVER after meshd, its install or `socialweb-pod-cni`: all three wait for the agent's Node, so ordering the agent after any of them is a deadlock (`internal/nodeagent/units.go`). 6. *Where the binary comes from.* The gateway's web app serves pinned downloads at `/downloads//` (`internal/gatewayhttp/downloads.go`, `LoadPinnedDownloads`): every file in its downloads dir with a `sha256sum`-format `.sha256` sidecar, held in memory, served only at its own hash and name with `cache-control: public, max-age=31536000, immutable` and `etag ""`; a sidecar that names another file, or bytes that do not hash to it, stop the gateway at startup. The meshd pin is the sidecar hash of the shipped `socialweb-meshd-x86_64` and `socialweb-meshd-aarch64` (`MeshdPinsOf`), and the download base is the GatewayAPI component's `publicOrigin`, or in a market network the web app's local address as a guest reaches it (`guestnet.HostAddress`, `192.0.2.1`, the address at which a guest reaches its host's loopback through pasta). The downloads come from the socialweb-computer binary: its asset `socialweb-computer-k8s-downloads` (versions.json, built by `scripts/build-web-app-downloads.sh` in the release build and pinned by its tree sha256) holds both meshd builds and both `socialweb-node` builds, each with its sidecar, and the Gateway root hands the extracted tree to the GatewayAPI component as its downloads dir. The NodeClaim mapper writes the node-agent pin into the contract's `spec.nodeJoin.nodeAgent`; the bootstrap fetches them by the same hash. The Gateway root names the two meshd builds as required downloads (`gateway.GatewayRequiredDownloads`), so a GatewayAPI component that would offer no meshd, or has no download base, fails to start (`internal/services/gatewayapi/config.go`). The guest image does not embed meshd; the install path that keeps an embedded binary is ready for when it does. Not carried by the mesh: node addresses. Traffic to another node's InternalIP (NodePort on another node, hostNetwork pods, kubelet ports) has no route; apiserver-to-kubelet and apiserver-to-pod traffic already rides k3s's own agent tunnel (`--egress-selector-mode cluster`). Every market node reports the same InternalIP (172.30.0.2, its pasta address), so the apiserver dials a kubelet by its Hostname address (`kubelet-preferred-address-types=Hostname`), which the k3s join module makes equal to the node name; the tunnel routes a host name to the agent session of that node. IPv6 pod CIDRs are routed and checked by the code but untested. ### Every market node resolves names on its own CoreDNS A Pod's resolver was the cluster's one CoreDNS on one node, so on a multi-node cluster every Pod off that node failed to resolve (NEXT.yaml limit, measured on five nodes). Now the resolver is on the Pod's own node, so it needs no cross-node path at all, the mesh included. - *The address.* `socialweb-node-dns-link` (oneshot) creates the dummy interface `swdns0` and puts `169.254.20.10/32` on it, idempotently. Decided by Claude, user may veto: a guest unit and not an init container, because the address is node plumbing that must exist before the kubelet starts Pods that resolve through it, and a tenant-side init container would need `NET_ADMIN` on the host network. - *The kubelet.* `socialweb-k3s-agent` passes `--kubelet-arg=cluster-dns=169.254.20.10` and `--node-label=socialweb.computer/node-local-dns=true`, and is ordered after the address unit. The server's `--cluster-dns` is not used: it is also the ClusterIP of k3s's `kube-dns` Service, which must be inside the service CIDR. `resolv-conf` is left at k3s's choice (`/etc/resolv.conf`, which the module writes with the provider's resolver and public fallbacks, and which has no loopback entry). - *The DNS.* `internal/tenantnet/manifests/node-local-dns.yaml`, applied with the node reader in every tenant: ServiceAccount, ClusterRole (list/watch endpoints, services, pods, namespaces, endpointslices), binding and Corefile ConfigMap `socialweb-node-dns` in `kube-system`, and a DaemonSet of the same name: host network, `dnsPolicy: Default`, `priorityClassName: system-node-critical`, tolerating every taint, selected by `socialweb.computer/node-local-dns=true`, image `rancher/mirrored-coredns-coredns:1.14.6@sha256:1ba6f472...`, k3s's CoreDNS security context, probes on `169.254.20.10:8080/health` and `:8181/ready`. The Corefile binds only `169.254.20.10`, serves `cluster.local` and the reverse zones from the apiserver through the `kubernetes` plugin (in-cluster config: the `kubernetes` Service's ClusterIP, which kube-proxy on the node DNATs to the advertise address the apiserver link carries on `lo`), and forwards everything else to `/etc/resolv.conf`. - *Why the label.* Decided by Claude, user may veto: the DaemonSet is applied in EVERY tenant regardless of substrate, and the label is what keeps it off a node that has no `169.254.20.10` (a guest-substrate node, a node that joined with an older user_data): there it would crash-loop on `bind`. The address, the label and the cluster CIDR must stay equal across the cloud-init module, `internal/nodeagent` and `internal/tenantnet`. - *Baseline, never intent.* The DaemonSet is applied by `socialweb-addons`, so the baseline measures it (read live at zero nodes), and a DaemonSet is never intent anyway (section 10); its pod leaves its node idle for scale-in. - *k3s's own CoreDNS stays.* Decided by Claude, user may veto: `--disable=coredns` is not passed. The `kube-dns` Service (`10.43.0.10`) keeps answering anything that names it explicitly, and its pod is a k3s add-on that never holds a node. Retiring it is a separate change. - *Stated limits.* A node that has just become Ready starts its DNS pod and its first workload side by side; a workload that resolves before the DNS pod is ready fails (the image is seeded, so it usually wins, but nothing orders the two). The fix is a startup taint the node lifts once its DNS answers; open. No live multi-node reading yet (ACC-MESH). **What changed on prod at the deploy after 2026-10-07** (history: the gateway then ran as a container and the web app was TypeScript). The provider builds each cluster's argv when it spawns the group, and a deploy restarts the container, so every market cluster's k3s server comes back with `--flannel-backend=none --cluster-cidr=10.42.0.0/16`, and the first Ready of each cluster in the new process re-runs the bootstrap, which applies the node reader and the per-node DNS and re-measures the baseline. Measured 2026-10-07 on hat-8 with k3s v1.36.4+k3s1: a `--disable-agent` server started with the default (vxlan) backend, stopped, and started again on the same datastore with `--flannel-backend=none --cluster-cidr=10.42.0.0/16` answered `/readyz` `ok` in 2.1 s, logged no `critical configuration` and no fatal line, and still held the ConfigMap written before the switch; at zero nodes there is no flannel state anywhere (the backend lives only in the agent and its Node annotations). A cluster with flannel nodes still up would keep them on vxlan with no path to new nodes; prod read zero nodes at its last check. The web app had to ship in the same deploy: its image carried the meshd pair (`build.sh k8s` built it then, which needs cargo, rustup's `x86_64-unknown-linux-musl` and `aarch64-unknown-linux-musl` targets and clang on the build host, or `KCP_MESHD_DIST`), and `PUBLIC_ORIGIN` is already set by `scripts/deploy.sh`. Image size: 1,711,865,393 bytes, +17.2 MB for the two binaries. ### `k3s server` argv ``` --disable-agent the keystone --disable=traefik leaves the manifest controller free otherwise --data-dir --datastore-endpoint REQUIRED; empty is a silent new cluster --egress-selector-mode cluster --advertise-address what a node dials --bind-address NOT the advertise value. k3s defaults it to 0.0.0.0, which is EVERY tenant's apiserver on every interface of the host. Bind is loopback; advertise is an address the GUEST can reach, and the two being separate is the whole point: a node that is a guest dials the pasta gateway, never the loopback. Making them one value is correct only for a node on the host, which is the mechanism this design refuted --tls-san repeated; carries the gateway address, so the certificate covers what the node actually connects to --advertise-port the `k3s-secure` hold the apiserver front owns (section 8). A separate advertise lease once existed for a forward that this cluster does not have, and it is retired: dropped from an old lease file on open, refused on allocate. Measured with the two split, before the front: the EndpointSlice reads : -- 63014, connection refused, while the listen port answers 401 -- coredns sits 0/1 for five minutes, local-path-provisioner and metrics-server exit 2 into CrashLoopBackOff, and kubectl logs answers `proxy error ... code 502`. With one port: three system pods 1/1 with zero restarts, and kubectl logs arrives through the apiserver's own tunnel --https-listen-port `k3s-slot-`, behind the front; never advertised --apiserver-port `k3s-apiserver-`, the slot's loopback apiserver listener --token-file <0600 path> --write-kubeconfig /server/cred/admin.kubeconfig --flannel-backend=none market clusters only: pods cross nodes through socialweb-meshd --cluster-cidr=10.42.0.0/16 market clusters only: named because the guest masquerades it --kube-apiserver-arg=kubelet-preferred-address-types=Hostname market clusters only: every guest reports InternalIP 172.30.0.2, so the apiserver dials the kubelet by node name --kube-apiserver-arg=audit-policy-file / audit-log-path / -maxsize / -maxage / -maxbackup --kube-apiserver-arg=service-account-issuer, api-audiences --kube-controller-manager-arg=secure-port, --kube-scheduler-arg=secure-port, --kube-cloud-controller-manager-arg=secure-port --kube-scheduler-arg=config /kube-scheduler.yaml market clusters only: podInitialBackoffSeconds and podMaxBackoffSeconds at 1 s and leaderElect false, from a file this host writes before the server starts ``` A direct `--audit-policy-file` on the server is SKIPPED by k3s with only a warning, which leaves the cluster SILENTLY UNAUDITED and starting normally. The audit flags must ride `--kube-apiserver-arg`. An unarmed kcp is the failure mode nobody sees from a green suite. The next three paragraphs were written for the retired local substrates (a local process and a guest of this host); a market node dials the advertise address on its own loopback through the apiserver link (section 9). The rules they state still hold. **The address a node DIALS belongs to the substrate, and it is not the same address for both.** A node that is a local process reaches an address this host carries. A node that is a guest reaches NONE of them: it sits behind a tap in its own network namespace, and its only path to the host is the gateway its default route names - which is why a guest reaches the host's loopback through that gateway and cannot reach the host's LAN address at all. So the guest's dial target is built from the substrate's gateway rather than from an operator-chosen host address; k3s advertises that same gateway for this substrate; and the gateway is in `--tls-san`, because the guest dials it and a certificate that does not cover what the node connects to reads as an x509 error rather than as a missing SAN. The host-carried-address check is therefore narrowed to the substrates where it is true rather than deleted, and what replaces it for a guest is not another address property but POSITIVE EVIDENCE: `AssertAgentDialledOurPort`, run against the node's own account of what it dialled, because a node joining is not evidence that it joined THIS cluster. That control had been written and unreached since before the substrate existed. **Every guest gets its own address, taken from the range its cluster leases.** The shared constant was this host's own LAN address, handed to every node - one node works and two collide into two Node objects with one InternalIP, with flannel on each treating the other's pod CIDR as its own. A range is compared against the host's own networks before it is leased, so no guest is ever handed this host's address; the gateway is the range's address 1 and the node is `1+ordinal`, and `Lease.Host` refuses a number outside the range rather than returning one. ### The QEMU substrate **Historical.** Commit 9d85916 ("market only: the local substrates were retired") deleted `internal/nodeqemu` along with the other local substrates; no code in the tree boots a QEMU guest, and nodes come only from the market. What follows is kept as the record of the 2026-09-24 measurements and describes code that no longer exists. `internal/nodeqemu` was a second substrate behind the same `nodeHost` seam, selected by `SOCIALWEB_CLUSTERPROVIDER_SUBSTRATE=qemu`. It boots the SAME image as firecracker - the kernel, the initramfs and the rootfs `internal/nodefirecracker.LoadImage` resolves from the builder's configuration - because a node image is a set of portable files, not a firecracker artifact. The guest is started with a `qemu-system-x86_64` command line: ``` -name the per-node token the record is matched by (see below) -m -smp the machine size, from the image manifest -machine q35,accel=kvm:tcg kvm where the host has it, and a plain fallback rather than a refusal where it does not: the host contract checks for kvm, and a boot that finds none is a boot that is merely slow -display none -serial stdio -monitor none -kernel -initrd -append "" -drive file=,if=none,id=rootfs -device virtio-blk-pci,drive=rootfs -drive file=,if=none,id=nodedata,readonly=on -device virtio-blk-pci,drive=nodedata -netdev tap,id=net0,ifname=,script=no,downscript=no -device virtio-net-pci,netdev=net0,mac= ``` The rootfs clone is writable and the per-node identity disk is read-only, exactly as under firecracker, so the guest still finds its token on the disk by label rather than on any command line. **The network is the SAME mechanism, not a QEMU mode.** pasta gives the namespace child a link carrying one of the host's addresses; `nodevm.WireNamespace` creates the tap and bridges it onto pasta's own link, and refuses to configure the VMM until the namespace has been observed to carry that link. Both substrates start pasta the same way - `nodevm.PastaArgv` - and both exec the same `kcpagent namespace ` inside it; the params file's `kind` field decides which VMM the agent runs. There is no `-netdev user`, no host-forwarding rule and no QEMU-only address: the guest dial address, the gateway, the tap, the bridge and the per-node address all come from the same range `internal/subnetallocator` leases to the cluster, so a QEMU guest is addressed exactly as a firecracker guest and reaches the host's apiserver only through the leased gateway. **The record's identity is the node name on the command line.** `internal/noderecord` matches a token SUBSET against `/proc//cmdline` after confirming the pid's start time. Firecracker's identity is its API socket; QEMU has no API socket, so its identity is `{"-name", }` - the node name is already per cluster and per node, qemu is started with it, and a record whose identity does not match is NOT OURS and is never signalled. A gateway that restarts reaps a QEMU guest the previous one booted by reading the same record and matching the same `-name`, which is the whole of the "reap by identity" contract. **What is refused, and why each refusal is a value that cannot be in force.** `qemu` demands the QEMU binary, the node agent and the image store and configuration, exactly as `firecracker` demands its own binary; `checkPlacement` refuses a qemu placement missing any of them, and it refuses a firecracker-only input under qemu (and a qemu-only input under firecracker) because a VMM binary for the substrate that was NOT selected is never executed, and an operator who set one and saw no complaint would believe it was carrying it. `--advertise-address` is refused under qemu for the same reason it is refused under firecracker: a guest reaches none of this host's addresses, so the value cannot be in force. **The shared half lives in `internal/nodevm`, and neither substrate imports the other.** The types that are true of both mechanisms (`Image`, `Guest`, `Network`, `NodeDocument`, `NamespaceParams`), the namespace plumbing that wires pasta's link to the guest's tap, the per-node identity disk, the rootfs clone, the console's fatal-marker watch and the `/proc` identity scan are one package. `cmd` is the only place that knows both substrates: it loads one image and builds one `nodeSubstrate`, and `nodeSubstrateFor` chooses the implementation. A cross-substrate import is a coupling this avoids rather than a convenience it takes. **A node image is built for one VMM, and refuses to boot under the other.** The builder's configuration carries `vmm` (`firecracker` or `qemu`, absent meaning firecracker), and the VMM is a recorded input: it is in `BuildInputs` and in `BuildConfig`, so two images of the same inputs for different VMMs have different fingerprints and never collide in the store. The image's manifest also carries `vmm`, so the image is self-describing, and a substrate refuses an image built for the other one - in `checkPlacement`, where the builder's own configuration is held to the selected substrate, and again in `nodeSubstrateFor`, where the loaded manifest is held to it. That refusal is the point: both VMMs boot the same kernel and hand the guest the same files, so a cross-substrate boot is not caught by the kernel, comes up, joins and reaches Running, and the only place the mismatch is written down is this field. What genuinely differs by VMM is the virtio TRANSPORT the guest's block device arrives on - firecracker hands it over virtio-mmio, qemu over virtio-pci - and the builder derives which transports the image must be able to load from the real module tree rather than asserting a rule: on kernel 7.2.7-200.fc44.x86_64 `virtio_pci` and `virtio_blk` are BUILT IN (present in `modules.builtin`, absent from `kernel/drivers/`), while `virtio_mmio` is a loadable module the firecracker configuration already stages. So a qemu image needs no transport module at all here and a firecracker image needs the one it already names; the artifact sets are otherwise identical. The builder refuses a VMM whose transport the named tree carries neither built in nor loadable, and refuses a VMM whose transport is a module the initramfs does not name. **Status: the QEMU argv, identity, placement and per-VMM image are exercised by unit tests, and the four-step proof runs under QEMU on this host.** `internal/nodeqemu` boots a real guest only when `SOCIALWEB_NODEQEMU_IMAGE`, `SOCIALWEB_NODEQEMU_AGENT` and a qemu binary are supplied along with pasta and `/dev/kvm`, and skips when they are not; a unit test boots a stand-in executable instead and asserts the command line and the published pid, so no test needs real QEMU. **The four-step proof was run under both substrates on 2026-09-24, from scratch, each one uninterrupted run, nothing deleted by hand.** The QEMU run used the qemu image at `kcp-image/out/sha256-3f9b63b082a3fbf7b92a49160c8e2697ed2b726dad8ca8048ef4171c9998e654` (built from `kcp-image/nodeimage-qemu.json`, `vmm=qemu`), an agent built from the same tree, and `/dev/kvm` present, so the guest was booted with `-machine q35,accel=kvm -cpu host`. All four transitions hold: the control plane reached Ready with zero nodes (STEP 1); `kubectl get nodes` was empty and the cluster stayed at zero across 10 passes 2 s apart over 20 s (STEP 2); a tenant workload applied at zero nodes gave `Provision, nodes=1`, the qemu guest booted and its k3s joined as a Ready node, and the tenant pod ran on `liveweb-node1` (STEP 3); deleting only the tenant workload gave `Teardown` after the hold, `kubectl get nodes` was empty again, and the control plane was still Ready and still listed its namespaces (STEP 4). The firecracker run was re-measured the same day against a freshly built firecracker image and the same agent, so the two numbers below share one method and one image. | measurement | firecracker | qemu | |---|---|---| | `INSTANCE_START_TO_READY_S`, host-timed from the VMM's instance record to the Node's Ready condition, resolution 250 ms | **5.460** | **7.256** | | wall time of the whole proof | 98.79 s | 114.39 s | | historical baseline (2026-09-23, older image, firecracker's `InstanceStart` timestamp) | 9.749 / 9.609 | - | The qemu guest is ~1.8 s slower to Ready than firecracker's on this host. The two start clocks are the same kind of host timestamp but not the same instant: firecracker's `StartNanos` is written when its API's `InstanceStart` succeeds, qemu's when the VMM process is started, so qemu's interval includes the process spawn and device setup that firecracker's does not - the real gap in guest boot is therefore somewhat smaller than the table's difference. One operational condition the first firecracker run exposed, and which is pre-existing rather than qemu-specific: the node's data directory must be short enough that `/fc.sock` fits the unix socket path limit (108 bytes), or firecracker's socket cannot be dialled and the guest never starts. The proof's default `TMPDIR` satisfies it; a very long temporary or state path does not. One prerequisite the qemu live boot exposed earlier: the host-side agent binary must be built from the same tree as the gateway, because the namespace parameters carry `kind` and `vmm` rather than firecracker's fields, and an older prebuilt agent refuses them. The guest-level measurements recorded above - the module set, the iptables failures, the Ready timing - were taken under firecracker and are not re-derived here; the qemu run above is the full-flow proof, not a re-measurement of those. ### The host platform Execution is Linux only; darwin and windows are seams that compile, vet and refuse by name. Plan 0006 M0-XOS-GATE made that true and keeps it so: `deploy/check.sh` builds and vets the tree for darwin/arm64, darwin/amd64, windows/amd64, windows/arm64 and linux/arm64 on every run (default tags, so no embedded blob is needed). The OS-neutral packages (`api/...`, `clusterprovider`, `cloudprovider`, `nodeprovider`, `nodemarket`, `materialiser`, `kcpclient`, `gateway`; the `vmruntime` and `hostsvc` interfaces, `kcpapply` and the ComputeVM controller core) import no go-systemd, netlink, firecracker SDK or `internal/fcapi`. An untagged OS-specific file fails the cross-OS build on the commit that adds it. `internal/hostos` is primitives only. Every darwin body that has no portable answer returns `" is unsupported on darwin; see plan 0012"` wrapping `errors.ErrUnsupported`; nothing implements macOS or Windows VM behaviour. | seam | Linux | darwin (plan 0012) | Windows (plan 0011) | |---|---|---|---| | `hostos` lock (`lock_unix.go`, `lock_windows.go`) | `flock` | `flock` (same file, `//go:build unix`) | `LockFileEx`/`UnlockFileEx` | | `hostos` process identity (`proc_*.go`) | `/proc//stat` start ticks and `/proc//cmdline`; `kill(pid, 0)`, `SIGTERM`, `SIGKILL`, `SIGHUP` | refuses: ErrUnsupported | `OpenProcess` + `GetProcessTimes`, the command line via CIM, `TerminateProcess`; no reload | | `hostos.BootID` (`bootid_*.go`) | `/proc/sys/kernel/random/boot_id` | refuses | refuses | | `hostos.LocalPortRange` (`portrange_*.go`) | `/proc/sys/net/ipv4/ip_local_port_range` | refuses | `netsh int ipv4 show dynamicport tcp`, falling back to 49152-65535 | | `hostos` process attributes (`sysproc_*.go`) | `DieWithParent` is `Pdeathsig: SIGKILL`; `NewProcessGroup` is `Setpgid` | `Setpgid`; `DieWithParent` is empty (ponytail: no Pdeathsig) | `CREATE_NEW_PROCESS_GROUP`; `DieWithParent` is empty (ponytail: a job object) | | `apiserverfront` hold (`fdstore_*.go`) | `[::]:0` dual-stack, `SO_REUSEADDR` set after bind | refuses (no hold) | refuses (no hold) | | `portscan` (`scan_*.go`) | `/proc/net` and `/proc//cmdline` | refuses | `netstat -ano` and the command line via CIM | | `guestdisk` rootfs clone (`clone_linux.go`, `clone_other.go`) | `FICLONE` reflink, else a `SEEK_DATA`/`SEEK_HOLE` sparse copy | plain copy | plain copy | | `check-host` facts (`internal/hostcontract`, `gather_linux.go`, `gather_other.go`) | the full host contract, firecracker included | the portable facts only | the same | The capability predicates `Firecracker`, `QEMU`, `NetworkMode`, `Accel` and `AccelWHPX` are gone from the tree: nothing declares or reads them. `check-host` names the VM runtime compiled into the build. The service-manager, VM-runtime and guest-network seams (`hostsvc`, `vmruntime`, `guestnet`) are Linux-only bodies behind interfaces; the Windows (HCS, HNS, SCM) backends are plan 0011 and macOS (vfkit, launchd) is plan 0012. See section 5, "Host processes and the VM lifecycle". What follows is the measurement of the four-step proof, on the local substrates retired since. **What was observed, and what was not.** On 2026-09-24 the four-step proof was run on this Linux host as the capability matrix: a Go test that plan 0026 deleted ran both substrates in one process, and both passed in 210.86 s -- firecracker `INSTANCE_START_TO_READY_S=5.423` and qemu `INSTANCE_START_TO_READY_S=6.915` (host-timed from each VMM's instance record to the Node's Ready condition, 250 ms resolution), each through all four transitions. The second substrate's run exposed a test-harness defect rather than a substrate one: `liveports.TakeForProcess` opened a slot lock and did not retain the handle, so a second subtest in the same process could take the first's slot and the two runs collided on ports. The handle was then retained for the process, and the runs used separate slots. A second, deeper defect turned a failed placement into an unbounded hang: the in-process child runner of the time (`processrunner.Process.Wait`) waited for the goroutines copying the child's stdout and stderr to reach EOF, not only for the direct child to exit, and a node's VMM is a descendant of pasta that inherits those pipes. Signalling pasta alone left the guest holding the write end, and the proof blocked for 37 minutes in `sync.Once.Do` inside `Wait` -- past its own context, which `Wait` does not observe. `WaitDelay` then bounded the drain to five seconds after the child exits, and a reaped process was not an error. MEASURED after both fixes: the matrix ran clean twice, both substrates in one process -- 210.86 s (firecracker 5.423 s, qemu 6.915 s) and 205.33 s (firecracker 5.166 s, qemu 13.489 s). Everything in this document that is measured was measured on Linux, and the Windows column above is **design and cross-compilation only**: no Windows binary was executed on this Linux host, so no Windows boot, no WHPX boot, no user-mode networking path and no Windows process-identity read has been observed to work. On 2026-09-24 `GOOS=windows GOARCH=amd64 go build ./...` and `go vet ./...` were clean, which proved the tree compiled and type-checked for Windows, and nothing more; 7c4f741's untagged reflink clone broke that, and plan 0006 M0-XOS-GATE restored it and gates it. The runtime paths that carry the most risk are the ones no compiler can check: the CIM/PowerShell command-line read (it shells out per lookup, and a localized or denied environment breaks it), `TerminateProcess` standing in for a graceful stop, `LockFileEx`'s byte-range lock against `flock`'s whole-file semantics, WHPX actually being available on a given Windows host, and slirp's `10.0.2.2` reaching the apiserver's loopback listener. Each is a place a Windows host must be measured before this document may claim it works. ## 10. The scale decision ``` observe -> decide -> perform ``` `internal/scale` is a pure `Decide(Observation, State, now, Options) -> Decision`. It asks whether ANYTHING is running, not whether a specific pod is. **The narrow predicate cannot scale to zero:** a predicate that can only fire upward cannot express it. So the intent test is deliberately coarse, and a list failure is an ERROR, never an empty observation - an unreadable cluster is not an idle one, and acting on that reading tears a node down under its own work. `internal/scaledriver` performs the decision and owns the one thing `Decide` cannot: the state a decision carries BETWEEN passes. `State.Nodes` is deliberately NOT remembered; it is re-read from the Node CRs, because those CRs are the idempotency record and a node that died must be seen to be gone rather than trusted from a variable. What is remembered is `IdleSince` - **without it there is no hold at all**, because `Decide` would see an idle cluster for the first time on every pass and never reach it. **The state advances only on success.** A driver that recorded a node as up because a call was attempted leaves a cluster with intent and no node, permanently. **Both halves of the observation come from ONE call.** The objects decide whether there is intent and the count decides whether a node exists; read at different moments they can disagree - a pod counted before its node's CR landed is a cluster that provisions twice, and the reverse tears a node down under its own work. **Node phase semantics, and where their boundary is.** `Pending` is written before the command exists and `Draining` on every teardown path, and `Running` is written only by `ConfirmJoined`, which the composition root calls after the tenant cluster's own node list reports the node. So `Running` means k3s registered it, not merely that a command is up. The scale decision does not read the phase at all - it counts the tenant cluster's Nodes - so a node stuck in `Pending` past the join timeout is a warning and not a wedge: the next pass re-observes and confirms it. The count is the tenant apiserver's own Nodes, which is what `kubectl get nodes` reads, so the number the decision acts on is the number an operator can check. That is the *dead* direction that matters: a node that died without its CR being updated would be counted alive by the CR route, which is the direction that stops a cluster ever reaching zero. **The tenant's Node object outlives the agent, so the teardown must delete it.** A k3s agent does not remove its own Node object when it stops - kubelet does that only when a cloud provider is configured, and none is. So a teardown that stops the process and skips the delete leaves the count above zero with no intent, and `Decide` calls teardown again on every pass for ever: the cluster is never observed at zero nodes, which is the objective. The delete belongs to the teardown, after the guest is stopped, tolerating NotFound as success; a provision deletes nothing. It cannot live in `nodelocal`, which has no tenant client, so it lives in the composition-level `scale.Nodes` that wraps the host - and a nil tenant client is an explicit refusal, never a skipped cleanup, because a skipped cleanup and a completed one look identical from a green suite. `SOCIALWEB_CLUSTERPROVIDER_SCALE_HOLD` is the down-direction grace. It is a parameter rather than a constant because the number has not been derived by measurement. **The decision carries a target total, and the boolean that reaches zero is untouched.** The narrow-predicate argument above is not weakened by counting: `Intent` stays a boolean, and a SEPARATE `Demand(o)` counts the work as a pod count, so an error in the count cannot make zero unreachable - it can only ask for the wrong number of nodes. `Target(o, opts)` is `ceil(Demand / PodsPerNode)` clamped into `[1, MaxNodes]` when there is intent, and `0` when there is not, which is the only place the ratio is applied. `Decision.Nodes` is therefore a target total rather than a delta, and the driver reconciles the cluster's actual count - re-read from the Node CRs, never remembered - to it. **Our own add-ons are a MEASURED baseline, and a cluster holding only them has nothing deployed.** Add-ons this provider installs into a tenant cluster are workloads like any other, and at zero nodes their pods are Pending, so they would otherwise be intent for ever: a cluster that had run one pipeline would never return to zero. The rule is a delta, measured rather than listed. The installer applies with its own server-side field manager, `socialweb-addons` (`clusterprovider.AddOnFieldManager`), and right after the apply the provider lists the Deployments, StatefulSets, DaemonSets and Jobs whose `managedFields` carry an `Apply` entry from that manager, hashes each one's LIVE spec as read back (so the apiserver's defaulting is inside the hash), and writes the result to `addon-baseline.json` in the cluster's data directory. That file is provider state; a tenant cannot edit it. `scale.Narrow` then drops, before `Intent` and `Demand` see them, every workload whose kind, namespace, name AND spec hash match an entry, and every pod whose controller resolves to one. A baseline workload whose spec has changed (the tenant scaled our operator to two) is NOT dropped: that is a tenant's request. A workload of the same name in another namespace is not ours. The baseline is re-measured on every add-on pass (each `Bootstrap`). There is no list of add-on names here for the same reason the k3s add-on rule has none. Non-workload objects (CRDs, RBAC, webhooks) are not recorded, because nothing the scale decision reads is one of them. The External Secrets operator now runs beside the control plane (section 5), the tenant holds its CRDs, ClusterRoles and webhook registration and no workload, and the measured baseline is EMPTY: the rule stays because it is what keeps any future add-on from pinning a node. **A pod and the workload that controls it are ONE unit of demand.** `Demand` counted a Pending pod and also its Deployment's replica, so three one-replica Deployments with their three Pending pods demanded six and, at one pod per node, asked for six nodes. `scale.Narrow` drops every pod whose controller chain (Pod -> ReplicaSet -> Deployment, or Pod -> StatefulSet / DaemonSet / Job) names a workload present in the observation; the workload's own want is the count. A bare pod, and a pod of a ReplicaSet no Deployment controls, still count as themselves, because nothing else counts them. The ReplicaSets are read only to resolve that chain; they are not a predicate kind and the `Observation` still ranges over exactly the five it did. **`MaxNodes` defaults to 10, and a ceiling below one node is REFUSED rather than clamped.** Named defaults are `DefaultMaxNodes = 10` and `DefaultPodsPerNode = 8`, resolved in `Options` the same way `hold()` resolves the grace, and settable per cluster as `SOCIALWEB_CLUSTERPROVIDER_SCALE_MAX_NODES` and `SOCIALWEB_CLUSTERPROVIDER_SCALE_PODS_PER_NODE`. The refusal is on `n < 1`, not on `n < DefaultMaxNodes`: a ceiling under one is not a smaller cluster, it is a configuration whose only consistent reading is "never place the work", while a ceiling of three is a smaller cluster and is taken. A five-job GitLab run measured what one node means for a runner with `concurrent = 5` (section 15); `scripts/deploy.sh` passes 10 and one pod per node, so a cluster serving this runner asks for the nodes the jobs need. **Scale-in is per node: a node holding nothing beyond the baseline for the hold is cordoned, drained and released, one per hold.** The rule has a pure half and a side effect half: - *What a node holds.* `scale.Loads(nodes, observation, replicaSets, baseline)` reads, per tenant Node, `Busy` (live pods bound to it that are not baseline), `Evictable` (live baseline pods bound to it: our add-ons' workloads by the measured baseline, and k3s's own add-ons by `AppliedByTheControlPlane`) and `Incoming` (a Pending pod with no node yet whose `status.nominatedNodeName`, `nodeSelector kubernetes.io/hostname`, or single-valued required hostname / `metadata.name` affinity names this node). Pods that FOLLOW the nodes are never counted on a node: a DaemonSet's pods and mirror (static) pods (`kubernetes.io/config.mirror`). Terminal pods are not counted. A Pending pod naming no node goes into `Placement.Unplaced` with its creation time. The observer computes this in the same call as the rest of the observation, so the per-node half and the intent half cannot disagree. - *The decision.* `scale.Decide`, with intent and the counted nodes above the target, keeps a per-node idle clock in `State.NodeIdleSince` (the first pass that saw the node with `Busy == 0` and not `Incoming`; a busy pass deletes it). It returns `Release` naming ONE node, `Nodes: count - 1`, when that node is held by this provider (`NodeLoad.Held`), idle, and idle for the whole hold (`SOCIALWEB_CLUSTERPROVIDER_SCALE_HOLD`, default 2 m). Among several it picks the emptiest (fewest `Evictable`), then the newest (latest `creationTimestamp`), then by name. A release restarts the idle clock of every node left, so three idle nodes over a target of one go one per hold: one at 2 m, the next at 4 m, the third kept. No release happens while a Pending pod with no node is younger than the hold (or has no creation time): the scheduler may be about to bind it to any idle node. A pod Pending longer than the hold with idle nodes present is not going to fit them, and holding every node for it would pin the cluster. The target is never undercut: the branch is only reached when `count > target`, and the target is at least one while there is intent. With NO intent the old path is unchanged: the whole-cluster hold, then `Teardown` to zero. - *The side effects (`internal/providermain/scalein.go`).* `scaledriver` performs `Release` through `scale.Releaser` (an actuator that is not one is `ErrNoReleaser`, never a whole-cluster teardown). `tenantNodes.Release` cordons the Node (`spec.unschedulable`), re-observes the cluster, and if the node took work between the decision's read and the cordon (Busy or Incoming) it uncordons and keeps it. Otherwise it evicts, through the Eviction API so disruption budgets hold, every live pod bound to the node that does not follow the nodes; a refused eviction uncordons and keeps the node. Then the release is `teardownHeld`, the path the node loss rule and the giving-up rule already use: claim deleted, the gateway's Requester releases with `vm.delete`, the node's vault role, policy and secret revoked, the tenant Node deleted. A node the provider uncordons is only one it cordoned itself; an operator's cordon is left alone. Evicted pods are not waited on: the tenant Node is deleted and their controllers recreate them elsewhere. - *Decided by Claude, user may veto:* one node per hold rather than every idle node at once (each release can move baseline pods onto the nodes left); the clock is in memory, so a provider restart restarts every node's hold (the safe direction); only nodes this provider process holds are released, and a market node that outlived a provider restart is held again by adoption (section 9, "A restarted provider never reaps; it adopts"), so it is released like any other; a failed drain uncordons rather than leaving a cordoned node counted. **A DaemonSet is never intent, and never demand.** A DaemonSet's desired count follows the nodes, so with one tenant DaemonSet any cluster that had a node kept intent for ever and never reached zero. `Intent` and `Demand` do not read `Observation.DaemonSets` at all, and skip every pod whose controller is a DaemonSet or that is a mirror pod, whether or not its DaemonSet was in the same read. A DaemonSet-only cluster is a cluster with no intent: it holds its nodes for the hold and goes to zero. The `DaemonSets` field stays in the observation because the baseline narrowing ranges over it. **A custom resource is intent only while an operator of ours that runs IN the tenant cluster has not reconciled it (the rule "workloads beyond the baseline plus our operators' CRs not yet reconciled" is the user's, the mechanism decided by Claude, user may veto).** `scale.Operators` is a registry of `{Group, Kind, ReadyCondition, InCluster}`. A CR counts only if its group/kind is registered with `InCluster: true` and it is not reconciled: its `ReadyCondition` is not `True`, or that condition or `status.observedGeneration` is older than `metadata.generation`, or it is being deleted behind a finalizer. Each such CR adds one to `Demand` and makes `Intent` true, so it asks for one node, on which its operator runs and reconciles it. An operator that runs beside the control plane reconciles at zero nodes, so its CRs need no node and are never registered as in-cluster. `scale.OurOperators` ships EMPTY: ESO runs beside the control plane, and no other operator of ours runs in a tenant cluster. Leftover namespaces, RBAC and ConfigMaps never hold nodes. `NodeObserver` lists each in-cluster kind through its `CustomResources` lister and refuses to observe at all when the registry names an in-cluster kind and no lister was supplied, so a future entry cannot silently never count. On the retired guest substrates the ceiling also met the cluster's own `/24`: every node took an address from the range `subnetallocator` leased. A market node takes no address of this host's. Plan 0026 deleted the Go tests that held these rules (three held market nodes over a fake tenant: the DaemonSet-only node went at 2 m, the node holding a k3s add-on pod at 4 m with that pod evicted, the node running work was kept). NOT yet read: a live market cluster releasing a node. The reading for this is a table case: the same observation that decided `Provision, Nodes: 1` before now decides `Provision, Nodes: ` with `MaxNodes` raised, and every existing case at `MaxNodes == 1` is byte-for-byte what it was. What is NOT yet read is a live cluster holding more than one node at once - `NEXT.yaml` AUTO-1 stays open for that, and `docs/plans/0002-scale-to-n-nodes.md` names the instrument: three concurrent pods, three joined nodes, and a return to zero. **A market that takes no bid backs the cluster off, exponentially, capped at ten minutes (decided by Claude, user may veto).** MEASURED on prod: with no bidder answering, the provider wrote a NodeClaim, the gateway's fulfiller got `no_bids` after its 30 s bid window, the claim ended Failed, `confirmHeld` gave the node up, and the next scale pass claimed again -- a new public `market.rfp` about every 50 s per missing node, 49 RFPs and 86 `no_bids` in one hour. The rule: - *What counts.* A claim that ends `Failed` with `status.contract.rfpUri` set and `status.contract.acceptUri` empty (`nodemarket.MarketDeclined`): the Requester asked the market and no bid was taken. That is the requester's whole `NO_CONTRACT_OUTCOMES` class -- `no_bids`, `bids_denied` and `policy_rejected` -- and all three loop identically, each re-ask a fresh public RFP, so all three back off. A claim that failed after an accept (the guest died, the join never forwarded) keeps today's behaviour: the market did answer, and the next pass asks again at once. So does every non-market substrate, which never reports a declined end. - *Why that shape and not a reason field.* The claim is classified from structured record URIs the NodeClaim controller already writes, never from `status.message` text. A new `status.reason` was rejected because `APIResourceSchema.spec` is immutable and, MEASURED (`internal/versions/versions.json`), a schema change does not reach existing APIBindings: every cluster home bound before the bump would prune the field and silently never back off. The Requester keeps `rfpUri` and writes no `acceptUri` on a `no_bids` end, and the NodeClaim controller copies both onto the claim. Status stays one writer per half: the NodeClaim controller writes the claim's status, the provider only reads it. - *The wait.* `scale.BackoffDelay(n) = 15 s * 2^(n-1)` for the n-th declined or failed end in a row, capped at `scale.DefaultBackoffCap = 5m`: 15 s, 30 s, 1 m, 2 m, 4 m, 5 m, 5 m ... One confirm pass that sees several declined claims counts ONE end, because they are one answer from one market at one moment. The wait runs from the pass that saw the end. `Options.BackoffBase` / `BackoffCap` default to `scale.DefaultBackoffBase` and `DefaultBackoffCap`. - *Where it lives.* `scale.State.Backoff` is pure state; `scale.Decide` returns `None` with `Decision.BackoffUntil` set while `st.Nodes < target` and the wait holds. The decision's reason says the cluster is backing off, until when (RFC 3339 UTC), how many ends in a row, and the claim's own message, and the provider's `cluster provider: node scale` line carries `backoffUntil`. `scaledriver.Driver.Declined(cluster, reason)` records an end (and logs `backing off market provisions for this cluster` with `until`, `wait`, `why`); `Driver.Joined(cluster)` clears it. The composition root (`wiring.go` `scale`) feeds both from `nodeHosts.confirm`'s outcome before the scale pass, `Joined` first so a pass that sees one node join and another declined still backs off. - *Resets.* A node of the cluster newly confirmed joined (the tenant cluster reports it and its claim reads Running; a node already joined does not re-reset on every pass), or the cluster's intent dropping to none (`State.Next` keeps the back-off only while the decision carries intent). A provision does NOT reset it: the count of ends outlives the provision it allowed, so the next end waits longer. - *Memory only.* The back-off is held in the provider process. A provider restart starts every cluster from no back-off, so the first pass after a restart may post one RFP at once; that costs one RFP per restart, not one per pass. `tenantNodes.retain` restores the primary slot when it is empty: dropping ordinal 1 unconditionally would leave a first node claimed again after the first was given up unheld, so its claim would then never be confirmed, never released, never counted in flight and never seen ending, and it could not back anything off. **A market node NotReady for `NodeLossGrace` is released, and its Node deleted (provider half; the user's number was 90 s, and 76c4c864 shortened it to 1 minute).** In `tenantNodes.confirmHeld`, after the given-up rule and the per-pass `SyncSecrets`, each held market node's tenant Node is read (`readTenantNode`). If its `Ready` condition is `False` or `Unknown` and `now - lastTransitionTime >= NodeLossGrace` (1 minute, `internal/providermain/nodes.go`), the node goes through `teardownHeld`, the same path a scale-down takes: the claim is deleted, its vault role, policy and secret are revoked, the NodeClaim controller has the Requester release the machine with `vm.delete` and drops its finalizer, then the tenant Node is deleted so the scheduler stops binding pods to it and evicted pods reschedule; the node is reported in `confirmOutcome.lost`, and the scale pass that follows in the same tick counts one node fewer and claims a replacement if the work still needs one. Decisions, made by Claude, user may veto: - *The clock is the Node's own `lastTransitionTime`*, written by the tenant cluster's node-lifecycle controller on the same host, so a provider restart does not restart the grace. A condition with a zero transition time falls back to the first pass that saw it not Ready, held in memory. - *Not covered:* a node the tenant cluster never reported (no Node object) -- the confirm and giving-up rules own it, however long it has been -- and a Node with no `Ready` condition yet (its kubelet has not posted a status). A Ready node is never lost, whatever its transition time. - *Held nodes only.* The rule acts on nodes this provider process holds. A market node that outlived a provider restart is held again by adoption at startup and on every pass (section 9, "A restarted provider never reaps; it adopts"), so the rule covers it from the first pass after a restart. - *It blocks the cluster's pass* for as long as `teardownHeld` waits for the NodeClaim controller (`ClaimReleaseTimeout`), exactly as the giving-up rule and the scale-down already do. **A market cluster's kube-scheduler runs with a 1 s pod backoff, written into the cluster's data dir before the server starts.** The scale-from-zero path has one wait this project does not own: kube-scheduler holds every pod it could not place -- and a cluster with zero nodes cannot place any -- for a backoff that starts at 1 s and doubles, capped at 10 s. The node that finally makes such a pod placeable does not cancel that wait, so the pod that waited for the first node can wait again after it arrives, which is the whole latency this milestone is about. Market clusters therefore name `--kube-scheduler-arg=config=/kube-scheduler.yaml`, rendered by `clusterstack.KubeSchedulerConfig` (pure) and written by `clusterprovider.WriteKubeSchedulerConfig` before the steps are built, exactly as the OpenBao config is; the file sets `podInitialBackoffSeconds` and `podMaxBackoffSeconds` to 1 and carries a `clientConnection`, because a `--config` file makes kube-scheduler IGNORE the deprecated `--kubeconfig` flag k3s still passes. A stack that dials no market has no zero-node window and its argv names no config file. **The backoff was not the 16 s; leader election was.** With the 1 s backoff deployed the first pod still waited 15 s after its Node was Ready. k3s `--disable-agent` starts kube-scheduler only once a node registers ("Starting controller" 02:53:17.22Z), and the new scheduler then waited out the leader lease its previous incarnation still held ("Attempting to acquire leader lease" 02:53:17.35Z, "Successfully acquired lease" 02:53:33.08Z, kube-scheduler's 15 s default lease). Each cluster runs exactly one k3s server, so there is nothing to elect: the same file now sets `leaderElection.leaderElect: false` (the `--config` file owns leader election, so a flag would be ignored). The 1 s backoffs stay, as they cost nothing and bound the wait once the scheduler runs. The two flags the plan preferred do not exist in this kube-scheduler: MEASURED against k3s v1.36.4+k3s1, `--pod-initial-backoff-seconds` and `--pod-max-backoff-seconds` are absent from kube-scheduler's own `--help` (which lists `--config` and the deprecated `--pod-max-in-unschedulable-pods-duration`), and a non-numeric value for the first is not rejected, so the flag form is not a flag this binary defines. The file form is read: an unreadable config (garbage bytes) makes the scheduler exit -- k3s's log records `scheduler exited` while the server itself stays up, so the failure an operator sees is a cluster whose pods are never placed, not one that failed to start. Both halves were measured against the real binary (a pod created at zero nodes was placed on the node that arrived, 7.7 s end to end including the cluster's start). What this does NOT claim: on this host a pod waiting for a node was requeued ~50 ms after the node became schedulable with the DEFAULT backoff in every shape tried (a node becoming Ready, a taint leaving it, a label arriving), so the 16 s read on prod is not reproduced here and this change bounds the worst case rather than an observed wait. ## 11. Teardown, and the gate Teardown is gated, because a teardown that races a running job is a lost write, and after rotation the objects are gone. **`status.nodeExpiry` decides whether the gate can compare:** ``` absent / unknown -> REFUSE. A gate that cannot compare must not proceed. explicit 0 -> PROCEED. There is no expiry to compare against. non-zero reachable -> refuse if the gate would outlive it ``` **Conflating absent with `0` makes a deliberately-unlimited cluster undeletable.** `0` means UNLIMITED (`clusterprovider.NodeExpiryUnlimited`). The trade: it removes the only automatic bound on an ORPHAN. Containment then rests on token-per-instance, the Requester's session accounting (`RequesterSettings.sessionCapSeconds`), and our own teardown path - **there is no timer backstop**. **A producer reports; it never guesses.** The producer surface cannot express an estimate: it returns a value *and whether it has one*, so "nothing to report" is a bit rather than a sentinel, and a source that forgets to report errs toward refusing rather than toward proceeding. `0` is the report of a node with no session cap at all (the local-process mechanism was one, before it was deleted), and it is a fact about that node, not a guess. That is why absent and `0` are both reachable and mean opposite things. A guessed deadline that is late reproduces exactly the silent loss the gate exists to prevent, and a guess is indistinguishable from a report in the schema, so it would remove the refusal the gate is. The report is written **only when the object carries no value**: the settled shape of the `Cluster` value is a roll-up over the node CRs, and a second writer that revises a report is the one direction that loses data. A stale earlier value can only refuse a teardown; it can never permit one that outlives the truth. **A gateway restart with live children: adopt by record, never by guess.** Every cluster member is a HostProcess unit, and its record is the HostProcess object, not a process scan: a restarted provider lists each cluster's member objects, adopts a Running one whose invocation matches the label `socialweb.computer/verified-invocation`, and re-places or re-gates the rest (section 5, "The per-cluster slice of member units"; `UnitsHost.Adoption`). An adopted unit has one owner, the object, so it cannot look alive to one owner and dead to another. Before the HostProcess units, a gateway restart reaped its children instead: a process group was found by its leased ports through `internal/portscan`, and a node by a `node.json` pid record; that path is gone with the local substrates. A market node is adopted the same way: its machine is the bidder's, its guest needs no session of ours, and its owner of record is the `NodeClaim` in the cluster's home workspace, which the NodeClaim controller alone releases and only on deletion. The claim is the single record, and a restarted provider holds it again from that record without writing it (section 9, "A restarted provider never reaps; it adopts"). A deleted or switched-off cluster is reaped by deleting its member objects (`UnitsHost.Reap`), and the HostProcess supervisor stops each unit; the provider waits until every member is Stopped before it drops the finalizers. Its Postgres object stays for an OFF cluster and goes with a deleted one. **Reap does not reach the store.** The cluster's store is its Postgres object (`pg-`), a unit of its own: deleting a cluster's members leaves it, and the purposes the members hold are the front's (section 8), none of them a node's, because a node binds no port of this host's. **One Host per cluster, for the life of the process.** A node actuator built fresh on each call cannot scale to zero: `Provision` in one pass and `Teardown` in a later pass act on different objects, so the teardown finds no handle, stops nothing, and the node keeps running. The composition root memoises one host per cluster and hands back the same one thereafter, under a mutex because passes may overlap. Memoisation also holds the cluster's `k3s-secure` lease for the process lifetime, which is what keeps `status.URL` meaningful after OFF: a port re-leased to another cluster is a silent misroute on this cluster's next ON. **A shutdown leaves the members running.** `Provider.Shutdown` (`internal/providermain/wiring.go`) stops no member unit and no Postgres object: the next provider adopts them. Its node-host reap skips the market substrate entirely: deleting a market claim is what makes the gateway destroy the machine, so a shutdown that reaped market hosts would release every rented node on every deploy (watched before the change: 2 claims deleted and both nodes' vault state revoked; none after). A cluster switched OFF or deleted still releases its market nodes through `reapCluster`. **Every cluster, not the one this process was told to join.** `SOCIALWEB_CLUSTERPROVIDER_JOIN` names a single cluster by its bare name and keeps working exactly as it did. `SOCIALWEB_CLUSTERPROVIDER_JOIN=*` is the mode the gateway needs, because its web app lets any atproto user create a cluster and NOTHING materialises one unless a process joins it: the provider lists `registry.Refs` - which walks root, the accounts, the homes and the clusters - and the Cluster controller (`internal/controllers/cluster`) reconciles EVERY cluster, each on its own watch and deadlines (`deploy/obligations.txt`), with at most `MaxConcurrentBringUps` (4) bring-ups at once (`internal/providermain/wiring.go`). A failure in one cluster is logged with the cluster's name, home and error and requeued on its own: one cluster that cannot come up must not hold back the others. Startup walks every `Cluster` CR to decide which members to adopt or bring up, tree-wide; shutdown leaves every member running for the next provider. **One process per host still holds ONE apiserver per store (invariant 1).** Every cluster gets its own state dir, its own port leases, its own Postgres object, its own slice of member units and its own `Host` entry, and the `Host` is memoised by the same key the state dir is named by, so a pass in one cluster and a teardown in a later pass act on the same object. Two clusters named `ci` by two different accounts are two entries in every one of those maps. **A market claim does not stall the pass.** `tenantNodes.Provision` writes the `NodeClaim` and returns. It does NOT poll the tenant cluster for the node first, and the distinction is the whole reason the all-clusters mode is usable: a market machine is absent for minutes, and one new claim would otherwise block every other cluster in the tree for the whole timeout, once per cluster, serially. Joining is confirmed on a LATER pass, by `nodeHosts.confirm`, which runs before that cluster's scale pass: a node whose tenant Node object has appeared is confirmed through the same `ConfirmJoined`, and a node whose claim reads `Failed` is torn down there - the claim released and the tenant Node deleted - so the failure is reported and the cluster may ask the market again rather than counting a machine it will never get. ## 12. Code layout This section maps the code as it is. Every directory under `api/`, `internal/` and `cmd/` appears once in the blocks below. The layout is the plan 0026 layers (`docs/plans/0026-abc-control-plane.yaml`, and "Architecture rules (plan 0026)" in AGENTS.md). A new transport is a new sibling package, not a branch inside an existing one. ### The layers, and how the checker assigns them `scripts/check-architecture.sh` runs `go run ./hack/archcheck`. The checker gives every package one layer from its path alone (`hack/archcheck/layers.go`). An import may only point down this order: ``` api <- contract <- kit <- transport <- impl <- controllers <- app <- cmd ``` | Path | Layer | Notes | | --- | --- | --- | | `api/**` | api (common) | types, contract status, refs; no logic, no I/O | | `internal/contract/**` | contract (abc) | imports api, other contracts and the standard library only; no third-party import | | `internal/kit/**` | kit (common helpers) | no concept | | `internal/controllers/runtime/` | transport | the dynamic client and unstructured objects are allowed only here and in `internal/apigen` | | `internal/controllers/**`, and the package `internal/controllers/runtime` itself | controllers (adapters) | one reconciler per kind | | `internal/app/**`, `internal/stack/**`, `internal/devcore/**`, `pkg/**`, `internal/main` | app (factory) | `main` is a package directly under `internal/` whose name ends in `main`; `pkg/**` is the factory another module imports (the internal rule keeps it out of `internal/`) | | `cmd/**`, `hack/**`, `internal/apigen/**`, `internal/lexgen/**` | cmd (CLI and tools) | | | every other `internal/**` | impl | this includes `internal/bus`, the component runtime and the adapters under `internal/services/**` | | a directory with no Go package | assets | rank 0, so any layer may embed it | The backend rule. A package under one of the backend prefixes of `hack/archcheck/layers.go` (`internal/compute`, `internal/vault`, `internal/pds`, `internal/atprotorepo`, `internal/events`, `internal/vmruntime`, `internal/hostsvc`, `internal/fcapi`, `internal/guestnet`, `internal/guestdisk`, `internal/guestimage`, `internal/policyengine/runs`, `internal/relay`, `internal/ingress`, `internal/ssh`, `internal/dispatcher`, `internal/gateway`, `internal/guestcp`, `internal/market`, `internal/lease`, `internal/postgresctl`, `internal/ttlctl`, `internal/clusterprovider`, `internal/tenantstore`) must not reach, directly or through any import, client-go, controller-runtime, multicluster-runtime, kcp-dev, unstructured objects, `internal/bus`, `internal/controllers`, `internal/app`, `internal/kcpserver`, `internal/kcpclient`, `internal/kcpdatastore` or `internal/stack`. Every backend package at the depth `internal//` must hold a compile-time assertion against an `internal/contract` interface (`var _ compute.Provider = (*Provider)(nil)`). The host libraries (`internal/vmruntime`, `internal/hostsvc`, `internal/fcapi`, `internal/guestnet`, `internal/guestdisk`, `internal/guestimage`, `internal/postgresctl`, `internal/ttlctl`, `internal/clusterprovider`) are exempt from the assertion. ### Abstract kinds, and where each lives A consumer refs only the abstract kind. The abstract kind names its concrete kind in `spec.backendRef`, and its controller mirrors the backend's contract status. | Abstract kind | Concrete kinds | Controllers | Backends | Contract | | --- | --- | --- | --- | --- | | AtprotoRepo | LocalRepo (on the Pds component), RemoteRepo (on a RemotePDS plus a credentialsRef) | `internal/controllers/atprotorepo`, `internal/controllers/localrepo`, `internal/controllers/remoterepo`, `internal/controllers/remotepds`; `internal/controllers/repos` registers them with `internal/controllers/pds` | `internal/pds/local`, `internal/atprotorepo/remote`, `internal/atprotorepo/scoped`, `internal/pds/remote` | `internal/contract/atprotorepo`, `internal/contract/pds` | | Vault | OpenBaoServer (local), RemoteOpenBao | `internal/controllers/vault`, `internal/controllers/openbaoserver`, `internal/controllers/remoteopenbao` | `internal/vault/openbao`, `internal/openbaoctl`, `internal/remoteopenbao` | `internal/contract/vault` | | ComputeVM | FirecrackerVM (HcsVM and VzVM wait for plans 0011 and 0012) | `internal/controllers/computevm`, `internal/controllers/firecrackervm` | `internal/compute/firecracker`, `internal/computevm`, `internal/vmruntime` | `internal/contract/compute` | | EventSource | PdsEventSource, RelayEventSource, JetstreamEventSource; an EventStream refs an EventSource | `internal/controllers/eventsource`, `internal/controllers/pdseventsource`, `internal/controllers/relayeventsource`, `internal/controllers/jetstreameventsource`, `internal/controllers/eventstream`; `internal/controllers/events` registers them | `internal/events/pds`, `internal/events/relay`, `internal/events/jetstream`, `internal/events/delivery` | `internal/contract/events` | The types are in `api/v1alpha1` (AtprotoRepo, LocalRepo, RemoteRepo, Vault, OpenBaoServer, RemoteOpenBao, ComputeVM, FirecrackerVM), `api/pds/v1alpha1` (RemotePDS) and `api/events/v1alpha1` (EventSource, its three backends, EventStream). ### The bus and the component runtime `internal/bus` is the declared bus. `bus.Resolve[T](ctx, b, from, to, capability)` returns a typed contract handle. The caller is the reconciled object or the component instance; System is allowed only for the callers in `deploy/bus-bootstrap-callers.txt`. A call that no accepted ComponentLink allows fails with `ErrLinkDenied` (`internal/bus/links.go`). `internal/component` is the component runtime: a running component instance, its Service, readiness and endpoints, and the registry of running components by workspace, kind and name. `internal/controllers/component` reconciles a Component object into it, and `internal/stack/componentcore` runs it with the controller manager in every core. ### The map api (common): ``` api/apiresource/ the APIResource and PrinterColumn metadata that each group's resources.go fills for apigen api/common/v1alpha1/ shared types: ObjectRef, LocalObjectRef, StrongRef, SecretRef, CredentialsRef, the contract labels api/v1alpha1/ the socialweb.computer group: application roots, components, market, identity, records, VM, vault, cluster, node and host kinds, and the deployments (Infra, EphemeralInfra, InfraCollection, EphemeralInfraCollection; plan 0029) api/events/v1alpha1/ events.socialweb.computer: EventSource, PdsEventSource, RelayEventSource, JetstreamEventSource, EventStream api/exposure/v1alpha1/ exposure.socialweb.computer: ComponentLink, Exposure, Service api/pds/v1alpha1/ pds.socialweb.computer: RemotePDS api/policy/v1alpha1/ policy.socialweb.computer: PolicyEngine, PolicyTrigger, PolicyWorkflow, PolicyWorkflowRun api/kcpgvr/ GroupVersionResource vars for the foreign groups (kcp tenancy, apis and core; Kubernetes coordination, authentication, rbac) ``` contract (abc): ``` internal/contract/atprotorepo/ Repo: record reader, writer and subscriber; commits, list pages internal/contract/clusterhome/ the account hash, home path, mount path and cluster key of a cluster's home workspace internal/contract/component/ component key, endpoints, Service, readiness, restart policy, phase internal/contract/compute/ Provider: shape, provision request, instance internal/contract/dispatcher/ Proxy internal/contract/events/ Source, Stream, Sink, Event, Filter internal/contract/gateway/ the web user, sign-in, sign-out, records and account agent of the gateway internal/contract/guestcp/ the guest control plane: inputs, guests, grants, onNetwork report, relay internal/contract/host/ host effects: unit backend, unit spec, status, events, identity, observation, limits internal/contract/ingress/ ingress Provider, Registrar, registration internal/contract/lease/ Slots, Rolls internal/contract/market/ market roles: identity, peers, offers, contracts, intents and their observer internal/contract/nodeclaim/ NodeClaim: claim, links, settlement, mapper (infra and entry names, entry state and fields), binder internal/contract/pds/ Accounts, Sessions, Sync, ServiceAuth, Authenticator internal/contract/policy/ Engine, Evaluator, Runner, verdicts internal/contract/relay/ Relay, Crawler internal/contract/ssh/ ssh Server, Runner, account, PTY internal/contract/tenantstore/ Eraser: removes one ServiceAccount and its token Secrets from a tenant cluster's datastore internal/contract/vault/ KV, Leases, PKI, PKI backend and provisioner, health ``` kit (common helpers): ``` internal/kit/authretry/ retries a call once after an auth error and invalidates the credential internal/kit/b64/ loose base64 and JSON normalisation internal/kit/binding/ an APIBinding target and its key internal/kit/childproc/ child process tail, secret filter, kill, backoff internal/kit/clock/ sleeps and time formatting internal/kit/conds/ condition set and status helpers internal/kit/debounce/ a debouncer internal/kit/fsx/ JSON file read and write, exists internal/kit/fswatch/ one inotify watcher per process, multiplexed across subscribers internal/kit/hostport/ host, port, URL and did:web helpers; listen on a host and port internal/kit/httpclient/ external HTTP and TLS clients internal/kit/httpjson/ JSON requests and responses with size and status errors internal/kit/jsonx/ JSON equality internal/kit/kcp/ service labels, service FQDN and handler URL in kcp internal/kit/kcprest/ a rest config for one logical cluster internal/kit/lease/ a renewing value source that fetches once per expiry internal/kit/logx/ log level parsing internal/kit/loopback/ a loopback bridge internal/kit/networkdial/ dialers and HTTP transports that resolve through a host map; TLS; websocket internal/kit/opt/ positive-or-fallback option values internal/kit/servetls/ serving TLS from files and refs internal/kit/sha256sum/ file and byte sha256 internal/kit/strs/ string helpers internal/kit/supervise/ a supervised goroutine group with restart backoff internal/kit/tlsreload/ reloading certificate pools and key pairs internal/kit/ttlcache/ a TTL cache with single-flight fetch internal/kit/wsconn/ websocket URL parsing and dial ``` transport: ``` internal/controllers/runtime/ the controller runtime every reconciler runs on: manager, workspaces, server-side apply under a field manager, finalizers, rate-limited backoff, shared status, events (ranks as controllers) internal/controllers/runtime/kube/ the typed kcp client: workspace clients, apply, get, list-watch awaits, the scheme internal/controllers/runtime/kcpapply/ applies a manifest set to kcp and reports each change, wait and phase internal/controllers/runtime/manifests/ loads, renders and rebases manifest sets: values, service suffix, name grants, waits ``` impl, the bus and the component runtime: ``` internal/bus/ the declared bus: typed handles, caller identity, the handler registry, the ComponentLink check internal/component/ the component runtime: instance, Service, readiness, endpoints, registry, credentials, HTTP client, trust internal/component/kcpclients/ the kcp client set a component gets, from the store or a kubeconfig internal/component/ports/ the port lease registry that a component's listeners bind through ``` impl, the backends of the abstract kinds and the contracts: ``` internal/atprotorepo/remote/ RemoteRepo: a repo on a remote PDS with its credentials internal/atprotorepo/scoped/ narrows a repo to the record routes it covers internal/pds/identity/ the Pds identity store and authenticator internal/pds/local/ the Pds component backend: accounts, sessions, sync, repo and service auth on internal/atproto/pds internal/pds/remote/ the RemotePDS backend: session, accounts, sync, service auth, credentials internal/vault/openbao/ the Vault backend: the OpenBao client, admin, PKI, the local server and RemoteOpenBao access internal/remoteopenbao/ probes a RemoteOpenBao (health, seal status, auth, mounts) and writes Reachable, Unsealed, Authenticated and Authorized internal/compute/firecracker/ the compute.Provider for FirecrackerVM internal/events/delivery/ the delivery loop, progress tracker and cursor store the event sources share internal/events/jetstream/ the JetstreamEventSource backend internal/events/pds/ the PdsEventSource backend: subscribeRepos through an AtprotoRepo internal/events/relay/ the RelayEventSource backend internal/relay/local/ the Relay component backend on internal/atproto/relay internal/ingress/iroh/ the iroh ingress provider internal/ingress/opentunnel/ the opentunnel ingress provider internal/ingress/xrpcsubscription/ the xrpc-subscription ingress provider internal/dispatcher/tlsproxy/ the DispatcherTLS backend: a TLS front that proxies a guest to the dispatcher by allowed host internal/guestcp/openbao/ the guest control plane on OpenBao: grants, issuer, store internal/guestcp/relay/ the guest control plane's relay internal/ssh/gateway/ the SshServer backend: the ssh gateway internal/ssh/ownersession/ owner ssh sessions into a guest internal/lease/local/ the lease contract on local files: slots, ports, subnets, front, rolls, holders internal/market/bidder/ the bidder role: offering, policy, bids, provision, guest report, submit, StateSyncs internal/market/participant/ what both roles share: inputs, crawl, comms, credentials, records, policy, service auth, and the StateSync outbox internal/market/requester/ the requester role: RFP, accept, inbox, events, sessions, secrets, runner internal/marketidentity/atproto/ the MarketIdentity backend: accounts, association, OAuth, vault state store internal/marketpresence/ bidder presence heartbeats and MarketPresence names internal/gateway/ the gateway's tenant API: cluster view, feed, store, revoker internal/gateway/custody/ the custodial web sign-in client internal/gateway/nodeclaim/ maps a NodeClaim to an entry of its Cluster's Infra: secrets policy, pins, lost messages internal/policyengine/runs/ the policy run backend: run client and decision parser ``` impl, atproto: ``` internal/atproto/cloudinit/ the Go cloud-init modules and BuildUserData: k3s join, secrets, k3s and iroh modules, the module registry internal/atproto/infra/ the computer.socialweb.infra and computer.socialweb.ephemeralInfra record types (generated by lexgen) internal/atproto/ingress/ the XRPC ingress protocol: service auth, directory, subscriber, tunnel over the relay, caller, and the relay server (the IngressProxy component and `internal ingress-relay`) internal/atproto/market/ the market lexicon types (generated by lexgen), records, signer and attestations, discovery, heartbeat, comms, repo store internal/atproto/oauth/ the atproto OAuth server side: client metadata, DPoP, JWT, scopes, server metadata, store internal/atproto/oauthclient/ the atproto OAuth client: PKCE, callback, discovery, DPoP, QR session transfer, session store internal/atproto/pds/ the PDS server: accounts, auth, blobs, repo handlers, sequencer, subscribeRepos, sync, well-known, service auth internal/atproto/pds/files/ the file-backed blob, key and secret stores of a PDS internal/atproto/pds/plcsubmit/ submits PLC operations over HTTP internal/atproto/pdsclient/ the PDS client: local and remote PDS, token and DPoP sources, resolver, service auth, subscribe internal/atproto/plc/ the PLC directory: documents, signed operations, genesis and tombstone, the server internal/atproto/relay/ the relay: slurper, sequencer, frames, store, verify internal/atproto/repo/ the atproto repository: MST, CAR, commits, codec, block store, sqlite record store internal/atproto/serviceauth/ service-auth tokens: sign, verify, admin, lxm from the path internal/atproto/xrpc/ the XRPC client: bearer, service-auth and DPoP auth, errors, sync internal/atprotoauth/ verifies an atproto service-auth token for one XRPC method: signature, audience, method, admitted DID methods, a 5-minute lifetime cap internal/didkey/ did:key encode, parse and verify (ed25519, secp256k1) internal/xrpcserver/ the XRPC server: lexicon routes and subscriptions internal/oauthcallback/ the registry of OAuth callback handlers internal/sshtransport/ ssh transport: key authorizer from PDS records, identity resolution, guest and owner sessions ``` impl, exposure, events and status: ``` internal/exposure/ the Exposure publisher, the ComponentLink link policy, the ingress registrar internal/eventstream/ EventStream matching, dial, the market sink, record watch sources internal/statusstate/ status sources: PLC identity, relay health, vault credentials and certificates internal/statesync/ the StateSync GVR; the outbox is in internal/market/participant ``` impl, compute, guests and nodes: ``` internal/computevm/ the ComputeVM host side: guest report, published state, host keys, guest login, termination, vault refs, supervisor internal/vmruntime/ the Linux VM runtime: prep and one transient VM unit per guest (swc-userns-stage -> pasta -> socialweb-vmexec -> firecracker); `internal vm start|inspect|stop|list` is its CLI internal/fcapi/ the firecracker API config and client internal/guestdisk/ clones and grows the guest rootfs and writes the seed internal/guestimage/ the pinned guest image store: names, pins, fetch and verify by sha256, GC internal/guestimagerelease/ picks the guest image source under the release base (guest-images//, then releases/, then main/) internal/guestnet/ guest network wiring: pasta argv, MAC, relay targets, the onNetwork URL internal/guestrootfs/ the guest rootfs build spec (bakes, preinstalls) behind `nodeimage guest-rootfs` internal/guestseed/ the guest seed image and its user_data modules: guest login, iroh transport, onNetwork reporter, accept bundle internal/guestidentity/ guest identity: bid config, grants, OIDC, guest login, RBAC sync, rate limit internal/nodeagent/ the guest node agent behind cmd/socialweb-node: join, CNI, diagnose, hostname, DNS link, masquerade, meshd install internal/policyaction/ the ComputeVM policy action: writes the ComputeVM that a policy run asks for internal/infra/ a deployment's declaration, its record codec and hash, the two-way sync decision, the contract lifecycle and the EphemeralInfra end rule ``` impl, clusters: ``` internal/clusterprovider/ the tenant cluster backend: per-cluster stack and units, bundle, front, holds, links, roll, scheduler, tenant API and credential, vault, postgres, ESO internal/clusterkcp/ the kcp and tenant API side of a cluster: grants, kubeconfig, CCM grant, ESO install, hardening, node observer, add-on baseline internal/clusterstack/ the file layout and spec of one tenant cluster stack: TLS, credentials, kubeconfigs, OpenBao, scheduler internal/tenantstore/kineetcd/ tenantstore.Eraser over kine's etcd v3 API: the pre-start erase of a revoked identity's ServiceAccount internal/clusterlink/ ClusterLink keys and dumbpipe tickets internal/cloudprovider/ the cloud-controller-manager logic: node patches from the instance snapshot, provider IDs, pressure internal/scale/ the scale decision: baseline, backoff, operators, scale-in, unschedulable pods internal/scaledriver/ performs the scale decision and keeps its state internal/nodemarket/ places a node as a NodeClaim in the cluster's home workspace, reads its status for the join wait, closes it by deleting the claim, keeps the k3s node-password file internal/nodeprovider/ derives a node's DNS-1123 name from the cluster key and the node id internal/tenantapply/ applies objects to a tenant cluster as its owner internal/tenantnet/ the tenant node network objects: node-local DNS, node reader RBAC internal/tenantnet/manifests/ their manifests internal/identitybundle/ the cluster identity bundle: harvest, sum, cluster-uid internal/kubeconfig/ builds a client-certificate kubeconfig internal/eso/ the External Secrets Operator: bundle, webhook TLS, install beside a cluster internal/esovault/ the ESO SecretStore for a cluster's vault internal/clientcert/ records and revokes tenant client certificates, named by the cluster key internal/clientcertfront/ the edge front that admits tenant client certificates to kcp internal/revocation/ the revocation plan: effects per cluster, deny, permit, settle internal/plcresolver/ resolves a did:plc or did:web to its key IDs for the revocation sweep; NXDOMAIN is an outage, not an absence internal/plcsweep/ the periodic pass that lists DIDs, resolves them and revokes those whose keys are gone internal/sweepkcp/ the kcp side of that sweep: the worklist from workspaces and AtprotoIdentity records, the deny, the fan-out ``` impl, vault: ``` internal/openbaoctl/ an OpenBaoServer's config, unseal keys, plugins, grants, atproto auth and client policies internal/baosync/ diffs and applies the desired OpenBao mounts, auth, policies and plugins internal/rootvault/ the root vault's paths: storage, config, TLS, client certificates internal/vaultcert/ vault TLS: the clients CA, client certificates, root and served vault TLS, reload internal/vaultlayout/ node secret paths, roles and policies (cert auth, node JWT) internal/credentials/ resolves a credential ref through vault clients built from the kcp store ``` impl, kcp, the store and the host: ``` internal/kcpserver/ runs kcp slots: admin and front kubeconfig, member tokens, roll, kcp version pairs internal/kcpclient/ the kcp client for homes, clusters, agents, client certificates, cluster links, MarketIdentity and the feed internal/kcpdatastore/ the kine and postgres datastore of kcp: argv, unit specs, backup retention internal/storemaint/ store maintenance in the core: kine autovacuum, sqlite and postgres readings internal/pgunit/ a postgres install and server: data dir, HBA, DSN, wire readiness internal/postgresctl/ runs the server of a Postgres object internal/apiserverfront/ the L4 front that holds kcp's listener fds across slot rolls, and its control socket internal/tlsfront/ a TLS front with a self-signed or given key pair internal/bootstrap/ the units that run before kcp exists: profile, observe, ensure, roll, drift internal/bootstrap/table/ the table of those units per app (gateway, bidder, dev) internal/hostcontract/ the host contract that check-host evaluates: facts and roles internal/hostlifecycle/ the file authority behind KcpInstallation, HostInstall and HostRelease internal/hostos/ the one platform seam (locking, process identity, liveness, termination, command line, boot id, port range, SysProcAttr); darwin refuses with ErrUnsupported internal/hostprocess/ reconciles a HostProcess into a unit spec, adopts it by name, mirrors remote objects internal/hostsvc/ the systemd user unit backend over D-Bus: units, identity, probes, listen ports, env files internal/hostsvc/treeaudit/ audits the process tree against the units internal/systemdunit/ the one unit-file model and writer, restart policy and backoff internal/systemdunit/notify/ sd_notify: ready, status, stopping internal/notify/ the core's sd_notify ready and watchdog internal/materialiser/ the per-cluster process reconciler: reap, allocate ports, spawn, identity guard internal/procgroup/ a process group: spec, members, steps internal/portallocator/ port ranges, purposes and leases, and the advertise address checks internal/portscan/ finds listener pids (procnet, lsof) internal/subnetallocator/ allocates guest subnets from a pool and checks host collisions internal/nsexec/ `internal ns-exec`: a private hostname and bind mounts internal/irohdial/ `internal iroh-dial`: dials an iroh ticket through dumbpipe internal/ttlctl/ the TTL expiry of finished objects internal/logcap/ log rotation caps internal/diskusage/ disk usage of a path internal/install/ install, upgrade, uninstall and GC of a gateway or bidder host internal/install/templates/ the unit, slice and drop-in templates it writes internal/instance/ the instance token and the datastore plan it implies internal/assets/ the release assets: embedded blobs, store, extract, GC, dev cache internal/assets/assetpack/ packs the asset blobs (a package main) internal/assets/blobs/ the per-platform blobs: linux_amd64/, linux_arm64/ internal/versions/ versions.json: component versions, release assets, guest image inputs internal/buildstamp/ the build identity: dev or release, commit, dirty state internal/capabilities/ the verb tree of socialweb-computer and what a platform lacks for a verb internal/clicfg/ the declarative CLI options: spec, env, config.json internal/progress/ progress lines and bars for a TTY internal/devcheck/ checks of a dev host internal/devinstance/ a dev instance: id, state and runtime roots, layout, units, stand-ins, gateway child internal/devsup/ `internal devsup`: the per-instance subreaper of every dev unit, and the offline sweep internal/gatewayhttp/ the gateway's HTTP: cluster API, downloads, OAuth, pins internal/appcli/ the CLI's MarketIdentity client, watch and prompt renderer ``` impl, the Go port of kcp-libs: ``` internal/kcplibs/common/clientlimit/ client rate limits on a rest config internal/kcplibs/common/ref/ a cluster ref: key, base host, cluster URL internal/kcplibs/impl/kcpstore/ the kcp object store client used across the code ``` impl, the policy engine: ``` internal/policyengine/atproto/ the policy record client internal/policyengine/bundled-actions/ the embedded policy actions and workflows: tangy// (TypeScript index.ts and its built dist/index.js) and workflows/ internal/policyengine/cache/ the record and verdict cache internal/policyengine/controller/ engine, workflow, run and trigger reconcile logic internal/policyengine/deno/ resolves the deno that runs the actions; deno_embedded/ holds its zip in an embed_deno build internal/policyengine/executor/ the executor: gha-lite and deny internal/policyengine/models/ requests, results, annotations, cache files, workflows internal/policyengine/registry/ the action registry internal/policyengine/status/ phases and conditions internal/policyengine/trust/ the trust snapshot and resolver internal/policyengine/workflow/ parses and runs a gha-lite workflow ``` controllers (adapters: kcp objects to contract calls): ``` internal/controllers/atprotoidentity/ AtprotoIdentity: status.state from the account's Clusters (RevocationPending, ColdDeferred, Revoked) internal/controllers/atprotorecord/ AtprotoRecord: writes the record to its repo and tracks the objects it references internal/controllers/atprotorecordwatch/ AtprotoRecordWatch: watches repos for matching records and writes the children internal/controllers/atprotorepo/ AtprotoRepo: binds the backendRef (LocalRepo or RemoteRepo) and mirrors its contract status internal/controllers/atprotosession/ AtprotoSession internal/controllers/childapplication/ ChildApplication: the child workspace and its components internal/controllers/cluster/ Cluster: the cluster pass, and the provider controller on internal/clusterprovider internal/controllers/clusterroll/ ClusterRoll internal/controllers/component/ Component: runs it in the component runtime and reports phase and endpoints internal/controllers/componentlink/ ComponentLink: reports Accepted and ResolvedRefs and puts an accepted link into the bus internal/controllers/computevm/ ComputeVM: the FirecrackerVM backend, the termination StateSync, the guest report internal/controllers/events/ registers the event source, event stream and record watch controllers together internal/controllers/eventsource/ EventSource: binds the backend kind; the helpers the three backend controllers share internal/controllers/eventstream/ EventStream: connects an EventSource to a sink internal/controllers/exposure/ Exposure: publishes a service through its ingress internal/controllers/exposurelocal/ Exposure, in process: Ready on a given origin, routes to the bus handler; no listener, no ingress (pkg/embedded) internal/controllers/firecrackervm/ FirecrackerVM: places and runs the guest through internal/compute/firecracker internal/controllers/guestcontrolplane/ GuestControlPlane: its inputs from ComputeVM, GuestIdentityGrant, MarketIdentity, OpenBaoServer and Exposure internal/controllers/guestidentitygrant/ GuestIdentityGrant internal/controllers/hostinstall/ HostInstall internal/controllers/hostrelease/ HostRelease internal/controllers/infra/ Infra and EphemeralInfra: their ComputeContracts (staged, requested, released), their AtprotoRecord and AtprotoRecordWatch, the pull of a record edited elsewhere internal/controllers/infracollection/ InfraCollection and EphemeralInfraCollection: the live entries of the account's index at infra/self into child objects; no listing internal/controllers/jetstreameventsource/ JetstreamEventSource internal/controllers/kcpinstallation/ KcpInstallation internal/controllers/localrepo/ LocalRepo: a repo on a local Pds component internal/controllers/marketidentity/ MarketIdentity: account, OAuth sign-in endpoints, verified associations internal/controllers/marketintents/ forwards Bid, ComputeVM, StateSync, ComputeContract, Exposure and MarketIdentity changes to the market roles as intents on the bus internal/controllers/marketpresence/ MarketPresence internal/controllers/nodeclaim/ NodeClaim: its entry of the Cluster's Infra (the Infra's only spec writer), its bundle, links and Exposure internal/controllers/openbaoserver/ OpenBaoServer internal/controllers/pds/ provides the Pds component's handles on the bus internal/controllers/pdsaccount/ PdsAccount internal/controllers/pdseventsource/ PdsEventSource internal/controllers/plcidentity/ PlcIdentity internal/controllers/policy/ registers the policy engine, trigger, workflow and run controllers together internal/controllers/policyengine/ PolicyEngine and its runner internal/controllers/policytrigger/ PolicyTrigger internal/controllers/policyworkflow/ PolicyWorkflow internal/controllers/policyworkflowrun/ PolicyWorkflowRun internal/controllers/portlease/ PortLease internal/controllers/postgres/ Postgres internal/controllers/relayeventsource/ RelayEventSource internal/controllers/relaystatus/ RelayStatus internal/controllers/remoteopenbao/ RemoteOpenBao: status only; it never writes the remote vault internal/controllers/remotepds/ RemotePDS, and its handles on the bus internal/controllers/remoterepo/ RemoteRepo, and the credentials it provides internal/controllers/repos/ registers the repo controllers (AtprotoRepo, LocalRepo, RemoteRepo, RemotePDS) together internal/controllers/service/ Service internal/controllers/subnetlease/ SubnetLease internal/controllers/tenantnode/ a tenant cluster's Node objects, through internal/cloudprovider internal/controllers/ttl/ deletes expired objects and reports what it deleted per kind internal/controllers/vault/ Vault: binds the backendRef (OpenBaoServer or RemoteOpenBao); roots and secrets on the bus internal/controllers/vaultmemory/ OpenBaoServer and Vault, in process: a memory KV per workspace on the bus, with snapshot and restore; no OpenBao (pkg/embedded) internal/controllers/vaultcertificate/ VaultCertificate internal/controllers/vaultcredential/ VaultCredential ``` adapters, XRPC and HTTP routes to contract calls (impl by the checker's path rule): ``` internal/services/bidder/ the Bidder component: routes over internal/market/bidder internal/services/dispatchertls/ the DispatcherTLS component over internal/dispatcher/tlsproxy internal/services/gatewayapi/ the GatewayAPI component: the gateway's tenant XRPC API over internal/gateway internal/services/guestcontrolplane/ the GuestControlPlane component's routes over internal/guestcp internal/services/ingressproxy/ the IngressProxy component over the internal/ingress providers internal/services/pds/ the Pds component over internal/pds/local internal/services/plc/ the Plc component over internal/atproto/plc internal/services/relay/ the Relay component over internal/relay/local internal/services/requester/ the Requester component: routes over internal/market/requester internal/services/sshserver/ the SshServer component over internal/ssh/gateway and internal/ssh/ownersession ``` app (factory): the roots and the composition code: ``` internal/app/ Register: the five root controllers on the controller manager (componentcore calls it once) internal/app/compose/ what the roots share: desired child specs, apply and drift, ownership and adopt, ComponentLinks, vault, repo, pds, records, streams, operator associations internal/app/gateway/ the Gateway root: web app account, requester identity, root vault internal/app/marketbidder/ the MarketBidder root internal/app/marketnetwork/ the MarketNetwork root: child roots, workspaces, vault, guest TLS, vouches, child VMs; dev up applies, waits for and deletes it internal/app/marketrequester/ the MarketRequester root internal/app/sshgateway/ the SshGateway root internal/stack/bidder/ `run bidder`: the bidder's spec, kcp and core internal/stack/componentcore/ the component core: the controller manager, every controller registration, providers, component supervision internal/stack/gatewaykcp/ `serve gateway kcp`: the gateway's kcp, the bootstrap cluster, the clusterprovider env internal/stack/gatewayssh/ `run gateway ssh`: the ssh gateway's spec, kcp and listener internal/stack/request/ `run request`: invocations, contract summaries, the requester's kcp pkg/embedded/ the requester an app embeds (the AT-IRC iOS app, plan 0038): the MarketRequester root and its Requester in the caller's in-process kcp, comms Firehose, a memory vault, in-process Exposures, no listener and no HostProcess; on ios the policy sandbox fails closed until plan 0038 I6. `Config.OnSnapshot` gets the whole vault (OAuth session and refresh token, DPoP key, contract secrets bundles, atircMinted tokens) as clear JSON with a rising `seq`: the app stores it encrypted (iOS Keychain or a Data Protection Complete file excluded from backups), keeps only the highest seq, and never logs it. `Start` returns once the provider sets are applied and the root is created; the root composes asynchronously (its bindings, then MarketIdentity `requester` in `root`, about 8 s later on a laptop): `Ready()` closes, and `WaitReady(ctx)` returns, once that MarketIdentity exists. The provider workspace `root:socialweb-provider` holds only APIExports and APIResourceSchemas, so its `/apis/socialweb.computer/v1alpha1` lists no resources; the kinds are served in `root` through its APIBindings internal/devcore/ `internal dev-core`: one dev instance's core: components, host processes, postgres, OpenBao, store maintenance, log caps, TTL scope internal/foldmain/ the Invocation a verb body runs with (args, env, stdio, the resolved bin// path, the verb words) and the re-exec argv it builds internal/ccmmain/ `internal cloud-controller-manager`; the clusterprovider re-execs it as ` internal cloud-controller-manager` with its env rebuilt internal/checkhostmain/ `check-host` internal/gatewaymain/ `serve gateway xrpc`: the gateway's config and wiring internal/atprotorelaymain/ `internal atproto-relay`: the standalone atproto relay of relay.mini-cloud-0002.chadig.com on internal/relay/local (state dir, known hosts crawled again at start, sd_notify, graceful stop); scripts/deploy-atproto-relay.sh installs it internal/ingressrelaymain/ `internal ingress-relay`: the standalone did-key ingress relay of xrpc.fedproxy.com (unix socket behind caddy, sd_notify, graceful stop); scripts/deploy-xrpc-relay.sh installs it internal/nodeimagemain/ `nodeimage guest-rootfs|guest-manifest`, the two Go steps of scripts/build-guest-image.sh, compiled only into a dev build internal/providermain/ `serve provider` and `migrate`: the clusterprovider's config, wiring, startup, adopt, roll, scale-in, teardown internal/tenantcredentialmain/ `internal tenantcredential` internal/vaultmain/ `vault apply-root` ``` Each `internal/main` is the body of a host verb behind `Main(ctx, foldmain.Invocation) int`; `cmd/socialweb-computer` dispatches the verb to it with the raw args. cmd (CLI) and tools: ``` cmd/socialweb-computer/ the multi-call CLI; every host verb lives here (dev, run, apply, install, upgrade, units, serve, vault, check-host, internal) cmd/socialweb-node/ the guest node agent (internal/nodeagent), built by scripts/build-web-app-downloads.sh cmd/socialweb-vmexec/ the VM-unit exec asset cmd/socialweb-apigen/ runs internal/apigen (`go generate ./api/...`) cmd/openbao-plugin-auth-atproto/ the OpenBao auth plugin for atproto OAuth login (its own module) cmd/openbao-plugin-auth-sshsig/ the OpenBao auth plugin for ssh signature login (its own module) internal/apigen/ generates the APIResourceSchemas and APIExports under deploy/ and the zz_generated Go files from api/ internal/lexgen/ generates the Go lexicon types (internal/atproto/market and internal/atproto/infra zz_generated.lexicons.go) from lexicons/ (a package main) hack/archcheck/ the architecture checker behind scripts/check-architecture.sh ``` Outside Go: ``` deploy/ manifests and manifest sets (apps/, market-network/, eso/), the generated schemas and exports, the audit policies, obligations.txt, bus-bootstrap-callers.txt, check.sh, the guest image and kernel inputs (guest-image/, guest-kernel/), swc-userns-stage/ rust/ a cargo workspace; rust/socialweb-meshd is the per-node pod mesh daemon that runs INSIDE a market guest (section 9), rust/build.sh its static musl build, rust/zig its cc and ar wrappers scripts/ the build, deploy and stage scripts; the only TypeScript left in the repository is scripts/*.ts (oauth-signin.ts, associate-bidder.ts, publish-spaces.ts) and the policy actions under internal/policyengine/bundled-actions/tangy/ ``` `internal/assets/blobs/.work/` is a scratch directory of the asset build, not tracked by git. **A cluster's key is derived once, from its home workspace.** `clusterhome.Key` (`internal/contract/clusterhome`) is the only spelling: it takes a home workspace path `root::` - `clusterhome.HomePath` builds it from the account DID and the cluster name - and returns `-`, which is a DNS-1123 subdomain. Every place a cluster is identified INSIDE a provider uses it and nothing else: the state dir (`pdr-`), the node data dir (`/nodes/`), the `Cluster` field of every port and subnet lease, the node name `nodeprovider.NodeName` derives, the `nodes.json` instance snapshot, the `scaledriver` state map, the tenant-credential mint file, the `NodeClaim` name and the process-group attribution in the startup sweep. Both sides reach it through `internal/contract/clusterhome`: the clusterprovider from a `Ref` (`Ref.Key()` in `internal/clusterprovider/registry.go`), the gateway from the `(did, cluster)` pair a mint binding carries (`clusterhome.Key(clusterhome.HomePath(did, cluster))` in `internal/clientcert`), so the file the clusterprovider writes and the file the gateway reads are named by one function. The bare name is still what a `Cluster` CR, a `Node` CR's `spec.cluster` and a `NodeClaim`'s `spec.cluster` carry, because those live inside one home workspace and are unique there; what the key keeps apart is everything OUTSIDE it. `internal/providermain/migrate.go` is why an existing single-cluster deployment keeps working. Before the first pass, and only when the CR tree names a given bare name exactly ONCE, it renames `/pdr-` to `pdr-`, `/` to `` (the per-node data dirs under it), the mint credential `/` to ``, and re-keys that cluster's port and subnet leases in place, preserving their ports and ranges. A name two accounts share is left alone and reported: which home it belongs to is not knowable from the name, and renaming it onto one of them would hand that cluster the other's data. It is a one-way move - a downgrade finds the keyed names and starts from an empty state dir - and the cost of the node names changing with the key is recorded in section 15. Binaries: `socialweb-computer` (the gateway, the clusterprovider, the vault and check-host are its verbs) and `socialweb-vmexec`; `socialweb-node` is the guest agent, built by `scripts/build-web-app-downloads.sh` and fetched by the guest's bootstrap. Outside `cmd/`, `rust/build.sh` builds `socialweb-meshd`, which runs inside a market guest beside the k3s agent. The one placement is `internal/nodemarket`, whose node is a claim rather than a process; the VM runtime (`internal/vmruntime`) is what the BIDDER runs. `check-host` is the host contract and nothing else (`internal/hostcontract`, `docs/host-contract.md`): `install` runs it as the account, and it probes the gateway or bidder host the binary runs on. It does not carry the retired local substrates' build report (the node image store, kernel, module tree, kvm, VMM and in-container postgres checks). `internal/versions/versions.json` pins component versions, the release assets and the guest image inputs; `scripts/stage-assets.sh` and `socialweb-computer` read it. ### The guest image, and the live gate A guest image is a kernel, a rootfs and `image.json` per hypervisor and arch. `scripts/build-guest-image.sh` builds it on any Linux host with docker or podman from the pinned inputs in `versions.json` `.guestImages`; its Go steps are the dev-build verbs `socialweb-computer nodeimage guest-rootfs|guest-manifest`. Each output is pinned by its sha256, never embedded in the binary, and fetched and verified by `internal/guestimage`. It is a TOOL, never a controller, and the image boots `root=/dev/vda` with no initramfs init. The bidder's VM runtime boots it; the gateway places no image, because its node is a claim. The live gate is a live run. A guest comes up only through the RFP flow on a market network (`deploy/market-network`, `internal/app/marketnetwork`). ### The market network A market network is one `MarketNetwork` application in its own kcp: the shared atproto services of a local market plus child application roots (bidders, requesters, gateways) that find each other over the atproto market protocol. `socialweb-computer dev up --profile market-network` brings one up. `internal/app/marketnetwork` is its root: it composes the network's own components (Plc, Relay, Pds, IngressProxy, DispatcherTLS, OpenBaoServer), the child roots in their `root:` workspaces, the network vault, the guest TLS and the vouches. `deploy/market-network/` holds the network's workspace, vault and RBAC manifests and the README for its operator. The vendored deno-kcp market example that this subsection once described, with its TypeScript services, is gone from the repository. ### Access links, state syncs and remote vaults **A `ComponentLink` is the rule that admits a component call** (`api/exposure/v1alpha1/types_componentlink.go`, `internal/bus/links.go`, `internal/exposure/links.go`, `internal/app/compose/links.go`). It lives in the workspace of its target. `spec.from` names a caller by ObjectRef or by a label selector, `spec.to` names the target, and `spec.capabilities` lists the calls it admits, each a name plus an optional scope; `spec.suspend` takes it out. `internal/controllers/componentlink` reports `Accepted` and `ResolvedRefs`, and only an accepted link admits a call on the bus; a call that no accepted link allows fails with `ErrLinkDenied`. `internal/app/compose/links.go` generates links from the concrete component refs in a root's desired specs: wildcard refs (`*`) are skipped, only component kinds are targets, and `PolicyEngine` is excluded. **A `StateSync` is the durable runtime outbox** (`api/v1alpha1/types_statesync.go`, the outbox in `internal/market/participant/outbox.go`). Its spec names the source, sink, direction, payload, `idempotencyKey`, conflict and deletion policies, optional ack paths, retry limits and suspend; its status carries the phase, observation, ack, cursor, attempts and conditions. Observe is the authority: the controller applies only when the observation differs, `Apply` is idempotent under the key, and the object becomes Acked only after an observation matches the desired hash. Deletion is `Retain`, `DeleteRemote` or `AckThenDelete`. The status holds refs, CIDs, hashes and metadata, never secrets or payload bytes. The bidder runs the controller with the `atproto.market.event` adapter. The ComputeVM controller creates a `vm.terminated` sync before teardown; its finalizer waits for Acked, then drops after the timeout so the sync outlives the VM. The adapter writes a deterministic payload and `market.event` rkeys. The bidder also writes the `onNetwork` and `hostKeys` report syncs. **A `RemoteOpenBao` is vault access separate from an `OpenBaoServer`'s lifecycle** (`api/v1alpha1/types_remoteopenbao.go`, `internal/vault/openbao/remote.go`, `internal/remoteopenbao`, `internal/controllers/remoteopenbao`). Both are concrete kinds of the abstract `Vault` (`internal/controllers/vault`). A remote ref is refused with `ErrRemoteDisabled` unless remote access is enabled, uses token or AppRole auth, requires https unless `AllowInsecureHTTP` allows http, and reads its credentials and CA through SecretRefs. The controller probes `sys/health` and `sys/seal-status`, authenticates, checks the declared mounts and capabilities, and writes only status and the conditions `Reachable`, `Unsealed`, `Authenticated` and `Authorized`; it never writes the remote vault. The roots compose a `Vault` whose `spec.backendRef` names their own OpenBaoServer (`internal/app/compose/vault.go`), so local OpenBao access remains the default. ### Composition roots: internal/app (plan 0026 W3-ROOTS) `internal/app/{marketbidder,marketrequester,gateway,sshgateway,marketnetwork}` are the five app roots; each is one composition controller on `internal/controllers/runtime`, registered by `internal/app.Register`, which `componentcore` calls once from its controller manager (every core runs the manager for its own workspaces, and a root reconciles only the roots of its core's base workspace). `internal/app/compose` is what they share: - **Writers.** A root applies every child spec, every ComponentLink it needs and its own status by server-side apply under the field manager `socialweb.computer/`. A child edited by another manager is re-applied: the root applies without force first, and on a field conflict records a `ChildEditReverted` Warning Event in the child's workspace (namespace `default`) and applies again with force. Root status is applied with the root's manager and never carries `status.market`, which the participant component reports under its own manager. Sub-managers of the root: `.../adopt` (owner references on intent objects), `.../identity` (the MarketIdentity OAuth endpoints), `.../records` (operator AtprotoRecord status), `.../restart` (the restart annotation). - **Ownership.** Every child in the root's own workspace carries a controller ownerReference to the root and the labels `apps.socialweb.computer/root`, `root-kind` and `root-cluster`; the root watches every child kind by those labels. `compose.Adopt` puts the root's ownerReference on unowned Bid, StateSync, MarketPresence and ComputeContract objects (and on infra its sets created). A ComputeVM is owned by its Bid, a won Bid stays open while its VM lives, and a root's deletion first releases its ComputeVMs, so no TTL on a short-lived object reaches a running guest. Cross-workspace children cannot carry an ownerReference: the Gateway's requester MarketIdentity, its association watch and records and its market streams live in `root:app`, and a MarketNetwork's child roots live in `root:`; they carry the root labels (including the root's logical cluster), are pruned by label and are torn down by the root's deletion before it lets go of its finalizer. - **Links.** A root writes the ComponentLinks its children use, each in the workspace of its target: spec refs between components (`xrpc.call`, `stream.dial`), Service and Exposure to their target, policy.run, vault.*, repo.*, pds.*, pds.serviceauth, events.* and its own calls as the caller (the root reaches its Vault and its Pds through the bus to bootstrap the app account). It waits for its own links to be Accepted before it calls. - **Suspend.** `spec.suspend` on a root stops participation, never a process: the Bidder and Requester components get `spec.suspendParticipation` (no new RFPs, bids, accepts or offerings; the bidder withdraws its offering), while releases of running VMs, termination StateSync acks and credential renewal continue. A suspended root applies only the children it already has, runs no account, claim or record bootstrap, prunes nothing and reports phase `Suspended`. A MarketNetwork suspend cascades into each child root's `spec.suspend`; `spec.apps[].suspend` and `spec.components[].suspend` can only add suspension (the latter stops that component's process). Each suspend field has one writer: the composer of that object. - **Operator associations (D21).** `spec.identity.approvedAssociations` on a root is composed into the requester or bidder MarketIdentity spec and into the Requester component's `operatorDID`; the MarketIdentity controller only reports `status.associations` (declared plus watch-verified). The prod gateway reads them from `SOCIALWEB_GATEWAY_APPROVED_ASSOCIATIONS`, so a restart keeps them. ## 13. Invariants A change must not break any of these. Each exists because something failed. 1. **One apiserver per store.** kine's revision cannot tolerate a second writer. 2. **`--disable-agent`.** Without it "zero nodes" is not a state. 3. **The token never rides argv.** k3s derives its bootstrap key from it; `/proc` is world-readable. 4. **The identity bundle is exactly 13 entries.** A short list makes k3s regenerate CAs and invalidate every credential. 5. **`--datastore-endpoint` is required and non-empty.** Empty is a silent new cluster. 6. **Audit flags ride `--kube-apiserver-arg`.** Direct flags are skipped with a warning, leaving the cluster silently unaudited. 7. **The advertise port comes from a lease.** A colliding one points a node at another cluster's apiserver and reads as an x509 error. 8. **A node's ports are its own, and no lease of the process group covers one.** The kubelet's is bound inside the guest's own network namespace, under its own kernel, and no port it binds is published on this host. So a node holds no lease to give back: a node's teardown releases nothing, and the cluster's own leases - its `k3s-secure` port first - outlive it, because `status.URL` stays live after OFF. This replaces "the kubelet lease is released on the NODE's stop, and is not in `DefaultPurposes`", whose purpose is retired: there is nothing left to release and nothing left to keep out of a list. 9. **Absent `nodeExpiry` refuses teardown; explicit `0` proceeds.** 10. **Nothing is read back out of a node.** The token and the name are ours. 11. **Never request-header auth against the mount, and no tenant request through it.** Tenant requests go through the client certificate front to the cluster's own front. 12. **A revocation needs a consultable deny record**, not just a deleted object. 13. **Secrets never ride argv or `user_data`, and a market node's credentials cross only OpenBao JWT over the cluster link.** `user_data` is readable from inside the guest. The NodeClaim controller's join bundle hands a market guest through the `--secrets` capability the material a guest may hold before it has authenticated to anything: the cluster OpenBao's CA, `claim.json`, and the two capability tickets - the OpenBao link's ticket, and the apiserver link's ticket with the node's advertise address and port. A capability ticket is one that is useless on its own: the cluster's apiserver still refuses a connection without the k3s token, and OpenBao still refuses one without the JWT login. Delivering the apiserver ticket with the OpenBao ticket is what lets the guest dial the apiserver while it is still exchanging its workload token for that login, instead of waiting for the ticket to come back out through the OpenBao it is still dialing. The join token, the node password and the server CA are credentials, and they are read by the guest itself from the cluster's OpenBao, after a JWT login bound to its workload subject and its claim's audience, over its own dumbpipe link to that OpenBao. The apiserver ticket is read from that same KV entry when the bundle does not carry one, and the guest then joins the serial way. Nothing reaches a guest over ssh: the gateway opens no session to any guest. The provider half (KV, policy, mount, role, revoke at release) is in section 7; the guest login is `auth/jwt-<12 hex>/login` as role ``, reading `socialweb-nodes/data/`. 14. **A list failure is an error, never an empty observation.** 15. **The state advances only on success.** 16. **No comments in code.** Reasoning belongs in this document. The only comments are the `ponytail:` form and functional comments a tool reads (`//go:` directives, generated-code headers, code generator markers), and every byte is ASCII. This is a written rule; nothing enforces it. 17. **A gate must be able to fail.** A check whose failure and success look alike is not a check, and a probe that cannot fail proves nothing. 18. **The substrate seam does not own the lifecycle.** Closing a node must not be the same act as destroying it, or the teardown semantics stop being observable. 19. **A child's output goes to a file, never through a pipe a caller may forget.** A pipe whose reader never reads blocks the child once the kernel buffer fills, and the symptom is a readiness timeout over a healthy process. A cluster member's output is its HostProcess log file, capped and read as a bounded tail (section 5). 20. **The provider's tenant credential is minted, never copied.** It is a client certificate signed by that cluster's ClientCA and valid only against that cluster. The k3s-written admin kubeconfig is never read: it is shared state on a multi-cluster host, and its server name is unverified against the address a node dials. 21. **A cluster's members are not stopped by the provider's own stop.** They are HostProcess units that outlive the core and the provider, and the next provider adopts them. Teardown is the explicit deletion of the member objects, and it is what stops the units and deletes the tenant-side Node object. A member stopped by the gateway's own SIGTERM would bypass both and leave a store needing crash recovery. 22. **A workspace ServiceAccount does not leave its workspace.** Its token's scope confines it to the logical cluster it was minted in and kcp refuses it everywhere else before RBAC is consulted, so a ClusterRoleBinding naming it elsewhere grants nothing and no subject string changes that. Any grant whose effect is expected in another workspace is mis-designed, not mis-spelled - which is why the runtime holds a credential kcp does not confine. 23. **A spawn failure says why.** The reconciler carried the op and dropped the error, so a process group that never came up read as "the cluster is Initializing" and the readiness gate's own reason - and the component log under the cluster's data dir - were the only account of it. 24. **A snapshot is a template, never an instance.** Every node boots its own clone of the rootfs. One writable image serving as both carries a guest's writes into the next boot, and it presents as a corrupted image rather than as shared state; two live VMs on one file corrupt each other. 25. **A boot command line is a console.** Anything on it is written to the guest's console at least twice - firecracker logs the body of its own boot-source request, and the kernel prints `Kernel command line:` - so no secret may ride it. A market guest's secrets come from OpenBao (invariant 13). 26. **A process's identity is read from the service manager and the kernel, and an empty identity matches nothing.** A unit is matched by its name and InvocationID, a VMM by its pidfd start ticks and its own id (section 5, "Adoption is a tri-state verdict"). Matching every token present, read literally, would match every process on the host. A pid whose identity does not match is NOT OURS: a reuse stays a refusal and never becomes a kill. 27. **A guest's own account of its own elapsed time is not evidence.** Time a guest by the host's clock. A reported 332-second boot was a literal `echo` in an init script, printed 0.29 s after the VM started, and it sent this work after a 30x non-issue. 28. **A failure already visible in a guest's console is not waited for.** Named fatal markers end the wait immediately, with the guest's own words. A timeout that runs to its end over a host that died at its first step reports the timeout instead of the cause, and the two are indistinguishable until the clock runs out. 29. **A market node's credentials never ride the claim or the join bundle.** The join token, the per-node password, the cluster's server CA and the apiserver ticket live in the cluster's OpenBao at `socialweb-nodes/`, and the guest fetches them itself (invariant 13). A `NodeClaim` is a cluster-scoped object in the owner's home workspace, readable by anything that can read that workspace, so it carries only what is not secret: the PATHS the provider reads and the grant that authorizes the guest's fetch (`issuer`, `subject`, `audience`, `role`). The join bundle carries the two capability tickets and nothing else: an OpenBao ticket alone buys a dial to OpenBao, and that still needs the JWT login, and an apiserver ticket alone buys a dial to the apiserver, and that still needs the k3s token, which only OpenBao holds. They also never ride argv, `user_data` or a boot command line, for the reasons invariants 13 and 25 already give. This replaces "a market node's secrets cross only the SSH session's stdin" (plan 0005 M3). 30. **A cluster is its home workspace, not the name inside it.** Two accounts may each name a cluster `ci`; they are two clusters, with two state dirs, two sets of port and subnet leases, two sets of node names, two mint credentials and two process groups. Anything that identifies a cluster OUTSIDE its home - a state dir, a lease, a node name, a snapshot, a mint file, a process group - carries the key `kcpclient.ClusterKey(home)` = `-` and never the bare name. A pass over one cluster must never key anything by a name two workspaces can share, because the failure mode is not a collision error: it is one cluster reading another's leases and state dir and reporting them as its own. 31. **A claim does not wait for the machine.** Provisioning a market node writes the `NodeClaim` and returns; the join is confirmed on a later pass. A blocking wait inside a pass that serves every cluster in the tree stops being a local delay and becomes a global one, and the thing being waited for is minutes away by construction. 32. **A node is counted by the cluster that has it, never by the claim that asked for it.** The count a scale decision reads is the tenant cluster's own node list, plus - under a substrate whose `Provision` returns before the machine exists - the claims this process holds whose nodes that list does not name. A claim's phase is not a count: the NodeClaim controller writes `Joining` when the guest has fetched its join secrets, and the Node object it is registering appears in the cluster's list only after its agent's connection through the apiserver link is served, so a phase-keyed count has a gap with a duration in it and every claim that crosses the gap together is counted nowhere at all (section 15). 33. **A guest kernel loads no module on demand.** The market guest's kernel is built with every feature the node needs built in (`deploy/guest-kernel`, `CONFIG_MODULES=n`). On the node images before it, a module the kernel could not autoload left iptables-nft's native `log` and `limit` rules failing with ENOENT and NetworkPolicies silently not enforced (section 15, plan 0005 M5-NETPOL). 34. **An ingress provider selected by `spec.provider` fails closed.** `xrpc-subscription`, `iroh` and `opentunnel` are distinct packages; a missing or non-executable dumbpipe, a missing opentunnel command, a missing `spec.iroh` or `spec.opentunnel` block for the selected provider, or an unknown provider is an error. The component does not silently start a different provider, and an `ingressRef` that resolves to a component with no registrar is an error rather than another provider. 35. **A `submit*` XRPC handler enqueues and returns.** The handler parses the body, writes the intent durably into kcp - a `StateSync` (`api/v1alpha1/types_statesync.go`, `internal/statesync`), a status field, or the requester's inbox - under an idempotency key, and answers. It resolves no DID or service, calls no PDS, relay, OpenBao or VM, provisions nothing, verifies no signature and delivers no event in the request path. A worker or controller (the StateSync outbox of `internal/market/participant/outbox.go`, the requester inbox of `internal/market/requester/inbox.go`) owns that work and retries it under the same key. The worker calls the counterpart's `submitEvent` XRPC only when the counterpart record names one: the bidder from the Accept record's `submitEvent`, the requester from the bidder Receipt record's `submitEvent`; with no ref the firehose carries the event. A timeout does not make a remote call a request-path operation (section 15). 36. **iroh is the default and required guest transport; a tunnel is an opt-in per contract.** Every guest gets the iroh module, the requester's ssh dials the iroh ticket in `status.contract.vmAddress` by default, and a contract is Held on the iroh address and the host keys alone. The other two transports exist only for the entries of `spec.tunnels`: transport `opentunnel` serves the guest's `127.0.0.1:` at an https URL (a `vm.onNetwork` kind `https`), and transport `webrtc` bridges a datachannel to it through socialweb-guest-agent (a `vm.onNetwork` kind `webrtc` whose address is the agent's DTLS fingerprint). Each comes back signed by the bidder into `status.contract.networks` with its tag and kind (plans 0032 and 0034). The requester's tunnel module carries config only; the bidder installs opentunnel, wootty and socialweb-guest-agent from its own GuestControlPlane `/v1/downloads` over the host-loopback map, checked by sha256, and a guest fetches no binary from the internet. A wootty shell rides an opentunnel tag only, needs `spec.secretsRef`, and does not start without a `WOOTTY_AUTH_TOKEN` of 32 characters or more from the secrets bundle. WebRTC signaling is age ciphertext in `compute.events.guest.signalWebRTC`, sealed to a `compute.events.guest.publicKey` of the other side; no SDP, ICE candidate, IP address or private key is in a record, kcp object or log, and the only STUN server is `stun:stun.cloudflare.com:3478`. 37. **A guest reaches its host's loopback only on its ComputeVM's `spec.hostPorts`, and reaches no LAN host and none of its host's own addresses but by UDP on high ports, and the host's own addresses only when `spec.guestNetwork.hostUDP` is on.** socialweb-vmexec loads the nf_tables bridge table `swcguest` in the guest's namespace before it wires the tap; a VM whose table does not load does not start (plan 0034). Besides TCP to `192.0.2.1` on `spec.hostPorts`, DNS to the pasta gateway and ICMPv6 neighbor discovery, the table drops every frame from the guest to the host's own addresses (listed by vmruntime in the host namespace at Prepare), to the guest link 172.30.0.0/30, to 0/8, 10/8, 100.64/10, 127/8, 169.254/16, 172.16/12, 192.0.0/24, 192.168/16, 198.18/15, 224/4, 255.255.255.255, ::1, fc00::/7, fe80::/10 and ff00::/8, and every ethertype but IPv4, IPv6 and ARP. TCP to the LAN is never open. The one allowance is UDP to 10/8, 172.16/12, 192.168/16 and fc00::/7 on destination ports 1024 and up except 1900, 5353 and 5355, for same-LAN WebRTC candidates; it is on by default (`spec.guestNetwork.lanUDP` unset or true on the ComputeVM), set by the bidder operator (`--no-guest-lan-udp`) and never by the requester, and it never covers the host's own addresses, 100.64/10, 169.254/16 or 127/8. The one exception for the host's own addresses is `spec.guestNetwork.hostUDP` on the ComputeVM (unset: false): UDP to them, but not to loopback or link-local, on the same ports, so WebRTC works when the requester runs on the bidder's host. The bidder always writes it as its effective value, its own `spec.guestNetwork.hostUDP` when set, else on exactly when its `spec.policy` is `bidder-only-me`; it is the bidder operator's choice (`--guest-host-udp`), never the requester's, and its Bidder's `status.guestNetwork` reports it with the reason. The rules are stateless: no inbound path into a guest exists but replies, which enter from the pasta side, and the one forwarded webrtc UDP port. That port is the only inbound path: for a guest with a webrtc tunnel the bidder leases one UDP port per ComputeVM (its own `spec.guestNetwork.webrtcUDPPorts {min, max}` on MarketBidder, Bidder and NetworkApp, default 31000-31999, a port registry of kind udp in its state dir) and writes it as the ComputeVM's and FirecrackerVM's `spec.guestNetwork.webrtcUDPPort`; pasta forwards that UDP port on the host's RFC1918 and ULA addresses only (`-u /`, never loopback, link-local, 100.64/10 or a public address) to the same port in the guest, where only socialweb-guest-agent listens (one pion UDPMux shared by every session, host candidates rewritten to those LAN addresses from `/etc/socialweb-guest-agent/bidder.json` `webrtc {udpPort, hostAddresses}`). The table accepts UDP from that source port to the host's own non-loopback, non-link-local addresses and, after the drop of the host's addresses and the guest link, to 10/8, 172.16/12, 192.168/16 and fc00::/7, on destination ports 1024 and up except 1900, 5353 and 5355; it is an explicit per-VM grant and holds with `lanUDP` and `hostUDP` off. It exists because a same-LAN peer (an iPhone offers only mDNS host candidates and the home's public srflx, and the router does not hairpin) can reach no other guest candidate. No address is in kcp: the ComputeVM carries the port only, and the addresses travel in the guest's user_data. ## 14. Open architectural gaps These are the things the architecture does not yet answer, and this list is the work. What is settled - a decision taken, a defect closed, a measurement recorded - is in section 15 instead, and the two are separated deliberately: a work list padded with history reads as complete, which is how a bullet that was already closed, or one whose subject the design has since deleted, survives re-reading. On 2026-10-23 every item was read again against the plan 0026 tree: the items that existed only because of the removed TypeScript runtime, and the items that later work had closed, moved to section 15 ("Gaps of the removed TypeScript runtime" and "Items closed by later work"). The items of the earlier batches, from 2026-09-24 to 2026-10-12, are closed; section 15 holds their readings (REL-7, PASTA-1, CHK-1, VAULT-1, VAULT-2, ESO-2, ESO-4, FLAKE-1, GL-1, GL-2 and MARKET-2 by name), and each plan's decisions are in its file under `docs/plans/`. **The live backlog is `NEXT.yaml`, which this section mirrors: an item is written here when it is opened and moves to section 15, with its readings, when it is closed.** An entry belongs here when the design does not answer a question the code already asks, when a reading the architecture depends on has no instrument, or when a settled decision turns out to have been recorded only in a comment. Every entry closed so far was found by reading the code against this document, or by running it, rather than by reading the document. The practice worth keeping: the artefact was run and each guard was watched to fail, rather than the code read against this document alone. - **Opened 2026-10-30 (plan 0036 A7): R2-START, the guest requester started with no command in the guest.** A contract with `spec.association.fromGuest` gets, in user_data, the user `r2` (group `socialweb-r2`, linger on), `socialweb-r2.service` (User=r2, not started at boot, `EnvironmentFile`-style `/run/socialweb-r2/r2-start.env` read by `/usr/local/bin/socialweb-r2-run.sh`, which runs `run request --name --guest-requester [--agent] [--hold]` and, with both, `run request agent `) and `socialweb-r2-release.service` (oneshot, User=r2, releases every guest requester invocation that is not Released, 120 s each, 600 s in all). `run request r2 [--r2-name ] [--agent] [--hold]` (or the page) creates an outbound ContractEvent with NSID `com.publicdomainrelay.temp.compute.events.guest.r2Start` and `spec.r2Start {name, agent, hold}`; the requester publishes it (`fromGuest` only), the bidder queues it, socialweb-guest-agent writes the env file (root:socialweb-r2 0640) and starts the unit. `run request --associate guest` adds `guestTools: [socialweb-computer]`. New requester scope token: `repo:com.publicdomainrelay.temp.compute.events.guest.r2Start?action=create&action=update`. Open: the page's r2Start button (wasm), and a live run of the unit. - **Opened 2026-10-30 (plan 0036 A8): R2-CASCADE, a parent release releases the guest requester's contracts.** (1) The requester, releasing a fromGuest contract whose guest requester is Verified, first publishes `guest.r2Release` (outbound ContractEvent `r2-release-`); the guest agent stops `socialweb-r2.service`, runs `socialweb-r2-release.service` and posts `guest.r2Released {ok, message}`; the requester waits for it at most 12 min (`GuestRequesterReleaseTimeout`, status step `releasingGuestRequester`), then sends vm.delete as before. New scope tokens: requester `repo:...guest.r2Release?action=create&action=update`, bidder `repo:...guest.r2Released?action=create`. (2) Bidder backstop, the smallest safe mechanism: at provisioning the bidder reads the requester's association mirror and its proof (one getRecord each); for a contract-scoped association it writes the record's expiresAt into the ComputeVM `spec.associationExpiresAt`, and its ComputeVM observer requeues to expiresAt plus 10 min (`AssociationExpiryGrace`) and then releases the VM. A guest requester's VM therefore cannot outlive its association by more than the grace, even when the cascade did not run (a destroyed parent guest, a lost event); a parent released earlier is caught by the cascade, not by the backstop, which bounds the worst case at expiresAt (24 h at most). (3) A denied accept with no receipt: `releaseDeniedAccept` waits for the receipt up to 4 times (`DeniedAcceptReceiptAttempts`, `ReceiptTimeout` each), each attempt in `status.release.lastError` and the `ReleaseFailed` condition; when it gives up the message names the manual path, `run request release ` (spec.release), which a Failed contract with an accept and a winner but no receipt now takes again (`RequesterContractDone` is false for it), finding the receipt at its stable rkey. Release revokes (A6) now run their first attempt before Released is written, the guest requester's token first, because a CLI requester stops its kcp right after Released (the rc126 V1 run lost the R2 revoke this way). Open: live runs of all three. - **Opened 2026-10-30: QR-NO-EXPECTED-ACCOUNT, an oauthQr requester with no expected account takes the first verified sign-in.** The device polls `com.fedfork.atprotoOauthQR` with the nonce in the `x-oauth-qr-nonce` header (not the query), and qr.fedfork.com (aceed9c) returns `verifiedDid` after it checked the posted session at the account's PDS. The MarketIdentity refuses a transfer whose `verifiedDid` is empty or differs from `userDid` (reason `QRSessionUnverified`, a new QR URL), so `checkAccount` compares a DID the PDS vouched for. A bidder in mode oauthQr must name its account (`spec.identity.did` or `spec.identity.handle`, `run bidder --handle`); the composer fails the MarketBidder otherwise, and hat-7 names `bobvmbuilder.bsky.social`. The CLI requester (`run request -- --atproto-oauth-qr`) may name none: then `checkAccount` has nothing to compare, and whoever completes a verified sign-in at the QR URL within its 5-minute window becomes the requester's account. The CLI prints `signed in as ()`, and `-- --expect-handle ` sets `spec.identity.handle` so `checkAccount` pins the account (`AccountMismatch` and a new QR otherwise). Open: whether a requester with no expected account should stay allowed, given the QR URL is printed to stderr and logs. - **Opened 2026-10-30 (plan 0036 V1): GUEST-SELF-BINARY, the socialweb-computer binary in a guest.** R2 (`socialweb-computer run request` inside a guest) needs the CLI there; the release binary (~560 MB of embedded assets) is too big for `socialweb-guest-downloads`. Decision: `spec.guestTools: [socialweb-computer]` on ComputeContractTemplate, so Infra and EphemeralInfra `spec.contracts[].guestTools` reach the ComputeContract the infra composer creates (the page requests only through them); the `computer.socialweb.infra#contract` lexicon carries `guestTools` (array of string, knownValues `socialweb-computer`). It is opt-in per contract and fixed at request time, because user_data is fixed at provisioning while `spec.association` is set later by `run request associate`, and tying it to `spec.agent` would put 560 MB on every agent VM. The requester writes the marker `/etc/socialweb-computer/guest-tools/socialweb-computer` into user_data; the bidder's `GuestBinariesNeeded` turns each marker into a pin; the bidder and its GuestControlPlane pin and serve `/proc/self/exe` (one sha256 per process, `internal/kit/selfexe`; the open fd stays the running inode across an upgrade's rename) as `socialweb-computer-x86_64`, streamed by `http.ServeContent`, never held in memory. `setup-guest-binaries.service` fetches, checks the sha256 and installs it like every other guest binary. A bidder accepts only the tools in `guestseed.GuestTools`. linux/amd64 bidders only: a bidder with no pin for a requested tool refuses the bid up front (`bidder_guest_tools_unserved`, Bid Failed GuestToolsUnserved, no bid record). Both run in the one core process of the bidder stack (`internal/stack/bidder/core.go`, componentcore), so the pin in user_data and the served file are one hash. The guest script now ends for good (exit 2, `RestartPreventExitStatus=2`) when a requested binary has no pin for the guest's arch, or when every attempt got HTTP 404 (the bidder's download set changed after provisioning, e.g. an upgrade swapped its binary); other failures still exit 1 and the unit retries. Open: no live run yet (instrument: a V1 guest that runs `socialweb-computer version`); a guest disk needs about 600 MB free; a non-release (dev) binary in a guest lacks its embedded assets. - **Opened 2026-10-25 (plan 0034 follow-ups; the text and closing instrument of each is in `NEXT.yaml`).** RELAY-SSH-ROUTE: fedproxy's atprp-ssh-relay lets a dead `-R` session take the Caddy route of a live one and deletes the live route when the dead one is reaped; fix on branch fix/ssh-session-drop. PAGE-CONTRACT-ALLOC: the browser page allocates ~1.3 GB over a contract's two busiest minutes (idle after release 16-36 MiB/min). PAGE-GOROUTINES: ~7,690 goroutines idle in the page. PAGE-PENDING-28S, PAGE-KUBECTL-EARLY, PAGE-REFRESH-E2E: page timing, ordering and the unobserved OAuth refresh through the JS worker. PAGE-STREAM-INGEST: A vs B2 is the user's decision. RELAY-MST-INVERT: our Go relay cannot verify two older hat-9 commits replayed from cursor 0. RFP-WINDOW-RKEY: the bid window reads the rkey before createdAt. RELEASE-NO-CONTRACT-WAIT: releasing a VM-less contract waits 3 m and exits 1. - **Closed 2026-10-30 (stable rkeys).** LEGACY-TID-CLAIMS: the 11 TID operator claims in john's repo and the 1 in bob's were re-written at their `trust.ClaimRKey` rkeys (same CID) and the TID copies deleted; `trust.legacyClaimListed` is gone, so the trust resolver reads the mirror, the operator claim and each vouch by one getRecord at a stable rkey and lists no repo. - **Plan 0013 soak paused 2026-10-19 06:28Z (third reading, partial).** `dev-soak-3` ran rc18 code 1 h warm-up + 2 h 43 min of the fit window at ~1.1 RFP/s: 15,311 RFPs created, 0 failed; 227 tenant cycles, 0 failed; 286 VM contracts, 21 failed around chaos; host OOM 0, MemAvailable 82 GB, swap 0. Flat by the fit rule: 183 of 209 series, including the gateway requester heap (40-47 MB), its sockets (41-47, burst peak 17,077), all repo KV files, OpenBao stores, object counts, record counts except three small drifts, kine rows at the network level, and the host disk. **Still growing**, all on the installed bidder's systemd-user side: `handles/3de3bcd364/kcp` 240 -> 507 MB (+110 MB in 2.7 h), its kine file 44 -> 80 MB and rows 5,336 -> 7,826, the bidder's kcp unit memory 1,110 -> 1,474 MB, its OpenBao HostProcess 63 -> 71 MB, and the guest-kcp pod 76 -> 80 MB. Three series the fit called NO are cumulative counters, not gauges: Bids Expired, Bids Lost and gc objects; the fit script needs counter differencing. Logs dirs and the host journal grow under their rotation caps. These are pre-plan-0026 readings: guest-kcp is now the GuestControlPlane component and runs in the bidder core's process, so its series is part of the core's memory. A full 3 h reading on a post-0026 build and the counter-differencing fix are needed before 0013 can close. - **Which bidders get a direct submitRfp (no plan attached; user, 2026-10-18).** A requester posts its market.rfp record, and relay/firehose watchers see it; the direct `submitRfp` XRPC is only a faster path and the path to bidders that do not watch the firehose. There is no recipient cap and no fill order today. A future policy-engine plan decides which bidders get a direct submitRfp and in what priority; plan 0017 no longer carries a cap (its M2 was removed). - **The kcp store keeps a bounded history by mechanism; the plateau reading is open (plan 0013 M3-STORE-COMPACTION, plateau read by M4-SOAK-READING on 2026-10-18).** Four mechanisms, each read live on hat-8 (test-0001, own `dev up` instances, no VM): - Compaction stays the apiserver's. kine compacts to the revision the apiserver wrote, and never within its last 1000 revisions (`compactIter`, kine v0.17.1), so a quiet store keeps a deleted object's rows until 1000 more revisions pass. Reading: 150 ConfigMaps of 400 KB created, patched and deleted left 300 rows of 400 KB in the store after two compaction ticks; 1300 small writes later the next tick removed them. The row count and file size therefore plateau at about the live set plus the last 1000 revisions, not at the live set. - sqlite (a bidder, and the dev instances that use kine-sqlite). The DSN carries `_auto_vacuum=incremental` (`kcpdatastore.SqliteParams`; the driver sets it before `journal_mode`, so a new file gets it on its first connection). Compaction deletes rows and leaves free pages; with `auto_vacuum=NONE` the file never shrinks. `internal/storemaint` runs in the core (`devcore.Run`, every core: bidder, gateway, dev), first pass after 30 s and then every 5 minutes: it opens the file as a second connection (modernc.org/sqlite, pure Go, busy timeout 30 s), runs `PRAGMA incremental_vacuum(2000)` until the freelist is empty, then `PRAGMA wal_checkpoint(TRUNCATE)`, which is what makes the main file shrink under WAL. Reading, the `quick` profile: a fresh store answers `auto_vacuum` 2 and `journal_mode` wal; after the churn above the pass at 11:35:33 freed 449 pages (247058432 -> 245219328 bytes), at 11:40:34 freed 162, and at 11:45:35, one tick after the 1000-revision window passed the churn, freed 59079 pages: `fileBytesBefore` 246345728, `fileBytesAfter` 4063232. - **Only a store created by this binary is incremental.** `auto_vacuum` of an existing file cannot change without a full `VACUUM`, and a prod deployment preserves nothing (AGENTS.md, Prod deployments), so no `VACUUM` runs: an old file reports `autoVacuum: none` in `status.store` and `incremental_vacuum` is skipped for it. The plan item's risk line, a one-time `VACUUM` at the next start, is not built. - postgres (the gateway's kcp store). `kcpdatastore.PostgresSettings` (`autovacuum=on`, `track_counts=on`) is in the store unit's argv and in `postgresctl.DefaultSettings` (tenant cluster postgres); the core sets the kine table's `autovacuum_vacuum_scale_factor=0.02` and `..._threshold=1000` on its first successful pass (`ALTER TABLE kine SET (...)`, once kine has made the table) and then reads `pg_total_relation_size`, `n_live_tup`, `n_dead_tup`, `autovacuum_count`, `last_autovacuum` over the unix socket through `pgunit.Query`. Reading, `prodlike` profile: first read `totalBytes` 253779968, deadTuples 54; the pass at 11:31:47 showed `autovacuums` 1, `lastAutovacuum` 18:27:24Z, deadTuples 0; the pass at 11:41:47 deadTuples 587 again after more churn. Vacuum keeps the table's size, it does not return it to the file system; the table size plateaus, it does not fall. - `status.store` (`v1alpha1.StoreUsage`: kind, path, bytes, freeBytes, rows, deadTuples, autoVacuum, vacuums, reclaimedBytes, lastVacuumAt, sampledAt) is on `ApplicationStatus`, so the root object of a MarketBidder, MarketRequester, Gateway and SshGateway carries it next to `status.stateDir`; the four schemas moved a revision. `socialweb-computer status` prints it for the MarketBidder, MarketRequester and Gateway roots. Reading: a `market-network` instance's bidder and gateway roots (both kine-sqlite in that profile) showed `autoVacuum: incremental`, bytes and rows; a Gateway object applied to the `prodlike` instance showed `kind: kine-postgres`, `autoVacuum: "on"`, bytes 254844928, rows 1685, `vacuums` 1. - kcp starts with `--event-ttl=1h` (`kcpserver.EventTTLFlag`; `CheckArgv` refuses a slot without it). Reading: the running `kcp start` command line carried it. The flag is accepted, hidden from `--help`. No controller or component in this tree creates an Event object (searched again at the plan 0026 tree for EventRecorder, `events.k8s.io` and a record broadcaster), so every controller writes status only. - Not read: the plateau of file size, kine rows and kine table size under M4-SOAK-READING load. This item's `done_when` is that reading, so the plan item stays `todo`. A child kcp of a market network is covered because its core also runs `devcore.Run`. - **Disk use of a long-running instance is bounded by rule; the soak reading is open (plan 0013 M3-DISK-RETENTION; the kcp audit logs that the 2026-10-18 soak found unbounded are bounded now, section 15).** The rules, by owner (the run directory rule of the removed runtime is in section 15, "Gaps of the removed TypeScript runtime"): - File logs. Rotation is copytruncate with one `.1` copy that keeps the newest `cap` bytes (`internal/logcap`), safe because every writer opens with O_APPEND (devsup, systemd `append:`). The core rotates every `*.log` in its logs dir and the devsup logs over 64 MiB (`logcap.DefaultCap`): `logcap.Keep` makes one pass at start and then a pass on file-system events, debounced to one a minute (`logcap.DefaultEvery`). The HostProcess supervisor enforces `spec.output.capBytes` (64 MiB when unset) on its watched output files. - The journal. A user manager cannot get its own journald size limit: journald keeps user units in `user-.journal` files that share the host's SystemMaxUse pool (default 10% of the file system, at most 4 GiB), and LogNamespace= is refused for units of a user manager. So `install` does not change the host's journald.conf (the bidder host is shared); it writes `LogRateLimitIntervalSec=30s` and `LogRateLimitBurst=10000` into the core unit templates, which caps the message rate of each core whatever the host's journald.conf says (journald still scales a burst by free disk space). Units that write to a file (kcp, kine, the front, postgres, OpenBao, the VMM) are under the file rules above. - Guest images (`/images///`). After it tears a VM down, the FirecrackerVM controller (`internal/controllers/firecrackervm`) asks the VM runtime to collect the store (`vmruntime` `CollectImages`). An image stays when a FirecrackerVM status names it, a VM work dir under `/handles/*/vms/*` boots it, this binary's versions.json pins it, or `bin/current` or `bin/previous` names it (`version --json` lists `guestImages`). Everything else goes, and `.fetch-` scratch dirs older than an hour go too. No VM teardown means no collection. - Unchanged: the asset cache GC and the VM work dir collection (an orphan work dir goes after `OrphanGrace`). - The reading. Each application's appctl (`internal/app/compose`) samples its state dir at most every 3 minutes (`StateDirSampleEvery`; bytes on disk from st_blocks, file count) and writes the last sample into `status.stateDir` of its root object with the next status write. M4-SOAK-READING records the series; the item closes when it is flat. - **PORT-1: the kcp process piles up loopback connections to the fixed-port front of an old tenant cluster until the gateway host has no ephemeral port left (opened 2026-10-15 from a live reading on the prod gateway host; `NEXT.yaml` PORT-1).** `listen tcp 127.0.0.1:0` failed host-wide with "address already in use" because about 28000 loopback connections from the kcp process were open to the fixed-port front of each of two old tenant clusters (canary on port 20000, goat-demo on port 20018), which exhausted the ephemeral range for every port 0 listener on the host. The relay in `internal/apiserverfront/front.go` closes both ends when either end closes, so the connections are not held by the front; they are held on the kcp side, as dials from kcp to the front of an old cluster. Deleting the two clusters released the ports and the host recovered. The origin is not found: which kcp component dials the front (the mount, the kubeconfig mount child, a controller informer) and why it opens a new connection instead of reusing or closing one is open. Every listener on the host binds port 0, so one cluster front that leaks connections takes the whole host down: a host-wide failure with a per-cluster cause. The instrument that closes it: the count of established loopback connections from the kcp process to each cluster front on the gateway host, read with `ss -tn state established '( dport = : )'`, stays flat across one hour with the cluster idle, for a healthy cluster and for a cluster whose apiserver is stopped, and the count of sockets in the ephemeral range shows headroom while a cluster is deleted and recreated. - **A HostProcess has a memory limit; a component has none, so one component's leak takes the whole core (plan 0013 M3-DENOPOD-LIMITS, rewritten for plan 0026 on 2026-10-23).** Every HostProcess unit gets a `MemoryMax` and `MemorySwapMax=0`: postgres 1 GiB (`pgunit.MemoryMax`), kine 512 MiB (`kcpdatastore.KineMemoryMax`), a kcp slot 3 GiB (`kcpserver.SlotMemoryMax`), the front 256 MiB (`kcpserver.FrontMemoryMax`), OpenBao 512 MiB (`openbaoctl.MemoryMax`), a stand-in 1 GiB (`bootstrap.StandInMemoryMax`); VM units carry the VM's memory plus overhead, and a HostProcess object that sets none gets 2 GiB (`hostprocess.DefaultMemoryMax`). The `dev` backend (devsup, `proc`) enforces no unit limit by design. The components (Pds, Plc, Relay, Bidder, Requester and the rest) run in the core's own process, in the component runtime of `internal/component`, and the core unit carries no `MemoryMax` (the installed templates set `Delegate=yes` and `OOMPolicy=continue` and no limit). A root's `spec.components` may carry `ComponentLimits`; appctl resolves and reports them, and writes `LimitsEnforced=False` (reason `ReasonComponentLimitsInactive` in `internal/app/compose/status.go`) because the in-process runtime cannot enforce them. So a component that leaks grows the core until the kernel or systemd stops it, and every component of that core restarts with it; nothing names the component that breached. The per-process limits that the removed runtime enforced, and their readings, are in section 15 ("Gaps of the removed TypeScript runtime"). What would close it: a per-component bound the runtime can enforce in process (a heap budget or a bounded queue per component), or a component kind that runs as its own HostProcess unit, and a reading of a breach on an installed core. - **Open after the 3 h soak (plan 0013 M4-SOAK-READING, 2026-10-18; the readings are in section 15).** Each is a series that kept growing, with its measurement and the smallest fix named. - The hono-pds block store, the requester's vm.delete and market.event records, the requester's orphan records, the bidder's bid record cap, the requester heap and the gateway requester's retention are closed (section 15, "hono-pds collects unreachable blocks" and "The requester's orphan records, the bidder's bid sweep, the requester heap and the gateway requester's retention"). Left open from the dev-bound-1 reading and the second 3 h soak on dev-soak-2 (2026-10-18, section 15, "Plan 0013 M4-SOAK-READING: second 3 h soak, systemd-user"); each is named with its slope over 20:35Z to 23:36Z: - Requester native memory (the bursts of ESTAB connections to the dispatcher) and the bidder PDS growing to its limit are closed (section 15, "The contract watch's getRecord bursts", read on gl-runner-1 after this soak). Left from that reading, taken before plan 0026: the soak requester's RSS read +75.6 MB an hour (300 to 362 MB) with its fds flat and its heapUsed at 75 to 119 MB while its ComputeContracts still rose (2441 to 2702). The Requester component (`internal/services/requester` over `internal/market/requester`) now runs in the core's process, so the series to read is the core's RSS over a window past the contract TTL. - Deleting a tenant cluster takes more than 4 minutes in some cycles under 8 lanes (the driver's 4-minute wait timed out 30 times; the clusters were gone later; the gateway core logged "a step of deleting this cluster has not completed" 67 times). - Every status write logs `Warning: unknown field "status.store"` (1063 of 3000 lines of the network core log): some root kind lacks `status.store` in its schema. At HEAD the MarketBidder, MarketRequester, Gateway and SshGateway schemas carry it and `deploy/marketnetwork-apiresourceschema.yaml` does not, nor does `MarketNetworkStatus`; not re-read since plan 0026. - The node claim index, the pod cgroups, the spec-change restart, the bidder's bid TTL and the relay backlog are closed (section 15, "Gaps of the removed TypeScript runtime" and "Items closed by later work"). - **A market network's children cannot run under the systemd-user backend (plan 0013 ACC13-NO-HOST-OOM, read on dev-soak-2, 2026-10-18; rewritten for plan 0026 on 2026-10-23).** `dev up` refuses `--host-backend systemd-user` (`bringUp` in `cmd/socialweb-computer/dev.go`: "dev places every unit under devsup (proc); systemd-user is for install"). So every HostProcess of a market network (kcp slots, kine, OpenBao, the fronts) runs under devsup with no `memory.max`, and only the VM units go to the user manager (`--vm-host-backend systemd-user`). `run bidder --host-backend systemd-user` runs only for an account that `install bidder` made (`bidder.CheckAccount` in `internal/stack/bidder`). The second soak therefore read a breach on a second bidder made with `install bidder` and pointed at the network with `--market-network-file`. The components of a market network run in its cores' processes and have no limit on any backend (see the item above). The change that closes it lets `dev up` place its children's cores and HostProcesses as user units (Delegate=yes, OOMPolicy=continue) and carries the backend into the child records. - **The bidder's kcp wait on the hot path: the prod reading is open (plan 0009 ACC9-LATENCY, 2026-10-17).** The cuts and the readings on dev-linux-1 are in section 15 (M4-LATENCY-READINGS item 10): the kcp wait went from 30 / 42 to 11 / 13 ms p50 / p95 (4 at a time, one series of two at 25.7) and from 30 / 93 to 12 / 16 serial. What is left is the prod bidder's own `status.market.kcpWait` over 256 RFPs after the deploy. If its p95 is over 20 ms at a host load under 16, the next cut is the Bid create beside the bid config and payload PDS writes. If it holds, this item closes into section 15. - **The Windows bidder: the seams as they landed (plan 0011 M0-REVISIT, 2026-10-16, read at 79f40adb; re-read against the plan 0026 tree on 2026-10-23).** `GOOS=windows GOARCH=amd64 go build ./...` and `go vet ./...` are green (`deploy/check.sh --fast` builds windows/amd64 and windows/arm64). The plan was written against names that did not land: - The VM seam is `vmruntime.Runtime` (Prepare, Placed, Reserve with a `Placer`, Await, Start, Inspect returning an `Observation`, Shutdown, Stop, List, Cleanup, Watch, Load, CollectImages). The verdict type is `hostsvc.Verdict` (an alias of `internal/contract/host`'s). The service-manager seam is `hostsvc.Manager`; its `UnitSpec` is an argv plus cgroup limits, so it cannot describe an HCS system. The FirecrackerVM backend (`internal/compute/firecracker`, driven by `internal/controllers/firecrackervm`) does not call the runtime alone: `Reserve` hands a `Placer` that creates a `HostProcess` child, the supervisor starts the unit through `Manager`, and the backend reads liveness, orphans, exits and events from `Manager`. A Windows VM backend therefore needs the backend to ask the runtime, not the manager (item M1-CONTROLLER-RUNTIME-OWNED). - `status.runtime` is `{backend, ref (the unit name), instance (the InvocationID), pid, startTime (start ticks as a string), hostBootID, vmmID}`; `Backend` is a free string in the API, but `hostsvc.ParseBackend` accepts only `systemd-user` and `proc`. - kcp is a pinned asset (at HEAD built here from kcp-dev/kcp main at 74e9fb9c, version `v1.36.0+kcp-v0.33.0-162-g74e9fb9c03f778` in `internal/versions/versions.json`) in its own unit behind kine on sqlite (cgo) and our front, not in-process. The assets table has no windows/amd64 kcp, kine or any other entry (`platforms["windows/amd64"]` is status seam with zero assets). The `proc` backend is `devsup`, which is Linux-only. The bootstrap layer is installed systemd units. - The guest image is a `guestImages` target built by `scripts/build-guest-image.sh`, never embedded, fetched and verified by `internal/guestimage`; `hyperv-x86_64` is a reserved target name. It has no initramfs: the kernel boots `root=/dev/vda` from the rootfs ext4 and cloud-init reads a pure-Go NoCloud cidata drive (`internal/guestseed`). There is no `socialweb-nodeinit` in the VM path. - The guest network is constant: guest 172.30.0.2/30, gateway 172.30.0.1, MAC 06:00:ac:1e:00:02, host address 192.0.2.1 (`internal/guestnet`, pasta maps 192.0.2.1 to the host loopback). A guest reaches the host loopback only on the TCP ports of its ComputeVM's `spec.hostPorts` (default none): socialweb-vmexec loads an nf_tables bridge table `swcguest` in the guest's namespace before it wires the tap, which accepts those ports to 192.0.2.1, resets every other TCP connection to it, drops all else to 192.0.2.1 and 127.0.0.0/8, and drops every frame from the tap to the namespace's own stack; pasta runs with `--no-map-gw` and no port forwards. The bidder sets `spec.hostPorts` from its `GuestHostPorts`, and in a market network adds the guest TLS port and the ports of the user_data's 192.0.2.1 and `*.localhost` URLs (`guestseed.HostPortsOf`). There is no network index allocator. An HNS NAT network per host cannot give every guest this plan; M0-HNS-NETWORK measures what Windows can do with it. - iroh is the default and required guest transport; guests have no JSR registry or websocat, and a tunnel module (opentunnel, wootty or socialweb-guest-agent, installed by the bidder from its own `/v1/downloads`) only when the contract's `spec.tunnels` asks for it (plans 0032 and 0034). - The core binary installs to `bin//` with `bin/current` and `bin/previous` (not `versions//`); assets are `runtime//`, collected when unreferenced. `install` is behind `install.System` and `UserManager`, which are systemd and account shaped (`system_other.go` refuses on !linux). The bidder core is `Type=notify` with a watchdog (`internal/notify`). - The components run in the core's process (`internal/component`), so they need no process seam on Windows. Plan 0011 names a runner that no longer exists. The policy engine still starts its embedded deno as a child process to run the policy actions (`internal/policyengine/workflow`), outside the HostProcess units. - Windows stubs today: `hostsvc.Open`, `OpenProcClient`, `ListenPort(s)`; `vmruntime.ForHost`; `hostos.BootID`; `install.NewSystem`, `SelfCheck`, `RestartOwnUnits`, `NewStagedSystem`; `hostcontract.Gather`; `guestnet.Wire`; `devsup`; `nsexec`; `apiserverfront` hold; `cmd/socialweb-vmexec`; the release asset blobs (`assets_release_unsupported.go`). `hostos` process identity, lock, port range and `DieWithParent` (empty) have Windows bodies. - Decision (decided by Claude, user may veto): on Windows every non-VM host process of a bidder (kine, kcp slot, front, OpenBao, the policy engine's deno) is a child of the bidder service inside a kill-on-close job object. Only VMs outlive the core, and HCS owns them. This replaces the plan's pre-installed vault service. - Decision from the user (2026-10-16): the Windows desktop is never rebooted, shut down or signed out. A reboot-only proof is `blocked` and its closest no-reboot equivalent is measured and labelled. - **An application owns its PDS: the PDS is the Pds component in its own kcp (plan 0018, opened 2026-10-17; rewritten for plan 0026 on 2026-10-23).** Bob's bsky.social account hit the hosted cap of 35000 write points per 24 hours because the prod bidder (MarketIdentity mode oauthQr) wrote every market record into Bob's own repo. Since plan 0018 the bidder's MarketIdentity is mode localRepo and its repo is a LocalRepo on the application's own Pds component (`internal/services/pds` over `internal/pds/local`; the repo kinds are in section 12). appctl writes the repo key once into the application's OpenBao (`identity/bidder/keys`, field `repoKeyHex`, `internal/app/marketbidder` and `internal/app/compose/vault.go`); the Pds component serves the repo on port 0 and, through the app's IngressProxy, on the public URL in the DID document's `atproto_pds` endpoint. Records the application declares are AtprotoRecord objects that appctl composes in Go (`internal/app/compose/records.go`: the operator vouch, the association record, `OnMissingReference: Delete`); `internal/controllers/atprotorecord` writes each to its repo, keeps it Pending with condition ReferencesResolved False, reason MissingReference, while a runtime reference is unresolved, rewrites it when a referenced value changes, and deletes it with the object (finalizer `atprotorecord.socialweb.computer/repo`). The association is dynamic (see "The operator comes from the event stream" below). The gateway requester side: the Gateway root composes the same Pds component for the MarketIdentity `requester` (mode localRepo, AtprotoRecordWatch `association` from `deploy/apps/repo-pds/association-watch..yaml`), with its key at `identity/requester/keys` (`internal/app/gateway/vault.go`), so rfp, compute.vm, accept and rbac records no longer go into alice's bsky.social repo; spec.ownerDid stays the account of a contract for the per-account limits. People still sign in to the web app through the atproto auth mount; only the identity that posts RFPs changed. Plan 0018 is marked done. Open: the live proof's readings were not written here, and it is not read whether prod requesters find the bidder DID through the relay index or need it in the Gateway's `bidderDids` (the operator's vouches name Bob, not the bidder DID). - **The operator comes from the event stream, not from the config (plan 0018 M4-DYNAMIC-ASSOCIATION, opened 2026-10-17).** Neither the Gateway spec nor the MarketBidder spec names an operator; `operator` (handle or DID) is an optional allow-list on both. Records are public and a new DID is visible on plc.directory, so a claim also carries a nonce: random, single-use, kept in the application's OpenBao (`identity//claim`), shown only in the MarketIdentity status `pendingAction.url` (`/#bdr|plc=&n=&s=`, admin-only kcp status). The operator writes, from any atproto client, one `com.publicdomainrelay.temp.badgeBlueKeys` record in its own repo: {keyId: the application's DID, challenge: the operator's DID, service: bidder_associate | requester_associate, nonce, createdAt}. The kind AtprotoRecordWatch (export marketidentities) is the mirror of AtprotoRecord: spec {repo, collection, match: [{fieldPath, value | valueFrom}], source: {eventSourceRefs}}, status {records: [{did, handle, uri, cid, value, verifiedAt}], cursors: [{source, cursor}], conditions: Verified}. The AtprotoRecordWatch controller (`internal/controllers/atprotorecordwatch`, in the core since plan 0026) reconciles each watch: it writes an EventStream child per source (`internal/eventstream` plans them; relay subscribeRepos with a client-side filter, Jetstream `wantedCollections` in `internal/events/jetstream`), takes the matches the streams report only as triggers, and verifies the records through its backend's `Verify`, which fetches each with com.atproto.repo.getRecord from the author's PDS; only a verified record stays in status. The condition Verified is False with reason MatchPending or StreamNotReady until the streams are ready, and VerifyFailed after a failed pass, which is retried after 30 s (`VerifyRetry`); a passing watch is verified again every 5 minutes (`VerifyInterval`; both listed in `deploy/obligations.txt`). Each source resumes from its cursor in status. `deploy/apps/repo-pds/association-watch..yaml` declares `association`: keyId equals valueFrom MarketIdentity status.did, service equals the application's service; its sources are the EventSources the root composes: the network relay (`relay-network`, from the network value's relay host) or, with no relay host, two public Jetstream instances (`jetstream-bsky-us-east`, `jetstream-bsky-us-west`). The MarketIdentity reconciler (mode localRepo, `spec.associationWatch`) waits on the watch: the first verified record whose author is its challenge and that carries the current nonce (or whose author is the allow-listed operator) wins; it is written to `spec.approvedAssociations` and the nonce is deleted. Then the identity is locked: a later claim needs an admin patch of `spec.approvedAssociations`. The association record (`onMissingReference: Delete`) follows `approvedAssociations[0]`. Revocation: once the watch is Verified and no record of an associated DID is left, the reconciler removes that DID from `spec.approvedAssociations` (condition Associated False, reason Revoked), the association record leaves the repo, and a fresh nonce is shown. A restart keeps the association because the watch re-verifies the operator's record. The associateConfirm call stays a fast path for an allow-listed operator. Open: the did-key-associator QR page (qr.fedfork.com) does not yet copy `n` into the record it writes. - **A prod tenant owner's repo through the web app session (plan 0031 REC9-PROD-OWNER-SESSION).** Built: the MarketIdentity mode `oauthSessionProxy` (`spec.did`, `spec.sessionRef` {Vault `root-vault`, mount `auth/atproto`, path `session/`}) reads the session the openbao-plugin-auth-atproto mount holds through the bus capability `vault.atproto.session` (contract `vault.AtprotoSessions`, the handle logs in as the root vault client `atproto-session-proxy`, policy `atproto-session-proxy`, scope the one session path the ComponentLink grants); its pds.Authenticator asks the plugin's proof endpoint for one access token and one DPoP proof per request, and only the plugin refreshes the session. No session, or an expired or revoked one, is phase SessionExpired with reason NoSession or SessionInactive and pendingAction signInAgain. A session must hold the scope that the sign-in which produced it requests today: for this mode that is the gateway web app sign-in, whose scope (`compose.GatewayAtprotoScope`, the same value the atproto plugin config requests) the Gateway root writes into `spec.sessionScope`; the plugin's recorded granted scope is compared with it (`oauth.MissingScopes`), and a session that lacks part of it is SessionExpired with reason ScopeChanged ("the web app session of lacks scopes the gateway web app now requests (...); sign in to the web app again"). The other OAuth modes compare with the scope their own flow requests (the core's role scope; for oauthQr the bidder scope the associator grants, not the QR page's union). A stale owner pauses only its own Infra writes: its RemoteRepo is not Ready with the identity's message, its Infra records are not Synced, and the Gateway root stays Ready with condition SessionOwnersReady False naming the owner. The identity returns ErrSignInAgain, so the runtime's rate-limited backoff (at most 5 minutes) reads the plugin again; a new sign-in makes it Ready with no other step (the plugin pushes no session change). The Gateway root composes it, as `owner-` in `root::app` with label `apps.socialweb.computer/session-owner`, for each DID that owns a Cluster (not deleting) and is not in the market network owner sessions file nor the requester's own, and keeps it while an infra AtprotoRecord names its repo; once the identity has reported the session's PDS (Ready, or SessionExpired after that) it adds the RemotePDS of the session's PDS, the RemoteRepo and the AtprotoRepo `repo-`. The web app scope carries `repo:computer.socialweb.infra` (create, update, delete), so owners sign in again once. Every Cluster's Infra is published to its owner's repo; there is no opt-in (the `spec.publishInfra` field, the setPublishInfra procedure and `POST /api/clusters/publish` are removed, 2026-10-24), because Infra is how the gateway and the web app manage compute contracts. Deleting the Cluster deletes its Infra and so its record. Open: the prod reading of done_when (a market network gateway has no public-origin, so no web sign-in). - **The market network runs its applications in one kcp: what is left (plan 0019).** The design as built and its readings are in section 15, "The market network is one kcp (plan 0019, 2026-10-18)". Open: the web app's account and home workspaces (`root::`, the tenant clusters) still sit at the root of the network's kcp, not under root:gateway (kcpclient, the clusterprovider registry, sweepkcp and the tenant credential paths take `root` as their base); the composer still copies the network value and the bidder DIDs onto the root objects (no record carries them); the per-child core holds the network's admin kubeconfig (the shared kcp isolates components by workspace RBAC, not the cores); the host map of a core's component runtime (`component.NetworkHostMap` in `internal/component/network.go`, built from the component registry) holds the names of every workspace whose components that core runs, each name carrying its workspace; its lookup (`Address`) takes no caller, so a component can resolve another workspace's names. ACC19-ONE-KCP is marked done in plan 0019, but the integration tests it named were deleted by plan 0026, so the one-kcp network has no standing check until plan 0027. ## 15. Settled decisions, and the defects they closed - **pkg/embedded keeps impersonating system:kcp:admin; a stalled workspace initializer is a kcp retry fix plus a self-explaining wait error (plan 0038, 2026-10-31).** Before: 2 of 7 iOS app launches stalled with `Workspace [app] in root for Ready after 1m0s (last error: Workspace app not Ready)`, CPU idle, requester never started. kcp's apibinder initializer creates a workspace type's default APIBindings as the workspace owner (kcpapply stamps the applying identity), records any create error other than a missing APIExport on the LogicalCluster (`WorkspaceAPIBindingsInitialized=False`, `WorkspaceInitializedAPIBindingErrors`) and returns nil, and its LogicalCluster informer has no update handler, so one transient error (the owner's RBAC not yet visible at boot) leaves the workspace Initializing for good. Upstream kcp v0.33 with a system:masters base: 0 of 14 harness boots stalled (8 after /readyz, 6 racing Start from "Serving securely"). A forced stall (non-masters base, so the impersonated identity is scoped) now reads `Workspace socialweb-provider phase Initializing initializers [system:apibindings]; APIBindingsInitialized=False APIBindingErrors: encountered errors: apibindings.apis.kcp.io is forbidden: User "system:kcp:admin" cannot create resource "apibindings" ... access denied`. Decided: no system:masters owner and no dropped impersonation (least privilege). The fix is the kcp fork's requeue on a failed default APIBinding create (kcp-wasm 7258b169d, rc135 kcp batch), and kcpapply's Workspace wait timeout now carries the Workspace phase, its initializers and every False LogicalCluster condition, so a remaining stall says why. - **A bidder bids only inside an RFP's bid window, and never twice on one RFP (2026-10-25).** Before (rc117 on hat-7): right after its core restarted, the bob bidder published a bid on each RFP of the gateway requester from the previous 29 h (39 in the first minute, each on a different RFP, contracts long finished). Path: a bidder market EventStream resumes from its stored cursor, rewound (a relay 1000 events, which on our low-traffic relay is about a day; a Jetstream 30 s), and a Persisted stream with no recency bound replays whatever lies behind it; every replayed `market.rfp` went through the sink into a new inbox StateSync and `enqueueBid`. No code lists a requester's `market.rfp` collection, and the gateway re-sends submitRfp only while a contract collects bids. The only dedup was the inbox StateSync of the first delivery; a01e01f7 prunes it 10 min after it is Acked, and an RFP the bidder never handled (it was out of the market) had none. The lexicon has no expiry. Now: `market.RFPBidWindow` (10 min from the RFP's `createdAt`, else from its TID rkey; the requester's default bid window is 30 s) gates every entry: the bidder sink drops a closed RFP before the record fetch and again after it (`market_rfp_closed_dropped`, with the source, every 100), submitRfp answers 409 `RFPClosed`, the inbox adapter skips it, and a Deciding Bid of a closed RFP fails `BidWindowOver`. The handled fact is the Bid object (named by `market.BidRKey`) and, after its 24 h TTL, the bidder's own `market.bid` record at that rkey (`bidder_rfp_already_bid`), never an inbox StateSync, and the bidder's inbox retention is the bid window, so a pruned entry is always of a closed RFP. Provision and release StateSyncs, the dedup of a replayed accept or event, are never pruned. Market network `rfpwin` (one bidder, comms Both, test-0001, UTC): four requests ran to Released (09:49 to 10:00, 4 bids); all Bid objects and the open RFP's inbox entry deleted (the state after a TTL and a prune), `dev restart rfpwin bidder-1/core` at 10:04:11: its market stream dialed the network relay at cursor 1 and replayed the backlog, `market_rfp_closed_dropped` from `relay/http://127.0.0.1:25569`, 0 bids; a new request then got its bid (10:05:40) and Held. With that request Held, its Bid deleted (orphaning the ComputeVM) and its inbox entry deleted, a second restart at 10:06:00 replayed the open RFP: `bidder_rfp_already_bid` from the repo record, 0 bids, and the held guest finished and released. A further request got its bid and Held. 6 `the bidder published a bid` lines in all, one per request. - **A role keeps market objects only for its own contracts (2026-10-25).** Before: the requester's market sink turned every record of the collection-filtered public streams into an Outbound, Retain StateSync in its root workspace, also other contracts' bids, payload records (`bids.free`, `compute.events.*`) and guest events; nothing deleted an Acked one, so the phone held 759 (+21 a minute), every StateSync LIST decoded all of them (3.4 GB a minute), and a `bids.free` payload reached the bid inbox as a bid (`requester_bid_inbox_invalid`). Now: a requester stream carries `market.bid`, `market.receipt` and `market.event` only, a bidder stream `market.rfp`, `market.accept` and `market.event`; the requester takes a record only when its strongRef names one of its ComputeContracts' RFP, accept or receipt (an event whose receipt is not in the status yet is followed to that receipt's accept), keyed by an in-memory index from the ComputeContract watch, and every object the intake creates is owned by that ComputeContract. Outbound + Retain stays the mode: the role adapter is the target that applies the payload (an Inbound StateSync never calls Apply), and the delete of an inbox entry has no remote side; what was missing was the addressee rule and the end of the entry's life, now the prune of a foreign entry at once and of a Released or Failed contract's Acked entry at 10 min. Compose adopts the object a watch event names instead of listing its kind, and a MarketRequester whose identity is SessionExpired, AwaitingScan or Failed, or which is Failed, parks until its identity or spec changes. Market network `inbox34` (3 bidders, two concurrent `run request --hold`, one Both and one Firehose, ssh over webrtc): - Before (da344ba9): each requester workspace held 37 StateSyncs, 8 of its own contract and 29 of the other one (payload records, events, the other receipt, 7 event mailboxes); 3.1 and 2.8 StateSync LIST a minute, 115 and 111 StateSync requests a minute while Held. - After: 6 StateSyncs each, all owned by `ComputeContract/contract`, 0 foreign; 1.9 and 2.2 LIST a minute over 6 objects, 32 and 28 StateSync requests a minute. Both Held, ssh over webrtc, release completed. - Upgrade: a held requester of the old binary resumed with the new one went from 37 to 9 StateSyncs, all its own, within seconds of the sync. A long-lived MarketRequester app with an Infra contract: its StateSyncs (ownership Infra, ComputeContract, StateSync) were deleted 10 min after their creation once the contract was Released. - **The controller runtime lists, then watches; it never asks kcp for a watch-list (plan 0034 ACC34-CLI-SSH, 2026-10-25).** Before: client-go 0.36 turns on `WatchListClient` by default, so every informer synced through a streaming watch with `sendInitialEvents`. On a kcp that just started, kcp answers each such watch with its initial-events-end bookmark only after about 1 s (kcp traces `"WatchList" ... total time: 1005ms`, one per resource). multicluster-runtime engages each logical cluster with every controller in turn, and each engagement blocks until its new informer synced, so the engagements of a resumed core advanced one per 1.1 s: the requester's OpenBaoServer was reconciled about 30 s after the controller manager started, and the Requester component started 35 s after it. `run request ssh` after a resume starts a stopped invocation's kcp again (cold), and the resumed MarketRequester still showed its old Ready status, so the CLI published its offer at once and waited for the core: ssh connect to return was 36.1 s (hat-7) and 39.5 s (local), against 2.4 s and 9.2 s for the first ssh. After: `internal/controllers/runtime` replaces client-go's feature gates with `WatchListClient` off, so a reflector lists (served from storage while the watch cache warms) and then watches. Readings (own network p34rsm, `dev down --keep-state` then `dev up`, cold kcp): the network core's OpenBao unsealed 32 s after the manager started and its last engagement came at 55 s; after, 5.3 s and 18 s, no `WatchList` trace over 500 ms (63 before), `dev up` 30 s instead of 64 s. Over webrtc on that network: ssh while the invocation runs 0.6 s (offer to answer 0.16 s); ssh after a resume 5.7 s, 7.3 s and 6.9 s, of which offer to answer is the core warming up (5.6 s: OpenBao placed 3.9 s and unsealed 6.2 s after the manager started); the guest answers each offer in 15 to 30 ms and ICE connects host to host in 0.3 s. MarketIdentity also watches the OpenBaoServers of its workspace, so an identity waiting on its vault reconciles when the vault is unsealed instead of on its error backoff. - **A tenant cluster's nodes are one Infra (plan 0031, 2026-10-24).** Before: every NodeClaim got a ComputeContract `nodeclaim-<20 hex>` that the NodeClaim controller applied, released (spec.release), deleted and orphan-swept itself, listing every ComputeContract of the gateway workspace on each reconcile and mapping contracts back through `spec.nodeClaimRef`; the Gateway root's teardown was DeleteAccountClusters, then a wait on `compose.HeldContracts`. After: the NodeClaim controller is the only spec writer of one Infra per Cluster with a claim (`nodes-` in the gateway workspace, `spec.clusterRef` from the claim's Cluster ownerReference, because the claim's `spec.clusterRef.name` is the cluster key), one entry `node-` per claim (Staged until the bundle is written, Requested, Released on the claim's deletion), the node fields in the kcp-only `spec.nodes`; the Infra controller runs the ComputeContract `infra--`; `status.infra` names the entry; `ComputeContract.spec.nodeClaimRef` is gone. A Cluster that is deleting gets its Infra deleted at once, and a root deletes its deployments (`compose.DeleteDeployments`) before its contracts and components: the Gateway after DeleteAccountClusters, the MarketRequester first (plan 0029 R8). An unbound Infra keeps its record (AtprotoRecord `deletionPolicy: Retain`, `status.record.direction: Retained`; plan 0029 U1). Readings (own network nodes31, `--bidders 1 --with-gateway`, dev builds of the branch over f8da10c9, 2026-10-24 UTC): - ACC31-SCALE-UP-TWO: a 2-replica Deployment with hostname anti-affinity (applied 05:32:12) made two claims; their entries went Requested at 05:36:12.55 and 05:36:12.59, both contracts Held and the Infra `nodes-bbba9dd001a611921f2f` Deployed "2 deployed" at 05:38:21.859, the claims Running at 05:38:32.752 and 05:38:36.814 with `status.infra` naming their entries, both tenant nodes Ready, each contract owned by the Infra with nodeJoin, ownerDid and the bundle secretsRef; no `nodeclaim-*` contract was written. - ACC31-SCALE-IN-ONE: replicas 1 at 05:42:15; after the hold the entry `node-8b685c737dff` went Released (05:44:15.725), its contract Released 05:44:15.819, the claim went 05:44:15.884, the entry was dropped 05:44:15.912 and the contract deleted 05:44:15.935; its guest was gone by 05:44:32; the other entry, contract (uid e6edc9eb), claim (uid b6f9ef8a) and node stayed Requested, Held, Running and Ready. - ACC31-CLUSTER-DELETE: Cluster `tenant` deleted 05:47:31.394; `node_claim_infra_deleted` 05:47:31.475, contract Released 05:47:31.528, Infra gone 05:47:31.567, claim gone 05:47:41.515, Cluster gone 05:47:42.741, the guest gone at 05:49:04 after its graceful stop. - ACC31-GATEWAY-ROOT-DELETE: cluster `busy` with a Running node and pod; Gateway root deleted 05:51:51.694; Deleting "1 Cluster(s) tearing down" 05:51:54.908, its Infra deleted and the contract Released 05:51:55.042, Infra gone 05:51:55.119, "13 component(s) stopping" 05:52:06.551, Requester deleted 05:52:06.727, root gone 05:52:07.712 (16.0 s; plan 0028 M4 read 17.7 s); the guest went at 05:53:26; no Gateway, Infra or ComputeContract was left. - ACC31-R8-ROOT-DELETE: MarketRequester root `root:infra-r` with Deployed Infras r8 (bound) and u1 (unbound) deleted 05:41:52.538; Deleting "2 deployment(s) releasing" 05:41:52.759, both contracts `requester_released` by 05:41:52.775, both Infras gone by 05:41:52.864, then "8 component(s) stopping" 05:41:53.319, root gone 05:41:55.482; the guests were gone at 05:41:56; the repo then held the u1 record only (r8's was deleted). - ACC31-U1-UNBIND: `spec.record` removed from the Deployed Infra u1 at 05:39:19.356 (and u3 at 05:47:01.413 on the final build): `infra_record_unbound` and `atprotorecord_retained` within 0.1 s, the AtprotoRecord, its watch and deliver link gone, the Infra Deployed with its guest Running, the record still listed by listRecords, `status.record.direction: Retained`. Open: a status apply that sets a field to null (`internal/controllers/runtime` `dropped`) left `status.record` in place on this kcp (the field's managedFields entry is an Update operation of the same manager), so a status field another pass removes may not go; read before relying on it. - **A deleted ComputeContract releases its guest (plan 0028, moved from section 14, 2026-10-24).** Before: a ComputeContract carried no finalizer, so a delete removed it at once and the requester saw only Gone: `Requester.Observe` dropped its caches and never started a release, the runner kept holding in `waitForRelease`, and no vm.delete was sent. A won Bid never expires, so the guest held a host core until someone removed it by hand. Two deleters reached this path: a direct delete of a held contract, and the deletion of a MarketRequester root, whose delete pass stopped the Requester before kcp garbage collection deletes the contracts it owns. R0-REPRO reading (own market network p28, `--bidders 1`, test-0001, dev build of c30c3dab): run 1 held `root:r0a` contract `contract` at 03:42:13Z and deleted it directly at 03:43:38Z; the object was gone at once, the requester logged nothing after the delete, and at 03:48:46Z (5 min 8 s later) the requester repo held 0 vm.delete records, Bid `bid-80d63ec51b896211ef8b` was Won, ComputeVM and FirecrackerVM `fc-xxyudiphteqt-d5414751` were Running and unit `socialweb-vm-20b17a6d47-fc-xxyudiphteqt-d5414751.service` active. Run 2 held `root:r0b` at 03:45:03Z and deleted its MarketRequester root `app` at 03:46:17Z; the Requester logged `requester_contract_left_for_restart` at 03:46:18Z, the contract went by garbage collection at 03:46:19Z, and at 03:51:21Z (5 min 4 s later) the repo held 0 vm.delete records, the Bid was Won and ComputeVM `fc-6hawypma76qe-46183fe5` Running. The market network applied a new root `app` in `root:r0b` within seconds; its Requester found no contract. Both guests were removed by hand (ComputeVM delete) at 03:51:33Z. After: the ComputeContract forwarder of `internal/controllers/marketintents` holds the finalizer `computecontract.socialweb.computer/release` (`v1alpha1.FinalizerComputeContract`) and returns Waiting on a deleting contract while the requester's `Requester.releaseWait` answers a wait: the requester lets go when the contract is Released, never wrote an accept and has no runner, or its `status.release.deadline` passed after at least one attempt, when it writes Failed "vm.delete was not acknowledged by (the contract TTL): " (the plan 0020 bound, which the Go port had left as Releasing). With no requester subscribed the finalizer stays only while `ComputeContract.HoldsGuest` is true, so a Released or never-accepted contract never blocks a workspace delete. Gone stops the runner (its context is cancelled and it writes and logs nothing more) and logs `requester_contract_gone_unreleased` when the finalizer was removed by hand. The MarketRequester delete pass deletes the root's ComputeContracts and reports Deleting " ComputeContract(s) releasing" before it stops a component; a deleting intent object now requeues the roots of its workspace (`intentWatch` in `internal/app/compose/root.go`), which it did not for an owned object. The Gateway delete pass stopped its Requester with no look at the NodeClaim contracts of its tenant clusters (M4 read: unsafe for a direct delete of the Gateway root; the network's teardown deletes the account workspaces first). It now first deletes every Cluster in the account workspaces (`compose.DeleteAccountClusters`; it writes no Cluster spec) and reports Deleting " Cluster(s) tearing down" while any is left, a Cluster event requeuing a deleting Gateway root (`compose.TeardownWatch`); each Cluster teardown deletes its NodeClaims, which the still-running Requester releases, each bounded by its release deadline; then it waits while `compose.HeldContracts` counts a contract that holds a guest, and only then releases its records and stops its components. The requester logs `requester_release`, `requester_release_retry`, `requester_released` and `requester_release_bound`. Readings (same network, dev builds of this branch, 2026-10-24 UTC): - ACC28-DIRECT-DELETE: `root:a1` Held 03:58:25; delete 03:58:37.628; Terminating with the finalizer 03:58:37.738, Releasing 03:58:37.763, Released 03:58:38.000, object gone 03:58:38.036; exactly one vm.delete record; ComputeVM `fc-cz5yawblk7nf-d871c971` gone 03:58:39.801 with its unit and work dir; the Bid has `status.termination` (03:58:38Z) and Complete=True ComputeVMGone. - ACC28-DELETE-BEFORE-ACCEPT: `root:a5b` (denyBidderDids names the one bidder) "collecting bids" 04:02:53; delete 04:02:59.813, gone 04:02:59.950; the repo holds one rfp, 0 accept and 0 vm.delete records; no ComputeVM of that requester appeared. - ACC28-DELETE-WHILE-STOPPED: `root:a2` Held with its core stopped (spec.apps[].stopped); delete 03:59:35, Terminating with the finalizer for 61 s while the guest ran; requester started 04:00:20; `requester_release` 04:00:36.244, Released 04:00:36.324, object gone 04:00:36.336, one vm.delete record, ComputeVM gone 04:00:37.647. - ACC28-ROOT-DELETE: `root:r1` with `contract` and `contract2` Held on bidder-1; root (uid 5265cb66) deleted 04:04:49.441; Deleting "2 ComputeContract(s) releasing" 04:04:52.028, "1 ..." 04:04:52.065, both Released by 04:04:52.041, "8 component(s) stopping" 04:04:52.466; both FirecrackerVM backends released (04:04:52.344, 04:04:52.442) before the Requester was deleted (04:04:52.785); the ComputeVM objects went 04:04:53.477 and 04:04:53.773; the root was gone 04:04:54.368; two vm.delete records. - ACC28-BOUND: bidder-1 stopped 04:06:30; `root:b1` deadline 04:09:49 by spec.ttlSecondsAfterFinished; delete 04:06:46; attempts 1 to 6 each HTTP 404 (`requester_release_retry`), attempt 4 after a requester restart; Failed "vm.delete was not acknowledged by 2026-10-24T04:09:49Z (the contract TTL): HTTP 404" 04:09:49.047, object gone 04:09:49.054. - ACC28-FORCED-REMOVAL: `root:f1` with bidder-1 stopped; delete 04:06:50.218 and a merge patch of `metadata.finalizers: null`; one `requester_release_retry` 04:06:50.324, then `requester_contract_gone_unreleased` (phase Releasing, its receipt) once at 04:06:50.353 and no further line for the contract until its core stopped at 04:07:05 (the 5 s retry did not run). - M4-GATEWAY-ORDER, before the Cluster teardown (network p28g, main 93f4de91): with the tenant cluster "tenant" holding one Running node and a running pod, the Gateway root deleted at 04:24:44.665 stayed Deleting "1 ComputeContract(s) still hold a guest" for 3 min 15 s (the wait had no bound of its own); it went on only after the pod delete and the scale-in deleted the NodeClaim (04:30:00.424): Released 04:30:00.654, Requester deleted 04:30:01.728, root gone 04:30:03.193. - M4-GATEWAY-ORDER, after (network p28h): the same tenant with a Running node and a running pod; the Gateway root deleted at 04:52:59.216; Deleting "1 Cluster(s) tearing down" 04:53:02.353, the Cluster Draining 04:53:02.348; NodeClaim deleted 04:53:12.526, `requester_release` 04:53:12.556, `requester_released` 04:53:12.742 (one attempt), contract Released 04:53:12.767 and gone 04:53:13.006; FirecrackerVM backend released 04:53:12.947; Cluster gone 04:53:15.421; Deleting "2 AtprotoRecord(s) ..." 04:53:15.522, "13 component(s) stopping" 04:53:15.857, Requester deleted 04:53:16.146, root gone 04:53:16.894 (17.7 s in all). The ComputeVM went at 04:54:44.847 after the k3s guest's graceful stop; no guest, VM unit or account workspace was left. Open: `run request --exec` ends its ssh session and stops the requester's core as soon as the contract leaves Held, so a contract deleted under a running `run request` is released only when the invocation runs again; a root waits for the bidder's release answer, not for the guest's stop, so a ComputeVM can outlive the root by its graceful stop. - **One live atproto session per account, owned by the MarketIdentity controller (plan 0030, 2026-10-24).** The bobvmbuilder oauthQr bidder on hat-7 lost its OAuth session at 2026-10-24T03:19:38Z (invalid_grant "Invalid refresh token"). The session secret at the MarketIdentity `spec.sessionRef` had five readers that each built their own live `oauthclient.Session` and could refresh and save it: the MarketIdentity controller (a new Open on every reconcile), the RemoteRepo credentials provider (a second Open in the same core), the Bidder (`spec.pdsAccessToken`), the RemotePDS (compose copied the ref into `spec.auth`) and the Requester (`spec.pdsAccessToken`, or `oauth-session.json`). The refresh token rotates on every refresh, so a copy holding a spent token that saved its data put the spent token back. Now `internal/marketidentity/atproto.Accounts` holds one `internal/pds/identity.Authenticator` per MarketIdentity; a reconcile, a sign-in and every bus resolver use that one session, and only it refreshes (`oauth_session_refreshed`, did and session id, no token). The controller provides `pds.Authenticator` (`pds.session`) and `pds.ServiceAuth` (`pds.serviceauth`) on MarketIdentity; the RemoteRepo signs its PDS requests through the first, the Bidder and the Requester mint service auth through the second (`participant.PDSServiceAuth`), each admitted by a ComponentLink its root composes. `pdsAccessToken` left the Bidder and Requester specs, RemotePDS `spec.auth` holds no identity session, and the per-process reload of e671cd9e went with the copies. The stored format did not change. The gateway web app session was not a copy: openbao-plugin-auth-atproto holds it and is its only refresher. Three defects of the market network surfaced in the live run and were fixed: the ingress tunnel dropped every response header but Content-Type (an OAuth token request never saw its DPoP nonce), the in-tree resource server checked DPoP proofs only against its issuer URL (a client on the PDS's loopback listener always got invalid_dpop_proof), and the MarketIdentity controller used a bare HTTP client that could not reach an authorization server behind the network ingress. Readings (market network oauth30, `--bidders 0`, bob's PDS `spec.oauth.accessTokenTTLSeconds` 60, a bidder outside the network by `run bidder --market-network-file` in mode oauthBrowser for an account on bob's PDS, signed in by a scripted POST to `/oauth/authorize` with an atproto localhost client_id because the PDS cannot fetch metadata behind the network ingress, UTC 2026-10-24): signed in 04:59:15; `oauth_session_refreshed` every 30 s from 04:59:45 to 05:19:15 (40 lines, one session id), one line per rotation, every line from the marketidentity-controller, no invalid_grant or SessionExpired; the offering was written at 04:59:18; `run request --network oauth30 --hold` got the oauth bidder's bid and accept at 05:07:54, the receipt at 05:08:11 and Held at 05:08:13; `run request release` reached Released at 05:08:51 and the bidder deleted the guest the same second. An earlier run logged 26 refreshes, two of its bidder core restarts reopening the one stored session, with no invalid_grant. The Bidder and Requester specs carry no `pdsAccessToken`, the composed RemotePDS has an empty `spec.auth`, and the only ComponentLinks to the MarketIdentity are the Bidder's `pds.serviceauth` and the RemoteRepo's `pds.session`. The bidder's OpenBao has no audit device, so who reads the path is shown by code, not by an audit log. - **The guest image is published once, under the key of its pins, on 2026-10-23.** Every tag and `main` push rebuilt `firecracker-x86_64` from source (about 15 minutes of the `guest-image` job) and uploaded the same 1.5 GiB rootfs again under `releases/` or `main/`, although the build is reproducible and the pins rarely move. The image now lives at `guest-images///`; `image.json` names the sha256 of the kernel and the rootfs, so its own sha256 names the whole image. The `guest-image` job reads that key without credentials: present, it fetches the three files, checks each against its pin and checks the input sha256s the published `image.json` records (kernel source, base, kernel config, builder Dockerfile, `deploy/guest-image/rootfs.json`) against the tree, and neither builds nor hands publish an artifact; absent, or when a fetch or a hash fails, it builds as before and publish uploads to the key with `publish-spaces.ts --content-addressed`, which writes only what the key does not already hold. `guestimagerelease.Source` tries the key first and keeps `releases/`, `main/`, `main/latest` and the newest release after it, so a binary published before the key and every image already under those prefixes still resolve; no consumer was found that needs a new per-release copy. Reading taken before the first CI run: the key answered 404, the three files under `main/fd1a644/guest-image/firecracker-x86_64` hashed to their pins, and a build of this commit resolved its image from a local server holding only the key, then from the public bucket through `main/latest`. - **Gaps of the removed TypeScript runtime, closed by plan 0026 removing it (moved from section 14, 2026-10-23).** Each was open in section 14 because of a mechanism that plan 0026 deleted: the TypeScript services ran as pods of a deno runtime in the core, each pod in its own run directory and, on an enforcing host, its own cgroup. Since plan 0026 the components run in the core's process, in the component runtime of `internal/component`, and host processes are HostProcess units. No Go code at HEAD names the run directories, the TypeScript tree variable or the restart annotation. - An upgrade did not restart the pods that ran the old assets (2026-10-19: after rc19 the gateway's requester still logged the rc17 `bidder_discovery` shape, because a core restart adopted every running pod and an upgrade changed neither the spec generation nor the restart annotation). The fix recorded the TypeScript tree in each run and replaced a run whose tree differed; its prod reading was pending. Now a component is a goroutine of the core, so the core that an upgrade restarts runs every component from the new binary. - A pod's spec change (its args or memory) did not restart it; only the restart annotation did. Now `Runtime` in `internal/component/runtime.go` stops and starts a component when its spec generation changes, and `dev restart` writes `spec.restartedAt` (`internal/stack/componentcore/source.go`, `internal/app/compose/apply.go`), which is a spec change and so a new generation. - The cgroup of an exited pod was left behind (62 empty `pod-` cgroups after the 60-restart breach; only a core start cleared them). Components get no cgroup of their own, so there is none to leave. The core's cleanup of the pod cgroups at exit and its `supervisor` leaf went with them. - The per-pod memory limit (plan 0013 M3-DENOPOD-LIMITS). Before: a pod had `spec.resources.memory`, ran with the V8 heap flag at 80% of it and, on systemd-user, in its own cgroup with `memory.max`; a breach restarted that pod alone and was counted in `status.restartReasons` (read on hat-8, 2026-10-15: a 128 Mi pod `hog` restarted 6 times in 40 s with `HeapLimitExceeded: 6` while its neighbour kept its run id and the core stayed up; on a real user unit the pod that allocated off-heap buffers was killed alone with `OOMKilled`). The mechanism is gone with the pods. The HostProcess unit limits stay, and the missing per-component limit is open again in section 14 in its current form. - Disk: the run directories under the state dir (removed with their object, swept after an hour, stdout and stderr rotated over 16 MiB), the per-closure `deno/` (6 GB on prod) and `ts/` (4.2 GB) trees that old builds seeded, and 26,645 exited run dirs on prod. Nothing at HEAD writes them. The one deno left is the policy engine's, which runs the policy actions: `internal/policyengine/deno` extracts its embedded binary once per deno version into `policy_engine/deno-embedded/` under the user cache dir. - The relay backlog fell to 10000 frames and the relay pod got 1Gi; the relay RSS read 253 to 316 MB (414 at most, +15 MB an hour), flat by the fit rule. That was a reading of the relay pod. The Relay is now a component in its core and has not been re-read. - **Items closed by later work (moved from section 14, 2026-10-23).** Each was still listed as open in section 14 after the work that closed it landed. - `scripts/validate-candidate.sh` could not pass. Before: lane b waited for the Go integration suite that plan 0026 deleted, and lane c upgraded from v0.0.0-rc15 across a change of shape and asserted a `runs other assets` log line that no code prints any more. Nothing called the script. Now it is deleted (2026-10-24, user decision): a candidate is validated by the CI run on its tag and by the acceptance runbook `docs/acceptance/0001-auth-modes.md` on prod and every bidder host. - PDS ingress per app (plan 0022). Before: the API carried `spec.ingress` and `Pds.spec.requireIngress`, but no app root created the IngressProxy and nothing gated account creation on a Ready ingress, so an app PDS could have a loopback service endpoint. Now plan 0022 is done (W1 and W2 items done, closed by 1ab24182 on 2026-10-22 with live PDS proofs on prod and every bidder host): the app roots compose the `ingress` IngressProxy (`compose.AppIngressName`), the PDS spec takes its public URL from it (`compose.AppPdsSpec`), and the Gateway root keeps the PDS waiting while `requireIngress` is set and no public URL is known (`internal/app/gateway/gateway.go`), with reason IngressProviderNotReady while the IngressProxy is not Ready (`internal/app/compose/pds.go`). - A retired kcp slot's audit logs were never removed (prod's state dir held 62 GB, 46 GB of it `kcp/slot-a/audit-*.log` of the slot that rc14's roll retired). Fixed 2026-10-18 by 1810e4a3: a roll removes the retired slot's `audit-*.log` files (`internal/bootstrap/rollinstalled.go`, log line "removed the audit logs of the retired kcp slot"). - kcpapply refused an object that `kubectl annotate` had touched, which was how an operator restarted one pod: prod's gateway apply failed ("the object was applied by kubectl and is never adopted") and the web app went 502 until the managed-fields entries were cleared by hand. Fixed by 8623a675 (2026-10-18): `refuseKubectl` in `internal/controllers/runtime/kcpapply` tolerates a kubectl manager that owns only metadata (`annotationsOnly`) and still refuses one that owns spec or status. A component restart is now `spec.restartedAt`, written by `dev restart`, not an annotation. - The object-count quota defaults were retuned by the soak (plan 0013 M2-QUOTAS, 2026-10-18): Bids 10000 (was 20000), because 10000 Bids put the bidder kcp at about 2.1 GiB of its 3 GiB limit (0.14 to 0.15 MB of RSS per Bid on a base of 610 to 680 MB). At HEAD `compose.QuotaDefaults` (`internal/app/compose/infra.go`) is Bids 10000 and ComputeVMs 64 on a MarketBidder, ComputeContracts 1024 on a Gateway and on a MarketRequester, PolicyWorkflowRuns 500 per application workspace, and Components 16 per component kind; the DenoRuns quota went with the DenoRun kind, and `NodeClaims` 64 is declared but no quota entry uses it. The root's `spec.quota` overrides each (`-1` removes the limit). - The web app's node claim index kept one ConfigMap per home ever created (201 at 22:33Z on 2026-10-18, +55 an hour in the network kine). Closed by 6768644f (2026-10-18): a deleted cluster's index ConfigMap is removed and a sweep removes entries whose cluster is gone. Not re-read. - An installed bidder took no bid TTL from its config.json (the second soak bidder ran with the 86400 s default until a unit drop-in passed one). Now the TTL is `Bidder.spec.bidTtlSeconds` (default 86400), and a MarketBidder root sets it through `spec.components`, key `Bidder`, `settings.bidder.bidTtlSeconds` (plan 0024 W0-W3, below), so no flag or config.json field is needed. - The requester retried the release with a new event (plan 0020). Before: one non-2xx answer to the direct submitEvent of a vm.delete (a 404 from the did-key-ingress-proxy while the winning bidder was away, job 618) made the runner write releaseFailed and stop, so the guest outlived its contract, and every release wrote a new vm.delete and a new market.event record. Now the release loop of `contractRunner` (`internal/market/requester/runner.go`) records the event once and resends the same uri and cid until a 2xx, writing Releasing with the attempt and the last answer, up to the contract's deadline. ACC20-BIDDER-BACK and ACC20-ONE-EVENT were marked done on 2026-10-17 (1526b2e9) and the plan closed (752df79d); their readings were not written here. - The node half inside the control-plane container (a STATED LIMIT: a containerized subnet allocator sees only the container's networks and would lease a range that collides with the host's), and two concurrent proofs in two containers taking the same ports (`liveports.LockDir()` is per container). Both bounded a container that no longer exists: 0006 M5-RETIRE deleted the container path and `deploy/control-plane-container/` (cd68b0ae, 2026-10-13). The allocator's one reach site is still the materialiser's teardown release (`OpReleaseSubnet`), and `subnetallocator.Config.HostNetworks` is still injectable. `internal/liveports` has no importer at HEAD, since plan 0026 deleted the Go tests. - **A revoked identity's ServiceAccount is erased from the datastore before k3s starts (plan 0026 A9, opened and closed 2026-10-23).** Before: revokeIdentity deleted the did- ServiceAccount at once in a running cluster, and a stopped cluster recorded ColdDeferred; the cluster provider deleted the account at wake through the tenant apiserver, so from the apiserver's first answer to that delete (about 5 s in the live run, 20:16:50Z to 20:16:56Z) the stale token got 200 directly at the loopback apiserver. Now the k3s step's Prepare, after kine is ready and before the k3s unit is placed, reads the owner's identity state and, when it is revoked, deletes `/registry/serviceaccounts//` and every `/registry/secrets//-token-*` key over kine's etcd v3 API (`internal/tenantstore/kineetcd` behind `internal/contract/tenantstore`, wired in `internal/providermain` as `StackConfig.Revoked`), each with a mod-revision compare. A failed read or delete fails the step, so bring-up retries on the runtime's backoff and k3s never starts with the account present. The at-wake API delete and RevocationSettled stay the confirmation: the provider then reads the account absent (reason ServiceAccountAbsent) and the atprotoidentity controller moves the identity to Revoked. A cluster that was running at revoke time and restarts later runs the same erase, which finds nothing. After, read live on this host (market network `fixwindow`, one gateway, two non-exempt accounts; probes every 150 ms with the stale token at the front's advertise port, the k3s slot port and the kube-apiserver port): cluster b stopped, revoked (ColdDeferred), woken by `spec.running`; kine ready 22:21:09.045Z, the erase 22:21:09.096Z (8.6 ms, deleted true), k3s placed 22:21:09.098Z, first answer 22:21:09.253Z (503), 401 from 22:21:09.972Z, RevocationSettled ServiceAccountAbsent and identity Revoked 22:21:16Z, Ready 22:21:17.6Z; 2538 probes from 22:21:02Z to the teardown at 22:23:17Z, no 200 (connection refused, a failed TLS handshake at the front, 503 from the k3s supervisor, then 401). Cluster a, revoked while running and restarted: erase deleted false, 401 from the first answer, no 200 in 1055 probes. The non-revoked account's cluster k woke in the same minute with no erase, its ServiceAccount the same uid, and its token 200 from the first apiserver answer (22:21:30.6Z). - **Per-component suspend and typed overrides on the app roots (plan 0024 W0-W3, 2026-10-22).** The five app roots and `NetworkApp` accept `spec.components`, keyed by `Kind` for a singleton or `Kind/name`. appctl resolves every entry against the root's desired component set before it writes a child, and fails the pass closed on an unknown or ambiguous key, a settings block for another kind, an unsupported field or a negative value. It writes the child spec with `root.spec.suspend OR override.suspend`, so root suspend stays the master kill switch. Typed settings reach the Bidder heartbeat, offering refresh, bid TTL and policy poll; the Requester concurrency, contract/bid/receipt timings, sessions, directory cache and OAuth scopes; the GuestControlPlane login mount, role, grant/issuer timings, host-key attempts, relay toggle and HostKeyRef; the PDS blob/request limits, contact email and OAuth TTLs/scopes; the SshServer limits; the GatewayAPI node poll; the IngressProxy timeouts; the Relay crawl interval; and the PolicyEngine limits, trust and cache. Root status carries `componentSummary`, `componentOverrides`, and the `ComponentsReady`, `ComponentSuspended`, `ComponentOverridesValid`, `ComponentDrift` and `LimitsEnforced` conditions. A per-component suspend does not propagate to that component's Service or Exposure or to the shared IngressProxy. `ComponentLimits` resolves and reports but the in-process component runtime does not enforce it; the `LimitsEnforced=False` condition says so. Live proof on this host, hat-7, hat-9 and the prod gateway is W4 and has not run yet. Not gaps. Each entry was open once, and each says what settled it and what the settlement was - and, where it matters, that it was settled AFTER the entry was written. That timing is kept because it is the case nobody re-reads: a gap still listed as open reads as honest incompleteness, which is why a bullet can outlive the work that closed it, and why one whose subject has been deleted can sit in a list of open questions for hours without anyone stopping on it. The measurements stay with the entries rather than being tidied away, because a number without its condition is how the 332-second boot outlived its own refutation. - **The pds sockets, the gateway's orphan MarketIdentity and the uncounted OOM kills of the second soak (2026-10-19, read on dev-sock-1, c-32, a one-kcp market network with the gateway under `soak load` at 1.1 RFP/s, 3 VM lanes and 4 tenant lanes, plus a bidder made with `install bidder --allow-dev-binary` and pointed at the network).** Three items of section 14 closed: - alice-pds and bob-pds did not leak; `soak sample` did (099ec6c5). `ss -tanp` per pid each minute, with the peer of each socket resolved to its process: with no sampler both held 16 to 20 ESTAB sockets (the cores' clients, the relay, the dispatcher); with the old `soak sample --every 1m` the sampler's own process held 1 more inbound keep-alive per PDS each minute (10 after 10 minutes, 00:10Z to 00:20Z: alice 6 to 15, bob 18 to 28). `pdsSample` built a new `http.Transport` per sample and never closed it, and a Deno.serve listener keeps an idle keep-alive connection while its client does. With `CloseIdleConnections` the series stayed flat for 63 minutes under the load (00:21Z to 01:23Z: alice 4 to 8, mean 5.6; bob 16 to 19, mean 17.8). The second soak's +59 an hour is that sampler; the steps down at its core kills were the cores' own pooled connections. - The orphan `root:gateway:app/requester` was a report without an owner link, not a leak (55682b85). The Gateway controller wrote that MarketIdentity into `root:app` with no label, and `adoptChildren` adopts only in the root's own workspace; the object lives as long as the `app` workspace, which carries the Gateway's owner reference. Every MarketIdentity that appctl writes now carries `apps.socialweb.computer/root`. Removing the label by hand brought back the soak's reading (orphanTotal 1, example `root:gateway:app/requester`); a gateway core restart wrote the label again and the next scan read 0. The Gateway's status read orphanTotal 0 in all 62 per-minute samples of the churn run (80 tenant cluster cycles, 121 VM contracts). ACC13-NO-ORPHANS stands. bidder-1's scan caught a ComputeVM between its create and its adoption 9 times (a new name each time, none left at the end). - An OOM kill of an adopted run now counts (a0e475c2). The pods of an installed core live in its runs unit and outlive a core restart; the new core adopts each run without a wait goroutine, so no done.json was written and the kill counted as `Restarting`. In the second soak the installed pds run of 19:11:45Z, adopted at the 19:27Z core restart, died at 20:22Z (`Restarting`), the run of 20:22Z, adopted at the 21:22Z kill, died at 21:43Z (`Restarting`), and only the run of 21:43Z died in the core that started it (`OOMKilled`). Observe now reads `oom_kill` and `memory.max` of an adopted run's pod cgroup and writes done.json, and the settled-cgroup sweep keeps the cgroup of a run that a DenoPod still names. Six forced kills (`memory.max` of the run's pod cgroup set to 8M) on the installed bidder, four of them after a `kill -9` of its core: the kernel logged 6 `oom-kill` events in `pod-pod-*` cgroups, and `restartReasons` read pds `OOMKilled: 4`, guest-kcp `OOMKilled: 1`, policy-engine `OOMKilled: 1`. - **The contract watch's getRecord bursts: the gateway requester's native RSS and the pds DenoPod's growth (plan 0013 M4-SOAK-READING, 2026-10-18, commits fdeb7697, b97422d3, 9b600a20).** Before: the gateway requester DenoPod's RSS rose 227 to 550 MB in 90 minutes with its heap flat at 45 MB, and in the second soak an installed bidder's pds DenoPod (512Mi) was OOM-killed at 496 to 499 MB anon 81 to 85 minutes into each run (the next entry). Cause: `openContractWatch` opens a dedicated relay watch per contract and calls getRecord on every bid, market.event and receipt the watch sees, before it can tell whether the record is its own. A dedicated watch started 1000 events back from the stored cursor (`resumeCursor`'s overlap), and a requester with no stored cursor (the gateway's) started with no cursor, which this relay answers with its whole backlog (10000 frames). So each new contract replayed hundreds of records and fetched each one at once: 830 concurrent getRecord calls per contract on the soak requester and 300 to 700 on the gateway requester (a fetch wrapper's in-flight peak, every 10 s). Each call took its own keep-alive connection to the network dispatcher (`ss`: 850 to 900 ESTABLISHED, idle about 60 s, reaped after 90 s, a sawtooth). The bidder's pds DenoPod served all of them through its ingress. The heap stayed small and the RSS was native: the requester's connection buffers and the glibc main arena (`[heap]` in smaps, which a burst raises and never returns). The change: a dedicated watch starts at the newest cursor that any watch of this process has seen, or the stored cursor with no overlap, and the shared stream, once it is set up again after a teardown, starts there too. The 1000-event overlap stays for the first connect after a restart. After the change the fetch peak is 4 to 9 in flight, apart from one burst at the start of each process (the first replay). hono-repo-pds logs `pds_memory` every minute (`--memory-log-ms`). Reading on droplet gl-runner-1 (48 vCPU, 94 GB), account test-0001: one `dev up --profile market-network --bidders 1 --with-gateway --vm-host-backend systemd-user`, the app values of dev-bound-1 (bid TTL 1800 s, gateway retention 900 s), `test/soak load` at 1.1 RFP/s, a VM contract every 40 s (3 lanes), 2 tenant lanes, contract TTL 40 minutes. Load from 20:59Z to 23:22Z: 9371 RFPs, 211 VM contracts held and released, 74 tenant cluster cycles, 0 failed. /proc was sampled every minute. Before is 21:00Z to 21:37Z (the pds to 21:10Z, before a heap snapshot raised its RSS by 140 MB). After is 22:20Z to 23:22Z (the requesters were restarted onto the change at 22:14Z and the pds at 21:41Z). The fit is the section 15 soak rule: | series | before (start, end, slope per hour) | after (start, end, max, slope per hour) | after plateau | | --- | --- | --- | --- | | gateway requester RSS (MB) | 147, 168, +29.8 | 146, 154, 155, +6.4 | yes | | gateway requester heapUsed / external (MB) | 38 to 46 / 6.8 | 42 to 43 / 6.8, flat | yes | | gateway requester fds | 42, 337, +334 | 45, 42, 48, -1.2 | yes | | soak requester fds | 185, 893, +850 | 41, 43, 49, +0.1 | yes | | bidder pds RSS (MB) | 172, 227 in 11 minutes, +322 | 218, 252, 252, +18.8 | yes | | bidder pds heapUsed (MB) | 36 to 60 | 46 to 76, +2.3 | yes | | bidder pds external (MB) | 6.5 to 8.7 | 19, 38, 38, +18.2 | NO (the firehose backlog filling, below) | | bidder pds fds | 30 to 33 | 30, 30, 34, +0.5 | yes | | bidder pds RSS, 00:15Z to 01:16Z (MB) | | 227, 273, 275, +11.3 | yes | | bidder pds external / brk heap, 00:15Z to 01:16Z (MB) | | 46.5 to 46.9, +0.3 / 79 to 81, +1.0 | yes | | gateway requester RSS, 00:15Z to 01:16Z (MB) | | 146, 155, 155, +5.2 | yes | The pds external rise is the hono-pds `FirehoseSequencer` backlog: a heap snapshot shows each retained Uint8Array as a backlog frame's `blocks`. The backlog is bounded at 10000 frames or 64 MiB of blocks, about 5 KB a frame here, so it fills in about 2.5 hours at 1.1 bids a second and stopped at about 47 MB (the 00:15Z to 01:16Z rows, load running to 01:16Z: 16895 RFPs, 382 VM contracts released, 132 tenant cycles). Heap snapshots (`node:v8` `writeHeapSnapshot`) raise the pod's RSS by about 140 MB that it keeps (glibc), so a reading that takes one must not be fitted across it. - **The four disk and record series of the second soak plateau (plan 0013 M3-DISK-RETENTION, M3-PDS-RECORD-RETENTION, 2026-10-19).** The second soak (below) left four series growing; each had one cause: - DENO_DIR `gen/blob` (+23.6 MB an hour): the bidder's scope gate (`--policy open`) runs the gha-lite workflow, whose node action (`dist/index.js`, 300 KB) ran from a `text/typescript` blob URL with a new uuid. Deno transpiled it and cached a 1 MB emit under `gen/blob` per run; the scope cache's 5 minute positive TTL times two requesters is 24 runs an hour. `runActionInWorker` now makes a `text/javascript` blob unless the action main ends in `.ts`, so a JavaScript action emits nothing (6daeb0f4). - One `pg-.log` per cluster ever made: nothing removed it. A Postgres deleted with `spec.deleteData` (every tenant cluster's) now removes `pg-.log` and its rotated `.1` with its data directory (33d370c6). - The repo kv files (+1.2 to +4.0 MB an hour): deno kv makes its sqlite file with `auto_vacuum` none, so the file keeps its high-water size. Every core now runs `storemaint.KeepDenoKv` over its `components` dir and the network state dir: a deno kv database (sqlite magic plus the `kv` and `data_version` tables) is converted once (`PRAGMA auto_vacuum=INCREMENTAL`, `VACUUM`), then `incremental_vacuum` and a `wal_checkpoint(TRUNCATE)` run every 5 minutes (9db73adb, eb5a164c). It runs in the core process, never in the deno process, because two sqlite libraries in one process break each other's posix locks. A conversion under a deno kv writer at about 14000 writes a second gave no writer error and a clean `integrity_check`. - com.fedproxy.rbac records of deleted clusters: only `releaseCompute` deleted them, and only for the capabilities it was given; the orphan scan's unclaimed release passes none, and a flow that throws after the accept or a contract deleted while its requester is down never releases. The requester's orphan scan now lists the rbac records too and removes those of a VM that is settled with no contract, and those whose VM no compute.vm record and no ComputeContract names (b1299ba9). Reading: droplet dev-disk-1 (c-32, sfo2), `dev up --profile market-network --bidders 1 --with-gateway --vm-host-backend systemd-user`, app values `bid-ttl-sec` 1800 and `record-retention-sec` 900, the `test/soak` load of the second soak (1.1 RFP/s, a VM contract every 40 s on 3 lanes held 20 s, contract TTL 40 minutes, retention 900 s, 4 tenant lanes) from 00:00Z, chaos every 7 minutes (`dev restart` over six DenoPods, `kill -9` of the gateway core every 42 minutes), `du` and sqlite pragmas every 5 minutes, fitted 01:05Z to 02:11Z (14 samples) with the second soak's fit rule. In the window: 4356 RFPs, 99 VM contracts, 58 tenant cycles (118 in all, none failed). | series (window 01:05Z to 02:11Z) | start | end | max | slope per hour | plateau | | --- | --- | --- | --- | --- | --- | | network DENO_DIR `gen` (MB); `gen/blob` files | 9.72; 0 | 9.92; 0 | 9.92; 0 | +0.20; 0 | yes | | gateway `pg-*.log` files (116 removed with their clusters) | 5 | 5 | 5 | -0.1 | yes | | soak requester repo kv + wal (MB) / rows | 62.4 / 59528 | 67.7 / 70444 | 80.3 / 113080 | +3.0 / +6519 | yes; the file follows the rows | | bidder-1 repo kv + wal (MB) / rows | 40.1 / 44094 | 48.4 / 52674 | 59.2 / 67100 | +2.0 / +1465 | yes | | gateway repo kv + wal (MB) | 4.3 | 1.6 | 6.4 | +1.5 | yes | | repo kv free pages after a pass | 0 | 0 to 4787 | 4787 | | freed every 5 minutes | | gateway repo com.fedproxy.rbac | 2 | 4 | 5 | +0.8 | yes: the live clusters; the scan removed 6 left by chaos | The bidder-1 repo file fell from 55.0 to 34.3 MB when its Bid records passed their TTL (00:50Z to 00:55Z), which the old file never did. The logs dirs still grow below their 64 MiB rotation caps; the network PDS and relay kv files were flat before. - **Plan 0013 M4-SOAK-READING: second 3 h soak, systemd-user (2026-10-18, fitted 20:35Z to 23:36Z; ACC13-NO-HOST-OOM holds, M3-DISK-RETENTION, M3-PDS-RECORD-RETENTION, M4-SOAK-READING and ACC13-BOUNDED do not).** Host: droplet dev-soak-2 (c-48, 48 vCPU, 94 GB, Ubuntu 24.04, sfo2), a dev build of v0.0.0-rc17 (fdf71e31). Two backends ran side by side, and every row below says which: - `[proc]`: the market network `soak2`, `dev up --profile market-network --bidders 1 --with-gateway --vm-host-backend systemd-user` as test-0001, app values `bid-ttl-sec` 1800 on the bidder and `record-retention-sec` 900 on the gateway. Its cores, kcp, kine, OpenBao, postgres and DenoPods run under devsup with no cgroup limit (`dev up` cannot place them under systemd-user, section 14); only bidder-1's VM units are user units. - `[sd]`: a second bidder `bidsd.bob.test`, made with `install bidder --allow-dev-binary` for the system account swc-bidder and pointed at the network (config.json `profile` market-network and `market-network-file` the network's MarketNetwork status; a unit drop-in adds `-- --bid-ttl-sec 1800`). It is the prod shape: the `socialweb-computer-bidder@` unit (Delegate=yes, OOMPolicy=continue), kcp, kine, the front and OpenBao as user units with MemoryMax, each DenoPod in its own `pod-` cgroup with `memory.max`, VM units in its user manager. It won 29 of the 60 VM contracts alive at 22:20Z (bidder-1 31). Load (the `test/soak` driver as user units, 18:55Z to 23:36Z): 1.1 RFP/s, a VM contract every 40 s on 3 lanes held 20 s, contract TTL 40 minutes, requester retention 900 s scanned every 60 s, 4 tenant lanes (create a cluster, a pod scales a node out through the RFP flow and in, delete the cluster). OpenBao tidy at 15 minutes on every server (`soak tidy` every 10 minutes for the network's, a patch for the installed bidder's); the other sweeps at their defaults, which are the first soak's (bidder record sweep 10 minutes, storemaint 5 minutes, run dirs every minute with a 1 h keep, guest images every 10 minutes). In the window: 11948 RFPs, 263 VM contracts (247 released, 17 failed `no_bids` around chaos), 165 tenant cycles (4 cut by the driver's stop, none failed). Chaos (a root unit): a `dev restart` every 7 minutes over seven network DenoPods (25 in the window, all rc 0), `kill -9` of the bidder DenoPods of both bidders every 42 minutes (4 each), `kill -9` of a core every 42 minutes (bidder-1's once, the gateway's twice, the installed core twice; systemd restarted the installed core). Sampled every minute: `soak sample`, `ss` and `ps` per pid, `smaps_rollup`, threads and fds of the requester and pds DenoPods, the installed bidder's unit and pod cgroups, objects and repo records; the state dir broken down by subdirectory every 10 minutes. Fit: least squares after warm-up (1 h after the installed bidder settled at 19:32Z), flat when |slope x window| <= max(2 x residual sd, 5 % of the mean). Memory breach (ACC13-NO-HOST-OOM), 19:18:58Z: the installed bidder's DenoPod `bidder` limit was set from 1536Mi to 96Mi with a `deno.computer/restart` (a spec change alone does not restart a pod). The pod was OOM-killed 60 times in 120 s (`status.restartReasons` `OOMKilled: 60`; kernel "Memory cgroup out of memory ... memory.oom.group set"), then the limit went back. 16 of the 18 swc-bidder processes kept their pids (the core, kcp, kine, the front, OpenBao and its plugin, the other three pods); only the bidder pod's two were replaced. MemAvailable read 82.9 GB before, 83.9 during and 84.8 after. Under load the installed pds DenoPod (512Mi) breached by itself three times (20:22Z, 21:43Z, 23:08Z) and was restarted alone each time. No global OOM in the run, swap 0, MemAvailable at least 76.5 GB, the host vmstat `oom_kill` counter counted only those pod cgroup kills (189 killed tasks, 3 per OOM kill of a pod). No quota was hit (0 refused creates); the M2-QUOTAS reading (a 10-Bid quota declined the 11th RFP with reason capacity) and the first soak's Bid quota hit stand for the quota clause. | series (window 20:35Z to 23:36Z) | start | end | max | slope per hour | plateau | | --- | --- | --- | --- | --- | --- | | [proc] network kcp RSS (MB) | 1691 | 1732 | 1839 | +21.0 | yes | | [proc] network kine RSS | 448 | 421 | 584 | -7.5 | yes | | [proc] network core / OpenBao | 95 / 119 | 98 / 120 | 100 / 121 | +0.7 / +0.4 | yes | | [proc] tenant clusters' postgres (under the network) | 2434 | 2872 | 2881 | -26.9 | yes | | [proc] relay / dispatcher DenoPods | 253 / 636 | 316 / 646 | 414 / 804 | +15.1 / +16.6 | yes | | [proc] alice-pds / bob-pds / PLC DenoPods | 139 / 137 / 140 | 146 / 143 / 140 | 147 / 144 / 147 | +2.4 / +2.2 / +0.1 | NO / yes / yes | | [proc] alice-pds / bob-pds sockets | 127 / 118 | 293 / 294 | 293 / 295 | +58.6 / +59.1 | NO (section 14) | | [proc] bidder-1 core | 139 | 75 | 224 | +2.7 | yes | | [proc] bidder-1 bidder / guest-kcp / policy-engine DenoPods | 252 / 154 / 116 | 229 / 149 / 118 | 330 / 164 / 119 | -7.1 / +0.2 / -0.2 | yes | | [proc] bidder-1 pds DenoPod | 325 | 273 | 338 | +1.7 | yes by the fit; +67 to +151 per run (section 14) | | [proc] gateway core / web app / pds DenoPods | 94 / 234 / 199 | 97 / 243 / 193 | 102 / 244 / 207 | +0.5 / +6.4 / -0.1 | yes | | [proc] gateway OpenBao (root and tenant vaults) | 704 | 706 | 712 | -8.8 | yes | | [proc] gateway requester DenoPod | 417 | 679 | 978 | +62.4 | NO: +345 / +388 per run, heap flat (section 14) | | [proc] soak requester DenoPod | 355 | 1552 | 2157 | +249 | NO: up to +2281 per run, heap 85 to 92 MB (section 14) | | [sd] installed kcp / kine / core / front / OpenBao RSS | 1066 / 85 / 85 / 39 / 120 | 1011 / 84 / 132 / 39 / 125 | 1116 / 158 / 230 / 40 / 126 | +2.1 / +1.6 / +4.6 / +0.0 / +1.5 | yes | | [sd] installed bidder / guest-kcp pod RSS | 232 / 158 | 303 / 159 | 327 / 161 | -1.5 / +0.8 | yes | | [sd] installed pds pod RSS | 267 | 270 | 531 | -17.3 | NO: OOM at 512Mi every 81 to 85 minutes | | [sd] installed kcp unit memory.current (with page cache) | 1225 | 1363 | 1500 | +65.2 | NO; its RSS is flat (row above) | | [sd] empty pod cgroups | 4 | 3 | 5 | -0.0 | yes; 62 after the breach, cleared only by a core start (section 14) | | [proc] network kine live keys | 6245 | 6344 | 6546 | +50.4 | yes | | [proc] network kine.db (MB) | 105 | 98 | 142 | +1.6 | yes | | [proc] Bids (bidder-1) / ComputeContracts (soak requester) | 2379 / 2701 | 2306 / 2702 | 2590 / 2940 | +1.6 / -3.7 | yes | | [proc] ComputeVMs / HostProcesses / NodeClaims | 4 / 109 / 3 | 3 / 117 / 4 | 6 / 120 / 4 | -0.2 / -1.4 / -0.1 | yes | | [proc] core/v1 objects (the node claim index ConfigMaps) | 291 | 454 | 454 | +54.7 | NO (fixed on main, 6768644f) | | [sd] installed Bids / gc objects | 2381 / 2396 | 2303 / 2329 | 2612 / 2630 | -2.0 / -1.1 | yes | | [sd] installed kine rows / file (MB) | 8001 / 60.4 | 6150 / 45.2 | 8087 / 61.4 | -128 / -1.3 | yes | | [proc] OpenBao storage bidder-1 / gateway (MB) | 17.1 / 17.0 | 17.3 / 17.0 | 17.6 / 17.1 | +0.1 / -0.0 | yes | | [sd] installed OpenBao storage (MB) | 16.6 | 17.3 | 17.5 | +0.1 | yes | | [proc] state dir total (MB) | 12168 | 12115 | 17193 | -144 | yes | | [proc] bidder-1 VM work dirs / gateway tenant cluster dirs | 7792 / 1318 | 7467 / 1392 | 12594 / 1393 | -227 / -10.2 | yes | | [proc] network DENO_DIR (gen/blob) | 211 | 282 | 282 | +24.3 | NO (section 14) | | [proc] network / gateway / bidder-1 logs dirs | 20.6 / 10.2 / 5.0 | 53.4 / 28.7 / 11.5 | 53.4 / 28.7 / 11.5 | +11.0 / +6.1 / +2.6 | NO: below the 64 MiB rotation caps; the gateway's keeps a pg log per cluster | | [sd] installed stateDir (status) | 8237 | 5437 | 12440 | +34.3 | yes | | [sd] installed kcp dir (audit logs) | 317 | 486 | 572 | +60.1 | bounded: reached maxbackup 4 x 100 MB, 572 then 486 | | [proc] soak requester repo kv (MB) / compute.vm / market.rfp | 56.0 / 1164 / 1167 | 64.7 / 1033 / 1039 | 64.7 / 1280 / 1253 | +4.0 / -12.3 / -7.1 | kv NO; records yes | | [proc] soak requester repo market.accept / market.event / vm.delete | 25 / 25 / 22 | 25 / 25 / 25 | 28 / 28 / 28 | -0.7 / -0.8 / +0.1 | yes | | [proc] bidder-1 repo kv (MB) / market.bid / market.event / market.receipt | 39.5 / 2854 / 135 / 49 | 42.7 / 2513 / 99 / 35 | 42.7 / 3067 / 170 / 60 | +1.2 / -7.4 / -4.3 / -1.7 | kv NO; records yes | | [sd] installed repo kv (MB) / market.bid / market.event / market.receipt | 35.4 / 2794 / 120 / 42 | 44.2 / 2766 / 136 / 47 | 44.2 / 3015 / 153 / 53 | +3.0 / +24.2 / +4.8 / +1.7 | kv NO; records yes | | [proc] gateway repo kv (MB) / compute.vm / market.accept | 6.2 / 16 / 16 | 6.3 / 17 / 17 | 6.3 / 20 / 20 | +0.1 / -0.4 / -0.1 | yes | | [proc] gateway repo com.fedproxy.rbac | 7 | 20 | 21 | +5.3 | NO: records of deleted clusters (section 14) | | host MemAvailable (MB) | 80127 | 77722 | 84193 | -293 | yes | | host load average 1 min / swap used | 16.6 / 0 | 15.8 / 0 | 25.2 / 0 | -0.1 / 0 | yes | | host journal (MB) | 44.4 | 77.0 | 77.0 | +11.2 | NO; the host's SystemMaxUse pool bounds it | Root status after the driver stopped (23:36Z): bidder-1 2286 objects, the soak requester 2676, the installed bidder 2329, each with `orphanTotal` 0 and `stuckTotal` 0; the gateway 18 objects with one orphan (the MarketIdentity `root:gateway:app/requester`, section 14) and 0 stuck. The TTL controller had deleted 3773 objects in bidder-1, 9867 in the soak requester and 437 Bids in the installed bidder (whose older Bids were annotated to 1800 s at 19:32Z). Done_when read: ACC13-NO-HOST-OOM holds (the breach above, three more breaches under load, each restarting only its pod; the host never short of memory). ACC13-BOUNDED and M4-SOAK-READING do not hold: the requester DenoPods' native memory, the pds DenoPods' memory, the network PDS sockets, the node claim index ConfigMaps, the gateway repo's rbac records, the DENO_DIR `gen/blob` and the repo kv files keep growing (section 14). M3-DISK-RETENTION does not hold: the state dir totals are flat in both backends, but by subdirectory `gen/blob` (+23.6 MB an hour), the per-cluster pg logs (one per cycle) and the repo kv files (+1.2 to +4.0 MB an hour) keep growing, and the logs dirs have not reached their caps yet. M3-PDS-RECORD-RETENTION does not hold for the gateway repo (20 rbac records of deleted clusters after about 12000 RFP-only, 263 VM and 165 tenant node contracts); the soak requester's, bidder-1's and the installed bidder's repos hold only live and retained records. - **The gateway kcp memory and the gateway OpenBao storage under tenant cluster churn (plan 0013 ACC13-BOUNDED, 2026-10-18).** Both were open in section 14 after M4-SOAK-READING (below): kcp 599 to 1134 MB in 3 h, OpenBao storage 17 KB per cluster cycle. Host: droplet dev-churn-1 (c-32, sfo2), account test-0001, `dev up --profile market-network --bidders 1 --with-gateway`, the soak driver's tenant lanes (6 lanes: create a cluster, Ready, a pod scales a node out through the RFP flow and in again, delete the cluster, wait until it is gone), samples every minute (`soak sample`, `ss` per pid, `/proc` RSS and fds per pid, kcp heap profiles from `/debug/pprof/heap`, the root vault listed with a root token generated from its unseal keys). Causes, each fixed on main: (1) The kcp memory was not the churn. kcp v0.33.0 (`pkg/server/localproxy.go`) proxies a request for a mounted workspace through a new `http.Transport` per request unless `--mount-proxy-client-cert-file` is set, and that transport never closes its keep-alive connection. The cluster provider's `MountStatus` probe (`GET /clusters//version`, every pass, about 25 a minute per cluster) therefore left one kcp connection, and one pair on the cluster's apiserver front, per probe; a long-lived cluster (the soak's bootstrap cluster, prod's `canary`) grew without bound, and prod ran out of ephemeral ports (112939 loopback connections). Every slot root now holds a self-signed client pair and `kcpserver` passes both mount proxy flags, so kcp uses one shared transport (90 s idle timeout); `CheckArgv` refuses an argv without them (c883e72d). Read side by side on the droplet with no churn and one tenant cluster: the main build's kcp went from 48 to 220 established connections in 9 minutes, its fds from 386 to 687 sockets and its RSS from 720 to 738 MB in 10 minutes (108 MB an hour, the soak's slope), and the tenant front reached 1260 sockets in 20 minutes; the fixed kcp held 17 to 27. (2) The apiserver front closes a connection pair after 5 minutes with no bytes either way, so a client that leaks silent keep-alive connections cannot exhaust the host again (d116cb41; a client-go HTTP/2 connection pings every 30 s, so this does not cut those). (3) The gateway core minted a new tenant client certificate about once a second (the node observer and the node reconcile), and client-go keeps one transport per distinct certificate for the life of the process: 90 open connections to the tenant front and 3 TLS handshakes a second. A tenant client is kept per cluster for 5 minutes, keyed by Cluster UID and instance (755e8bea): 2 connections after. Every `openbao.Client` and vault admin client also built its own transport with no idle timeout (63 core connections per cluster vault); they close an idle connection after 90 s (47134c70). (4) The OpenBao storage: the requester runner writes `contracts//secrets-key` into its vault mount for every contract (in the gateway, the root vault's `requester/` KV mount, one entry per node a tenant cluster ever rented, metadata and versions) and nothing removed it. Every 5 minutes the requester controller now lists `contracts/` and removes the key of a name that is no ComputeContract, runner or waiting contract (96a9b60d); its first pass removed 6. The per-cluster vault directories (`openbaoservers/c--bao`) and data directories are purged with the cluster (read: only the live clusters' directories exist after 122 deletions). Tidy (token and lease) covers what is left: 3 token accessors and 2 cert login leases. (5) An upgrade did not apply a new kcp argv: the core adopted the running slot, so prod ran slot a from Oct 12 without the mount proxy flags after rc15 until a hand `units roll kcp`. An installed core now compares the active slot's `/proc` cmdline and executable with the slot spec it renders and the kcp asset link, and on a difference runs the installed roll under the roll lock (an executable change checks the kcp compat records first), then adopts the new slot; a refusal is logged with the `CheckArgv` reason (558cba23). Read on droplet dev-churn-2: `install gateway` with a9fdef3d, `upgrade` to 8b96e1ca, the core logged `argvChanged` with the refusal and rolled a to b in 12 s, slot b runs with both flags, and a second core restart adopted b without a roll. Readings after the fixes, fitted 03:05Z to 04:18Z (73 minutes after 15 minutes of warm-up; 128 clusters created and 122 deleted since 02:46Z, 0 failed; the same fit rule as M4-SOAK-READING): | series | start | end | max | slope per hour | plateau | | --- | --- | --- | --- | --- | --- | | kcp RSS (MB) | 913 | 999 | 999 | +48.5 | NO | | kcp RSS, 03:35Z to 04:18Z (MB) | 974 | 999 | 999 | +33.3 | yes | | kcp in-use heap, pprof (MB) | 349 | 337 | 351 | (3 profiles) | yes | | kcp established connections | 27 | 23 | 27 | +0.3 | yes | | kcp socket fds | 164 | 160 | 166 | -0.2 | yes | | tenant front (bootstrap cluster) established | 8 | 6 | 8 | +0.4 | yes | | tenant front socket fds | 26 | 24 | 28 | -0.2 | yes | | lane cluster fronts established (6 clusters) | 136 | 208 | 214 | +0.2 | yes | | gateway core RSS (MB) | 94 | 102 | 114 | +5.3 | yes | | gateway core socket fds | 279 | 162 | 402 | +1.7 | yes | | OpenBao processes established | 450 | 413 | 511 | -2.7 | yes | | gateway openbaoservers dir (KB) | 17648 | 17712 | 17736 | -3.3 | yes | | kcp logical clusters | 26 | 26 | 28 | +0.0 | yes | | kcp workspaces | 29 | 31 | 33 | -0.2 | yes | | kcp APIBindings | 97 | 97 | 103 | +0.1 | yes | | Clusters / ClusterLinks / HostProcesses | 7 / 10 / 138 | 7 / 14 / 160 | 7 / 14 / 160 | +0.0 / -0.2 / +1.5 | yes | The root vault's file storage read 129 files and 104.8 KB at 02:50Z and 118 files and 90.9 KB at 04:18Z; its KV entries were the 6 live contracts' keys, 5 live bundles and the identity and kcp entries. The kcp RSS climbs toward a steady state while its live heap stays at 340 to 350 MB (the Go runtime keeps freed pages); the last 44 minutes are flat. The network kine's live keys rose 1206 to 1454 (Bids +104 and Events, which end by their TTLs; the gateway's own kinds are flat). Not settled in the window and not part of these two series: the gateway requester DenoPod RSS (restarted at 03:48Z for (4)) and the web app DenoPod RSS (179 to 243 MB); the gateway requester's connections to the dispatcher are a sawtooth (50 to 750, 2200 in the burst after a restart) that the fetch pool drains, and an unread body on a non-2xx getRecord no longer holds one (ed00b815). Also found: on a fresh droplet the `socialweb-computer-k8s-downloads` asset stages to tree baf256f7 (the socialweb-meshd binaries differ by 240 bytes), not the pinned 01030c9a, so a droplet's dev build needs that pin changed in its own checkout. Cause: ring's C was compiled by the host clang (Fedora 22.1.8, Ubuntu 18.1.3). Closed by 13473a6b: rust/build.sh compiles it with the zig cc and zig ar of a zig it pins by sha256; the tree is acb85490 on Fedora 44 and on Ubuntu 24.04. ACC13-BOUNDED's gateway kcp and OpenBao parts hold on this reading. - **The requester's orphan records, the bidder's bid sweep, the requester heap and the gateway requester's retention (plan 0013 M3-PDS-RECORD-RETENTION and M3-DISK-RETENTION, 2026-10-18, commits 8dad7806, b517683f, 1be95509, feccd06d, a60721ac, efa7e5cd).** The four series that section 14 kept open after dev-gc-1. Causes and changes: **Historical.** This is the TypeScript-era record retention. The Go rewrite (963f7980) deleted it with `ts/`, and no record retention exists: nothing deletes a market record by age, the options `record-retention-sec` and `record-retention-scan-ms` are gone from `deploy/apps/option-map.yaml`, and the owner's rule is that no component deletes records for people. What follows describes code that no longer exists. (1) Orphan records. Every 30 s the controller's resync lists the ComputeContracts and called `forget()` on every runner whose name the list lacked. A contract created while the list was in flight had already been launched from the watch, so it was taken for deleted: the runner set deleted and released, skipped every later status write and ended with "the contract was released before a VM was held", and the object kept no `status.contract`. Its compute.vm and market.rfp were named by nothing, so retention never removed them. That is one orphan per resync under load (45 in 50 minutes on dev-bound-1 before the fix, +110 an hour on dev-gc-1). A name that `observe()` saw at or after the list began is no longer forgotten (8dad7806). The same pass found that the gateway's requester, whose records are in its application's repo PDS since plan 0018, compared each ref with the local requester PDS's DID and removed nothing at any retention; refs are matched with the DID that `deleteRepoRecord` writes to. A request that throws now hands its state to the runner (`ComputeRequestFailed`), and an at-capacity round removes its rfp and compute.vm before the next round (b517683f). As a backstop, the orphan scan removes a settled compute.vm that no ComputeContract claims once it is older than the record retention, with the rfp, accept and events that name it. It reads one listing per pass, stops after 60 s, logs `requester_orphan_records_removed`, and covers a deleted NodeClaim's contract (1be95509). (2) Bid sweep. The cap of 100 deletes a pass was set for the bsky.social write budget, which no longer applies since plan 0018. The sweep now deletes every due record each pass in `applyWrites` batches of 100, with a 5-minute bound per pass, and lists every page (oldest first) until that bound (feccd06d). (3) Requester heap. `createATProtoEventStreamsClient` pushed each dedicated watcher's stream watchers into the client's `watchers` list and never took them out on close. Every contract watch after the first batch stayed reachable with its `openContractWatch` closure (a60721ac). The unfixed requester (network "bound", same host) read heapUsed 266 MB at 06:59Z, 326 MB at 07:09Z and 368 MB at 07:19Z, with its ComputeContracts flat at 2640: about +300 MB an hour, which reaches the 792 MB limit in about 2 h. The fixed one reads flat (below). (4) Gateway retention. A Gateway spec option mapped `to: requester` in `deploy/apps/option-map.yaml` goes to the gateway's requester DenoPod. `record-retention-sec` is the first such option; the prod default of 7 days stays (efa7e5cd). The requester also logs `requester_memory` every minute (its V8 heap and the sizes of its per-contract maps), and the soak driver gained `orphans` and `release`. Reading on droplet dev-bound-1 (c-48, sfo2): one `dev up --profile market-network --bidders 1 --with-gateway --vm-host-backend systemd-user`, with app values setting the bidder's `bid-ttl-sec` 1800 and the gateway's `record-retention-sec` 900. The `test/soak` driver ran at 1.1 RFP/s, a VM contract every 40 s (3 lanes), 2 tenant lanes, contract TTL 40 minutes, requester retention 900 s scanned every 60 s, sampled every minute. The requesters were restarted at 07:00Z and 07:22Z onto the DID and resync fixes, and the fit runs from 07:40Z to 09:10Z (the section 15 soak rule). Load from 06:30Z to 09:10Z: 10559 RFPs, 239 VM contracts (237 released, 1 failed), 79 tenant cluster cycles (2 failed while the gateway requester restarted). `run request --network` reached the guest at 06:30Z and at 09:10Z. After the resync fix no contract ended "released before a VM was held"; the backstop's 39 removals were the orphans written before it. Readings: | series | start | end | max | slope per hour | plateau | | --- | --- | --- | --- | --- | --- | | requester repo kv file (MB) | 63.2 | 63.2 | 63.2 | +0.0 | yes | | requester repo: compute.vm | 1019 | 1039 | 1044 | +15.1 | yes | | requester repo: market.rfp | 1020 | 1031 | 1042 | +14.3 | yes | | requester repo: market.accept / market.event / vm.delete | 25 / 25 / 24 | 24 / 24 / 23 | 25 / 25 / 24 | +0.0 | yes | | requester DenoPod heapUsed (MB) | 101 | 98.8 | 119 | +2.2 | yes | | requester ComputeContracts (controller) | 2701 | 2701 | 2702 | -0.2 | yes | | requester DenoPod RSS (MB) | 410 | 454 | 472 | +21.6 | NO (heap flat; section 14) | | bidder repo kv file (MB) | 48.7 | 51.5 | 51.5 | +1.5 | yes | | bidder repo: market.bid | 2491 | 2457 | 3040 | -25.4 | yes | | bidder repo: market.event / market.receipt | 177 / 62 | 196 / 68 | 251 / 88 | +6.7 / +0.2 | yes | | bidder DenoPod RSS (MB) | 351 | 323 | 354 | -2.5 | yes | | bidder pds DenoPod RSS (MB) | 312 | 389 | 454 | +28.8 | NO (section 14) | | gateway repo kv file (MB) | 5.5 | 5.5 | 5.5 | +0.0 | yes | | gateway repo: compute.vm / market.rfp / market.accept | 6 | 10 | 10 | +0.3 | yes | | gateway requester DenoPod RSS (MB) | 227 | 550 | 698 | +283 | NO (native, heap flat at 45 MB; section 14) | | state dir total (MB) | 5354 | 13662 | 13662 | +568 | yes | The bidder's sweep deleted about 680 market.bid records and 125 to 160 trail records each pass (scanned about 3050), so Bids stay at bid rate times TTL. The repo kv files sit at their SQLite high-water marks (section 15, hono-pds entry). - **hono-pds collects unreachable blocks, and the requester's retention removes the vm.delete and market.event records (plan 0013 M3-DISK-RETENTION and M3-PDS-RECORD-RETENTION, 2026-10-18, commits 508d79fc and 58d5bc38).** Before: the 3 h soak read the bob PDS kv file at +213 MB an hour and the requester repo at +171 MB an hour, because `DenoKvStorage` never deleted a block, and the requester's compute.events.vm.delete and market.event records at +86 an hour each, because `retention.ts` removed rfp, payload and accept only. The change: `DenoKvStorage.put` writes a key-only index entry (`["blockidx", cid]`, time and size) beside each block; `Repo` runs a mark and sweep at most every 5 minutes after a write: it marks each head's commit, every MST node and every record CID the nodes name, and removes each indexed block outside that set that was written more than 10 minutes ago. A firehose frame carries its own blocks and `getRepo` serves only the head, so older revisions are not kept; the 10 minutes cover a write in flight. Each delete checks the index entry's versionstamp, so a block written again during the sweep stays; the first pass indexes the blocks of a store written before the index. The factory logs `repo_block_gc` (blocks, bytes, removed, kv file bytes before and after, pass ms). This covers every repo built on `createRepoFactory`: hono-pds, the requester's repo, each application's pds DenoPod (hono-repo-pds) and the ssh app PDS. The requester's retention also removes `state.release.payload`, `state.release.event` and `state.submitEventRef` (refs in its own repo only) and skips a contract whose runner is running, so a release that is still retrying keeps the event plan 0020 resends. Reading on droplet dev-gc-1 (c-32, sfo2): one `dev up --profile market-network --bidders 1 --with-gateway`, the `test/soak` driver at 1.1 RFP/s, a VM contract every 40 s (3 lanes), 2 tenant lanes, contract TTL 40 minutes, requester retention 900 s scanned every 60 s, sampled every minute, fitted from 03:50Z (one hour of warm-up) to 06:04Z with the section 15 soak rule. Since plan 0018 the bidder's and the gateway's repos are their pds DenoPods' `components/pds/repo.sqlite3`, so the network's bob-pds and alice-pds kv files are idle (0.2 and 0.4 MB, flat). Load to 06:04Z: 12869 RFPs, 254 VM contracts (239 released), 86 tenant cluster cycles with a node out and in through the RFP flow, and two `run request --network` runs that reached the guest. Readings: | series | start | end | max | slope per hour | plateau | | --- | --- | --- | --- | --- | --- | | requester repo kv file (MB) | 64.6 | 66.2 | 66.2 | +0.8 | yes | | requester repo: compute.events.vm.delete | 24 | 21 | 25 | -10.1 | bounded, 18 to 25 (the fit reads a step down as NO) | | requester repo: market.event | 24 | 22 | 26 | -8.9 | bounded, 19 to 26 (as above) | | requester repo: market.accept | 25 | 22 | 26 | -6.8 | bounded, 19 to 26 (as above) | | requester repo: compute.vm | 1097 | 1307 | 1454 | +111 | NO (orphans, section 14) | | bidder repo kv file (MB) | 43.0 | 80.9 | 80.9 | +17.5 | NO (live market.bid, section 14) | | gateway repo kv file (MB) | 5.9 | 9.2 | 9.2 | +1.5 | NO (7-day retention, section 14) | | state dir total (MB) | 11472 | 7889 | 13629 | -2533 | yes | The requester's last pass: 28306 blocks (20.0 MB) kept, 3621 of them reachable from the head, 9772 blocks (7.2 MB) removed, kv file 66.2 MB before and after, 6.8 s. The bidder's: 22612 kept (18.7 MB), 16532 reachable, 2284 (1.6 MB) removed, 2.4 s. The kv file does not shrink: SQLite reuses the freed pages, so the file stays at its high-water mark, about 3.5 times the kept bytes. A local check of 6 rounds of 300 creates and 300 deletes of 20 KB records read the file flat at 20.6 MB from round 2 on, with every record readable after a reopen. No `repo_block_gc_failed` line and no missing block in 3 h 30 min. The 125 hono-pds tests pass. The soak driver was ported to the one-kcp network (5d72f50d) and samples each application's repo PDS (8718ddc2). - **Plan 0013 M4-SOAK-READING: a 3 h accelerated soak of a market network with a gateway, read on a DigitalOcean c-32 droplet (2026-10-17 22:12:48Z to 2026-10-18 01:13Z; M4 stays `doing`, ACC13-NO-ORPHANS is `done`).** Host: droplet dev-soak (c-32, 32 vCPU, 64 GB, Ubuntu 24.04, sfo3, /dev/kvm), account test-0001, a dev build of the tree, one `dev up --profile market-network --bidders 1 --with-gateway` plus a long-lived MarketRequester child (`soakreq`) that the driver `test/soak` (Go: `load`, `sample`, `tidy`, `inspect`, `hammer`) adds to the network and feeds with ComputeContract objects. The driver and the sampler ran as user units on the droplet; the sampler wrote one JSON line per 5 minutes to `/var/tmp/soak-data/samples.jsonl` (not `/var/tmp/soak`, which the dev build mirrors over). Accelerated settings (the user changed the item to 3 h on 2026-10-17): Bid TTL 1800 s plus the 300 s bid window (a Bid lives about 35 minutes, so it turned over 5 times), ComputeContract TTL 40 minutes, requester record retention 900 s scanned every 60 s, the bidder's record sweep every 10 minutes, storemaint every 5 minutes, run directories swept every minute with a 1 h keep, guest images collected every 10 minutes. OpenBao tidy: the default interval is 6 h, so at 23:12:52Z `soak tidy` set `spec.tidy.interval` (token and lease tidy) and `spec.declared.pkiTidy[].interval` (pki auto-tidy) to 15 minutes on every OpenBaoServer of the network, the three applications and the tenant vaults; the bidder's server status read `lastRunAt` 23:12:53Z and runs every 15 minutes after. kcp store backups are written only on a kcp roll, and a market network never rolls a one-slot kcp, so M1-BACKUP-RETENTION was not exercised here. Load over the three hours: 11909 RFP-only contracts (a ComputeContract annotated `socialweb.computer/rfp-only` stops after its RFP is submitted; the bidder still bids), 270 VM contracts (263 held and released, 7 failed with `no_bids` or a `vm.delete` 404 across bidder restarts), 226 tenant cluster cycles (the driver makes a cluster, a pod scales a node out through the RFP flow, the node scales in, the driver deletes the cluster; 8 lanes). That is 12.4k RFPs and about 500 contracts, above the item's counts scaled to 3 h (6250 RFPs, 250 contracts). Phase 1 (22:12:48Z to 22:13:48Z): 0.6 RFP/s, one tenant lane, a VM contract every 40 s (3 lanes). Phase 2 (22:13:48Z to the stop at 01:14Z): 1.1 RFP/s and 8 tenant lanes, which is per hour 3970 RFP-only contracts, 90 VM contracts and 75 cluster cycles. Chaos: 25 DenoPod restarts through `dev restart` (5 each of bidder, guest-kcp, gateway requester, gateway web app and requester), then at 01:09Z `kill -9` of the bidder DenoPods and at 01:11Z `kill -9` of the bidder core and the gateway core; each came back (the provider placed the pods again, the network placed the cores again and read Ready). `dev restart core` was not used: see the limit below. Host: vmstat idle 55 % on average (25 to 80 %), memory used 18 GB on average (24 GB at most), MemAvailable at least 38 GB, load average 17 (31 at most), no swap, oom_kill 0. Fit: least squares over the samples from 23:12:48Z (one hour of warm-up, longer than a Bid lives); a series is flat when |slope x window| <= max(2 x residual standard deviation, 5 % of its mean). The record rows are fitted from 00:20Z, after the two retention fixes below. The tenant clusters' processes (k3s, kine, postgres and OpenBao of each cluster) are counted under the network's application; the driver keeps about 8 clusters at a time, so those rows are noise around a lane count, and the state dir total is dominated by the VM work dirs that come and go, which is why it reads flat while its parts below do not. Readings: | series | start | end | max | slope per hour | plateau | | --- | --- | --- | --- | --- | --- | | bidder application, all processes (RSS MB) | 1941 | 1999 | 2117 | +30.3 | yes | | bidder kcp | 989 | 995 | 1080 | +10.2 | yes | | bidder kine | 91 | 106 | 136 | +10.2 | yes | | bidder DenoPod | 249 | 278 | 324 | +2.3 | yes | | guest-kcp DenoPod | 154 | 152 | 157 | -1.8 | yes | | bidder core | 157 | 163 | 297 | +9.2 | yes | | bidder OpenBao | 127 | 131 | 132 | +2.5 | yes | | requester application, all processes | 2122 | 2243 | 2454 | +58.2 | yes | | requester kcp | 959 | 1036 | 1054 | +48.2 | NO | | requester DenoPod | 619 | 698 | 932 | +42.5 | yes | | requester core | 167 | 128 | 219 | -34.9 | NO | | gateway application, all processes | 3390 | 3641 | 3936 | +94.2 | yes | | gateway kcp | 943 | 1108 | 1134 | +104.1 | NO | | gateway requester DenoPod | 499 | 352 | 636 | -70.0 | yes | | gateway web app DenoPod | 226 | 237 | 243 | +7.7 | yes | | gateway core | 272 | 300 | 379 | +9.1 | yes | | gateway OpenBao (root vault and tenant vaults) | 861 | 963 | 1078 | +21.2 | yes | | gateway postgres | 221 | 259 | 259 | +8.6 | yes | | network kcp | 659 | 656 | 660 | -1.0 | yes | | network core | 110 | 112 | 115 | +0.9 | yes | | bob PDS DenoPod | 744 | 779 | 845 | +22.6 | yes | | alice PDS DenoPod | 185 | 213 | 254 | +18.5 | NO | | relay DenoPod | 429 | 718 | 781 | +139.3 | NO | | PLC DenoPod | 137 | 137 | 138 | -0.0 | yes | | dispatcher DenoPod | 158 | 161 | 162 | +2.0 | yes | | network OpenBao | 121 | 122 | 122 | +0.7 | yes | | Bid objects (bidder kcp) | 2398 | 2353 | 2405 | -28.5 | yes | | ComputeContract objects (requester kcp) | 2702 | 2686 | 2715 | -10.2 | yes | | ComputeVM objects | 4 | 5 | 9 | +0.4 | yes | | bidder gc scan: objects | 2426 | 2388 | 2455 | -25.9 | yes | | requester gc scan: objects | 2711 | 2696 | 2723 | -9.7 | yes | | gateway gc scan: objects | 96 | 93 | 120 | -5.3 | yes | | bidder kine rows | 6438 | 6741 | 8492 | +177.1 | yes | | bidder kine file bytes | 42188800 | 43956907 | 54951936 | +1002092.0 | yes | | requester kine rows | 6832 | 6908 | 7684 | +33.5 | yes | | requester kine file bytes | 41483605 | 42893995 | 47624192 | +812285.6 | yes | | gateway kine rows | 3157 | 3217 | 4435 | +34.1 | yes | | gateway kine file bytes | 19934549 | 19554304 | 27082752 | -343698.2 | yes | | bidder OpenBao storage (MB) | 18 | 19 | 19 | +0.2 | yes | | gateway OpenBao storage (MB) | 19 | 21 | 22 | +1.2 | NO | | requester OpenBao storage (MB) | 0 | 0 | 0 | +0.0 | yes | | network OpenBao storage (MB) | 1 | 1 | 1 | +0.0 | yes | | state dir total (MB) | 18561 | 27695 | 41607 | +4484.5 | yes | | bidder VM work dirs (MB) | 8856 | 11669 | 25682 | +980.4 | yes | | gateway kcp dir, audit logs (MB) | 3291 | 7891 | 8435 | +2812.8 | NO | | bidder kcp dir, audit logs (MB) | 302 | 607 | 650 | +187.5 | NO | | requester kcp dir, audit logs (MB) | 201 | 393 | 414 | +117.9 | NO | | network kcp dir, audit logs (MB) | 149 | 324 | 346 | +106.8 | NO | | gateway tenant cluster dirs (MB) | 1648 | 1926 | 2239 | +58.5 | yes | | bob PDS kv file (MB) | 224 | 585 | 628 | +222.2 | NO | | requester repo kv file (MB) | 173 | 470 | 505 | +182.8 | NO | | relay kv file (MB) | 0 | 0 | 0 | +0.0 | yes | | host MemAvailable (MB) | 49129 | 45789 | 54093 | -1396.9 | yes | | host load average 1 min | 17 | 17 | 31 | -0.2 | yes | | host oom_kill counter | 0 | 0 | 0 | +0.0 | yes | | host swap used (MB) | 0 | 0 | 0 | +0.0 | yes | Records in the PDS repositories (counts by collection, fitted from 00:20Z): | series | start | end | max | slope per hour | plateau | | --- | --- | --- | --- | --- | --- | | bob repo: compute.config.wif.simple | 2552 | 2527 | 2788 | -31.2 | yes | | bob repo: compute.events.vm.hostKeys | 98 | 99 | 115 | +5.5 | yes | | bob repo: compute.events.vm.onNetwork | 197 | 199 | 231 | +9.2 | yes | | bob repo: compute.events.vm.terminated | 97 | 96 | 110 | +1.6 | yes | | bob repo: market.bid | 2540 | 2485 | 2782 | -60.3 | yes | | bob repo: market.bids.free | 2552 | 2512 | 2793 | -55.5 | yes | | bob repo: market.event | 378 | 384 | 450 | +16.9 | yes | | bob repo: market.receipt | 100 | 101 | 116 | +4.4 | yes | | requester repo: compute.events.vm.delete | 195 | 260 | 264 | +86.4 | NO | | requester repo: compute.vm | 1917 | 1910 | 1939 | -7.9 | yes | | requester repo: market.accept | 116 | 115 | 118 | -0.5 | yes | | requester repo: market.event | 195 | 260 | 264 | +86.4 | NO | | requester repo: market.rfp | 1905 | 1900 | 1931 | -4.8 | yes | Seven defects the soak found and the tree now fixes, each proven by a short run on the droplet, in the order found. (1) hono-pds loaded and rehashed its whole record tree on every read and write (`collectBlocks` serialised and stored every MST node on each `set`, `getRecord` and `listRecords` rebuilt the tree, `diff` walked two whole trees), so at 1000 bids its heap hit the 512 Mi limit, it restarted on a new port and the relay lost it. Bench of 4000 records: write 117 ms to 4.4 ms, getRecord 79 ms to 0.1 ms. Nodes loaded from the store or already put are marked persisted and skipped, a repo keeps its latest tree for reads, and a write forks it. This is the only change to `ts/hono-pds/lib/abc/atproto-repo/mst.ts`; the need is that the soak could not run a PDS at 0.6 RFP/s without it; the 125 hono-pds tests pass. (2) The relay deno was at 25 GB after 30 minutes: hono-pds sends a commit frame as JSON with `blocks` as an object with one key per byte, a requester's compute.vm record carries the whole cloud-init user_data so a frame is about 4 MB of JSON, the relay kept 50000 parsed frames and stringified each again for each of 40 subscribers, and a closed `subscribeRepos` socket left its live subscription collecting every later frame. No subscriber reads `blocks` (they read the ops and fetch the record by key), so the relay forwards a frame with empty blocks and ends the live loop of a closed socket, and both sequencers also stop at 64 MiB; relay RSS after the change: 115 MB at 9 minutes against 2137 MB. (3) The ingress proxy closed a subscriber after any single request went 30 s without an answer, so one slow `submitRfp` evicted the bidder's websocket (89 reconnects in 9 minutes, a third of the RFPs answered 404, 20 % of VM contracts failed); it now evicts only a subscriber that answered nothing in the window (0 `ws_close` in the bidder pod afterwards). (4) A bidder pod killed one second after an accept left the ComputeVM running with no owner and the Won Bid with no `status.computeVM`; the ComputeVM controller now binds the two by the `socialweb.computer/accept` annotation (the gc scan then read 0 in 32 of 38 samples and 1 or 2 ComputeVMs without an owner in the other 6, once for three samples in a row around 23:10Z, and 0 after the driver stopped). (5) The bidder removed bids, payloads and bid configs but never the receipt, event and vm event records of an ended contract, which grew by 130 receipts and 500 events an hour; the sweep now removes a receipt older than the record TTL whose bid record is gone, and every event whose receipt is not live, with its payload (the bidder repo went to a sawtooth of 90 to 110 receipts and 370 to 450 events). (6) The requester's record retention removed 50 contracts' records a pass, below 100 contracts a minute, and a contract object deleted by its TTL is never swept; with 500 a pass the repo went from 2553 to 1874 compute.vm records and stayed there. (7) kcp's audit log keeps every rotated file (`--audit-log-maxbackup=0`, `--audit-log-maxage=0`): the gateway slot wrote 100 MB every two minutes under cluster churn and held 8.4 GB after 3 h (35 GB after 8 h in the earlier run), and the other slots 330 to 650 MB; `kcpserver` now passes maxage 1 and maxbackup 4. Proof on a fresh `dev up --profile quick` instance with the change: 5.49 million requests from 64 clients rotated the log about 40 times and the directory held the current file plus four of 100 MB, 488 MB in all. That fix is read on one quick kcp, not on the soak. What did not plateau, each a section 14 item: the kcp audit logs above (fixed, not re-read in a soak), the hono-pds kv files (the store never deletes a block: bob 628 MB and the requester's repo 505 MB after 3 h, +213 and +171 MB an hour), the gateway kcp RSS (599 to 1134 MB in 3 h, 827 to 1740 MB in 7 h of the earlier run, 100 to 130 MB an hour at 75 cluster cycles an hour, about 2 MB per cluster made and deleted; the slot limit is 3 GiB), the gateway OpenBao storage (17.7 to 21.2 MB, 1.2 MB an hour with tidy every 15 minutes, 17 KB per cluster cycle), the requester's vm.delete and market.event records (kept for ever, +86 an hour at 90 VM contracts an hour), and the relay and alice-pds RSS inside the window (the relay stepped from 430 to 720 MB at 00:55Z while its backlog filled; bob-pds peaked at 845 MB, above its 512 Mi limit, which the dev backend does not enforce). After the reading the market network set raises bob-pds to 2Gi and the relay to 1Gi, and the relay backlog cap falls from 50000 to 10000 frames; those three are numbers from this reading, not re-read. Readings for plan 0018 (each application's repo PDS becomes its own DenoPod): the PDS DenoPod is single-threaded and is the first ceiling. At 1.1 RFP/s the bob-pds process used 14 % of one core in the first hour and 36 to 43 % after, against the bidder DenoPod at 4 to 6 %, the requester at 11 to 20 %, the relay at 3 to 10 %, the network core at 148 to 244 % (cluster churn), the gateway core at 93 to 114 %, the bidder core at 72 to 93 % and the kcp processes together at 161 to 205 %. A probe every 6 s from the droplet (a new TLS connection each time; `describeServer`, `listRecords` with limit 1 and `getRecord`, 1577 of each) read p50 46 to 51 ms and p95 54 ms in the first hour and 134 to 166 ms after, maximum 7.4 s. The bidder's `status.market.rfpToBid` over its last 256 RFPs read p50 0.2 to 0.36 s and p95 0.3 to 14.8 s, and `kcpWait` p95 27 to 34 ms. An earlier pass on the same tree at 2.7 RFP/s with 13 tenant lanes put the bob-pds main thread at 85 % and `rfpToBid` p95 at 54 s, and the bidder could not answer a `submitRfp` within the requester's 10 s, so the rate one bidder and one hono-pds sustain on this host is about 2 RFP/s (1.1 was held for the readings above). ComputeVMs peaked at 12 of the 16 VM slots. Limit found, not a product defect: `dev restart core` on a market-network instance restarts every network DenoPod on a new port and empties the PLC, so the children lose their relay and every DID; the first chaos step that restarted the network core ended that run at 22:02Z (after 8 h of healthy readings, which are kept). The soak therefore restarts pods and kills the child cores. M2-QUOTAS retune from the plateaus: per Bid the bidder kcp holds 0.14 to 0.15 MB of RSS and the store 10 to 15 KB and 2 rows (base 610 to 680 MB), per ComputeContract the requester kcp holds 0.13 MB and the requester DenoPod 0.23 MB. At the 3 GiB kcp limit 20000 Bids would be 3.4 GiB, so `QuotaDefaults.Bids` is 10000 (about 2.1 GiB). ComputeVMs 64, ComputeContracts 1024 and 64 and NodeClaims 64 stay: the peaks read were 12 ComputeVMs, 5715 contracts in a requester whose quota the driver raised, and 1 NodeClaim per cluster. DenoRuns and PolicyWorkflowRuns stayed at 0 in the soak and stay at 200 and 500, unread. No quota hit happened in the soak; the earlier M2-QUOTAS reading (a quota of 10 Bids declined the 11th RFP, kcp RSS unmoved) stands. Done_when read: ACC13-NO-ORPHANS holds (at 01:18Z, after the driver stopped: bidder 2061 objects, requester 2435, gateway 20, each with `orphanTotal` 0 and `stuckTotal` 0). ACC13-NO-HOST-OOM holds for its last clause only: the host never ran out of memory (MemAvailable at least 38 GB, oom_kill 0 in 3 h and in 8 h); the DenoPods ran under the dev backend, which sets the V8 heap flag and no cgroup limit, so a memory breach was not re-read, and no quota was hit. ACC13-BOUNDED does not hold (the unflat series above), M4-SOAK-READING does not hold, M3-DISK-RETENTION's reading is contradicted (the kcp audit logs and the PDS kv files are state that never ends), and M3-PDS-RECORD-RETENTION holds for the bidder repo (every collection flat after the fixes) and not for the requester repo (vm.delete and market.event stay). M3-STORE-COMPACTION holds for the sqlite stores (the bidder kine file 40 to 59 MB and 6.4k to 8.5k rows, flat while 2400 Bids turned over; the gateway store flat in 3 h; in the 8 h run the gateway store went from 1428 to 3660 rows) and is unread for the gateway postgres, which a market network does not run (the tenant clusters' postgres is the one in the rows above). The archived 14:01Z to 22:02Z run (8 h, 8 to 13 lanes, 0.6 to 2.7 RFP/s, before fixes 4 to 7) shows the same shape: bidder, requester and network kcp flat (1.0, 1.05 and 0.65 GB), the gateway kcp at 1.76 GB and rising. - **Plan 0016 ACC16-TIMING and ACC16-NO-REGRESSION: createCluster to Ready was waiting on the kcp client rate limit, not on the CPU (2026-10-17 UTC).** Host: a DigitalOcean s-4vcpu-8gb droplet (4 vCPU, 8 GB, Ubuntu 24.04, sfo2) shaped like the prod gateway, one market network with a gateway and one bidder, the bootstrap tenant cluster as the createCluster, read in the gateway core log from "the bootstrap cluster exists" to the first `cluster provider: status` with phase Ready. Before (main at ebedcc74): 61.5, 62.0, 57.4, 57.5, 57.3 s, median 57.5 s. With the five-answer /readyz streak of ROLL-1 (main at 1b87e578): 65.1, 61.3, 63.3 s. After (this change): 35.9, 34.5, 33.4, 37.4, 32.1 s, median 34.5 s. Cause: the clients of the cluster provider, the HostProcess supervisor, the Postgres controller and the tenant add-ons were built from a rest.Config without QPS, so client-go limited them to 5 requests a second. Each of the 15 port holds (a Get and an Update of the front HostProcess) took 0.4 s, 6 s in all, and every burst of ESO CRD applies and tenant add-on calls queued behind the same limiter. `kcpserver.FrontRestConfig` and the tenant and admin clients now set 50 qps and burst 100. Four smaller changes: the Postgres controller wakes on a new Postgres object and polls at 200 ms while one is not Ready (it ticked every 3 s, three ticks per store, 10 s from object to Ready, now about 3 s); the cluster's Postgres object is created before the front is placed and awaited after it, so initdb runs beside the front; the ESO webhook and controller share one tenant-half install (`stepNeeds`: the controller no longer needs the webhook ready, only the install, which both Prepares run once) and its CRDs are applied 8 at a time; a pass that spawned is followed by one at once, and `place` polls at 100 ms. A k3s step gates on the unauthenticated and controllers checks only: the five-answer streak stays in `WaitSlot`, which only the roll calls, so the roll guarantee of ROLL-1 is the code it was. Member timeline before and after (seconds after createCluster; P placed, R ready): front P 8.1 R 11.1, kine P 17.7 R 21.2, k3s P 21.3 R 33.8, ccm R 35.9, esowh P 46.6 R 51.2, eso R 56.3, Ready 57.4; after front P 2.9 R 5.4, kine P 6.3 R 10.5, k3s P 10.6 R 25.7, esowh and eso P 30.1, ccm R 30.8, eso R 32.8, Ready 33.4. One run with every change but the 50 qps: 53.3 s. CPU: `taskset` on the `dev up` process tree (the units of an instance are not in the user slice, so a cgroup cpuset on the user slice changed nothing). 2 vCPU: main with the streak 99.9 and 87.4 s, the change 62.8 s; this matches prod's 78 to 117 s, so prod behaves like 2 vCPU. 4 vCPU: 57.5 (63.3 with the streak) before, 34.5 after. On a c-16 droplet (dedicated cores) 16 vCPU: 28.1 and 23.4 s after, 59.6 s before; 8 vCPU: 28.1 and 23.4 s after. Before the change CPUs did not help (59.6 s on 16 vCPU against 57 to 63 s on 4); after it, 4 to 8 to 16 vCPU buys about 25 percent. Resizing the prod droplet is not needed for ACC16-TIMING. Fail fast: with kine's argv replaced by /bin/false in a scratch build (not committed), kine@a was placed at 10:18:23.3 and the pass reported "kine@a of 4r81tsr8-tenant is Failed: hostprocess: ... kine@a.service failed after placement (result start-limit-hit, status 1)" at 10:18:32.9, 9.6 s later (the unit's own crash-loop budget, 5 starts at RestartSec 2); k3s and every member after kine were never placed, the next passes retried at the 2 s tick with the same message, and `dev down` left no unit, process or state. Killing k3s with SIGKILL during its start instead is restarted by its unit and is not a failure (the gate waited for the new k3s, Ready 33.3 s). Not done, and why: the hold of each port records itself on the front HostProcess and in the lease file one purpose at a time; batching that is a change of the held-port logic, which plan 0016 stop_and_ask reserves, and it now costs about 0.5 s; initdb --no-sync saves under 1 s and is left out. `deploy/check.sh --fast` on the droplet: gofmt, build, apigen, vet and deno check pass; the windows cross build stops in test/soak (`syscall.Stat_t`), a file of main that this change does not touch. Not read on prod; the prod reading is the lead's. - **ROLL-1 closed: a rolled k3s slot got the apiserver front before its own `/readyz` was green (2026-10-17 UTC).** Opened by the acceptance check ACC6-CP-SWAP. Reading before, on dev-linux-1, a market network with a gateway and one bidder, a tenant cluster with one market node and one running pod, two pollers every 200 ms through the cluster front (one with a kept-alive connection, one with a new connection and a client certificate per poll), `units roll cluster --idle-bound 5s`: greenReadyAt 20.4 s after the request, front flipped 0.006 s later, old slot stopped 1.2 s after that; the new connections got 500 from 07:32:26.6 to 07:32:30.9 (22 of 481), the first three also naming `poststarthook/rbac/bootstrap-roles` (the first two `poststarthook/scheduling/bootstrap-system-priority-classes` as well), all of them `poststarthook/apiservice-discovery-controller`; the kept-alive poller failed 8 of 479 from 07:32:28.5, after its connection to the old slot closed. Cause: `WaitSlot` (`internal/clusterprovider/front.go`) passed when `waitK3s` (`internal/clusterprovider/steps.go`) saw 200 or 401 from an anonymous `/readyz` on the slot port, and when `waitK3sControllers` found a Deployment in kube-system, which the old slot had already written to the shared store. Neither says the new apiserver ran its post-start hooks, and the roll flipped the front on that. Fix: `WaitSlot` also calls `waitK3sReadyz`, which needs `/readyz` to answer 200 five times in a row (250 ms apart) with the slot's admin client certificate before the front is pointed at the slot. This holds for the first start (the k3s step gate in `steps.go`) and for the roll. The old slot serves until the flip, and the front keeps established connections on it until it is drained, as before. Reading after, same run, three at once on one host: 0 failed of 532 and 0 of 528, 0 of 537 and 0 of 533, 0 of 527 and 0 of 528 polls from 3 s before the request to 10 s after the report; the roll took 95.1 s, 96.1 s and 95.1 s of which 10 s the idle bound and 60 s the watch; greenReadyAt 28.6 s, 29.1 s and 28.7 s after the request (about 8 s more than before because three clusters started at once, and 1.3 s for the five answers); flip to old stop 0.95 s, 1.6 s and 1.6 s; the node's Ready transition time, the NodeClaim phase and receipt, and the ComputeVM were the same before and after. Plan 0006 ACC6-CP-SWAP is done. - **The plan-level acceptance checks of plans 0006 to 0009 and 0013 to 0016, read on 2026-10-17 UTC (main at 80998ef9).** Host: dev-linux-1 (Ubuntu 24.04, 48 vCPU), up to six market networks at once, VM account test-0001; the gateway networks needed the web app downloads asset staged unpinned (its tree sha256 differs from the pin off Fedora), and the sign-in test needed Chrome and kubectl on the host. Integration tests, each PASS: `TestARentedVMSurvivesABidderKillAndRestart` 226 s, `TestRequestSecretsReachTheGuestThroughTheRFPFlow` 217 s, `TestTenantClusterScalesFromZeroOnMarketVMsAndBack` 389 s (0 ssh lines), `TestBidderAnswersEachRFPOnceAcrossARestart` 308 s, `TestKubeconfigClientCertificateWithoutCredentialHelper` 170 s with kubectl, `TestWebAppSignInThroughThePluginKeepsTheCookieAcrossARestartAndIssuesACertificate` 384 s. The hostsvc contract suite passed on systemd-user and on proc. One check failed (ROLL-1, closed in the entry below). Manual runs on top of the harness. Certificates (market network, gateway, no bidder): `getKubeconfig` without `ttlSeconds` gave 8 h, 86400 gave 24 h, 60 gave 1 min; 59, 86401, 0, -5 and 999999 were 400 InvalidRequest; k3s saw the caller DID as user and `socialweb.computer:cluster-owners` as group; injected `X-Remote-User`/`X-Remote-Group`/`X-Remote-Extra-*` headers were ignored; the k3s `client-admin` certificate of the same client CA was refused (403); `revokeIdentity` refused the old certificate 0.08 to 0.17 s after the call, `deleteCluster` 0.16 s; the revoked serials were in `pki/certs/revoked`; OpenBao builds no CRL (`pki/crl/rotate` warns that the imported issuer lacks CRL signing key usage), so the plan text now says the revoked list. The issued client key was in no file under the network state dir, no journal line, no process argument or environment and no kcp object (8 kinds). Sign-in (market network, gateway, bidder): after `kill -9` of the gateway core (back 0.6 s later, 4 DenoPods restarted, 9 member processes kept their pids) the same cookie answered `GET /session` 5.9 to 6.5 s later and the claim kept its uid and Running phase; `DELETE /session` made the PDS log a call to `/oauth/revoke` (log line added in the mirror only), the cookie then got 401, and a downloaded certificate was refused 0.32 s after `revokeIdentity`; the cookie token was in no journal line and no process, the user's repo held 14 records in 4 collections and none held a token or key, and the only text files with `PRIVATE KEY` were 28 TLS and kcp `*.key` files of the infrastructure. The web app's DenoPod environment holds only projected file paths; its vault client `client-socialweb-computer-k8s` reaches `socialweb-computer-k8s/*` only and that mount held only `contracts//bundle`. Vaults (gateway network, one bidder, one node): with a node the cluster vault held one `jwt-` mount, its role, the policy `node-` and the `socialweb-nodes/` entry, equal to the OpenBaoServer declaration; the same after a restart of the bidder DenoPod, after raising `signingKeyRotation` to 7 (three keys published) and after `kill -9` of the gateway core; 45 s after the pod was gone the cluster vault held approle, cert and token only, the bidder vault held no `guests/` entry and no `guest-` policy, the root vault held no `contracts/` entry. Only the root vault (atproto plugin) and the bidder vault (sshsig plugin) set a plugin directory. Bidder (market network, two guests): each guest's pasta and firecracker ran in their own user namespace under host uids 721896 and 656360 inside test-0001's range 655360 + 1048576, no file under the state dir was owned by those uids, and neither uid could read the bidder's state, kcp, vault keys or the other guest's work dir; `kill -9` of a firecracker made its ComputeVM Lost 0.21 s later, the unit was inactive, the ComputeVM was released by the bidder and the `vms/` directory held the live guest only; `units audit` passed after boot and after a bidder restart; ssh with the reported host key worked, with another key and with no entry it failed `Host key verification failed`; no log of the network held an ssh session line. Deno argv: 42 service processes, none with `-A`, all with `--deny-read` and `--deny-write` of `/dev/kvm` and `/run/user/`, the policy engine with no `KCP_TOKENS` and no projected read, no process with `--allow-ffi`, and `--allow-run` only on the 20 launchers of `deploy/denopods/service.yaml`. Installed bidder (staged install of account swc-acc joined to a market network, two held guests, UTC 07:03 to 07:04): `systemctl --user kill -s KILL` of the bidder, `systemctl --user restart` and `upgrade --binary` to a build with another closure each brought it back Ready in 12 s, 4 s and 4 s with three DenoPods once each, the two VM units kept their InvocationID, pid and start ticks, both ComputeVMs stayed Running, `units audit` passed each time, after the upgrade every DenoPod ran closure 109e962f and the closure 013dc7eb stayed on disk; `run request release` then gave vm.delete 200 for both with no ComputeVM, VM unit or work dir left. The qr.fedfork.com client metadata scope covers every bidder and requester scope token (25 against 24). Not shown by these runs: a second bidder process on the same kcp (the relay ingress sent every RFP to the second process, so no duplicate arrived) and a gateway `upgrade` with a signed-in cookie. - **Plan 0012 is paused by the user (2026-10-16); what M0 settled and what it did not, as of 2026-10-17 UTC.** Done and committed: M0-REVISIT, M0-HOST-BASELINE and M0-VFKIT-ASSET (entries below). One pre-probe was run before the pause, outside any item: a system-domain LaunchDaemon with `UserName` the non-admin account `socialweb-m0012` ran the ad-hoc-signed thin vfkit (virtualization entitlement only) with `--bootloader efi,variable-store=...,create`, one vCPU, 512 MiB, a virtio-serial log file, a REST socket and no disk, no kernel and no network; it is a framework probe and not a guest, and the RFP rule that every guest is born from cloud-init was not touched. vfkit logged `virtual machine is running` and, in the same second, `VM is stopped` (the firmware had nothing to boot), and launchd read last exit code 0. So Virtualization.framework accepted a VM start in a job with no GUI session and a non-admin user while the owner's console session stayed open. It does not say whether a VM survives the death of its submitter, runs with no console session at all (the Mac may not be signed out, so that cannot be measured here), runs in the user/ domain of an account with no login, or what its networking does; those stay M0-VZ-LIFETIME and M0-LAUNCHD-DOMAIN. Findings that change later items: a bidder on a Mac needs kcp, kine (sqlite, cgo, built here in 13 s) and bao as darwin assets and a launchd `hostsvc.Manager`, and `dev up` (the market network) is Linux only because `devsup` is, while the child bidder's `vmHostBackend` option already lets a network place VMs with another backend; a Mac market network therefore needs a darwin `devsup` (about 1000 lines on Linux: subreaper, pidfd and /proc become kqueue, process groups and sysctl) or a remote network that this Mac joins, and the second is not supported today because every network service binds 127.0.0.1:0 and peers find it by a per-application DNS shim; a darwin guest kernel needs virtio-pci (the firecracker configs have virtio-mmio only) and `vz-arm64` is the name reserved for it in versions.json, to be built on Linux by `scripts/build-guest-image.sh` from `firecracker-aarch64.config` plus PCI; no firecracker-aarch64 image was found in the Spaces bucket at main/35e91d21 (only x86_64 under `main//guest-image/`; other commits were not listed), so the arm64 rootfs would be built too. A c-16 build droplet was created for that build and deleted again before it did any work. - **Plan 0012 M0-VFKIT-ASSET: vfkit as an asset, shipped as the Red Hat universal file, extracted by the core's own store and run by a daemon user, read on 2026-10-17 UTC (05:25 to 05:40) on the Mac of M0-HOST-BASELINE.** Source: `https://github.com/crc-org/vfkit/releases/download/v0.6.4/vfkit`, 66431936 bytes, sha256 `0ed83fc8ca7aa708598835480dba1362406aa7cd1dab3b27464eb76327d9652d`, equal to the digest GitHub shows for the asset; the release also has `vfkit-unsigned`, not used. It is pinned in `internal/versions/versions.json` as asset `vfkit` (platform darwin/arm64, `kind file`, `archive raw`, tree sha256 28d3b7e3..., rawBytes 66431936, zstdBytes 29401851 measured with `assetpack tar` and `zstd -q -19 --long=27 -T0`). The file is a universal Mach-O (x86_64 + arm64) signed `Developer ID Application: Red Hat, Inc. (HYSCB8KRL2)`, hardened runtime (flags 0x10000), entitlements network.client, network.server and com.apple.security.virtualization; `codesign --verify --strict` passes and `spctl --assess --type open --context context:primary-signature` answers "accepted, source=Notarized Developer ID". Decision from the readings: ship it as is. It needs no signing step, keeps Red Hat's provenance and notarization, and already carries the virtualization entitlement; the cost against the arm64 slice alone is 66.4 MB against 31.9 MB raw and 29.4 MB against 15.1 MB zstd. Alternatives measured and not chosen: the arm64 slice (offset 34504704, size 31927232, sha256 84d9dd06...) is byte-identical to `lipo -thin arm64` (run twice) and keeps a valid signature that spctl also accepts as Notarized Developer ID, so a Linux build could cut it without any Apple tool; an ad-hoc re-signed thin file (`codesign --force -s - --entitlements `, sha256 d5582077..., flags 0x2, no team id) runs but spctl rejects it. Extraction used the real `internal/assets.Store.Resolve` (a throwaway program, not committed) with a pin built from `assets.TreeDigest`: 45 ms for the thin file and 96 ms for the universal one, a new inode inside a `.tmp-` directory renamed to `//vfkit`, store root mode 0700, no extended attribute on the result, `codesign --verify` valid after extraction. Run as the non-admin account `socialweb-m0012` (uid 480), the extracted ad-hoc file ran as a system-domain LaunchDaemon with `UserName` that account: `vfkit version: v0.6.4`, last exit code 0. Quarantine, three outcomes with the `com.apple.quarantine` attribute set by `xattr -w` (values `0083;