version: 1 generated: 2026-09-24 repo: /home/johnandersen777/src/publicdomainrelay-kcp/socialweb-computer-kcp objective: one_sentence: > A tenant cluster whose k3s NODES scale to zero while its kcp + `k3s server --disable-agent` control plane stays live. why_disable_agent_matters: > A single-node k3s where the server IS the node can never produce zero nodes. `--disable-agent` is what makes "zero nodes" a real state rather than a contradiction. scope: in: kcp, k3s, k3s-node-in-firecracker, rustfs, kine, openbao, postgres out: > Every other repo. Provisioning guests through the market/RFP flow. The market's SSH door (its report surface is in scope, the door is not). Multi-node per cluster (v1 is 0-or-1). standing_constraints: - No comments in code, ever. Decisions live in docs/ARCHITECTURE.md. An enforced hard rule. - Always update docs. The doc is the decision record; code carries none. - Launch subagents for all work; the main thread coordinates and verifies. - Node runs as a guest VM (firecracker). Nothing on the L0 host needs root. - Scope subagents so they do not need to talk to each other. When they must, tell them how. state_at_handoff: branch: refocus-core-process-substrate head: 4db84a9 worktree: clean architecture_doc_lines: 2814 go_packages: 50 tests: > `deploy/check.sh --fast` green (about 9 s); `--local` green (about 75 s, names the 14 artifact-gated skips); `--full` needs the live artifacts and exits non-zero naming what is missing without them. The privileged rig that used to carry the live gate (`deploy/test-rig`) was deleted on 2026-10-07 with the local substrates (plan 0005 M1-MARKET-ONLY); the market network is what boots a real guest now. deployment: > The deployment is ONE container: `deploy/run-container.sh` builds the image and starts it detached as `socialweb`, and `deploy/wait-ready.sh socialweb 300` returns 0 when the cluster reaches `phase: Ready` (about 30 s, as `uid=1000`). Inside it: kcp, the gateway, the credential plugin, the clusterprovider, each cluster's postgres/kine/k3s, a root OpenBao and a per-cluster OpenBao. Nodes are firecracker guests. The host runs only the container runtime. kcp_on_host: > The host systemd units are RETIRED: `deploy/install-kcp.sh` and `deploy/socialweb-gateway.service` are deleted, and kcp now runs inside the container. A stale /etc/systemd/system/kcp.service may still exist on this host from before; it is not the deployment and nothing on the host's 6443 is claimed. A test that needs kcp starts one with its own root directory, as the proofs do. proven: four_step_proof: test: TestTheFourStepProofRunsThroughTheCompositionRoot location: cmd/socialweb-clusterprovider/live_fourstep_test.go result: PASS 97.65 s reverified: > PASS 103.10 s on 2026-10-05, same host, same command, all four steps, nothing deleted by hand. Re-run because the GitLab and cloud-controller-manager work leans on this proof and a reading that names no date cannot be told from a stale one. The 97.65 s entry is kept rather than replaced: the two are different builds, and the newer one is the one that was watched. transitions: - Cluster CR reached Ready at its leased apiserver port with ZERO nodes - "`kubectl get nodes` empty and staying empty across passes" - a node joined ON DEMAND as liveweb-node1 Ready (Provision, nodes=1) - back to zero with the control plane still Ready and still listing its namespaces timing: INSTANCE_START_TO_READY_S = 9.749 / 9.609 (host-timed, firecracker InstanceStart to Node Ready) timing_2026_10_05: INSTANCE_START_TO_READY_S = 5.430 (host-timed from the firecracker VMM's instance record to the Node's Ready condition, resolution 250 ms) gitlab_job_pod_proof: test: TestAGitLabJobPodIsPlacedOnAProvisionedNode location: cmd/socialweb-clusterprovider/live_gitlab_ci_test.go result: PASS 84.40 s (a second run of the same test reported 86.16 s), 2026-10-05 transitions: - control plane Ready with zero nodes; `kubectl get nodes` empty - namespace gitlab-runner created - the Pod shape GitLab Runner's kubernetes executor creates is applied: one build container and one helper container in one Pod, RestartPolicy Never, labels job=gitlab-ci and runner.gitlab.com/job-id, no control-plane annotation - the scale decision reads Provision with nodes=1 - the node joins on demand and the Pod reaches Succeeded - the Pod's spec.nodeName equals the node the scale decision provisioned (liveweb-node1) - the Pod is deleted; the decision reads Teardown; `kubectl get nodes` is empty again with the control plane still Ready reads: > This is the cluster half of "GitLab CI works here", measured. It is the runner's actual Pod shape, not a Deployment standing in for it. The GitLab instance half - a pipeline going green in GitLab's UI - is documented in docs/gitlab/ and NOT run on this host, because gitlab/gitlab-ce is a multi-gigabyte image the gate never pulls. three_node_scale_proof: test: TestThreeGitLabJobPodsScaleToThreeNodes location: cmd/socialweb-clusterprovider/live_scalemax_test.go result: PASS 90.01 s (an earlier run of the same test reported 95.65 s), 2026-10-05 env: SOCIALWEB_CLUSTERPROVIDER_SCALE_MAX_NODES=3, SOCIALWEB_CLUSTERPROVIDER_SCALE_PODS_PER_NODE=1 transitions: - control plane Ready with zero nodes at its own leased port; `kubectl get nodes` empty - three GitLab Runner job pod shapes applied at once (each one build + one helper container) - "the scale decision reads `Provision, Nodes: 3` with reason `intent and no node`" - three nodes joined on demand; `kubectl get nodes` reports three - all three job pods reached Succeeded - the pods are deleted; the decision reads Teardown past the 1m hold; `kubectl get nodes` is empty again with the control plane still Ready reads: > Autoscaling to more than one node, measured. What it does NOT show, stated because the numbers read stronger than they are: it asserts the CLUSTER reached three nodes, not that three guests each hosted work. The scheduler placed all three pods on liveweb-node1, which the design permits, so the evidence for three simultaneous guests is three Provision calls plus `kubectl get nodes` listing three, together with an independent observation of two firecracker processes during the run. No negative control was run: that the test would fail with MAX_NODES unset is argued from Target's ceiling rather than measured. Partial scale-in is not implemented (plan 0002 section 4.4), so the target only grows while intent exists and the down direction remains the all-or-nothing teardown past the hold. gitlab_pipeline_proof: test: TestAGitLabPipelineRunsOnATenantCluster location: cmd/socialweb-clusterprovider/live_gitlab_pipeline_test.go result: PASS 34.12 s, 2026-10-05, against a real GitLab CE 19.4.1 and a real gitlab-runner 19.4.1 requires_these_env: > SOCIALWEB_LIVE_GITLAB_URL, SOCIALWEB_LIVE_GITLAB_TOKEN (a project access token), SOCIALWEB_LIVE_GITLAB_PROJECT (numeric id), SOCIALWEB_LIVE_GITLAB_RUNNER_TOKEN (the project runner authentication token) and SOCIALWEB_LIVE_GITLAB_RUNNER_BIN. Unset, the test SKIPS with the list, which is not a pass. transitions: - a tenant cluster reached Ready with ZERO nodes at its own leased apiserver port - a tenant credential was minted from that cluster's own ClientCA and written as ca.crt, tls.crt, tls.key - config.toml with executor = "kubernetes" was written and the real gitlab-runner started against it - a .gitlab-ci.yml was committed through the GitLab API and a pipeline triggered - "the job pod's own Pending state drove the scale decision (`intent and no node`) and a node joined" - GitLab's runner logged `Waiting for pod ... to be running on the node liveweb-node1` - two job pods were observed carrying spec.nodeName = liveweb-node1 - the pipeline reported success and the job's trace carried the marker this test's own .gitlab-ci.yml writes reads: > This is the whole GitLab integration end to end, and it is the strongest reading in this file: GitLab itself scheduled the job, GitLab's own runner created and drove the pod, and the pod ran on a node this cluster provisioned from the job's own Pending state. Three findings are carried in docs/ARCHITECTURE.md section 15 rather than here: the runner must be pointed at the apiserver's LOOPBACK address with a client certificate (the kcp mount carries bearers only and the exec path is not guaranteed to survive it); the live harness drives the scale decision only when a test asks it to, so the first attempt left its pod Pending for ever; and a job Pod runs INSIDE a guest, so CI_REPOSITORY_URL naming the GitLab host's loopback resolves to the guest's own and the clone fails with exit code 128, which the proof avoids with GIT_STRATEGY: none. not_run: > The GitLab container and the tokens are throwaway and live on this host only. A runner OFF the host still cannot reach a tenant apiserver (NEXT.yaml GL-2), and the minting CLI itself has not been run against a real data dir (NEXT.yaml GL-1). control_plane_in_container: image: socialweb-control-plane:local files: deploy/control-plane-container/{Dockerfile,build.sh,validate.sh} result: > TestTheControlPlaneReachesReadyAtZeroNodes PASSES INSIDE the container: 9.50 s, 7.78 s, 7.67 s across runs against a ~6 s host baseline. Runs as USER 1000:1000. NO --privileged, --security-opt, --cap-add, --device, --network host, and no userns remap. honest_verdict_inside: "9 HOLDS, 1 MISSING, 10 MUST BE PASSED IN, exit non-zero (control-plane half only)" mount_gate_question: question: Does a Draining CR shut the mount BEFORE its process group dies? answer: YES, measured against a real kcp started inside a test test: TestTheMountShutsBeforeTheTenantProcessGroupDies location: internal/clusterprovider/live_mount_test.go skip_behaviour: no env gate; skips only if kcp/kine/k3s/postgres are absent, with a skip message saying a skip means the question stays open. Runs in ~27 s, needs no root and no /dev/kvm. three_answers: serving: "401 with the tenant apiserver's own k8s Status body (a REACH, proven by bodies)" group_SIGTERMed: "502 empty body -- the reverse proxy with no backend" CR_Draining: "503 `Not available.` -- kcp's own refusal, backend never contacted" timeline: Draining moves the mount off 401 at +206 ms with the group still serving; reap comes on a LATER pass at +260 ms reading 503; group dies last. host_contract: binary: cmd/socialweb-computer-kcp-check-host host_result: "19 checks, 19 HOLDS, 0 MISSING, exit 0 (verified by running it, not by report)" discrimination: broken module source -> 2 MISSING exit 1; no config -> 5 MISSING exit 1 categories: HOLDS / MISSING / MUST BE PASSED IN why_three: > On a host the last two collapse to "this host cannot run this". Inside a container, MISSING means the image is wrong while MUST BE PASSED IN means the container is correctly declaring its edge. One binary, two readings. how_to_run: deployment: "deploy/run-container.sh ; deploy/wait-ready.sh socialweb 300" gate: "deploy/check.sh --fast (the loop) ; deploy/check.sh --local (this host's proofs) ; deploy/check.sh --full (needs the live artifacts)" live_gate: "SOCIALWEB_REQUIRE_LIVE=1 go test ./... (a live tenant cluster and a real guest; a skip is a failure)" all_tests: "deploy/check.sh --fast" control_plane_zero_node_proof: host: "SOCIALWEB_LOCAL=1 go test ./internal/clusterprovider -run TestTheControlPlaneReachesReadyAtZeroNodes -v -count=1" in_container: "deploy/control-plane-container/validate.sh" mount_proof: "SOCIALWEB_LOCAL=1 go test ./internal/clusterprovider -run TestTheMountShutsBeforeTheTenantProcessGroupDies -v -count=1" host_contract: "go run ./cmd/socialweb-computer-kcp-check-host -config /home/johnandersen777/kcp-image/nodeimage.json" four_step_proof_env: note: > These variables gate the four-step proof, and a skip is not a pass. The rig derives them from the image store and the builder's configuration; by hand, they need the BUILDER'S OWN CONFIGURATION, not just a built image directory -- an image is only valid relative to the configuration whose inputs name it. vars: - SOCIALWEB_CLUSTERPROVIDER_LIVE_IMAGE_OUT_DIR - SOCIALWEB_CLUSTERPROVIDER_LIVE_IMAGE_CONFIG - SOCIALWEB_CLUSTERPROVIDER_LIVE_FIRECRACKER - SOCIALWEB_CLUSTERPROVIDER_LIVE_NODE_AGENT - SOCIALWEB_CLUSTERPROVIDER_LIVE_K3S - SOCIALWEB_CLUSTERPROVIDER_LIVE_KUBECTL - SOCIALWEB_CLUSTERPROVIDER_LIVE_TENANT_IMAGE optional: SOCIALWEB_CLUSTERPROVIDER_LIVE_SCALE_HOLD (proof default 1m; scale.DefaultHold is 2m) live_environment: image_store: /home/johnandersen777/kcp-image/out store_contents: 3 images, 4.0 GiB total (older names kept as rollback) newest_fingerprint: sha256-e3f1e594e7aefb8a7c51e370a6ebd3f5e34f6364ceaf4eb97d7ef0d33f4d24c3 builder_config: /home/johnandersen777/kcp-image/nodeimage.json builder_config_notes: > 48 modules, 5 images pinned as @, agent_go_flags carries -buildvcs=false, firecracker and base_image repointed under artifacts/. This file is OUTSIDE the repo and is the operator's, not the code's. cross_check: > With -buildvcs=false the host's own builder answers "up to date ... nothing built, nothing pulled" for the directory the container's stage produced. Both builders compute one name from one input set; that agreement is the proof the fix worked. open_work: operator_actions_not_repo_work: - what: nodeimage.json gained -buildvcs=false; the next build produces a NEW fingerprint once. where: docs/ARCHITECTURE.md:1991 and :1995 recorded_in_memory: dir: /home/johnandersen777/.claude/projects/-home-johnandersen777-src-publicdomainrelay-kcp-socialweb-computer-kcp/memory/ items: - file: cleanup-backlog-close-each-tick.md line: 53 what: >- STALE as of 2026-09-24: the salvage-list numbers ARE in the doc now (docs/ARCHITECTURE.md:728, :731, :735, :736). Closed by DEV-6; do not re-report. - file: cleanup-backlog-close-each-tick.md line: 62 what: whether section 7 already carries the auth-paths conclusions; 1107 untracked experiment files were lost and never compared section_14_stated_limits: note: >- UPDATE 2026-09-24 (DEV-8): section 14 no longer claims "no open gaps". It now lists the open items, is the doc-side mirror of NEXT.yaml, and keeps the two stated limits below. entries: - line: 1348 what: > The allocator's host-network overlap refusal is a property of the NETNS it runs in -- host sees 15 networks, container 3 -- so a containerized allocator would lease a colliding range without refusing. Bounded because the allocator and the node substrate are ONE PROCESS behind SOCIALWEB_CLUSTERPROVIDER_SUBSTRATE=firecracker, whose validation already demands /dev/kvm and a VMM. Conditional: costs nothing only because this host's LAN/bridges sit outside 10.100.0.0/16, and a collision would land on the HOST. Seam exists: subnetallocator.Config.HostNetworks is injectable. - line: 1368 what: > liveports.LockDir() is /var/tmp/socialweb-liveports inside each container's own filesystem, so two proofs in two containers cannot see each other's slots. No production path depends on it -- internal/liveports is imported only by tests and the host-contract probe. not_recorded_anywhere_closed: - what: dangling /run/k3s containerd shm mounts needing root status: > CLOSED and NOT written down. Verified: 0 k3s entries in /proc/self/mountinfo and /run/k3s does not exist. It dated from the refuted local-process substrate. Do not add it to a list. do_not_rederive: - Cold boot to an init marker is 0.300 s. Boot is NOT the problem; the 10 s goal is met 30x. - > SNAPSHOT RESTORE IS REFUTED as the provisioning primitive. restore-to-Ready 9.974 s against cold 9.079 / 9.158 / 10.768 s, so restore buys nothing in time; and a restored guest cannot make a NEW node because restore does not re-run boot-time units, so the identity adoption never runs and a snapshot frozen as one node comes back as that node. Nodes boot COLD from their own rootfs clone. - Kernel 6.18.48 PANICS on this AMD host at restore (Bad FPU state, XRSTOR/AMX). Validate an image by RESTORING it, never by comparing versions. - The reported "332 s boot" was FABRICATED: a literal echo in run/initrd/init. Never trust a guest's self-report; the harness times on the host. - pasta + a tap inside the pasta netns works; an unprivileged tap in the HOST netns fails. The apiserver stays on loopback; --tls-san carries the address the guest dials. - k3s check-config reads the kernel CONFIG file and never checks whether a module is LOADED. It returns 0 over a missing module. This is why the module set is MEASURED, not checked. - iptables-restore is all-or-nothing, which is why the module list is 48 and not 43. - A named build context is mounted READ-ONLY, so a stage cannot write the store through one. /boot cannot even BE a context (/boot/efi is 0700). - The agent binary carries Go's VCS stamp, so a commit changing no agent source still moves every image's name. -buildvcs=false fixes it; agentBinaryDigest still identifies the source by content. - The advertise port and the k3s secure port are ONE lease answering both --apiserver flags, not two. known_traps: stale_gopls: symptom: > gopls repeatedly reports undefined symbols, unused imports and missing interface methods that the compiler disagrees with. It happened 6+ times in one session, including a large dump claiming a dozen files were missing a method that exists. rule: The COMPILER is the authority. Run `go build ./...` before acting on any diagnostic. root_cause_update: > 2026-09-24 (DEV-2): the tools were built with go1.23.7 and could not type-check this go1.26.4 module. gopls/staticcheck/govulncheck are now rebuilt with go1.26.4 and staticcheck has no `(compile)` lines. The trap was a stale tool build, not gopls flakiness. subagent_claims: rule: > Verify the artefact, not the summary -- including a subagent's. Read the deployed file rather than the doc's description of it; run the binary rather than its report. fake_registries: rule: > Every test that runs the reconciler substitutes a fake whose MountStatus answers from Spec.Running. A fake that answers the question underneath the question is how a vacuous gate survived a green suite for as long as it did. status_update: > 2026-09-24 (REL-6): the three live-proof fakes now answer 401 live / 503 Draining / 502 reaped from host liveness and DeletionTimestamp, not from Spec.Running. validate_sh: rule: > Container names are fixed, so a killed run blocks the next one; each run now removes its own name first. The host-contract step is gated on a POSITIVE assertion (the summary line must be present), not an exit-code allowlist, because check-host exits 2 without printing one. status_update: > 2026-09-24 (REL-3): the final `go test` is now also gated on a positive `--- PASS: $TEST` line, because go test exits 0 on SKIP and on `[no tests to run]`. caveman_builder_agent: rule: > caveman:cavecrew-builder has Read/Edit/Write/Grep/Glob ONLY -- no Bash. It cannot compile, test, or commit. Use it only when correctness is established by reading; otherwise dispatch general-purpose or run the verification yourself. the_recurring_defect: statement: > The dominant failure in this repo is a thing that reads GREEN while being WRONG. Prefer instruments that can fail. When adding a guard, watch it fail before trusting it. instances: - k3s check-config returning 0 over a missing module - pods 1/1 over an incomplete firewall - an advertise port nothing bound - TestEveryVerbIsBound asserting not-404 (presence) while getKubeconfig was permanently dead - check-host printing "holds 212" from free/budget when the store held 2 - an EMPTY /etc/rancher/node/password that opens read-write while k3s 400s for ever - validate.sh's `|| true` making the host-contract step unable to fail - the shut gate reading 401 as shut, so a SERVING cluster was reaped - a list outliving its work -- three times in one day - "2026-09-24: the teardown gate reaping on 300/429/500/504 (REL-2); Probe claiming Ready on a 503 (REL-1); the ClientCA pair check hashing openssl's empty stdin so junk compared equal (SEC-5); validate.sh exiting 0 on SKIP (REL-3); the live proofs' fakes answering 200 (REL-6)" practice: > Run the artefact. Watch the guard fail. Report the reading, not the intention. If a claim is greener than the fact, the claim is the bug. orientation: layout: | api/v1alpha1/ wire types only. No logic, no I/O internal/${concept}/ one package per concern; the interface declared where consumed cmd/${binary}/ the CLI layer; nothing else builds a binary deploy/ install scripts, manifests, audit policy, the control-plane container dependencies: "api <- internal <- cmd, one way. A new substrate or transport is a NEW PACKAGE, not a branch inside an existing one." binaries: [socialweb-computer, socialweb-vmexec, socialweb-clusterprovider, socialweb-gateway, socialweb-vault, socialweb-computer-kcp-check-host] key_packages: internal/vmruntime: the Linux VM runtime a bidder boots each firecracker guest with (socialweb-computer internal vm); internal/nodefirecracker keeps only the old image store check-host reads internal/nodemarket: the one node path, a NodeClaim the door fulfils under the owner's atproto session internal/noderecord: reads the node records a retired local substrate left, for the startup sweep internal/kernelmodules: the builder's module check, shared by the builder and check-host internal/subnetallocator: pool 10.100.0.0/16 -> /24 per cluster internal/liveports: flock port rendezvous for concurrent proofs internal/materialiser: the reconciler; shut()/reached() gate a teardown internal/kcpclient: kcp client; ServiceAccountName is the DID mapping's one spelling internal/livegate: the SOCIALWEB_REQUIRE_LIVE / -short gate for live tests start_here: docs/ARCHITECTURE.md sections 1 (objective), 3 (zero state), 9 (a node), 11 (teardown and the gate), 12 (code layout), 13 (invariants), 14 (the open list), 15 (settled decisions, with the readings)