0
0
version: 1
generated: 2026-09-24
repo: /home/johnandersen777/src/publicdomainrelay-kcp/socialweb-computer-kcp
objective:
one_sentence: >
A tenant cluster whose k3s NODES scale to zero while its kcp + `k3s server --disable-agent`
control plane stays live.
why_disable_agent_matters: >
A single-node k3s where the server IS the node can never produce zero nodes. `--disable-agent`
is what makes "zero nodes" a real state rather than a contradiction.
scope:
in: kcp, k3s, k3s-node-in-firecracker, rustfs, kine, openbao, postgres
out: >
Every other repo. Provisioning guests through the market/RFP flow. The market's SSH door
(its report surface is in scope, the door is not). Multi-node per cluster (v1 is 0-or-1).
standing_constraints:
- No comments in code, ever. Decisions live in docs/ARCHITECTURE.md. An enforced hard rule.
- Always update docs. The doc is the decision record; code carries none.
- Launch subagents for all work; the main thread coordinates and verifies.
- Node runs as a guest VM (firecracker). Nothing on the L0 host needs root.
- Scope subagents so they do not need to talk to each other. When they must, tell them how.
state_at_handoff:
branch: refocus-core-process-substrate
head: 4db84a9
worktree: clean
architecture_doc_lines: 2814
go_packages: 50
tests: >
`deploy/check.sh --fast` green (about 9 s); `--local` green (about 75 s, names the 14
artifact-gated skips); `--full` needs the live artifacts and exits non-zero naming what is
missing without them. The privileged rig that used to carry the live gate
(`deploy/test-rig`) was deleted on 2026-10-07 with the local substrates (plan 0005
M1-MARKET-ONLY); the market network is what boots a real guest now.
deployment: >
The deployment is ONE container: `deploy/run-container.sh` builds the image and starts it
detached as `socialweb`, and `deploy/wait-ready.sh socialweb 300` returns 0 when the cluster
reaches `phase: Ready` (about 30 s, as `uid=1000`). Inside it: kcp, the gateway, the
credential plugin, the clusterprovider, each cluster's postgres/kine/k3s, a root OpenBao and
a per-cluster OpenBao. Nodes are firecracker guests. The host runs only the container runtime.
kcp_on_host: >
The host systemd units are RETIRED: `deploy/install-kcp.sh` and `deploy/socialweb-gateway.service`
are deleted, and kcp now runs inside the container. A stale /etc/systemd/system/kcp.service may
still exist on this host from before; it is not the deployment and nothing on the host's 6443 is
claimed. A test that needs kcp starts one with its own root directory, as the proofs do.
proven:
four_step_proof:
test: TestTheFourStepProofRunsThroughTheCompositionRoot
location: cmd/socialweb-clusterprovider/live_fourstep_test.go
result: PASS 97.65 s
reverified: >
PASS 103.10 s on 2026-10-05, same host, same command, all four steps, nothing deleted by hand.
Re-run because the GitLab and cloud-controller-manager work leans on this proof and a reading
that names no date cannot be told from a stale one. The 97.65 s entry is kept rather than
replaced: the two are different builds, and the newer one is the one that was watched.
transitions:
- Cluster CR reached Ready at its leased apiserver port with ZERO nodes
- "`kubectl get nodes` empty and staying empty across passes"
- a node joined ON DEMAND as liveweb-node1 Ready (Provision, nodes=1)
- back to zero with the control plane still Ready and still listing its namespaces
timing: INSTANCE_START_TO_READY_S = 9.749 / 9.609 (host-timed, firecracker InstanceStart to Node Ready)
timing_2026_10_05: INSTANCE_START_TO_READY_S = 5.430 (host-timed from the firecracker VMM's instance record to the Node's Ready condition, resolution 250 ms)
gitlab_job_pod_proof:
test: TestAGitLabJobPodIsPlacedOnAProvisionedNode
location: cmd/socialweb-clusterprovider/live_gitlab_ci_test.go
result: PASS 84.40 s (a second run of the same test reported 86.16 s), 2026-10-05
transitions:
- control plane Ready with zero nodes; `kubectl get nodes` empty
- namespace gitlab-runner created
- the Pod shape GitLab Runner's kubernetes executor creates is applied: one build container and one helper container in one Pod, RestartPolicy Never, labels job=gitlab-ci and runner.gitlab.com/job-id, no control-plane annotation
- the scale decision reads Provision with nodes=1
- the node joins on demand and the Pod reaches Succeeded
- the Pod's spec.nodeName equals the node the scale decision provisioned (liveweb-node1)
- the Pod is deleted; the decision reads Teardown; `kubectl get nodes` is empty again with the control plane still Ready
reads: >
This is the cluster half of "GitLab CI works here", measured. It is the runner's actual Pod
shape, not a Deployment standing in for it. The GitLab instance half - a pipeline going
green in GitLab's UI - is documented in docs/gitlab/ and NOT run on this host, because
gitlab/gitlab-ce is a multi-gigabyte image the gate never pulls.
three_node_scale_proof:
test: TestThreeGitLabJobPodsScaleToThreeNodes
location: cmd/socialweb-clusterprovider/live_scalemax_test.go
result: PASS 90.01 s (an earlier run of the same test reported 95.65 s), 2026-10-05
env: SOCIALWEB_CLUSTERPROVIDER_SCALE_MAX_NODES=3, SOCIALWEB_CLUSTERPROVIDER_SCALE_PODS_PER_NODE=1
transitions:
- control plane Ready with zero nodes at its own leased port; `kubectl get nodes` empty
- three GitLab Runner job pod shapes applied at once (each one build + one helper container)
- "the scale decision reads `Provision, Nodes: 3` with reason `intent and no node`"
- three nodes joined on demand; `kubectl get nodes` reports three
- all three job pods reached Succeeded
- the pods are deleted; the decision reads Teardown past the 1m hold; `kubectl get nodes` is empty again with the control plane still Ready
reads: >
Autoscaling to more than one node, measured. What it does NOT show, stated because the numbers
read stronger than they are: it asserts the CLUSTER reached three nodes, not that three guests
each hosted work. The scheduler placed all three pods on liveweb-node1, which the design
permits, so the evidence for three simultaneous guests is three Provision calls plus
`kubectl get nodes` listing three, together with an independent observation of two firecracker
processes during the run. No negative control was run: that the test would fail with MAX_NODES
unset is argued from Target's ceiling rather than measured. Partial scale-in is not implemented
(plan 0002 section 4.4), so the target only grows while intent exists and the down direction
remains the all-or-nothing teardown past the hold.
gitlab_pipeline_proof:
test: TestAGitLabPipelineRunsOnATenantCluster
location: cmd/socialweb-clusterprovider/live_gitlab_pipeline_test.go
result: PASS 34.12 s, 2026-10-05, against a real GitLab CE 19.4.1 and a real gitlab-runner 19.4.1
requires_these_env: >
SOCIALWEB_LIVE_GITLAB_URL, SOCIALWEB_LIVE_GITLAB_TOKEN (a project access token),
SOCIALWEB_LIVE_GITLAB_PROJECT (numeric id), SOCIALWEB_LIVE_GITLAB_RUNNER_TOKEN (the project
runner authentication token) and SOCIALWEB_LIVE_GITLAB_RUNNER_BIN. Unset, the test SKIPS with
the list, which is not a pass.
transitions:
- a tenant cluster reached Ready with ZERO nodes at its own leased apiserver port
- a tenant credential was minted from that cluster's own ClientCA and written as ca.crt, tls.crt, tls.key
- config.toml with executor = "kubernetes" was written and the real gitlab-runner started against it
- a .gitlab-ci.yml was committed through the GitLab API and a pipeline triggered
- "the job pod's own Pending state drove the scale decision (`intent and no node`) and a node joined"
- GitLab's runner logged `Waiting for pod ... to be running on the node liveweb-node1`
- two job pods were observed carrying spec.nodeName = liveweb-node1
- the pipeline reported success and the job's trace carried the marker this test's own .gitlab-ci.yml writes
reads: >
This is the whole GitLab integration end to end, and it is the strongest reading in this file:
GitLab itself scheduled the job, GitLab's own runner created and drove the pod, and the pod ran
on a node this cluster provisioned from the job's own Pending state. Three findings are
carried in docs/ARCHITECTURE.md section 15 rather than here: the runner must be pointed at the
apiserver's LOOPBACK address with a client certificate (the kcp mount carries bearers only and
the exec path is not guaranteed to survive it); the live harness drives the scale decision only
when a test asks it to, so the first attempt left its pod Pending for ever; and a job Pod runs
INSIDE a guest, so CI_REPOSITORY_URL naming the GitLab host's loopback resolves to the guest's
own and the clone fails with exit code 128, which the proof avoids with GIT_STRATEGY: none.
not_run: >
The GitLab container and the tokens are throwaway and live on this host only. A runner OFF the
host still cannot reach a tenant apiserver (NEXT.yaml GL-2), and the minting CLI itself has
not been run against a real data dir (NEXT.yaml GL-1).
control_plane_in_container:
image: socialweb-control-plane:local
files: deploy/control-plane-container/{Dockerfile,build.sh,validate.sh}
result: >
TestTheControlPlaneReachesReadyAtZeroNodes PASSES INSIDE the container: 9.50 s, 7.78 s, 7.67 s
across runs against a ~6 s host baseline. Runs as USER 1000:1000. NO --privileged,
--security-opt, --cap-add, --device, --network host, and no userns remap.
honest_verdict_inside: "9 HOLDS, 1 MISSING, 10 MUST BE PASSED IN, exit non-zero (control-plane half only)"
mount_gate_question:
question: Does a Draining CR shut the mount BEFORE its process group dies?
answer: YES, measured against a real kcp started inside a test
test: TestTheMountShutsBeforeTheTenantProcessGroupDies
location: internal/clusterprovider/live_mount_test.go
skip_behaviour: no env gate; skips only if kcp/kine/k3s/postgres are absent, with a skip message saying a skip means the question stays open. Runs in ~27 s, needs no root and no /dev/kvm.
three_answers:
serving: "401 with the tenant apiserver's own k8s Status body (a REACH, proven by bodies)"
group_SIGTERMed: "502 empty body -- the reverse proxy with no backend"
CR_Draining: "503 `Not available.` -- kcp's own refusal, backend never contacted"
timeline: Draining moves the mount off 401 at +206 ms with the group still serving; reap comes on a LATER pass at +260 ms reading 503; group dies last.
host_contract:
binary: cmd/socialweb-computer-kcp-check-host
host_result: "19 checks, 19 HOLDS, 0 MISSING, exit 0 (verified by running it, not by report)"
discrimination: broken module source -> 2 MISSING exit 1; no config -> 5 MISSING exit 1
categories: HOLDS / MISSING / MUST BE PASSED IN
why_three: >
On a host the last two collapse to "this host cannot run this". Inside a container, MISSING
means the image is wrong while MUST BE PASSED IN means the container is correctly declaring
its edge. One binary, two readings.
how_to_run:
deployment: "deploy/run-container.sh ; deploy/wait-ready.sh socialweb 300"
gate: "deploy/check.sh --fast (the loop) ; deploy/check.sh --local (this host's proofs) ; deploy/check.sh --full (needs the live artifacts)"
live_gate: "SOCIALWEB_REQUIRE_LIVE=1 go test ./... (a live tenant cluster and a real guest; a skip is a failure)"
all_tests: "deploy/check.sh --fast"
control_plane_zero_node_proof:
host: "SOCIALWEB_LOCAL=1 go test ./internal/clusterprovider -run TestTheControlPlaneReachesReadyAtZeroNodes -v -count=1"
in_container: "deploy/control-plane-container/validate.sh"
mount_proof: "SOCIALWEB_LOCAL=1 go test ./internal/clusterprovider -run TestTheMountShutsBeforeTheTenantProcessGroupDies -v -count=1"
host_contract: "go run ./cmd/socialweb-computer-kcp-check-host -config /home/johnandersen777/kcp-image/nodeimage.json"
four_step_proof_env:
note: >
These variables gate the four-step proof, and a skip is not a pass. The rig derives them from
the image store and the builder's configuration; by hand, they need the BUILDER'S OWN
CONFIGURATION, not just a built image directory -- an image is only valid relative to the
configuration whose inputs name it.
vars:
- SOCIALWEB_CLUSTERPROVIDER_LIVE_IMAGE_OUT_DIR
- SOCIALWEB_CLUSTERPROVIDER_LIVE_IMAGE_CONFIG
- SOCIALWEB_CLUSTERPROVIDER_LIVE_FIRECRACKER
- SOCIALWEB_CLUSTERPROVIDER_LIVE_NODE_AGENT
- SOCIALWEB_CLUSTERPROVIDER_LIVE_K3S
- SOCIALWEB_CLUSTERPROVIDER_LIVE_KUBECTL
- SOCIALWEB_CLUSTERPROVIDER_LIVE_TENANT_IMAGE
optional: SOCIALWEB_CLUSTERPROVIDER_LIVE_SCALE_HOLD (proof default 1m; scale.DefaultHold is 2m)
live_environment:
image_store: /home/johnandersen777/kcp-image/out
store_contents: 3 images, 4.0 GiB total (older names kept as rollback)
newest_fingerprint: sha256-e3f1e594e7aefb8a7c51e370a6ebd3f5e34f6364ceaf4eb97d7ef0d33f4d24c3
builder_config: /home/johnandersen777/kcp-image/nodeimage.json
builder_config_notes: >
48 modules, 5 images pinned as <tag>@<sha256:...>, agent_go_flags carries -buildvcs=false,
firecracker and base_image repointed under artifacts/. This file is OUTSIDE the repo and is the
operator's, not the code's.
cross_check: >
With -buildvcs=false the host's own builder answers "up to date ... nothing built, nothing
pulled" for the directory the container's stage produced. Both builders compute one name from
one input set; that agreement is the proof the fix worked.
open_work:
operator_actions_not_repo_work:
- what: nodeimage.json gained -buildvcs=false; the next build produces a NEW fingerprint once.
where: docs/ARCHITECTURE.md:1991 and :1995
recorded_in_memory:
dir: /home/johnandersen777/.claude/projects/-home-johnandersen777-src-publicdomainrelay-kcp-socialweb-computer-kcp/memory/
items:
- file: cleanup-backlog-close-each-tick.md
line: 53
what: >-
STALE as of 2026-09-24: the salvage-list numbers ARE in the doc now
(docs/ARCHITECTURE.md:728, :731, :735, :736). Closed by DEV-6; do not re-report.
- file: cleanup-backlog-close-each-tick.md
line: 62
what: whether section 7 already carries the auth-paths conclusions; 1107 untracked experiment files were lost and never compared
section_14_stated_limits:
note: >-
UPDATE 2026-09-24 (DEV-8): section 14 no longer claims "no open gaps". It now lists the open
items, is the doc-side mirror of NEXT.yaml, and keeps the two stated limits below.
entries:
- line: 1348
what: >
The allocator's host-network overlap refusal is a property of the NETNS it runs in -- host
sees 15 networks, container 3 -- so a containerized allocator would lease a colliding range
without refusing. Bounded because the allocator and the node substrate are ONE PROCESS
behind SOCIALWEB_CLUSTERPROVIDER_SUBSTRATE=firecracker, whose validation already demands
/dev/kvm and a VMM. Conditional: costs nothing only because this host's LAN/bridges sit
outside 10.100.0.0/16, and a collision would land on the HOST. Seam exists:
subnetallocator.Config.HostNetworks is injectable.
- line: 1368
what: >
liveports.LockDir() is /var/tmp/socialweb-liveports inside each container's own filesystem,
so two proofs in two containers cannot see each other's slots. No production path depends
on it -- internal/liveports is imported only by tests and the host-contract probe.
not_recorded_anywhere_closed:
- what: dangling /run/k3s containerd shm mounts needing root
status: >
CLOSED and NOT written down. Verified: 0 k3s entries in /proc/self/mountinfo and /run/k3s
does not exist. It dated from the refuted local-process substrate. Do not add it to a list.
do_not_rederive:
- Cold boot to an init marker is 0.300 s. Boot is NOT the problem; the 10 s goal is met 30x.
- >
SNAPSHOT RESTORE IS REFUTED as the provisioning primitive. restore-to-Ready 9.974 s against cold
9.079 / 9.158 / 10.768 s, so restore buys nothing in time; and a restored guest cannot make a NEW
node because restore does not re-run boot-time units, so the identity adoption never runs and a
snapshot frozen as one node comes back as that node. Nodes boot COLD from their own rootfs clone.
- Kernel 6.18.48 PANICS on this AMD host at restore (Bad FPU state, XRSTOR/AMX). Validate an image by RESTORING it, never by comparing versions.
- The reported "332 s boot" was FABRICATED: a literal echo in run/initrd/init. Never trust a guest's self-report; the harness times on the host.
- pasta + a tap inside the pasta netns works; an unprivileged tap in the HOST netns fails. The apiserver stays on loopback; --tls-san carries the address the guest dials.
- k3s check-config reads the kernel CONFIG file and never checks whether a module is LOADED. It returns 0 over a missing module. This is why the module set is MEASURED, not checked.
- iptables-restore is all-or-nothing, which is why the module list is 48 and not 43.
- A named build context is mounted READ-ONLY, so a stage cannot write the store through one. /boot cannot even BE a context (/boot/efi is 0700).
- The agent binary carries Go's VCS stamp, so a commit changing no agent source still moves every image's name. -buildvcs=false fixes it; agentBinaryDigest still identifies the source by content.
- The advertise port and the k3s secure port are ONE lease answering both --apiserver flags, not two.
known_traps:
stale_gopls:
symptom: >
gopls repeatedly reports undefined symbols, unused imports and missing interface methods that
the compiler disagrees with. It happened 6+ times in one session, including a large dump
claiming a dozen files were missing a method that exists.
rule: The COMPILER is the authority. Run `go build ./...` before acting on any diagnostic.
root_cause_update: >
2026-09-24 (DEV-2): the tools were built with go1.23.7 and could not type-check this go1.26.4
module. gopls/staticcheck/govulncheck are now rebuilt with go1.26.4 and staticcheck has no
`(compile)` lines. The trap was a stale tool build, not gopls flakiness.
subagent_claims:
rule: >
Verify the artefact, not the summary -- including a subagent's. Read the deployed file rather
than the doc's description of it; run the binary rather than its report.
fake_registries:
rule: >
Every test that runs the reconciler substitutes a fake whose MountStatus answers from
Spec.Running. A fake that answers the question underneath the question is how a vacuous gate
survived a green suite for as long as it did.
status_update: >
2026-09-24 (REL-6): the three live-proof fakes now answer 401 live / 503 Draining / 502 reaped
from host liveness and DeletionTimestamp, not from Spec.Running.
validate_sh:
rule: >
Container names are fixed, so a killed run blocks the next one; each run now removes its own
name first. The host-contract step is gated on a POSITIVE assertion (the summary line must be
present), not an exit-code allowlist, because check-host exits 2 without printing one.
status_update: >
2026-09-24 (REL-3): the final `go test` is now also gated on a positive `--- PASS: $TEST` line,
because go test exits 0 on SKIP and on `[no tests to run]`.
caveman_builder_agent:
rule: >
caveman:cavecrew-builder has Read/Edit/Write/Grep/Glob ONLY -- no Bash. It cannot compile,
test, or commit. Use it only when correctness is established by reading; otherwise dispatch
general-purpose or run the verification yourself.
the_recurring_defect:
statement: >
The dominant failure in this repo is a thing that reads GREEN while being WRONG. Prefer
instruments that can fail. When adding a guard, watch it fail before trusting it.
instances:
- k3s check-config returning 0 over a missing module
- pods 1/1 over an incomplete firewall
- an advertise port nothing bound
- TestEveryVerbIsBound asserting not-404 (presence) while getKubeconfig was permanently dead
- check-host printing "holds 212" from free/budget when the store held 2
- an EMPTY /etc/rancher/node/password that opens read-write while k3s 400s for ever
- validate.sh's `|| true` making the host-contract step unable to fail
- the shut gate reading 401 as shut, so a SERVING cluster was reaped
- a list outliving its work -- three times in one day
- "2026-09-24: the teardown gate reaping on 300/429/500/504 (REL-2); Probe claiming Ready on a 503 (REL-1); the ClientCA pair check hashing openssl's empty stdin so junk compared equal (SEC-5); validate.sh exiting 0 on SKIP (REL-3); the live proofs' fakes answering 200 (REL-6)"
practice: >
Run the artefact. Watch the guard fail. Report the reading, not the intention. If a claim is
greener than the fact, the claim is the bug.
orientation:
layout: |
api/v1alpha1/ wire types only. No logic, no I/O
internal/${concept}/ one package per concern; the interface declared where consumed
cmd/${binary}/ the CLI layer; nothing else builds a binary
deploy/ install scripts, manifests, audit policy, the control-plane container
dependencies: "api <- internal <- cmd, one way. A new substrate or transport is a NEW PACKAGE, not a branch inside an existing one."
binaries: [socialweb-computer, socialweb-vmexec, socialweb-clusterprovider, socialweb-gateway, socialweb-vault, socialweb-computer-kcp-check-host]
key_packages:
internal/vmruntime: the Linux VM runtime a bidder boots each firecracker guest with (socialweb-computer internal vm); internal/nodefirecracker keeps only the old image store check-host reads
internal/nodemarket: the one node path, a NodeClaim the door fulfils under the owner's atproto session
internal/noderecord: reads the node records a retired local substrate left, for the startup sweep
internal/kernelmodules: the builder's module check, shared by the builder and check-host
internal/subnetallocator: pool 10.100.0.0/16 -> /24 per cluster
internal/liveports: flock port rendezvous for concurrent proofs
internal/materialiser: the reconciler; shut()/reached() gate a teardown
internal/kcpclient: kcp client; ServiceAccountName is the DID mapping's one spelling
internal/livegate: the SOCIALWEB_REQUIRE_LIVE / -short gate for live tests
start_here: docs/ARCHITECTURE.md sections 1 (objective), 3 (zero state), 9 (a node), 11 (teardown and the gate), 12 (code layout), 13 (invariants), 14 (the open list), 15 (settled decisions, with the readings)