johnandersen777.bsky.social/socialweb-computer-kcp

Switch repository

Loading…

# 0003 - Install the cloud controller manager into the per-cluster process group

Status: done. Landed 2026-10-05; `NEXT.yaml` CCM-1 closed to `docs/ARCHITECTURE.md` section 15 with
its readings. `NEXT.yaml` CCM-2 was split out of it.
Opened: 2026-10-05
Depends on: 0001 (the CCM package and binary, built and unit-tested). Closes `NEXT.yaml` CCM-1
together with the live assertion in its instrument.
Repo: socialweb-computer-kcp.

## 1. Goal

Make `cmd/socialweb-cloud-controller-manager` a member of a tenant cluster's own process group, so
it is up whenever the control plane is up - including at zero nodes - and is reaped with the group.

Today the binary exists and its decisions are unit-tested, and nothing starts it. A cloud
controller manager that nothing runs is a design, not an installation.

## 2. What exists

- `internal/procgroup/procgroup.go`: `Component` (`ComponentPostgres`, `ComponentOpenBao`,
  `ComponentKine`, `ComponentK3s`), `Step{Name, Spec, Ready}`, `Spawn`, `Reap`. `Spawn` starts in
  order and requires every step to have a `Ready` gate - a step with no gate is an error, not a
  skipped check. `Reap` walks the order in reverse.
- `internal/clusterstack/clusterstack.go`: `Spec` and one argv builder per component
  (`ServerArgv`, `KineArgv`, `OpenBaoArgv`), plus `Spec.Argv(component)` as the dispatcher.
- `internal/clusterprovider/steps.go`: `StepsFor`-adjacent step assembly. `steps.go:114` shows the
  idiom: `procgroup.Step{Name: ..., Spec: processrunner.Spec{Argv: argv, Output: log}, Ready: gate(...)}`,
  with waiters `waitDial`, `waitK3s`, `waitK3sControllers`, `waitOpenBaoTLS`.
- `internal/clusterprovider/stack.go:hasStepBeforeK3s` and `ErrNoGuardSeam`: the identity guard
  rides a step before k3s, and a stack that puts nothing before k3s makes the guard silently not
  run. **A new step placed BEFORE k3s changes what that guard means. This step goes AFTER k3s.**
- `processrunner.Spec{Argv, Dir, Env, Prefix, Output}` - the CCM's environment rides `Env`, never
  argv (invariant 3 and 13: no secret on argv).
- The binary's own contract, from plan 0001: `SOCIALWEB_CLOUDCONTROLLERMANAGER_TENANT_KUBECONFIG`,
  `CLUSTER`, `INSTANCES_FILE`, optional `MAX_NODES`, `PODS_PER_NODE`, `RESYNC`; it prints a
  readiness line; it refuses a k3s admin kubeconfig by name.

## 3. Decisions

- **D1 - A new `procgroup.ComponentCloudControllerManager`, ordered after `ComponentK3s`.** After,
  because the CCM is useless without an apiserver and because putting it before k3s would move the
  guard seam `ErrNoGuardSeam` exists to protect.
- **D2 - No port lease.** The CCM is an out-of-cluster controller: it dials the apiserver and
  listens on nothing. `portallocator` is untouched, and invariant 8's bookkeeping about who holds a
  lease does not grow.
- **D3 - The credential is minted into the cluster's own data dir, by the provider, at startup.**
  The CCM cannot read the k3s admin kubeconfig (invariant 20). The provider already mints tenant
  credentials (`internal/clusterprovider/tenantcredential.go`) and already knows the data dir and
  the apiserver URL, so it writes `<DataDir>/cloud-controller-manager.kubeconfig` before the step
  runs - the same shape as `WriteOpenBaoConfig`, which is already the precedent for "put a file on
  disk before a step that needs it, and refuse loudly if it cannot be written".
  The identity is a scoped one, not the superuser: the CCM needs to read and patch Nodes and
  nothing else, and its own credential is what should bound it when RBAC is added.
- **D4 - The instance source is a file the provider writes, at `<DataDir>/nodes.json`.** The CCM's
  `Instances` port is already file-backed. The provider is the thing that knows the guests, so it
  writes the snapshot; the CCM re-reads it per call and never caches.
  CORRECTED WHILE IMPLEMENTING, because the original wording named a mechanism that does not exist.
  It said a missing file is the delete path and therefore the provider must write `[]` rather than
  remove the file. The first half is true and the second half protects nothing:
  `internal/cloudprovider/fileinstances.go` answers `(known=false, err=nil)` for an ABSENT file and
  for an EMPTY array alike, so the two are the same answer - the delete path - and `[]` is not safer
  than a deleted file. What protects a live Node is that the snapshot LISTS it. The real asymmetry,
  and the one worth carrying: a snapshot that exists and cannot be PARSED is an error and patches
  nothing (invariant 14), while an absent one is not - so the file is load-bearing infrastructure
  rather than a cache, and the ordinary path is what keeps the absent case out of reach, since the
  provider writes the file before the controller is asked anything. The dangerous reading is kept in
  `NEXT.yaml`'s limits section rather than deleted along with the wrong mitigation.

## 4. Acceptance

- `deploy/check.sh --fast` green.
- The four-step proof still passes, with the process group now carrying a fifth component: the
  bring-up order, the teardown order and the timings are unchanged in kind.
- A new live assertion, in the four-step's own harness: after the node joins, `kubectl get node
  <name> -o jsonpath='{.spec.providerID}'` reads `socialweb://<cluster>/<node-name>`, and the Node
  carries `node.socialweb.computer/cluster`. Without this the CCM is installed and doing nothing,
  which is the failure this repo calls "reads green while wrong".
- A unit test that a process group containing the CCM is reaped in reverse order, with the CCM
  stopped before `k3s server` - a CCM that outlives the apiserver it reconciles against is a
  controller looping on connection errors during every teardown.
- The provider writes `nodes.json` as an empty array, never absent, and a test asserts the
  difference between the two as the CCM sees it.

## 5. Risks

- **Teardown ordering.** `Reap` reverses the spawn order, so a component appended last is stopped
  first. That is the behaviour wanted here, and it is worth an explicit test rather than relying on
  it being true by construction.
- **`ReapSurvivors`.** `internal/materialiser` reaps survivors after the group; a fifth component is
  a fifth thing it must find. The existing live teardown assertions cover this if the component is
  in the group's order, and not if it is started outside it.
- **A CCM that cannot reach its instance file must not delete Nodes.** Invariant 14, and the reason
  D4 states the empty-array rule: absent means gone, so an unwritten snapshot is not a benign
  default. Making the write atomic (temp file plus rename, the `WriteOpenBaoConfig` shape) is part
  of this, not a nicety.
- **The node name.** The CCM derives the provider ID from the Node object's name, which the
  composition root already builds with `nodeprovider.NodeName(cluster, nodeID)`. If the two
  derivations ever disagree the provider ID names a node that does not exist, and the failure looks
  like an autoscaler that cannot map its own Nodes.

## 6. Log

- 2026-10-05 Opened. Not started: it overlaps `cmd/socialweb-clusterprovider/{main,config}.go` with
  plan 0002, and the two are sequenced rather than run together so that neither change can be
  confused for the other's regression.