0002 - The scale decision to N nodes

Status: done. Live reading taken 2026-10-05; NEXT.yaml AUTO-1 closed to docs/ARCHITECTURE.md section 15 with its before-and-after readings. Opened: 2026-10-05 Depends on: 0001 (the cloud controller manager). Continues NEXT.yaml AUTO-1. Repo: socialweb-computer-kcp.

1. Goal

Make internal/scale able to demand more than one node, and make the driver reconcile the cluster's actual node count to that demand, without breaking the zero direction.

The request this serves: two concurrent GitLab CI jobs must both be placed, and the cluster must still return to zero when they finish.

2. What is true today, measured

internal/scale/scale.go:157 Decide(o Observation, st State, now time.Time, opts Options) Decision.

• Observation (:34) carries Pods, Jobs, Deployments, StatefulSets, DaemonSets.
• Intent(o) (:42) is a boolean: any non-control-plane-owned Pending/Running/Unknown pod, unfinished Job, replicas>0 Deployment/StatefulSet, or a DaemonSet wanting pods. AppliedByTheControlPlane (:22) excludes k3s's own addons, which is what lets an empty cluster reach zero.
• Decide (:157): intent and st.Nodes == 0 -> Provision, Nodes: 1; intent and st.Nodes > 0 -> None with the ADR 0014 reason; no intent and st.Nodes == 0 -> None; no intent and st.Nodes > 0 -> hold DefaultHold = 2m -> None, past it -> Teardown, Nodes: 0.
• State.Nodes is re-read from the Node CRs each pass and deliberately not remembered (section 10); State.IdleSince is the only remembered value, because without it the hold never arrives.

The consequence: Decision.Nodes is 0 or 1 by construction. One pending pod and fifty pending pods decide identically.

3. What must not break

• Invariant 14: a list failure is an error, never an empty observation. The new demand computation reads the same one observation; a failure still aborts the pass.
• Invariant 15: the state advances only on success.
• Section 10's coarse intent test: "a predicate that can only fire upward cannot express" zero. The boolean stays a boolean. The count is a SEPARATE function, so a bug in the count cannot make zero unreachable.
• Section 10's one-call rule: both halves of the observation come from one call, so a pod counted before its node's CR landed cannot provision twice.
• ADR 0014: a second Provision for a cluster that already has its node is a second node. This is resolved by reconciling a TARGET against a re-read count, not by refusing to provision whenever a node exists.
• deploy/check.sh --fast and the four-step proof.

4. Design

4.1 A separate demand function, not a boolean made countable

func Demand(o Observation) int

Demand is a pod-work COUNT, not a node count: the ratio is applied only in Target (4.2). It counts the work that needs a node: non-terminal, non-control-plane-owned pods that are Pending or Running (one each), plus the pods an unfinished non-suspended Job still needs, plus the replicas of Deployments and StatefulSets, plus the DaemonSet desired count. It never returns a negative, and Demand == 0 must agree with Intent(o) == false for every input the table tests cover; a case where they disagree is a bug in one of them and gets its own table entry.

A pod left Unknown by a departed node counts here, because it is Intent that says so and the two must agree; the "Pending or Running" reading is too narrow to cover it.

Intent is then re-expressed as Demand(o) > 0, and a table test asserts the two agree on every existing Intent fixture. That keeps one definition of "is there work" instead of two that drift.

The sum double-counts a workload whose pods are also in the observation, and that is a decision. A Deployment of N replicas contributes N as replicas AND its N pods contribute N more if they are Pending or Running, so a Deployment workload reports roughly twice its pod count. A bare-pod workload is exact: N pods report N. The over-count is bounded by MaxNodes, so it over-provisions within the cap rather than unbounded, and a later pass corrects as pods terminate. The live acceptance uses bare pods for that reason, and the readings it produces are exact. If the double-count is judged too costly, the fix is to subtract pods already owned by a counted Deployment or StatefulSet, which needs OwnerReferences the observation does not carry today.

4.2 The target

type Options struct {
    Hold        time.Duration
    MaxNodes    int  // ceiling; 0 means the default
    PodsPerNode int  // 0 means the default
}

func Target(o Observation, opts Options) int   // 0 when there is no intent

ceil(Demand / PodsPerNode), clamped to [1, MaxNodes] when there is intent, and 0 when there is not. The clamp's lower bound of 1 when intent exists preserves today's behaviour exactly at MaxNodes == 1, which is how the existing tests keep passing unchanged.

Defaults live in scale as named constants (DefaultMaxNodes, DefaultPodsPerNode) and are resolved by Options accessors in the existing o.hold() style, so a zero value is never a surprise: it is the default.

4.3 The decision

Decision keeps Action and Nodes, and Nodes becomes the TARGET TOTAL rather than a delta:

• intent, st.Nodes < target -> Provision, Nodes: target
• intent, st.Nodes > target -> not reachable in this iteration: see 4.4, where the target while intent exists only grows. The branch is written to None, Nodes: st.Nodes with a reason naming 4.4, so a future partial scale-in has one place to change rather than a silent gap. (Since 2026-10-07 this branch is the per-node scale-in of plan 0005 M4-SCALE-IN: Release of one idle node, or None while no node has been idle for the hold.)
• intent, st.Nodes == target -> None, Nodes: st.Nodes
• no intent -> unchanged: hold, then Teardown, Nodes: 0

4.4 The one genuinely new action

Teardown today means "take the cluster to zero". Scaling in from three nodes to two is a different act with different consequences (which node, is it draining, does its work move), and conflating them is how a scale-in becomes a lost job. So the first iteration does NOT scale in partially: while intent exists the target only grows to the cap, and the down direction remains the all-or-nothing Teardown past the hold, exactly as today. Partial scale-in is a follow-on item with its own plan, and it is only worth doing once something measures which node to drain.

This is a deliberate limit, stated rather than left implicit: the cluster scales out to MaxNodes and in to zero, and never to an intermediate count.

SUPERSEDED 2026-10-07 by plan 0005 item M4-SCALE-IN (docs/plans/0005-zero-ssh-market-nodes.yaml): scale-in is now per node. With intent and more nodes counted than the target, a node this provider holds that has held nothing beyond the baseline (DaemonSet, mirror and baseline pods aside) for the whole hold is cordoned, drained and released, one node per hold, emptiest then newest, never below the target; with no intent the all-or-nothing teardown above is unchanged. The "which node, and does its work move" question this section left open is answered by scale.Loads (which node holds what) and by the drain (evicted baseline pods move; a node running tenant work is never chosen). ARCHITECTURE section 10 carries the rule.

4.5 The driver and the actuator

internal/scaledriver compares the re-read node count against Decision.Nodes and calls Provision once per missing node (for counted := observedNodes; counted < dec.Nodes; counted++). The actuator interface is unchanged - scale.Nodes still has Provision(ctx) error - because the driver, not the actuator, owns the reconcile, so no host package changes. This corrects an earlier claim in this section that tenantNodes.Provision already loops per node; it did not and does not: cmd/socialweb-clusterprovider/nodes.go:328 provisions exactly one node per call, which is why the loop lives in the driver.

The node id is a fixed value today: cmd/socialweb-clusterprovider/main.go declares const nodeID = "node1", and newNodeHost passed it to nodeprovider.NodeName(cluster, nodeID) for the cluster's one node. It becomes distinct per node: newNodeHost takes the id as a parameter, the first node is still node1, and each further node is node2, node3, ... The next id is derived by reading the tenant cluster's own Node list and taking the lowest ordinal not present, so it is re-derived from the cluster's state on every provision rather than remembered from a variable (per section 10's rule), and a repeated provision after a slow join reuses the same node instead of booting a second one under a new name. tenantNodes holds every node it has started - the pre-built first node plus a map of the later ones - and its Teardown stops all of them and deletes all of their tenant Node objects, so the cluster still reaches zero nodes. The actuator is still memoised once per cluster, for the lifetime of the process.

4.6 The cloud controller manager's part

internal/cloudprovider/pressure.go (plan 0001) already computes Pressure and Target from the tenant cluster's own pods and nodes. Where the two disagree, the observation wins: scale.Decide is the single decider, and the CCM's numbers are the instrument that shows the decision was right.

5. Acceptance

• deploy/check.sh --fast green.
• Table tests: Demand and Intent agree on every existing Intent fixture; Target returns 0 at zero demand, 1 at one pod with PodsPerNode == 8, 3 at 17 pods, and MaxNodes at 100 pods with MaxNodes == 4; Decide at MaxNodes == 1 reproduces every existing case byte for byte.
• The four-step proof still passes unchanged, because MaxNodes defaults to 1.
• A new live case, in the same harness, where MaxNodes is 3 and three concurrent pods are applied: three nodes appear, all three pods reach Running, and past the hold all three nodes are reaped and kubectl get nodes is empty with the control plane still Ready.

6. Risks

• The demand count reading pods that are being garbage-collected inflates the target for one pass. The clamp bounds the damage at MaxNodes, and the next pass corrects it; the test for a transiently-inflated observation is worth having.
• MaxNodes interacts with port leases: each node takes addresses from the cluster's /24 (subnetallocator). The default ceiling must be small enough that the default /24 holds it, and the code should refuse a MaxNodes the cluster's range cannot address rather than failing at the third node's boot.
• Raising MaxNodes changes the cost profile, not just the behaviour: each node is a booted guest. It defaults to 1 for that reason, and the operator raises it deliberately.

7. Log

• 2026-10-05 Opened from plan 0001's discovery. NEXT.yaml AUTO-1 is this plan. Design settled at section 4; the partial-scale-in limit is stated in 4.4 rather than left latent.
• 2026-10-05 Implemented, then corrected twice against the code rather than against this document. Section 4.5's claim that tenantNodes.Provision already looped per node was WRONG - it provisioned exactly one - so the driver owns the reconcile and calls Provision once per missing node; section 4.5 is corrected to say so. Section 4.1 asked for Demand(o, podsPerNode) while 4.2 applied the ratio in Target; the unused parameter was removed and the missing sentence ("Demand is a pod-work COUNT, not a node count: the ratio is applied only in Target") added, because a parameter that is never read is a second way of saying something false. The Deployment double-count (its pods plus its replicas) is now recorded in 4.1 with its bound rather than living only in a report.
• 2026-10-05 MEASURED. TestThreeGitLabJobPodsScaleToThreeNodes PASSES in 95.65 s at MaxNodes=3, PodsPerNode=1: the decision reads Provision, Nodes: 3 with reason intent and no node, kubectl get nodes reports three, all three GitLab Runner job pods reach Succeeded, and past the 1m hold the nodes are reaped with the control plane still Ready at https://127.0.0.1:62102. The two prior live proofs were re-run after the harness change and still pass - GitLab job pod 84.93 s, four-step 100.53 s. The one honest weakness is recorded in section 15 too: the scheduler placed all three pods on liveweb-node1, so the reading proves the CLUSTER reached three nodes, not that three guests each hosted work.
• 2026-10-05 A harness defect was found while wiring the knobs and fixed: fourstepBringUp built scale.Options{Hold: cfg.ScaleHold} and dropped the new ceiling fields, so an env passthrough alone would have been inert and the three-node test would have failed for a harness reason rather than for the defect it exists to catch. The live harness now forwards them, which is what production main.go already did.
• 2026-10-07 Section 4.4 superseded by plan 0005 M4-SCALE-IN: per-node scale-in, one idle node per hold, never below the target; the all-or-nothing teardown remains the no-intent path.