# 0001 - GitLab on the virtual k3s clusters, with a cloud controller manager and node autoscaling

Status: done. Closed 2026-10-11; each phase's outcome and where it landed is in the last log entry.
Opened: 2026-10-05
Repo: socialweb-computer-kcp (the k3s product and the CCM). Sibling repos referenced:
publicdomainrelay (RFP / market / request-vm-ssh), gitlab-autoscale-socialweb-computer
(GitLab install prior art).

## 1. Goal, verbatim

> validate and document with copy pasteable examples in docs/gitlab/ how we can use our setup to
> spin a VM and configure it to run gitlab, then configure gitlab to run against one of our
> virtual scale to 0 kubernetes clusters (kubernetes executor). Make absolutely sure that gitlab
> CI works using our k3s setup. Also implement a cloud controller manager we install on the k3s
> clusters so that we know when to add more k3s nodes and make it so that autoscaling to add more
> nodes works.

Split into four deliverables:

- D-A `docs/gitlab/` - copy-pasteable: (1) spin a VM and install GitLab on it, (2) point a
  GitLab Runner using the **kubernetes executor** at a virtual tenant cluster.
- D-B A **proof** that GitLab CI works on our k3s setup (not a claim, a run).
- D-C A **cloud controller manager** installed per tenant cluster.
- D-D **Node autoscaling**: more than one k3s node when the work needs it, back to zero after.

## 2. What already exists, measured

Tenant cluster (this repo):

- `api/v1alpha1/types_node.go:7` - `Node` CR: `spec.cluster`, `status.phase`
  (`Pending|Running|Draining`), `status.nodeExpiry`, `conditions`, finalizer
  `node.socialweb.computer/teardown`. This CR is the guest-VM record. It is NOT the Kubernetes
  `Node` object; nothing links the two today.
- `internal/clusterstack/clusterstack.go:189` `ServerArgv` - `k3s server --disable-agent
  --disable=traefik --datastore-endpoint <kine> --egress-selector-mode cluster
  --https-listen-port <leased> ... --kube-cloud-controller-manager-arg=secure-port=<leased>`.
  k3s's own embedded CCM is not disabled, but no `--cloud-provider-name` is set.
- `cmd/socialweb-nodeagent/agent.go:105` - `k3s agent --server --node-name --token-file
  --data-dir --node-ip --flannel-iface`. No kubelet args.
- `internal/scale/scale.go:157` `Decide(o, st, now, opts) Decision`. Observation already carries
  `Pods, Jobs, Deployments, StatefulSets, DaemonSets`; `Intent()` is true for a pending/running
  pod, an unfinished Job, replicas>0, a DaemonSet wanting pods - all excluding objects stamped
  `AppliedByTheControlPlane`. The decision is: intent and 0 nodes -> `Provision, Nodes: 1`;
  intent and >0 -> `None` (ADR 0014); no intent -> hold `DefaultHold = 2m` -> `Teardown`.
  **`Nodes` is hardwired to 0 or 1.**
- `internal/scaledriver/scaledriver.go:48` drives `Observe` -> `Decide` -> actuator.
- `cmd/socialweb-clusterprovider/nodes.go:328` `tenantNodes.Provision` (confirmJoined polls the
  tenant Node list for `NodeJoinTimeout = 30s`), `:372` `Teardown`; guests come from
  `internal/nodefirecracker/node.go:48` or `internal/nodeqemu/node.go:46`.
- `internal/kubeconfig/kubeconfig.go:34` `Build` - the tenant kubeconfig carries an **exec
  credential** (`client.authentication.k8s.io/v1`, `interactiveMode: Never`, device credential in
  an env var). `AssertClean` (`:103`) refuses `bearer_token`, `token:`,
  `client-certificate-data`, `client-key-data`, `admin.kubeconfig` on any user-facing render.
- Gate: `deploy/check.sh --fast` (~9 s), `--local` (~75 s, names its skips), `--full` (fails on a
  skip). Four-step proof PASSES end to end.

GitLab prior art (`../gitlab-autoscale-socialweb-computer`):

- GitLab CE in docker-compose on one host, `setup/gitlab/{docker-compose.yml,setup.sh,
  gitlab-omniauth.rb,sshd-reconfig.sh,tailscale-serve.service}`; hostname
  `gitlab.dorper-matrix.ts.net`, UI on `:8443` over tailscale, sshd moved to 1997.
- Runner is a host user systemd unit (`systemd/gitlab-runner.service`), `executor = "instance"`,
  autoscaled by the Fleeting plugin `fleeting-plugin-socialwebcomputer`, which execs
  `request-vm-ssh` per instance. **Nothing in that repo touches Kubernetes.**

Vendored reference: `gitlab-kubernetes-executor.md` (2811 lines, GitLab's kubernetes executor
page). Facts taken from it: the runner creates one Pod per CI job through the Kubernetes API;
RBAC for `pods` (create/delete/get/list/watch), `pods/exec`, `pods/log`, `secrets`,
`serviceaccounts`, `services`, plus `apps/deployments` and `scheduling.k8s.io/priorityclasses`
(cluster-scoped, so a ClusterRole) when pre-warmed pause pods are used; pause pods are
Runner 18.10+ and exist to hide node scale-up latency, not to create nodes.

Host facts (measured on this machine):

- `kubectl`, `k3s`, `kine`, `docker`, `podman`, `deno`, `yq`, `jq` present. `gitlab-runner`
  MISSING. `crane` MISSING. `/dev/kvm` and `/dev/net/tun` present. 1.3 T free on /home.
- `/home/johnandersen777/kcp-image/` present (nodeimage.json, out/, kcpagent).
- `~/.kube/k3s.yaml` is a STALE symlink into a deleted `/tmp/TestTheMountShuts...` directory.
  Do not read it, do not trust it.
- Module cache has NO `k8s.io/cloud-provider` and NO `k8s.io/autoscaler`. `GOPROXY` is
  proxy.golang.org, so a new module is a network dependency.

## 3. The gaps

- G1 **Reachability.** What a GitLab Runner on a VM puts in `host =` is not established. The
  tenant apiserver binds a leased port on a private bind address and advertises a host-carried
  address; the tenant kubeconfig authenticates through an exec plugin. Whether any path exists
  from a VM to that apiserver, and what must be added, is the first thing to settle. See the
  reachability verdict in the log.
- G2 **No cloud controller manager.** Nothing stamps `spec.providerID`, sets addresses or
  labels on the tenant's Kubernetes `Node` objects, and nothing deletes a `Node` when its guest
  is gone. The `Node` CR and the `Node` object are unlinked.
- G3 **Scale is binary.** `Decision.Nodes` is 0 or 1 by construction, so two concurrent GitLab
  jobs cannot both be served. There is also no notion of capacity: "one pending pod" and "fifty
  pending pods" decide identically.
- G4 **No GitLab documentation at all.**

## 4. Decisions

Each decision names the alternative it rejected.

- **D1 - The CCM runs in the per-cluster process group, not as a pod.** Alternative: a
  Deployment in `kube-system`. Rejected because a Deployment needs a node, and the whole point of
  this cluster is that it has zero nodes; a component that cannot run at zero nodes cannot be the
  component that decides a node is needed. The CCM is a peer of kine, openbao and `k3s server`:
  always up. It is "installed on the cluster" in the only sense that survives the zero state -
  one process group per tenant cluster.
  **It authenticates with a MINTED tenant credential, never the k3s-written admin kubeconfig**
  (invariant 20: that file is shared state on a multi-cluster host and its server name is
  unverified against the address a node dials). The minting path already exists at
  `internal/clusterprovider/tenantcredential.go`.
  For the same reason the CCM must not read an unreadable instance source as "the guest is gone":
  invariant 14. `Instances.Get` returning an error means patch nothing and delete nothing.
- **D2 - Do NOT set kubelet `cloud-provider=external`.** Alternative: the textbook external-CCM
  handshake (kubelet taints the node, CCM initializes and removes the taint). Rejected for two
  reasons: the taint is applied so that no workload schedules before cloud-init has been
  confirmed, and our guests are fully initialized and
  `confirmJoined`-confirmed before they join, so the guarantee buys nothing; and the handshake
  makes every node unschedulable until a CCM that may itself need a node runs. We take only the
  half we need: the CCM stamps providerID and addresses on nodes that joined normally.
- **D3 - Hand-rolled node controller, not `k8s.io/cloud-provider`.** Alternative: pull in the
  upstream framework and implement `cloudprovider.Interface`. Rejected on design, not on
  availability - `proxy.golang.org` answers 200 from this host, so the module could be added. The
  framework requires a LoadBalancer/Zones/Clusters surface with no meaning here, its node
  controller is written for the in-cluster handshake D2 rejects, and this repo's whole idiom is a
  pure decision plus a thin actuator. The ceiling and the upgrade path are recorded as a
  `ponytail:` comment on the node controller, which is the repo's one permitted comment form.
- **D4 - The scale decision extends to N and stays pure.** Alternative: import the Kubernetes
  cluster-autoscaler and implement its external gRPC provider. Rejected for this iteration: it
  adds a large dependency and an out-of-process protocol to answer a question the repo already
  answers in a pure function. `scale.Decide` gains a capacity input and returns a target count;
  `scaledriver` reconciles actual against target. The external-gRPC shape stays available as a
  later adapter over the same decision - that is the upgrade path, not a fork of it.
- **D5 - The GitLab Runner runs outside the tenant cluster, on a host that can reach BOTH the
  kcp mount URL and the credential mint URL.** Alternative: the runner as an in-cluster
  Deployment (GitLab supports it). Rejected for the zero-node reason and because the tenant
  kubeconfig authenticates through an exec plugin that must be present where the runner runs.
  See D7 for what "can reach both" costs.
- **D6 - Reachability is stated as it is, not as it would be convenient for it to be.** Measured
  (see the log): a tenant cluster is reached at `https://<kcp-host>/clusters/<mountPath>`, kcp
  terminates TLS and reverse-proxies to `Cluster.status.URL` (`https://127.0.0.1:<k3s-secure>`),
  and the credential is an exec plugin that POSTs a device key to the gateway's
  `/v1/credential`. **Only a bearer survives the mount** (a client certificate does not), which
  is exactly why the credential is minted per request. kcp binds `127.0.0.1:6443` and the
  gateway `127.0.0.1:8080`; the only published port is `127.0.0.1:8443`, and its sole upstream is
  the gateway. **No reverse ingress into the host exists anywhere in the tree.**
  Therefore:
  - Story A, which works today and is what the docs will show: GitLab and the runner run **on
    the kcp host**, where both loopback endpoints are reachable. This is also what the existing
    GitLab prior art does (the GitLab host is the host that runs the requester).
  - Story B, GitLab on a separate VM: needs an authenticated front for the mount and the mint.
    That widens the trust boundary the repo deliberately keeps at loopback ("Nothing else is
    exposed"), so it is documented as an operator decision with its tradeoff spelled out, NOT
    silently implemented. Outward-facing exposure is not this plan's to grant.
- **D7 - `deno-kcp` and `kcp-libs` were read for reuse, and the applicable surface is empty.** The
  request said to reference them and reuse libraries as needed, so this is the answer rather than a
  silence. What was read: `kcp-libs`' 21 library packages and 11 `deno-kcp` internal packages, both
  their `README`/`AGENTS` and their exported API.
  - `kcp-libs` is kcp-shaped throughout. `impl/kcpstore` is a typed REST store over kcp's
    `/clusters/*` paths; `impl/informerwatch` is dynamic informers over `/clusters/*`;
    `common/kcp` builds kcp service labels and FQDNs; `common/denocomputer` is the
    `deno.computer` group vocabulary; `factory/admission` and `abc/queue` are a per-parent
    concurrency cap for kcp workloads. The cloud controller manager and the scale path work
    against a plain `k8s.io/client-go` client on a tenant apiserver and against this repo's own
    registry types, so none of those reach sites is the same shape.
  - `abc/reconcile`, `abc/store`, `abc/probe`, `common/ttl`, `common/condition` and
    `common/statuspatch` are the closest to generic, and they are small enough that adopting them
    would mean either a `deno.computer`-typed dependency for a `socialweb.computer` cluster, or a
    refactor of `kcp-libs` itself. Neither is this change's to make.
  - A grep for cloud-provider surfaces across both repos returns **zero hits in their own code**
    (every match is vendored OpenBao, and about Vault namespaces): no `cloudprovider.Interface`,
    no node controller, no provider ID, no node lifecycle. There is nothing to reuse for the part
    of this work that is new.
  - What WAS reused is the org-root idiom rather than the code: a pure decision plus a thin
    actuator, a port with one file-backed implementation for tests, and error strings that name
    the failure. That is the same shape as `kcp-libs`' `abc/` + `impl/` split, applied to a
    different subject.
- **D8 - The new documents live in this repo.** `docs/gitlab/` and `docs/plans/` under
  socialweb-computer-kcp, because this repo holds the k3s product and the CCM, and the GitLab
  material is an integration against them. `docs/ARCHITECTURE.md` stays the single design
  document; these two directories are the plan series and the operator recipes, and section 15
  records that as a decision.

## 5. Phases and acceptance

- **P1 - CCM.** `cmd/socialweb-cloud-controller-manager` plus `internal/cloudprovider/`
  (node-init: providerID `socialweb://<cluster>/<node>`, addresses and labels from the Node CR;
  node-lifecycle: a Node whose CR is gone or Draining is deleted; a scale-pressure reporter that
  counts Pending-but-unschedulable pods and node allocatable). Wired into the per-cluster process
  group as a new `procgroup` component. Acceptance: `deploy/check.sh --fast` green, table tests
  for every decision, one live test that a joined guest's Node object carries the providerID and
  disappears when the guest is reaped.
- **P2 - Autoscaling N>1.** `scale.Observation` gains capacity; `Decision.Nodes` becomes a
  target in `[0, MaxNodes]`; `internal/nodeprovider` names nodes per index; `scaledriver`
  provisions the difference and reaps the excess. Acceptance: `--fast` green, table tests for
  the new decision including the zero and the max cases, and the four-step proof still passing.
- **P3 - `docs/gitlab/`.** `README.md` (the story), `01-vm-gitlab.md`,
  `02-runner-kubernetes.md`, `03-ci-pipeline.md`, `04-ccm-and-autoscaling.md`, plus
  `examples/` holding the literal files (argocd-free: `config.toml`, `rbac.yaml`,
  `deployment-or-processgroup.yaml`, `.gitlab-ci.yml`). Every command block must be either
  executed or explicitly marked as not executed on this host, with the reason.
- **P4 - The proof.** `deploy/gitlab-ci-proof.sh`: bring a tenant cluster up through the
  composition root, apply a GitLab-shaped job pod (helper + build containers, the runner's labels,
  annotations, tolerations and resource requests), assert the node is provisioned, the CCM
  stamped it, the pod reaches Running, a second concurrent job gets a second node, and past the
  hold the nodes go back to zero with the control plane Ready throughout.

## 6. What is measured versus what is documented

The distinction is load-bearing (see docs/ARCHITECTURE.md section 15, "reads green while wrong").

- Measured on this host: everything under P1, P2, P4 that does not need GitLab CE itself;
  `config.toml` syntactic validity if `gitlab-runner` can be installed.
- Documented, not executed here, unless the network allows: pulling
  `gitlab/gitlab-ce` and running a real pipeline. The doc will say so at the top of the section
  rather than implying a run.

## 7. Risks and traps

- The stale `~/.kube/k3s.yaml` symlink (2). Ignore it.
- `test/nocomments_test.go` fails any byte above 0x7F anywhere it scans. Every new file ASCII.
- Changing `ServerArgv` or the agent argv touches the invariants in ARCHITECTURE.md section 13
  and at least two live tests. Read section 13 before editing either.
- Adding a `procgroup` component touches the teardown ordering that
  `internal/materialiser` and `ReapSurvivors` depend on. A component that is up when the group
  dies is a leak.
- The scale decision's idempotency note (ADR 0014) exists because a second `Provision` for a
  cluster that already has a node is a second node. Extending to N must not reintroduce it.

## 8. Log

- 2026-10-05 Opened. Discovery done against this repo, deno-kcp, kcp-libs, deno-kcp-specs,
  gitlab-autoscale-socialweb-computer, the vendored GitLab executor doc, the k3s advanced docs
  and the cluster-autoscaler external-gRPC provider docs. Decisions D1-D6 taken.
- 2026-10-05 G1 CLOSED - reachability verdict, measured, not reasoned:
  - `internal/kcpclient/names.go:27-33` MountPath = `root:<accountHash>:<cluster>:clusters`;
    `internal/kcpclient/kcpclient.go:397-408` renders the kubeconfig with `server` =
    `https://<kcp-host>/clusters/<mountPath>`. Shipped `<kcp-host>` = `https://127.0.0.1:6443`
    (`deploy/container-entrypoint.sh:26`).
  - `internal/kubeconfig/kubeconfig.go:34-93`: the tenant kubeconfig carries an exec credential
    only - no client certificate, no token, and for a device grant no CA either.
  - `cmd/socialweb-credential/main.go:70-98` POSTs the device key to `SOCIALWEB_MINT_URL`;
    `internal/xrpcserver/xrpcserver.go:31` MintPath = `/v1/credential`;
    `internal/tenantclient/tenantclient.go:164-196` issues the tenant SA token.
  - `docs/ARCHITECTURE.md:308-314`: only a bearer crosses the mount; a client certificate does
    not. `:302-306`: the minted kubeconfig's own server is loopback and works only from the host
    that wrote it.
  - `cmd/socialweb-gateway/config.go:145-156` `requireLoopbackAddr`: the gateway must bind
    loopback. `deploy/container-entrypoint.sh:285`: the only published port, 8443, terminates
    into the gateway and nothing else.
  - Verdict: **no path exists from an off-host VM to a tenant apiserver.** Nothing in
    `internal/`, `cmd/` or `deploy/` provides reverse ingress. D7 records the two stories.
- 2026-10-05 New constraint found before the CCM was written: invariant 20 forbids the provider
  reading the k3s admin kubeconfig, and invariant 14 forbids reading an unreadable instance
  source as an empty one. The CCM brief was corrected to mint its credential and to treat
  `Instances.Get` errors as "patch nothing, delete nothing".
- 2026-10-05 Baseline re-measured before anything was built on it:
  `TestTheFourStepProofRunsThroughTheCompositionRoot` PASSES in 103.10 s, all four transitions,
  against a node image already in the store. STEP 1 Ready at `https://127.0.0.1:62002` with zero
  nodes; STEP 2 empty and still empty across 10 passes 2 s apart over 20 s; STEP 3
  `INSTANCE_START_TO_READY_S=5.430` (host-timed, resolution 250 ms); STEP 4 node gone, control
  plane still Ready, still listing `default kube-node-lease kube-public kube-system users`. The
  leased port in the log confirms the port is not a constant, which is why the docs read it from
  `status.url` instead of printing a number. The reading is recorded in `handoff.yaml` with its
  date; the earlier 97.65 s entry is kept, because the two are different builds.
- 2026-10-05 Reaching the tenant apiserver from a runner: the direct loopback URL with a minted
  client certificate is the route that works, not the kcp mount. The mount is a reverse proxy and
  carries bearers only, so a CI runner's exec and attach paths are not guaranteed to traverse it.
  This is why `cmd/socialweb-tenantcredential` exists rather than a documented use of the mount.
- 2026-10-05 Opened `docs/plans/0002-scale-to-n-nodes.md` for the autoscaling half, because the
  decision between zero and one is a separate change from the CCM and carries its own invariants
  (sections 10 and 13) and its own limit, stated in that plan at section 4.4.
- 2026-10-05 P4 MEASURED. `TestAGitLabJobPodIsPlacedOnAProvisionedNode`
  (`cmd/socialweb-clusterprovider/live_gitlab_ci_test.go`) PASSES in 84.40 s on this host, a second
  run of the same test having reported 86.16 s against a different leased port (62002 vs 62302).
  It applies the Pod shape the kubernetes executor actually creates - one build container and one
  helper container, `RestartPolicy: Never`, the runner's own labels, no control-plane annotation -
  and asserts the scale decision reads `Provision` with `nodes=1`, the pod reaches `Succeeded`,
  `spec.nodeName` equals the node the decision provisioned, and that deleting the pod returns the
  cluster to zero with the control plane still `Ready`. What this proves is the CLUSTER half of
  GitLab CI; the GitLab instance half is documented in `docs/gitlab/` and not run here.
  A sibling of the four-step harness rather than a modification of it, so the existing proof is
  untouched: `fourstepBringUp` and its helpers are reused as-is, and the test consumes one more of
  the eight `liveports` FourStepProof slots, which is why the two live tests are sequenced and
  never run concurrently.
- 2026-10-05 P1a landed: `internal/cloudprovider/` (providerid, instances port, pure `Desired`,
  pure `Measure`/`Target`, the node controller, a file-backed instance source, the node writer) and
  `cmd/socialweb-cloud-controller-manager`, with tests. **Not installed**: nothing starts it.
  `docs/plans/0003-install-the-ccm-in-the-process-group.md` is that step and `NEXT.yaml` CCM-1
  stays open until it lands.
- 2026-10-05 GL-1's precondition closed: `MintTenantCredentialForClusterAs` mints an arbitrary
  CommonName and Organizations from the cluster's own ClientCA, because the vault's
  `tenant-superuser` PKI role pins `organization: system:masters` and can therefore issue only the
  superuser. A runner with cluster-admin would have made every RBAC rule in
  `docs/gitlab/02-runner-kubernetes.md` decorative, so the docs' recipe depends on this.
- 2026-10-11 Closed. Every phase landed, two of them in a different shape than planned:
  - P1 CCM: built here (P1a), installed by plan 0003 into the per-cluster process group; the live
    STEP 3 reading is `spec.providerID = socialweb://liveweb/liveweb-node1` (ARCHITECTURE section
    15). Three tenant CCMs ran on prod on 2026-10-11.
  - P2 autoscaling N>1: plan 0002 (Demand, Target, Decision.Nodes, per-node scale-in); plan 0004
    section 6a scaled five concurrent jobs to five nodes and back to zero.
  - P3 docs/gitlab/: shipped as README.md, 01-kcp-server.md, 02-gitlab-runner.md, 03-jobs.md and
    04-spa-cluster-runner.md rather than the file names listed in section 5; the VM-and-GitLab
    install is the existing GitLab on hat-10, not a recipe this repo runs.
  - P4 the proof: `deploy/gitlab-ci-proof.sh` was never written. A real GitLab pipeline replaced the
    GitLab-shaped pod: plan 0004 job 183 (streamplace `test`, 1104 tests, 0 failures) on a local
    substrate, then plan 0005 pipelines 78 and 79 on market nodes of an SPA-created prod cluster
    (0 nodes idle, scale up, per-node scale-in back to 0, control plane Ready, no ssh to guests).