# M0-KCP-SPIKE results (plan 0006), 2026-10-11

Branch `kcp-spike` (from main 8aa3ef1), host hat-8 (16 cores, 46 GiB, Fedora, go 1.26.6). Stand-in
processes only (kcp, kine, postgres, socat on 127.0.0.1); no guest, no container. Another agent's kcp
blue/green proof (kcps on 16449/16450, kine on postgres :25432) was running on the same host during every
reading, and the route A build overlapped the slot run (B2), so absolute timings carry that noise.

The user decided the route on 2026-10-12, while the spike was running: kcp runs **out of process** as
alternating blue/green slot units on kine -> a dedicated postgres, one kine per slot,
`--enable-leader-election`, `--shard-base-url` at our own L4 front on 127.0.0.1:6443, and the lease
handover is a delete of `kube-system/kcp-controllers` after the old slot exits. Route A was then capped
at the readings below and kept as a record. Route B was steered to three questions: port read-back,
shared root material, and the release binary as an embedded asset.

## Reproduce

| script | what |
|---|---|
| `scripts/kcp-spike/vendor-kcp.sh` | clones kcp v0.33.0 (commit 1a4cb725, checked) with `staging/` into `third_party/kcp` and copies embeddedetcd 2cc5c5cce35e into `third_party/embeddedetcd`. Both are gitignored on this branch; run this script first. `--print-replaces [prefix]` prints the replace block (kcp's 33 `k8s.io/*` lines verbatim + kcp, the five kcp-dev staging modules, embeddedetcd) |
| `scripts/kcp-spike/route-a-build.sh <out>` | go build / vet / GOOS=darwin,windows build / go test -json / binary sizes on the fork |
| `scripts/kcp-spike/fork-siblings.sh <scratch> <org-root>` | deno-kcp and kcp-libs built and tested on the fork through a scratch go.work (those repos untouched) |
| `scripts/kcp-spike/route-a-inproc.sh <bin> <work>` | runs `cmd/socialweb-kcp-spike` (kcp in-process, the cmd/kcp/kcp.go:95-162 sequence) |
| `scripts/kcp-spike/route-b.sh <work> [runs]` | release kcp + kine (sqlite, postgres on a unix socket), port 0, cold start / restart / RSS |
| `scripts/kcp-spike/route-b-slots.sh <work>` | two slots on one postgres, one kine each, leader election, shard base URL at a socat front: port read-back, shared material, lease handover |

## Route A: in-process on kcp's fork (record only)

The go.mod on this branch carries the full replace set; `go mod tidy` resolved it with no manual edits
(go.mod +148 lines, go.sum +269/-32). `github.com/kcp-dev/sdk` moves from v0.33.1 to kcp's staging sdk.

| reading | main 8aa3ef1 | fork |
|---|---|---|
| `go build ./...` (linux) | ok | **ok, 0 compile errors in 0 packages** (37 s cold) |
| `go vet ./...` | ok | ok |
| `GOOS=darwin go build ./...` | fails: processrunner, portallocator, subnetallocator (undefined hostos.*) | identical (pre-existing, M0-XOS-GATE) |
| `GOOS=windows go build ./...` | fails: nodevm | identical |
| `go test ./... -count=1` (offline) | 60 pkgs: 54 ok, 6 no-test/skip; 1438 pass, 0 fail, 64 skip | 61 pkgs (+ the spike cmd): 1438 pass, **0 fork-caused failures**. The one failure seen, `test.TestNoCommentsAndASCII`, was the spike file's own header comment, removed. One skip-vs-pass difference (`marketnetwork.TestTheRecordMatchesTheDenoKcpCommitItNames`) is the baseline worktree's location, not the fork |
| deno-kcp 25d10f92 | - | build ok, 10/10 test pkgs ok |
| kcp-libs f6911d2 | - | build ok, 37/37 test pkgs ok |
| socialweb-gateway binary | 26,823,317 B | 26,841,109 B (+17.8 KB) |
| socialweb-clusterprovider binary | 40,753,165 B | 40,787,767 B (+34.6 KB) |
| in-process kcp (`cmd/socialweb-kcp-spike`) | - | 177,440,498 B (release kcp: 176,825,356 B) |

In-process run (kine sqlite, port 0 via `--experimental-bind-free-port`, `SecureServing.Listener.Addr()`
read directly, 0.23-0.34 s after start):

| reading | value |
|---|---|
| cold start to /readyz | 8.9 s; RSS 415 MB at ready, 417 MB settled; 3,639 goroutines |
| ctx cancel -> `Run` returns | 8 ms (first), 1.1 s (later stops); secure port freed; goroutines 3,632 -> 181 left behind |
| second start in the same process, same root | ready 7.0 s; RSS 608 MB (first heap not returned) |
| two kcps in one process (two kine stores) | both ready (9.0 s, 9.3 s), distinct ports, RSS 656 MB, both stop in ~1.1 s, 359 goroutines left. Only error: `logging configuration was already applied earlier, changing it is not allowed` (klog/logs registry is process-global; the second kcp silently inherits the first's logging config). Feature-gate and Prometheus registries are shared without a panic; metric cross-talk was not checked |

Not measured (capped): client-go 1.37.0-alpha apply paths against a live k3s 1.36 tenant (the offline
suites of tenantclient, eso and clusterprovider pass); RSS on a prod-shaped root with gateway and
clusterprovider running in the same process; two embedded-etcd kcps (2379/2380 collide by construction).

Gate as written in M0-KCP-SPIKE: fork breakage fixed within a day - yes, there was none; go test green on
the fork - yes (offline suite); two in-process kcps - start, serve and stop. **The in-process route is not
blocked by breakage.** It is blocked by the 2026-10-12 decision: in-process ties kcp's lifetime to the core
process, so kcp blue/green would mean two whole core processes overlapping.

## Route B: release kcp v0.33.0 in its own process

Binary `/usr/local/bin/kcp` = `v1.36.0+kcp-v0.33.0`, sha256 0929a16bedf3275c..., same as
`/var/tmp/socialweb-control-plane-binaries/kcp`; static ELF; kine `/usr/local/bin/kine` (v0.17.1 build).
Flags: `--etcd-servers=<kine> --bind-address=127.0.0.1 --secure-port=0 --experimental-bind-free-port
--feature-gates=WorkspaceMounts=true` + today's audit flags. 3 runs per store, fresh root then restart
on the same root; RSS 30 s after ready.

| store / phase | port known (admin.kubeconfig / log line) | /readyz (min/med/max) | kcp RSS | kine RSS | SIGTERM -> exit |
|---|---|---|---|---|---|
| sqlite fresh | 0.38-1.78 s | 8.5 / 9.2 / 12.2 s | 382-396 MB | 57-59 MB | 1.1 s |
| sqlite restart | 0.27-0.37 s | 5.9 / 6.0 / 6.1 s | 378-404 MB | 62 MB | 0.05-1.2 s |
| postgres 18.6 fresh | 0.54-0.88 s | 8.7 / 8.8 / 8.9 s | 389-399 MB | 54-57 MB | 1.1 s |
| postgres restart | 0.27-0.40 s | 6.1 / 6.2 / 6.7 s | 392-409 MB | 60-61 MB | 1.1 s |

Each kcp holds exactly one TCP listener (the secure port) when the store is external. kine binds
`--listen-address 127.0.0.1:0` and logs `Kine available at http://127.0.0.1:<port>`. Postgres RSS was not
recorded (multi-process). kcp on postgres is within noise of sqlite.

### 1. How a port-0 slot reports its bound port (no kcp patch needed)

- Without `--shard-base-url` (today's prod flags), `--experimental-bind-free-port` is reported twice:
  `<root>/admin.kubeconfig` names `https://127.0.0.1:<bound>` (written 0.3-1.8 s after exec, long before
  /readyz), and the log line `secure_serving.go Serving securely on 127.0.0.1:<bound>`. The shard object's
  baseURL / virtualWorkspaceURL follow the bound port and are rewritten on every restart. Caveats: a stale
  admin.kubeconfig from the previous start sits there until rewritten, and `clientcmd.WriteToFile` is not
  an atomic rename.
- With the decided `--shard-base-url=https://127.0.0.1:<front>` (B2): admin.kubeconfig and the shard
  URLs name **the front**, never the bound port. What still reports the slot's own port: the log line
  (0.9-2.3 s) and the socket itself: the slot's MainPID has exactly one LISTEN socket, visible at
  0.17-0.30 s (`ss -ltnp` / `/proc/<pid>/net/tcp` matched against `/proc/<pid>/fd` socket inodes, same uid,
  no privilege). The listener is created in `Options.Complete` (options.go:355-361) before anything
  serves, so it is the right socket; readiness is then `/readyz` == 200 on that port.
- Cheapest: `--secure-port=0 --experimental-bind-free-port` + the core reading the unit MainPID's single
  LISTEN socket from /proc. Zero kcp patches; no port file. Fixed alternating slot ports A/B also need no
  patch and no read-back, but every concurrent kcp on a host (tests, `run request`, bidder-local kcp, a
  second core) then needs its own leased pair, which is the port-allocation problem port 0 removes.
  Keep fixed A/B only as the fallback if /proc read-back is rejected.
- Patches still needed on the decided shape: none. The embeddedetcd port-0 patch disappears with the
  external store; the port-file patch is replaced by the /proc read-back; ReleaseOnCancel is replaced by
  the lease delete (below).

### 2. Shared root material per slot (B2, measured)

Slot a fresh; slot b got copies of a's files; slot c got all but admin.kubeconfig; slot d a fresh root.
All on one postgres database, one kine each.

| probe | result |
|---|---|
| b (copied): a's shard-admin token, `/clusters/root/api/v1/namespaces` | 200 |
| b (copied): SA token minted by a | 403 (authenticated; same 403 as on a, no RBAC grant) |
| b (copied): a's kcp-admin token (`root`/`base` contexts) | 401: volatile, regenerated per process (authentication.go:109) |
| b rewrote admin.kubeconfig | shard-admin token unchanged, server = front, new kcp-admin token |
| c (crt, key, sa.key, .admin-token-store, no admin.kubeconfig) | **refuses to start**: `cannot create the 'admin.kubeconfig' file with an empty token for the shard-admin user` |
| d (fresh root): a's shard-admin token / a's SA token | 401 / 401 |
| d with a's CA | TLS failure (new self-signed apiserver.crt) |

A new slot needs `apiserver.crt apiserver.key sa.key .admin-token-store admin.kubeconfig` copied in
before start; admin.kubeconfig is mandatory whenever .admin-token-store is present (kcp reads the
shard-admin token back from it). The core keeps them in `<state>/kcp/shared/`, copies into the new slot
root before starting the unit, and consumers use only the `system:admin`/`shard-base` contexts
(shard-admin, persistent) against the front URL.

Handover (B2, under build load): a's SIGTERM -> exit 3.1 s (`--shutdown-delay-duration=2s`); lease
`kube-system/kcp-controllers` in `system:admin` deleted with b's shard-admin context through the front;
b held it 0.46 s later; `serviceaccount/spike` read through the retargeted front. Slot cold start
under that load: a 11.2 s, b 7.3 s (b on an existing store), RSS 357-402 MB.

### 3. Release kcp as an embedded asset (linux/amd64)

| form | bytes | decompress |
|---|---|---|
| raw static ELF | 176,825,356 | - |
| gzip -9 | 70,054,216 | - |
| zstd -19 --long=27 | 59,189,153 | 0.15 s |
| xz -9e | 57,065,516 | 0.80 s |

Embedded zstd adds ~59 MB to the core binary; extracted once into the content-addressed asset store and
exec'd from there. Cold start and RSS are the route B table: ~9 s fresh / ~6 s restart to /readyz,
~380-410 MB kcp + ~55-62 MB per kine.

## Recommendation

Out-of-process, as decided: release kcp v0.33.0 shipped as a zstd asset (~59 MB), started as blue/green
slot units with `--secure-port=0 --experimental-bind-free-port`, the core reading the bound port from the
slot MainPID's single LISTEN socket, shared material copied from `<state>/kcp/shared/` (all five files),
lease delete after the old slot exits. No kcp patch and no vendored kcp source are required on this
shape. Route A is recorded as feasible (0 compile errors, 0 test regressions, two kcps in one process),
and not chosen because it cannot do kcp blue/green without overlapping whole core processes.