# Dev run mode: the whole stack without systemd, N instances per host Research + design, 2026-10-12 (session clock 2026-10-11). Answers the user's request of 2026-10-12: > we should maintain a way of running all this without systemd involved for testing / lighter weight so > that we can run parallel dev flows in worktrees etc. and not need to re-install on the system every > time. systemd units are for production deployments / installs ... we like kcp because it could act like > systemd in a way and we can nest them infinitely to orchestrate anything we need. Plus the standing rule from the same day: production setup is the explicit root step `socialweb-computer install bidder|gateway`; unix accounts are never created on demand. Tags: [C] code read at main d6e1881 (file:line), [P] plan text, [R] research doc, [W] web source, [M] measured here on hat-8 (Fedora 44, kernel 7.2.9, go 1.26.2) with stand-in processes only (`/bin/sleep`, `/bin/sh`, a 130-line Go program), scratch dir deleted afterwards. No guest, no container, no kcp was started. --------------------------------------------------------------------------------------------------- ## 0. Answer in one screen 1. **Add a second `hostsvc.Manager` backend, `proc`, and make it the dev and test backend.** It is not an orphan-adoption scheme: it is a tiny per-instance supervisor process, `socialweb-computer internal devsup`, that is the real parent of every unit (so it reads exit statuses, like systemd does), is a child subreaper (so it catches descendants that change process group), puts each unit in its own process group, sets `Pdeathsig=SIGKILL` on each unit's main process, and serves the `Manager` calls over a unix socket. The core (`serve gateway kcp`, `run bidder`, `run request`) talks to it exactly as it talks to the user manager over `$XDG_RUNTIME_DIR/systemd/private`. Prior art: execrunner's Setpgid + state.json + pid/start-tick identity (kcp-libs supervisor.go:141, :20-28, :394), s6's one stable parent per daemon [W s6], and our own hostsvc identity code (identity_linux.go:47-71). 2. **Dev survival semantics (picked):** units survive a restart, crash or rebuild of the CORE (it is only a client of devsup, so adoption is the same code path as prod); every unit dies with DEVSUP (`dev down`, crash, `kill -9`, test process exit). That is exactly prod's shape, where units survive the core and die with the user manager (0006 M2-HOST-UNITS risks, plan 0006:485). [M] section 6. 3. **kcp is the API, not the supervisor.** Something that is not kcp must hold kcp's own process and its store (systemd in prod, devsup in dev). Above that root, long-lived host processes become kcp objects (the plans already have `Postgres`, `OpenBaoServer`, `ComputeVM`, DenoPod) whose `status.runtime.backend` names the placement backend. A generalized `HostProcess` kind is phase 2, after the backend and its contract suite exist. 4. **One kcp per instance** (process isolation; port 0 for everything, front included), with every role addressing its workspaces relative to a configurable base path (`root` in prod, `root:dev:` or `root:t:` where a parent kcp is shared). A host-level "dev parent" kcp holding each instance's `HostProcess` objects (kcp inside kcp) is phase 3 and optional. 5. **Backend is an explicit option, no default, never auto-detected:** `host-backend = systemd-user | proc`. `install` writes `systemd-user` into the account's config.json; every `dev` verb and every test harness passes `proc`. A unit's status always records the backend that placed it. 6. **One contract suite, two backends.** `internal/hostsvc/hostsvctest.Run(t, open)` is run against `proc` on every `check.sh --fast` (no root, no systemd, CI-able) and against `systemd-user` behind livegate on hat-8; differences are declared in `Capabilities()` and each capability-gated case states which backend skips it. A reflection test fails if a `Manager` method has no contract case. 7. **`systemd-run --user` (option c) is not the dev mode.** It is already what the systemd backend does in the login session's manager (the M2-HOST-UNITS live tests ran there, plan 0006:493); it stays the prod-fidelity check, with an instance-scoped prefix, not the default. 8. **s6 / runit / supervisord / process-compose: rejected** as runtime dependencies (one-binary rule, no pid+start-tick identity, no adoption contract); ideas borrowed (readiness/depends_on from process-compose, one-parent-per-daemon from s6, supervisord's grandchild problem as a test case). --------------------------------------------------------------------------------------------------- ## 1. What exists today [C][P] | Piece | Where | Relevance to dev mode | |---|---|---| | `hostsvc.Manager {Start, Inspect, Stop, ResetFailed, List, Identify, Check, Close}` | internal/hostsvc/hostsvc.go:126-135 | The seam. One implementation: systemd user manager over the private socket (systemd_linux.go:40-44, busdial_linux.go:42-48). `unsupported_other.go` for !linux. | | `UnitSpec` (Exec, Env, WorkDir, Output files, Limits, OOMPolicy, KillSignal, StopTimeout, After, Wants, Slice) | hostsvc.go:52-66 | Everything a proc backend must honour or declare unenforced. | | Identity = unit + InvocationID + PID + StartTicks + BootID | hostsvc.go:110-116; identity_linux.go:47-71 (pidfd_open -> /proc stat ticks -> pidfd_send_signal(0)) | Backend-neutral already except `GetUnitByPIDFD` (systemd_linux.go:375). | | Name rule `socialweb-` prefix + `.service`; app.slice refused | hostsvc.go:12-16, :137-146, :165-167 | Proc backend keeps the same names (namespaced by its own devsup, so no cross-instance clash). | | Unit properties: CollectMode=inactive, KillMode=control-group, PropExecStart(argv,false) | systemd_linux.go:132-155 | Semantics the proc backend must emulate: failed units stay readable until ResetFailed; stop kills the whole tree. | | processrunner `DieWithParent` (Pdeathsig SIGKILL from a locked OS thread) | processrunner.go:120-127, :138-147; hostos/sysproc_linux.go:10-15 | Pdeathsig fires when the creating THREAD exits, not the process [W pdeathsig]; the existing locked-thread trick is reused in devsup. | | procgroup (per-cluster processes die with the provider) | procgroup.go:104 | Being replaced by unit slices (0006 M3-CLUSTER-UNITS, plan 0006:760-817). | | execrunner (DenoPods): Setpgid, state.json {pid, started, ticks, stopped}, recover, ownedByUs(pid, ticks), kill(-pgid) | kcp-libs impl/execrunner/supervisor.go:20-28, :141, :249, :308-322, :394 | The adoption-by-ticks pattern; note it has no Pdeathsig and its children are in their own groups, so they outlive a SIGKILLed parent unless something else kills them. | | DenoPod pure reconciler: Observed -> Result{Phase, Ops} | deno-kcp internal/denopod/denopod.go:105-140; provider calls PodRunner Start/Observe/Stop/Probe (reconcile_pod.go:21-60; kcp-libs abc/runner/runner.go:18-63) | The shape a generalized `HostProcess` reconciler copies: pure state machine + a runner port. | | kcp for local dev: kine on sqlite + kcp, fixed 23791 and 6443, pid files, stale-port refusal | deno-kcp deploy/start-kcp.sh:4-6, :57, :74 | Fixed ports: one per host. Dev mode replaces it with port 0 (kcp-spike.md:82-103 shows the bound port is readable with no kcp patch). | | Port leases: carve 20000-29999, one lease file, key (cluster, purpose); Lease already has Instance and Account fields | portallocator.go:18-19, :99-109, :145-148, :169-172, :346-357 | Two instances with two lease files can lease the same not-yet-bound port (probe-then-bind race). Must key by instance and share one per-user lease file in dev. | | Live-test port blocks locked under /var/tmp/socialweb-liveports | liveports.go:67, :92-118 | Already parallel-safe for proofs; unchanged. | | Market network on fixed 2583-2589, accept.sh | internal/marketnetwork/live_test.go:18, :147; 0008 M4-MARKET-NETWORK-PORT ("a second market network on the host is refused by a lock file", plan 0008:287-289) | The one thing that forbids N market networks per host. | Production decisions that dev mode must not undercut [P][R]: units in the USER manager of a lingering account (user-decisions.md:7); one shared bidder account (:10); kcp out of process as blue/green slots on kine -> postgres behind our own L4 front (:22; kcp-spike.md:144-151); AppArmor profile at install on Ubuntu 24.04+ (:13); no socialweb code in a VM's process tree (0006 M2-VM-TREE-AUDIT); deno has no access to `/dev/kvm` or `$XDG_RUNTIME_DIR` (0007 M5-RUN-BIDDER, plan 0007:612-614); RFP flow is the spine (0006 state rule, plan 0006:74-78). --------------------------------------------------------------------------------------------------- ## 2. Requirements for dev mode R1. No root, no `install`, no systemd, no lingering account, no unit files. Runs as the invoking user. `dev` refuses to run as uid 0 (postgres refuses root anyway, plan 0006:97). R2. N instances on one host from N worktrees (or N instances from one worktree with `--instance`), with no collision on: - ports: port 0 everywhere a port is not part of a durable contract (kcp slots, kcp front, kine, postgres, bao, door, PDS/relay/market network services, k3s slot ports, front ports). Where a port must be stable across restarts inside an instance (a cluster's front port: guest tickets and node secrets name it, 0006 M3-APISERVER-FRONT), it is leased from ONE per-user lease file keyed by (instance, cluster, purpose), so instances see each other's leases; - paths: state `$XDG_STATE_HOME/socialweb-computer/dev//` (default `~/.local/state`, XDG spec [W xdg]); sockets `$XDG_RUNTIME_DIR/swc//` (short, < 108 bytes; 0700 by spec [W xdg]); a gitignored symlink `/.swc-dev -> ` for discovery. Not inside the worktree itself: long worktree paths break the 108-byte socket limit, `git clean -fdx` would delete live state, and `git worktree remove` refuses untracked files; - names: unit names are per devsup, so prod names (`socialweb-vm--...`) are reused unchanged; kcp workspaces are per kcp process; - assets: content-addressed, so one shared per-user cache `$XDG_CACHE_HOME/socialweb-computer/runtime/ /` is safe across instances (re-hash before use, 0006 M1-ASSETS) and avoids extracting ~400 MB of kcp/k3s/bao/deno per worktree. `` = `-`, or `--instance `. R3. Fast: `dev up` on a warm instance (kcp root and store exist) is bounded by kcp's restart-to-readyz, ~6 s on kine (kcp-spike.md:73-76); `dev down` stops everything in reverse order within the largest TimeoutStopSec (10 s for VMs, 0007 M2-VM-RUNTIME). Core rebuild + restart does not restart kcp, postgres, cluster members or guests. R4. Usable from `go test` (hermetic: a crashed test leaves nothing running) and from live market-network runs (RFP flow: requester -> RFP -> bid -> accept -> cloud-init user_data; guests reached only through the relay). Dev mode adds no provisioning path of its own: a market network instance runs `run bidder` with the proc backend and its guests come up only through `runComputeContract`. R5. Same code paths as prod above the `Manager` seam: the ComputeVM verdict table (0007 M2-CONTROLLER), cluster member adoption (0006 M3-CLUSTER-UNITS), kcp slot swaps (M3-CP-SLOTS) and vault unseal on a new InvocationID (M2-OPENBAO-SERVER) run unchanged on `proc`. R6. Survival semantics (picked; alternatives in section 4.1): - core restart / crash / rebuild: every unit survives and is adopted (Running, same InvocationID); - devsup exit (`dev down`, crash, `kill -9`, `--die-with` process exit): every unit dies; the next `dev up` finds every recorded identity Lost(UnitGone), exactly as prod after a user-manager restart or reboot (plan 0006:485); - host reboot: Lost(HostRebooted) by boot_id, same as prod. R7. Honest about what dev does not give: no cgroup limits (MemoryMax, MemorySwapMax, TasksMax, CPUQuota, CPUWeight, OOMPolicy are accepted and reported unenforced), no systemd-oomd, no journal. A VM's memory is still bounded by firecracker's own `machine-config` mem size. R8. Host prerequisites dev needs but never performs (reported by `dev check`, never fixed by it): `/dev/kvm` rw for the user (kvm group, already true on a dev box or not - dev never chmods), unprivileged user namespaces for pasta (on Ubuntu 24.04+ the AppArmor restriction needs a root-installed profile or the sysctl; Fedora has neither restriction), `/dev/net/tun`. --------------------------------------------------------------------------------------------------- ## 3. Options ### (a) A second hostsvc backend, `proc` Two sub-shapes: **(a1) Orphan adoption (execrunner shape).** The core starts each unit detached (setsid, no Pdeathsig), writes state.json {pid, pgid, ticks, bootID}; a restarted core re-reads and adopts by pid + start ticks. - Pro: no extra process; execrunner and hostsvc identity code already do the identity half. - Con [M, section 6]: an adopter is not the parent, so `waitid(P_PIDFD, ...)` on an adopted pidfd returns ECHILD; the exit status of a unit that dies while adopted is lost. That breaks the verdict rows that need Result/ExecMainStatus (Lost(UnitFailed | OOMKilled | VMMExited), CollectMode=inactive semantics, plan 0007:359-361) unless a wrapper process writes it - and a wrapper is our code in the VM's tree (forbidden, 0006 M2-VM-TREE-AUDIT). - Con: nothing kills the tree on `dev down` if the core is already dead; "everything dies with the dev session" needs a sweep by recorded pgid, which is only safe with the leader-ticks rule (section 6). - Con: Restart=on-failure needs someone to notice an exit promptly; the core may be down. **(a2) A stable per-instance supervisor, `devsup` (recommended).** `socialweb-computer internal devsup --state /sup --socket /sup.sock [--die-with ]`. - It is the parent: it reads every unit's exit via wait4, so Result / ExecMainCode / ExecMainStatus / CollectMode=inactive / ResetFailed / Restart=on-failure|always behave like systemd's. - It calls `prctl(PR_SET_CHILD_SUBREAPER)`: any descendant that leaves its unit's process group is reparented to devsup when its parent dies, instead of to PID 1 [W subreaper]; devsup attributes it by the inherited `SWC_UNIT=` environment variable and kills it if that unit is not active (the KillMode=control-group emulation). [M] shows the orphan otherwise lands on PID 1. - Each unit: own process group (`Setpgid`), `Pdeathsig=SIGKILL` set from a locked OS thread (the processrunner.go:138-147 trick; Pdeathsig is per creating thread [W pdeathsig]), stdout/stderr to the `Output` append files, `Env`, `WorkDir`, an InvocationID (128-bit random per start). - Stop = `KillSignal` to `-pgid` and to every attributed descendant (snapshot from `/proc//task/*/children` before signalling), wait `StopTimeout`, then SIGKILL the same set. - Identity/Check: the same Pin sequence (identity_linux.go:47-71); `GetUnitByPIDFD` is replaced by "pid's pgid == unit pgid, or pid attributed to the unit" answered by devsup. - Slices: `Slice` is a label; `StopSlice(prefix)` stops every unit whose slice equals or nests under it (systemd slice names nest by dash, hostsvc.go:16; 0006 M3-CLUSTER-UNITS escapes cluster keys). - Ordering: `After`/`Wants` are validated (named unit exists in this devsup) and otherwise advisory, as for transient units started one by one; the callers already start in order (M3-CLUSTER-UNITS). - Watch: devsup pushes unit state changes on the socket stream (the systemd backend uses D-Bus PropertiesChanged; 0007 M2-VM-RUNTIME needs Watch, plan 0007:297). - fd store: devsup holds a listening fd a unit hands it and passes it back to the next start of that unit with the `LISTEN_FDS` convention, so `M3-APISERVER-FRONT`'s "a front restart refuses no connection" (plan 0006:834-836) also holds in dev. Optional: `Capabilities().FDStore` gates it. - Private hostname (k3s slot lease identity, plan 0006:884-888): the `internal uts-exec` one-shot (user+UTS namespace, sethostname, execve) that M3-CP-SLOTS already names as the fallback becomes the proc backend's implementation of a `PrivateHostname` field; no resident code. - State on disk: `/sup/units/.json` {spec, InvocationID, pid, pgid, leader ticks, bootID, devsup instance, last Result}; `dev status` reads it without the socket. After a devsup crash, the next devsup sweeps leftovers by the safe pgid rule (section 6) before accepting calls. - Size: comparable to execrunner's supervisor.go (425 lines) plus a net/rpc-over-unix-socket server. - Pro: exact exit semantics; tree kill; same failure model as prod; one devsup per instance = isolation. - Con: a resident process of ours that is the parent of pasta in dev. It is NOT in the unit's process group, exactly as systemd is not in a unit's cgroup, so the VM tree audit (process group members + attributed descendants = pasta, firecracker only) still holds; the audit becomes backend-aware. - Cross-OS: the same contract maps to a Windows Job Object hierarchy (KILL_ON_JOB_CLOSE, nested jobs) and to darwin's kqueue NOTE_EXIT + process groups; proc_linux.go first, stubs return ErrUnsupported until plans 0011/0012 decide (darwin has no Pdeathsig, hostos/sysproc_darwin.go:10). ### (b) kcp-native: host processes are kcp objects; systemd is one placement The user's framing: kcp "acts like systemd" and nests. What it buys and what it cannot do: - Already true in the plans: `Postgres` and `OpenBaoServer` (export hostprocesses, status.runtime {backend, ref, instance, pid, startTime, hostBootID}, plan 0006:618-621, :664-667), `ComputeVM` (RuntimeRef {backend systemd-firecracker, ...}, plan 0007:281-283), DenoPod (execrunner). Every one of them is "a kcp object reconciled into a host process". The backend field is already in their status. - Generalize: a `HostProcess` kind (spec: asset ref + argv, env, workDir, output, limits, restartPolicy, stop signal + grace, after/wants, slice, privateHostname, readiness probe; status: phase, runtime {backend, unit, invocationID, pid, startTicks, hostBootID}, lastResult). A pure reconciler in the denopod.go:105-140 style (Observed -> Result{Ops}) and a placement port that IS `hostsvc.Manager`. Postgres/OpenBaoServer/kcp-slot/cluster-member controllers either own a HostProcess child or call the same library; ComputeVM keeps its own controller (its verdict table is richer) but places through the same Manager. - Cannot replace (a): somebody must run kcp, its kine and its postgres before any object can be read, and must hold processes while kcp or the core is down (the whole point of units outliving the core). A reconcile loop is not a process parent: it does not reap, kill trees, hold fds or restart within milliseconds. So the root of every tree is imperative - systemd in prod, devsup in dev - and the core "mirrors" what it placed into kcp after kcp is up, the way the kubelet creates a mirror Pod in the API server for each static Pod and keeps running it even if the mirror is deleted [W static pods]. - Nesting: a `HostProcess` whose argv is another core instance (`socialweb-computer dev core ...`) with its own kcp is "kcp inside kcp": the parent's reconciler places the child's processes; the child's kcp holds the child's own objects. Recursion depth is bounded only by ports (all 0) and memory (~0.4-0.5 GB per kcp + kine, kcp-spike.md:73-76). - Pro: one model, `kubectl get hostprocesses` per instance, status/history in objects, the same apply path (0006 M2-APPLY), placement swappable per object. - Con: more API surface before dev mode exists; HostProcess schema churn while M3 items are still being designed; kcp outage = no reconcile (units keep running, acceptable). - Verdict: the direction, staged after (a). Podman Quadlet is the inverse prior art (declarative specs compiled to systemd units, [W quadlet]); we keep the declaration in kcp and the placement pluggable. ### (c) `systemd-run --user` transient units, no install - What it is: the existing systemd backend pointed at the login session's user manager. Transient services are real units, listed by `systemctl --user list-units`, and failed ones stay loaded until reset-failed unless `--collect` [W systemd-run]. The M2-HOST-UNITS live tests already ran this way (plan 0006:493, "this host has Linger=no so the runs used the login session's user manager"). - Pro: identical semantics to prod, cgroup limits enforced (where delegated), no new code. - Con: still systemd - absent in CI containers, WSL without systemd, macOS, SSH sessions without pam_systemd; every instance shares ONE user manager, so names must carry the instance (`socialweb-dev--...`) or worktrees adopt each other's units; leaks outlive a crashed `go test` until GC; without linger units die at the last logout; units land wherever the slice says, and Fedora's user app.slice is under systemd-oomd kill (plan 0006:469-470). - Verdict: keep as `host-backend=systemd-user` with `--unit-prefix socialweb-dev--` for prod-fidelity runs on hat-8 and for the systemd half of the contract suite. Not the dev default. ### (d) s6 / runit / supervisord / process-compose - process-compose: Go, docker-compose-like YAML for non-containerized processes, depends_on conditions, readiness/liveness probes, restart policies [W process-compose, process-compose health]. An app, not a library; another config language beside cli-args-env/options.json; no pid+start-tick identity or InvocationID; no "adopt, never re-create". Borrow: readiness gating and depends_on conditions. - s6: one long-lived `s6-supervise` parent per daemon, `s6-svscan` supervising the supervisors [W s6]. This is the devsup shape; we take the idea, not the C toolchain. - supervisord: Python; cannot kill grandchildren of the process it spawned and needs `pidproxy` for daemons that hand signals to a child [W supervisord]. Exactly the hazard the subreaper + tree kill fixes; becomes a contract test. - runit: same class as s6, C, service dirs. - Verdict: rejected as runtime dependencies (ACC6-ONE-BINARY: one binary plus its own assets). --------------------------------------------------------------------------------------------------- ## 4. Design details ### 4.1 Survival semantics: why "die with devsup, survive the core" | Choice | Core rebuild cost | go test hygiene | Prod parity | |---|---|---|---| | Everything dies with the dev process (one process, Pdeathsig + killpg) | kcp ~6 s + postgres + cluster members + every guest re-rented through a new RFP | best | none: adoption code never runs in dev | | Units adopted by orphan ticks (a1) | none | needs sweeps; leaks on crash | partial: exit status lost after adoption | | **devsup parent (a2)** | none | `--die-with `: test exit kills all | full: same verdicts, same failure model (manager death = every unit Lost) | `dev up` runs devsup detached (`setsid`, survives closing the terminal; logind's KillUserProcesses is a host setting, `dev check` reports it). `dev up --fg` keeps devsup in the foreground and multiplexes unit logs; Ctrl-C = `dev down`. ### 4.2 VMs in dev (pasta -> firecracker, no systemd) Unchanged prep (0007 M2-VM-RUNTIME, plan 0007:263-281): clone, seed drive, fc.json, params.json; work dir `/handles//vms/`; API socket under `/swvm/.sock`. The unit spec is the same `UnitSpec`; on proc: ExecStart pasta -> `/bin/sh` trampoline -> socialweb-vmexec -> execve firecracker; devsup is pasta's parent; pasta, sh->vmexec->firecracker share the unit's process group unless one of them calls setsid, in which case the subreaper attribution catches it. Stop: SendCtrlAltDel grace in the controller (unchanged), then StopUnit = SIGKILL to the group + attributed descendants. Not enforced in dev: MemoryMax/MemorySwapMax/TasksMax/CPUQuota (declared by `Capabilities().EnforcesLimits=false`; the VM's RAM is still capped by firecracker's machine-config). Still required: `/dev/kvm` access (kvm group), userns for pasta (Fedora: yes; Ubuntu 24.04+: needs the root-installed AppArmor profile or sysctl - dev reports, never fixes). Guests exist only through `runComputeContract` against the instance's bidder. ### 4.3 kcp slots, fronts and clusters in dev - kcp: blue/green slot units `socialweb-kcp@a|b` (prod shape, user-decisions.md:22) on kine. Store in dev: kine -> sqlite in the instance dir by default (kcp-spike.md:73-76: within noise of postgres), kine -> a postgres unit with `--profile prodlike` and in the market-network profile. Slots bind `--secure-port=0 --experimental-bind-free-port`; the core reads the slot's single LISTEN socket from /proc (kcp-spike.md :96-100, no kcp patch). The kcp front binds port 0 in dev (prod keeps 127.0.0.1:6443 behind caddy) and reports its port through the same OnListen callback; `--shard-base-url` names the front's bound port, so the front must bind BEFORE the first slot starts (it does: front first, then slot a). - Default dev runs ONE slot; `dev roll kcp` exercises the swap (start b -> own /readyz -> front flip -> stop a -> delete kube-system/kcp-controllers lease, kcp-blue-green.md:17-64). - Tenant clusters: member units in slice `socialweb-cluster-.slice` (plan 0006:771-774), front on a lease from the shared per-user lease file, k3s slots on leases or 0, `controlplane.json` under the instance's data dir. k3s `--disable-agent` still needs `/etc/rancher/node/password` (plan 0006:97); in dev the same `internal ns-exec` one-shot that gives slots a private hostname also bind-mounts `/rancher` over `/etc/rancher` in a user+mount namespace, so dev needs no root-owned file (see Q6). - Name/slice mapping: identical names on both backends (`socialweb-c--.service`, `socialweb-pg-.service`, `socialweb-vm--.service`, `socialweb-kcp@`); the proc backend's namespace is its devsup, the systemd backend's is the user manager (+ the dev prefix when run in a login manager, option c). ### 4.4 Nesting: one kcp per instance, base-path-relative everywhere Options weighed: - N1, one kcp per instance (recommended as the instance boundary): crash and write-storm isolation (kcp-blue-green.md:34-38 measured ~120-150 conflicts/s/process in an un-elected overlap), each worktree may change APIResourceSchemas/APIExports/WorkspaceTypes/kcp flags or the kcp pin, the M3-SERVE-GATEWAY-KCP bring-up order (kcp first) and slot swaps are exercised. Cost ~0.4-0.5 GB + ~6-9 s cold start per kcp. - N2, one shared kcp with `root:dev::...`: cheap (workspace Ready in 0.3-0.7 s, kcp-blue-green .md:50), but kcp version, flags, feature gates and blast radius are shared; anything testing kcp itself cannot run there. Good for `go test` packages that only need "a kcp workspace": TestMain-scoped workspace `root:t:-` on a shared test kcp, deleted at the end. - N3, kcp inside kcp: a host-level dev parent kcp (`dev host`, one per user) whose workspace `root:dev:` holds the instance's HostProcess objects, including the instance's own kcp slot; the parent core places them on its devsup. Gives `kubectl get hostprocesses -A` over every worktree and is the same pattern prod can use for tenant clusters (members as objects in the cluster's home workspace). Phase 3. Requirement that makes all three cheap: no role assumes it owns `root`. Every component takes `kcp-base-workspace` (default `root`; prod unchanged) and creates/binds under it (`:bidder`, the gateway's provider workspaces, the market network's workspaces). kcp workspace paths nest `/clusters/:` and WorkspaceTypes constrain allowed parents/children (limitAllowedParents/limitAllowedChildren) [W kcp workspaces, kcp workspace types], so a `dev-instance` WorkspaceType with `limitAllowedChildren` keeps instance trees from being nested into each other by mistake. Inside one instance, roles: prod runs gateway and bidder kcps on separate hosts. Dev default runs them as two kcps (prod topology; ~1 GB per instance); `--kcp-shared-roles` puts both roles on one kcp under `:gateway` / `:bidder` to halve memory when the flow under test does not care. ### 4.5 Ports and leases - Port 0 + OnListen for every server we own (org rule; 0006 M2-KCP-PORT0 already says "run bidder, run gateway ssh, run request and every test use 0", plan 0006:594-595). - portallocator: leaseKey becomes (instance, cluster, purpose) - the `Lease.Instance` field exists (portallocator.go:106) but the key ignores it (:145-148); dev opens ONE per-user lease file `$XDG_STATE_HOME/socialweb-computer/ports/leases.json` under its existing flock (:227-243), with a carve per user that does not overlap the live-test blocks (liveports.go:33-56). Prod keeps one lease file per account. subnetallocator gets the same key change. - Market network services: the vendored manifests' fixed 2583-2589 (marketnetwork/live_test.go:18) become template fields rendered to 0, each service reporting its port via a port file/onListen that the market network reads; the market network's lock file goes. ### 4.6 Go interfaces ```go package hostsvc type Backend string const ( BackendSystemdUser Backend = "systemd-user" BackendProc Backend = "proc" ) type Capabilities struct { Backend Backend EnforcesLimits bool OOMKillReported bool FDStore bool PrivateHostname bool Journal bool } type UnitEvent struct { Name string Status UnitStatus } type Manager interface { Start(ctx context.Context, spec UnitSpec) (UnitStatus, error) Inspect(ctx context.Context, name string) (UnitStatus, error) Stop(ctx context.Context, name string) error StopSlice(ctx context.Context, slice string) error ResetFailed(ctx context.Context, name string) error List(ctx context.Context, prefix string) ([]UnitStatus, error) Identify(ctx context.Context, name string, pid int) (Identity, error) Check(ctx context.Context, want Identity) (Observation, error) Watch(ctx context.Context, prefix string) (<-chan UnitEvent, error) Capabilities() Capabilities Close() error } type Config struct { Backend Backend UnitPrefix string SystemdPrivateSocket string ProcSocket string } func Open(ctx context.Context, cfg Config) (Manager, error) ``` - `Open` refuses an empty `Backend` (no default, no probing). `UnitPrefix` (default `socialweb-`) lets the systemd backend run namespaced dev units (option c); `ValidateName`/`ValidatePrefix` check against it. - `Identity` gains `Manager string` (systemd: the user manager's own InvocationID; proc: devsup's instance id) so "the manager restarted" is distinguishable from "the unit restarted". - `UnitSpec` gains `Restart` (no | on-failure | always), `PrivateHostname`, `FDStore int`; the systemd backend maps them to properties (Restart=, ProtectHostname=private: where supported else the ns-exec wrapper, FileDescriptorStoreMax=). - Layering: `hostsvc.go` (types, validation) stays OS-neutral; `systemd_linux.go`, `proc_linux.go` (client), `internal/devsup` (the server; uses `hostos` and the identity code), `unsupported_other.go`. - Status everywhere records `runtime.backend` so an object placed by one backend is never "adopted" by the other (a mismatch is Lost(IdentityMismatch) with a message naming both backends). ### 4.7 CLI - `internal/clicfg/options.json` (0006 M1-CLI) gains `host-backend` (no default; env `SOCIALWEB_HOST_BACKEND`), `instance`, `kcp-base-workspace` (default `root`), `state-dir` (default resolved by install/dev), `asset-cache`. - `socialweb-computer dev up [--profile quick|market network|prodlike] [--instance x] [--fg]`: resolve instance -> flock `/instance.lock` -> start or adopt devsup -> place the role units (core role(s) as units of devsup, like the prod core is a unit of the user manager) -> wait readiness -> print `dev env`. - `dev down [--keep-state]`, `dev status` (units, ports, kcp URLs, from devsup or the on-disk records), `dev ls` (all instances of this user), `dev logs `, `dev env` (exports `KUBECONFIG`, base workspace, ports, for shells and TS tools), `dev restart core [--binary path]` (rebuild loop: units adopted), `dev roll kcp|cluster `, `dev check` (host prerequisites of R8, no fixes), `dev gc` (asset cache, only when no devsup of this user is running). - `market-network up|accept|down` (0008 M4-MARKET-NETWORK-PORT) = `dev up --profile market-network` + the accept checks. - Running `serve gateway kcp --host-backend=proc --instance x` by hand is allowed (that is what devsup runs); there is no hidden mode switch. ### 4.8 Tests - Tier 0 (offline): fakes as today (fake bus, fake hostsvc). - Tier 1 (contract, `internal/hostsvc/hostsvctest`): `Run(t, func(t *testing.T) Manager)`. proc runs on every `check.sh --fast` with `--die-with` the test pid; systemd-user behind livegate with a `socialweb-test--` prefix. Cases (stand-ins only): start/inspect/stop; adopt-not-recreate on a second Start; failed unit readable until ResetFailed; exit status of a killed unit (signal 9); survives the starter's restart (proc: a second client process; systemd: E1); stop kills a descendant that called setsid (`sh -c 'setsid sleep 1000 & exec sleep 1001'`); Restart=on-failure restarts with a new InvocationID; Watch reports an exit within 2 s; StopSlice stops nested slices and nothing else; List never crosses prefixes; identity mismatch on a reused pid; manager restart makes every unit Lost(UnitGone); limits cases gated on `EnforcesLimits`; fd store gated on `FDStore`; `TestEveryManagerMethodHasAContractCase` (reflect over the interface). - Tier 2 (kcp integration): `internal/devstack.Start(t, profile)` -> devsup (die-with test) + kcp slot on port 0 + base workspace; or `SWC_TEST_KCP=` to reuse a shared test kcp with a per-test `root:t:` workspace. - Tier 3 (market network): `dev up --profile market-network` per worktree; guests only via `runComputeContract`. - Parallel proof: `TestTwoInstancesOnOneHostShareNothing` (two devstacks in one test: distinct ports, dirs, sockets, kcps; stopping one leaves the other's InvocationIDs unchanged). ### 4.9 Prod-only (never in dev) `install bidder|gateway` and `uninstall`, account creation, kvm group membership, `loginctl enable-linger`, the `user@.service.d` Delegate drop-in, the AppArmor pasta profile, `/etc/rancher/node/password`, the installed user unit files, `bin/current|previous` and `upgrade` / `rollback` (dev uses `dev restart core --binary`), caddy, fixed prod ports (6443, 8788), check-host's lingering and delegation checks (dev check reports the dev subset). `dev` refuses to run as root; `install` refuses `host-backend=proc`. --------------------------------------------------------------------------------------------------- ## 5. Plan deltas (proposed; plans not edited by this research) ### 0006 - NEW `M2-HOST-BACKEND-PROC` - "hostsvc proc backend: devsup, a per-instance parent for dev and tests". depends_on [M2-HOST-UNITS]. Decision: section 3(a2) and 4.6, decided by Claude, user may veto (questions Q1, Q4). files: internal/devsup/, internal/hostsvc/proc_linux.go, hostsvc.go (Backend, Capabilities, Config, Watch, StopSlice, Restart, Identity.Manager), cmd/socialweb-computer internal devsup. test_first: the proc half of M2-HOST-CONFORMANCE red against a stub, plus TestADevsupCrashLeavesNoProcessAndTheNextDevsupSweepsByLeaderTicks, TestAPgidWhoseLeaderPidWasReusedIsNeverSignalled, TestDieWithStopsEverythingWhenTheWatchedPidExits, TestAnOrphanReparentedToDevsupIsKilledWhenItsUnitIsInactive. - NEW `M2-HOST-CONFORMANCE` - "one Manager contract suite, run against proc always and systemd-user behind livegate". depends_on [M2-HOST-UNITS]. Decision: section 4.8 tier 1; check.sh --fast runs the proc half. test_first: TestEveryManagerMethodHasAContractCase red first; the systemd half re-homes the M2-HOST-UNITS live tests (live_linux_test.go) into the suite. - NEW `M1-DEV-INSTANCE` - "`dev up|down|status|ls|logs|env|check|restart|roll|gc`, instance identity, XDG layout, shared asset cache, per-user lease file". depends_on [M1-CLI, M2-HOST-BACKEND-PROC, M1-ASSETS]. test_first: TestTwoWorktreesGetDistinctInstanceIDsAndShortSocketPaths, TestTheSocketPathOfTheLongestWorktreeIsUnder108Bytes, TestDevRefusesRoot, TestTwoInstancesOnOneHostShareNothing (stand-ins), TestALeaseIsKeyedByInstance (portallocator). - CHANGE `M1-CLI`: add the `dev` tree and options `host-backend` (required, no default), `instance`, `kcp-base-workspace`, `asset-cache`; test TestServeWithoutAHostBackendExits2NamingTheOption. - CHANGE `M1-INSTALL`: see section 5.4; writes `host-backend: systemd-user` into the account's config.json. - CHANGE `M1-ASSETS`: extraction root is an option; dev points it at the per-user cache; GC of the dev cache only by `dev gc` with no devsup of the user running. - CHANGE `M2-VM-TREE-AUDIT`: backend-aware listing (systemd: cgroup.procs; proc: process-group members + attributed descendants); add TestTheAuditPassesOnTheProcBackendWithDevsupOutsideTheGroup. - CHANGE `M2-KCP-PORT0` / `M2-KCP-UNIT` (kcp slots): the kcp front binds 0 outside prod and reports via OnListen; slot argv takes the front's bound URL; runs on either backend. - CHANGE `M3-CLUSTER-UNITS`: slices through `StopSlice`; the proc backend emulates; portscan attribution reads unit membership through the Manager, not cgroup files directly. - CHANGE `M3-APISERVER-FRONT`: fd store through `UnitSpec.FDStore`; proc implements it in devsup. - CHANGE `M3-CP-SLOTS`: private hostname through `UnitSpec.PrivateHostname`; `internal ns-exec` (hostname + optional bind of `/rancher` over `/etc/rancher`) is the proc implementation and the systemd fallback. - NEW (phase 2) `M3-HOSTPROCESS-KIND` - "HostProcess objects with a pure reconciler; systemd and devsup are placements". depends_on [M2-APPLY, M2-HOST-BACKEND-PROC, M2-HOST-CONFORMANCE]. Decision: section 3(b); Postgres/OpenBaoServer/kcp-slot/cluster-member placement goes through it; mirror records for the imperatively started root (kcp, its store, the core). test_first: TestAHostProcessIsPlacedOnTheBackendItsStatusNames, TestTheSameObjectsReconcileIdenticallyOnBothBackends (conformance harness), TestDeletingAMirrorRecordDoesNotStopTheRootUnit. Q3. - NEW (phase 3, optional) `M4-DEV-PARENT-KCP` - "`dev host`: a per-user parent kcp whose root:dev: workspaces hold each instance's HostProcess objects (kcp inside kcp)". depends_on [M3-HOSTPROCESS-KIND, M1-DEV-INSTANCE]. Q2. ### 0007 - CHANGE `M2-VM-RUNTIME`: Runtime composes `hostsvc.Manager` of either backend; Watch via `Manager.Watch`; `Capabilities().EnforcesLimits=false` recorded in the ComputeVM status condition in dev; the stand-in live tests run through the conformance harness on both backends (TestTheVMUnitsTreeIsPastaAndTheStandInOnly, TestAVMMThatExitsOnItsOwnEndsItsUnitAndLeavesNoProcess). - CHANGE `M2-CONTROLLER`: add TestACoreRestartUnderTheProcBackendAdoptsEveryVMUnit (stand-in VMM) and TestAnObjectPlacedByAnotherBackendIsLostNotAdopted. - CHANGE `M5-RUN-BIDDER`: `run bidder --host-backend=proc --instance x` runs without the shared account (the account check is prod-only, keyed on `host-backend=systemd-user`); the deno denies cover the devsup socket because it lives under `$XDG_RUNTIME_DIR` (plan 0007:612-614); test TestDenoInTheBidderPodCannotDialTheDevsupSocket. ### 0008 - CHANGE `M4-MARKET-NETWORK-PORT`: drop the fixed 2583-2589 and the one-network-per-host lock (plan 0008:287-289); market network service ports render to 0 and are read back; `market-network up` = `dev up --profile market-network`; test TestTwoNetworksOnOneHostEachRentAGuestThroughTheRFPFlow (two instances, two requesters, two bidders; guests only via runComputeContract). - CHANGE `M2-DENOKCP-INPROC`: "deno processes die with the core" is proved on both backends; on proc via devsup's subreaper attribution (execrunner children are in their own process groups and have no Pdeathsig, supervisor.go:141). test: TestAKilledCoreLeavesNoDenoProcessOnTheProcBackend. - CHANGE `M2-DENOKCP-FORK`: the runner_unix.go seam it already plans (plan 0008:154-158) also sets the `SWC_UNIT` attribution env so devsup can attribute DenoPod processes. ### 5.4 `install bidder|gateway` item check (M1-INSTALL, plan 0006:408-442) What it creates today, per the plan: | | gateway | bidder | |---|---|---| | account | `socialweb` | `swc-bidder` (one for all handles) | | groups | - | `kvm`, added BEFORE linger and before user@ first starts | | linger | `loginctl enable-linger` | same | | delegation | `user@.service.d/socialweb-delegate.conf` `Delegate=cpu memory pids` where the host delegates less | same | | AppArmor | - | Ubuntu 24.04+: profile granting userns to `/runtime/*/pasta` | | host file | `/etc/rancher/node/password` 0600 owned by the account | - | | user unit | `~/.config/systemd/user/socialweb-computer-gateway.service` | `socialweb-computer-bidder@.service` | | state | `bin//`, `bin/current`, `bin/previous`, `runtime//` | same | | final step | check-host as the account | same | Gaps to add (decided by Claude, user may veto): 1. **Idempotent by plan, not by luck.** `install` computes the step list from probes (account exists with the expected shell/home/groups? linger on? drop-in content equal? profile loaded with equal text? unit file equal?) and performs only missing steps; `--dry-run` prints it. Each mutating step is recorded in `/var/lib/socialweb-computer/install-.json` (root-owned: what install changed, versions, timestamps). An account that exists but was not created by install (no record) is refused unless `--adopt-account`, so a typo never takes over a human's account. 2. **Order rule preserved and tested:** kvm before linger before first user@ start (existing test TestInstallBidderAddsKvmBeforeEnablingLinger); if user@ is already running without kvm, install refuses and names the restart (it would stop every VM), never restarts it itself. 3. **Enabling units:** root writes the unit file; `systemctl --user -M @ daemon-reload` and `enable` the gateway unit; per-handle bidder units are enabled by the ACCOUNT later (`socialweb-computer bidder enable --handle h`), so adding a handle needs no root. 4. **config.json** for the account: `host-backend: systemd-user`, state dir, ports (prod 6443/8788 for the gateway) - the only place a non-dev default is set. 5. **`uninstall bidder|gateway [--purge] [--force]`:** refuses while any `socialweb-vm-*` unit or ComputeVM is Running unless `--force` (stopping it Loses tenant VMs); stops the core unit and every `socialweb-*` unit of the account; disables linger; removes the drop-in, the AppArmor profile, the unit file, the node password file; keeps the account and its state unless `--purge` (then `userdel -r` and the record). Every step reverses one recorded install step; unrecorded things are left alone. 6. **Tests (fake command runner):** TestInstallTwiceRunsNoMutatingCommandTheSecondTime, TestInstallRefusesAnExistingAccountItDidNotCreate, TestUninstallReversesEveryRecordedStep, TestUninstallRefusesWhileAVMUnitIsActive, TestInstallRefusesTheProcBackend, TestInstallWritesSystemdUserAsTheHostBackend. Live on hat-8: a throwaway account installed, installed again (no-op), uninstalled, `--purge`d; readings recorded. --------------------------------------------------------------------------------------------------- ## 6. Measurement [M] A 130-line Go stand-in supervisor (stdlib only) on hat-8: sets PR_SET_CHILD_SUBREAPER, starts three units - `attached` (`sleep 1001`, Setpgid + Pdeathsig SIGKILL), `attached-tree` (`sh -c 'sleep 1002 & exec sleep 1003'`, Setpgid + Pdeathsig), `detached` (`sleep 1004`, Setsid, no Pdeathsig) - writes state.json {pid, pgid, ticks, boot_id}; a second invocation adopts and sweeps. | Step | Result | |---|---| | adopt while supervisor alive | all three Running(adopted) by pid + start ticks; `waitid(P_PIDFD)` from the adopter: ECHILD ("no child processes") for every unit | | `kill -9` supervisor | `attached` and `sleep 1003` gone (Pdeathsig); `sleep 1002` (grandchild, same pgid) SURVIVED, reparented to PID 1 (`ppid=1 systemd`, not the dead subreaper, not the user manager: session processes do not descend from user@); `detached` survived (own session) | | adopt after supervisor death | attached: Lost(UnitExited) (pidfd_open ESRCH); detached: Running(adopted), still ECHILD for exit status | | adopt with start ticks off by one | detached: Lost(IdentityMismatch) - pid reuse is detected | | sweep: `kill(-pgid, SIGKILL)` by recorded pgid | killed the orphaned grandchild (its pgid outlived its leader) and the detached unit; nothing left | Consequences used above: (1) an adopter never gets exit statuses, hence devsup as the stable parent; (2) Pdeathsig alone leaks grandchildren, hence process groups + subreaper + tree kill; (3) a pgid sweep after a crash is safe only by this rule: signal `-pgid` when (a) a process with pid == pgid exists and its start ticks equal the recorded leader ticks, or (b) no process has pid == pgid but some process still has that pgid (the kernel does not reuse a pid while it names a live process group, so the group is still ours); never when a process with pid == pgid exists with different ticks (reused). Cleanup: all stand-ins killed, scratch dir removed. --------------------------------------------------------------------------------------------------- ## 7. Questions for the user (recommended answers) - Q1. Dev survival: units survive a core restart and die with a per-instance devsup (recommended), or everything dies with one dev process (simpler, no adoption in dev)? **Recommended: devsup** - it keeps guests and kcp across rebuilds and runs prod's adoption code in every dev loop. - Q2. Nesting: one kcp per instance with base-path-relative roles (recommended), one shared dev kcp with `root:dev:` trees, or a parent kcp holding instances as HostProcess objects (kcp in kcp)? **Recommended: per-instance kcp now; shared test kcp only for go test packages; kcp-in-kcp (dev host) as optional phase 3.** - Q3. `HostProcess` kind (kcp as the systemd API, placements pluggable): now, or after the proc backend and conformance suite? **Recommended: after (phase 2)**, so its schema is shaped by two working backends. - Q4. `host-backend` with no default (required everywhere; install writes systemd-user, dev passes proc), or default systemd-user? **Recommended: no default** - a forgotten flag in a worktree must not drop prod-named units into the login user manager. - Q5. Dev kcp store: kine-sqlite by default and kine-postgres in the market network/prodlike profiles, or postgres always (prod parity)? **Recommended: sqlite for quick, postgres for market network and prodlike.** - Q6. k3s's `/etc/rancher/node/password` in dev: bind-mount an instance-owned file in a user+mount namespace via `internal ns-exec` (no root), or require a one-time root-created host file shared by all instances? **Recommended: ns-exec** (it is the same one-shot M3-CP-SLOTS needs for the hostname). - Q7. Dev on Ubuntu 24.04+ (AppArmor blocks unprivileged userns for pasta): document the host prerequisite and have `dev check` report it, or add a root `install dev-host` step? **Recommended: report only; dev never escalates.** - Q8. Dev runs gateway, bidder and requester as one uid in one instance (prod: separate accounts/hosts). **Recommended: accept** - account separation is a prod property tested by install/check-host on hat-8. --------------------------------------------------------------------------------------------------- ## Sources Web: - [W systemd-run] systemd-run(1), `--collect`, transient services: https://www.freedesktop.org/software/systemd/man/latest/systemd-run.html , https://man.archlinux.org/man/systemd-run.1.en - [W systemd.exec] ProtectHostname=private:: https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html - [W delegation] cgroup delegation (only sub-trees delegated; systemd owns the delegated cgroup's attributes): https://systemd.io/CGROUP_DELEGATION/ - [W subreaper] PR_SET_CHILD_SUBREAPER(2const): https://man7.org/linux/man-pages/man2/PR_SET_CHILD_SUBREAPER.2const.html - [W pdeathsig] PR_SET_PDEATHSIG(2const), caveat "the parent is the thread": https://man.archlinux.org/man/PR_SET_PDEATHSIG.2const.en - [W pidfd] pidfd_open(2): https://man.archlinux.org/man/pidfd_open.2.en - [W xdg] XDG Base Directory Specification (XDG_STATE_HOME, XDG_RUNTIME_DIR 0700, lifetime): https://specifications.freedesktop.org/basedir-spec/latest/ - [W kcp workspaces] https://docs.kcp.io/kcp/main/concepts/workspaces/ ; [W kcp workspace types] limitAllowedParents/limitAllowedChildren: https://docs.kcp.io/kcp/main/concepts/workspaces/workspace-types/ - [W static pods] kubelet static Pods and mirror Pods: https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/ - [W quadlet] Podman Quadlet (specs to systemd units): https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html - [W process-compose] https://github.com/F1bonacc1/process-compose ; health/readiness: https://f1bonacc1.github.io/process-compose/health/ - [W s6] s6 overview (s6-supervise per daemon, s6-svscan): https://skarnet.org/software/s6/overview.html - [W supervisord] subprocesses and pidproxy: https://supervisord.org/subprocess.html Repo (main d6e1881): internal/hostsvc/{hostsvc.go, identity_linux.go, busdial_linux.go, systemd_linux.go}, internal/processrunner/processrunner.go, internal/procgroup/procgroup.go, internal/hostos/sysproc_*.go, internal/portallocator/portallocator.go, internal/liveports/liveports.go, internal/marketnetwork/live_test.go, cmd/socialweb-clusterprovider/main.go:117-124, docs/plans/0006, 0007, 0008, docs/research/2026-10-11-vm-lifecycle/{user-decisions.md, kcp-spike.md, kcp-blue-green.md, linux-proofs.md}; ../kcp-libs impl/execrunner/supervisor.go, abc/runner/runner.go; ../deno-kcp internal/denopod/denopod.go, internal/provider/reconcile_pod.go, deploy/start-kcp.sh.