johnandersen777.bsky.social/socialweb-computer-kcp

Switch repository

Loading…

# User decisions 2026-10-11 (plans 0006/0007/0008 fold)

- All code/docs/arch/plan items changeable; best solution wins.
- 0006/0007/0008 = Linux only, with cross-OS seams (interfaces + GOOS build-tagged files + per-OS embedded assets compiled in only for target OS). Windows -> plan 0011 (Windows host). macOS -> plan 0012 (macOS host). Both start with M0-REVISIT.
- Headless servers; VMs survive core crash/upgrade + logout; reboot = Lost; native per-OS mechanism; VMs only in lifecycle scope.
- Invariant: no socialweb-computer code in a running VM's process tree (VMM launched directly by service manager; our code = one-shot prep).
- Gateway lifecycle manager: USER manager of a lingering account (same mechanism as bidder); cluster groups are transient services in that user manager. Accept: only cpu/memory/pids delegated.
- Host-process CRDs: KEEP BOTH Postgres and OpenBaoServer CRDs (own units). Must add explicit ordering/readiness between a cluster's postgres unit and its cluster-group unit (start, adoption, restart).
- Identity mismatch on a running VM (unit active, pid/start ticks/firecracker id mismatch): mark Lost and STOP THE UNIT IMMEDIATELY. (Unknown = unreachable bus/socket still retries, never Lost.)
- Unix accounts: ONE SHARED lingering account for all bidder handles (units namespaced by handle in name).
- Guest console: capped file in the VM work dir (StandardOutput=append:<workdir>/console.log), size cap + rotation by the controller, deleted with the work dir on release. Tenant data never enters the host journal.
- Minimum systemd: 253+ (GetUnitByPIDFD); check-host refuses older. Features also probed directly.
- Ubuntu 24.04+ AppArmor userns restriction: ship an install-time AppArmor profile granting userns to the pinned pasta asset path; check-host verifies.
- Guest network: IDENTICAL address plan for every guest inside its private pasta user+net namespace; delete the network-index allocator; the host address a guest dials = pasta's mapped gateway, constant.
- Guest images (kernel + rootfs) built in CI on a Linux build host per hypervisor x arch, pinned by sha256, shipped as assets; bidders need no root/chroot/apt/go toolchain.
- Bidder deno: explicit deno permissions (no -A, no --allow-run, no /dev/kvm, no D-Bus socket path) enforced in the DenoPod template and tested.
- VM stop: graceful first. Controller sends firecracker SendCtrlAltDel via the API socket, waits a grace window, then the unit stops with KillSignal=SIGKILL (fixes the 45 s pasta-PID1 SIGTERM hang). Grace window measured on the market network (live, through the RFP flow) and recorded. SendCtrlAltDel is x86-only in firecracker; on arm64 the stop is SIGKILL directly until a guest-side shutdown path exists.
- 2026-10-12: NO backwards compatibility, migration, cutover, rollback or prod-downtime planning until the finished solution exists. The gateway runs no containers. Plan 0006 M5-CUTOVER (container volume copy, rollback to container, 7-day retire) is void: the finished binary is deployed fresh and the container path is simply deleted. Control-plane continuity applies only to upgrades of the finished system.
- 2026-10-12: tenant control-plane upgrades are blue/green behind a front: each cluster is a systemd slice of member units; a per-cluster TCP front owns the apiserver port; a new k3s+kine starts beside the old on the same postgres, the front swaps, the old stops (control-plane-handover.md design; scheduler and k3s leader election fixed; idle-first swaps).
- 2026-10-12: in-flight claims (Bidding/Provisioning) resume across a door restart: per-claim secrets-server key in OpenBao + served-state on the NodeClaim.
- 2026-10-12: kcp upgrades should follow the same pattern: caddy already reverse-proxies kcp, so start the new kcp, switch caddy over, stop the old. Research pending (kcp replicas on a shared etcd, controller leader election, caddy upstream swap).
- 2026-10-12 kcp (from kcp-blue-green.md section 9): kcp runs OUT OF PROCESS as alternating slot units (blue/green), not linked in-process; store = kine -> a dedicated postgres unit started before kcp (not a Postgres CRD object), one kine per slot, pg_dump before each kcp upgrade; our own L4 front holds 127.0.0.1:6443 (same code as the tenant k3s fronts) and does the swap after the new slot's own-port /readyz; caddy stays public TLS only and is never reconfigured by a swap (any generated caddy config sets install_trust false); kcp controller lease handover = delete kube-system/kcp-controllers after the old slot exits (no ReleaseOnCancel patch); slots run --enable-leader-election and --shard-base-url pointing at the front. Minor-version roll rule (Q4: equal storage versions/bootstrap schemas else planned short outage) decided by Claude, user may veto.
- 2026-10-12: new market lexicon com.publicdomainrelay.temp.compute.events.vm.terminated, bidder-authored, carried in the existing market.event envelope (receipt + payload strongRefs + bidder signature), reporting why/when a guest ended (guest_poweroff, vmm_exited, oom, host_reboot, identity_mismatch, deleted_on_request) and acknowledging requested deletes; requesters end the contract on it (door: claim Lost now, re-RFP; CLI: end session, skip redundant vm.delete); bidder keeps tombstones so a late vm.delete gets 200. The plans' "lexicons not owned" clause is lifted for this NSID. A 0007 item; implemented now against the firecracker provider.
- 2026-10-12 (0007 questions): VM isolation without on-demand accounts = per-VM subuid user namespaces: `install bidder` (root, once) reserves one /etc/subuid+subgid range for the shared account; each VM unit runs pasta+firecracker in its own userns mapped to a distinct sub-range (rootless-podman pattern via newuidmap), so a VMM escape lands in an unprivileged uid owning nothing. Unix accounts are never created on demand; the only account is created by the explicit install step.
- 2026-10-12: guest seed = a separate pure-Go NoCloud cidata drive; rootfs copied as-is; the guest grows its own root (growpart/resize in guest); debugfs/e2fsck/resize2fs removed from hosts.
- 2026-10-12: guest SSH host keys are guest-reported, replicating how onNetwork works: new atproto record(s) + plumbing so the requester receives the host key(s) and the ssh command; request-vm-ssh and the gateway/door validate against the sent key as the known host key (no TOFU, no host-side scan).
- 2026-10-12: production setup is one explicit command, `socialweb-computer install bidder|gateway` (root step: account, subuid range, kvm group, linger, delegation drop-in, AppArmor profile, node password, every user unit); systemd is for production installs only. A systemd-free dev/test run mode (no root, no install, parallel instances per worktree; kcp as the supervisor, nestable) must exist; research in dev-run-mode.md.
- 2026-10-12: orphan receipts (VM alive, NodeClaim lost) are ADOPTED FIRST with a usage check: claim phase Adopting; match a tenant Node (node name = guest name, vault role node-<name>, guest-reported host-key record as signed identity); joined + Ready -> adopt into that cluster; not joined but its user_data targets a cluster that still exists -> wait the join grace, adopt on join; cluster gone or grace passes with no join -> release (vm.delete once).
- 2026-10-12: dev run mode is KCP-NATIVE NOW (overrides dev-run-mode.md's phased recommendation): long-lived host processes are kcp objects (a HostProcess kind generalising DenoPod; ComputeVM, kcp slots, tenant cluster members) reconciled by an in-binary supervisor; systemd-user is one placement backend for production installs, plain supervised processes the dev backend; instances nest as kcp workspaces. Defaults accepted: host-backend option required (no default; install writes systemd-user, dev passes proc); dev kcp store sqlite for quick runs, postgres for market network/prod-like; k3s node password via per-instance userns bind-mount (no root); AppArmor in dev = report only; one uid for all roles in dev.
- 2026-10-12: the kcp market network (deploy/market-network, converting to the production shape in these plans) is THE vehicle for parallel iteration: each workstream / task tree brings up its own fully isolated atproto network on one host - fake PLC, relay, PDS, ingress proxy, multiple bidders, requesters - via 0008 M1-MARKET-NETWORK-PARALLEL / 0006 M1-DEV-INSTANCE (`dev up --profile market-network`) with M1-PORT-LEASE-INSTANCE. Workstream live tests never use real accounts or the shared hat-8 bidder; the real bidder (bobvmbuilder) + alice are a final soak only. This is wave 0, before other plan work.
- 2026-10-12 PARADIGM (organizing principle of 0006-0008): "what we are doing is a paradigm shift, we are using kcp (the kubernetes control plane) as a core part of applications, it's probably not how it's traditionally used. With the exception of our virtual kubernetes clusters project that is a more traditional usage of it." An application instance (bidder, requester, gateway, market network, dev instance) = its own kcp + manifest set + profile; state = kcp objects; components = DenoPods/HostProcesses reconciled from them (moving today's multi-server single deno processes into kcp); modes = profiles (QR-mode bidder: no PDS pod; market network bidder: own PDS pod); composition = applying manifest sets / nesting workspaces; applications talk only over the atproto market protocol, never by sharing a kcp. Host/cluster topology is not a design axis. Only the virtual Kubernetes clusters project (tenant k3s behind kcp) uses kcp traditionally.
- 2026-10-12 port 0 rule: every listener behind the edge binds port 0 and reports its bound address into its kcp object status; peers resolve by service name through kcp; the address is application state, not config. Reasons: N instances per machine, no allocator / bind-close-rebind race, nesting, blue/green, isolation. Only stable public edge entry points (caddy 443, an application's front) keep known addresses.
- 2026-10-12 scope now: atproto-market bidder and requester, and the gateway kcp; gateway ssh after this work. Fold into the existing plans (no new plan). Multi-server-to-kcp decomposition research pending (multi-server-to-kcp.md).
- 2026-10-12 (multi-server-to-kcp.md section 7): (Q1) STRICT - every application instance has its own kcp everywhere, including inside a market network; the market network is itself an application (MarketNetwork) that composes child applications (each with its own kcp) by applying their root objects; no nested-workspace sharing of a kcp between applications. (Q2) every CLI option hono-bidder and request-vm-ssh support today must keep working, transformed into the kcp-native paradigm: QR/OAuth auth becomes a CRD (identity object) that sets the sign-in up and surfaces what the user must do (URL to visit, etc.) as status; for testing, drive hono-pds's OAuth session minting support to complete it automatically; ephemeral-PDS mode (in-process repo + association records) stays supported for both bidder and requester. (Q3) kcp ONLY: no standalone `deno run hono-bidder` / `request-vm-ssh` without kcp; every run goes through socialweb-computer + kcp. (Q6) requester = a fresh kcp per invocation (not a persistent requester app).
- 2026-10-12 (section 7 cont.): requester per-invocation kcp KEEPS its state dir (kcp root + vault under state/<invocation>); `run request release|resume <name>` restarts that same kcp to finish the contract (ephemeral process, durable state). Bidder guest-facing endpoints (OIDC issuer, /v1/on-network) split into their own DenoPods after M4-BIDDER-VAULT and the per-instance OIDC fix. STRIP ALL NON-IROH GUEST TRANSPORTS as legacy (cloud-init-common tunnel / fedproxy-ssh / fedproxy-web / wootty modules, websocat ProxyCommand paths, relay-fqdn guest addresses) - this removes the guest JSR registry. OAuth sessions and every key live in the app's own OpenBao, never kcp Secrets. Cross-app discovery only through values the composer (MarketNetwork) copies onto a child's root object; the DNS shim resolves only within its own app.
- Cutover / control-plane continuity (superseded for cutover by the line above): User wants: the gateway tracks durably which clusters own which market-rented VMs, so after any restart/upgrade nodes re-join their control planes; ideally ZERO downtime for clusters by bringing up a new k3s control-plane process and shutting down the old one (blue/green). Research pending; user to be asked again.
- kcp in-process fork: decide after the M0 spike; plan carries both paths (in-process fork vs exec'd kcp asset with port-0 + port-report patch), gated on the spike's breakage count/effort.
- 2026-10-17: Q1 is amended, by profile. A market network profile (dev, CI, soak) runs its applications (PLC, relay, Alice's requester or gateway, Bob's bidder, their repo PDS DenoPods) in ONE kcp, each application in its own workspace, found by service name, as in the deno-kcp market example (`deno-kcp/deploy/examples/atproto/market`). The prod profiles keep a strict kcp per application (the gateway on its host, Bob's bidder on its host). Applications still talk only over the atproto market protocol. Atproto records an identity needs (association, vouch) are kcp objects (`AtprotoRecord`) kept in git in the manifest sets; variables are render-time root-object values or runtime `valueFrom` references to another object's status, and a reconciler holds a record Pending until its references resolve (dependency chaining without ordering scripts). Each application's PDS is a DenoPod in its kcp, never a server inside another process (plan 0018).