~ knotless Search/
↑↓ move↵ open⇧↵ all results

johnandersen777.bsky.social/socialweb-computer-kcp
Switch repository
Loading…

0
0
main
Switch branches or tags
383 lines · 30.8 KB Raw History

Linux VM-lifecycle proofs for plans 0006/0007

Host hat-8, Fedora 44, kernel 7.2.9-200.fc44, systemd 259.9-1.fc44, Go 1.26.2, pasta 0^20261002.gcba3570, Firecracker v1.13.1. Run 2026-10-11 23:12-23:25 UTC.

  • Scratch code: linux-proofs/gosrc/ (module swproof). Binaries are in linux-proofs/bin/.
  • Raw outputs: linux-proofs/e*.txt. Scripts: linux-proofs/e*.sh.
  • Every test unit was named swproof-*. All were stopped or garbage-collected at the end: systemctl --user list-units --all 'swproof-*' returns 0 units, and no stand-in processes are left.
  • The repo was not modified. go.mod and go.sum have the same sha256 before and after, and git status is unchanged by me. docs/plans/0011-windows-bidder-hcs.yaml appeared while I worked, but I did not create it.
  • No guest was booted. Firecracker was only run with --help or --version, and in two non-boot modes:
    • the pre-boot API with no configuration;
    • --config-file pointing at a kernel that does not exist, which fails while parsing, before any VM is created.
  • The real boot proof is left to the RFP-flow live tests (runComputeContract -> bid -> accept -> cloud-init).
  • Grounding note: scratchpad/lifecycle-linux.md and lifecycle-synthesis.md do not exist in this scratchpad, so I grounded on the repo sources only.

Summary table

#Claim testedVerdict
E0Host has cgroup v2, delegation, a private socket, /dev/kvm, pasta and firecrackerPASS, with caveats: Linger=no on this host; Fedora Delegate=cpu io memory (+pids); SELinux enforcing
E1A VM unit started over D-Bus by a bidder unit survives bidder restart, kill -9 and stop (same InvocationID and MainPID)PASS, over both the session bus and the private socket
E1-negPlain children of the bidder (setsid+Pdeathsig, or setsid only) die with the bidderPASS: they die in all 3 cases
E2The user manager accepts Restart/KillMode/OOMPolicy/MemoryMax/CPUQuota/TasksMax/ManagedOOMPreference/CollectMode, and they are enforcedPASS for memory, cpu and pids; AllowedCPUs is silently ignored (no cpuset delegated); IOWeight is written but has no effect (no BFQ/io.cost); MemoryMax without MemorySwapMax=0 lets the VM swap
E2-buggo-systemd PropExecStart(argv, true) means "unclean exit is a failure"DISPROVEN: true sets ignore_errors=yes, so a SIGKILLed VM reports Result=success and the unit is GC'd
E2-quirkIOWeight and ManagedOOMPreference can be combinedFAIL on systemd 259.9: ManagedOOMPreference=avoid|omit resets IOWeight to 100 (reproduced with plain systemd-run)
E3Identity check (pidfd + start ticks + GetUnitByPIDFD + InvocationID) works, and an unreachable bus maps to UnknownPASS. The pidfd of a dead process returns UnixProcessIdUnknown from GetUnitByPIDFD; this must map to Lost, not Unknown
E4pasta -> namespace verb creates a tap, then execve; the tap survives; only pasta and the VMM remainPASS: CAP_NET_ADMIN in pasta's userns is enough and /dev/net/tun is usable. The same works inside a unit, and the unit survives bidder restart
E4-selpasta can exec our binary directlyFAIL under SELinux: pasta_t is denied exec of user_tmp_t/bin_t on nosuid mounts. The /bin/sh -c 'exec "$0" ...' trampoline (as in PastaArgv) is required
E4-killKilling pasta tears down the namespace and the VMMDISPROVEN outside a unit: the child survives, orphaned. Inside a unit it holds because KillMode=control-group reaps it
E4-termsystemctl stop stops the VM promptlyFAIL for firecracker-like processes: the VMM is PID 1 of pasta's PID namespace and firecracker has no SIGTERM handler, so SIGTERM is ignored and stop waits for TimeoutStopSec (45s here) before SIGKILL
E5--config-file boots immediately while --api-sock stays up; host_dev_name attaches to a pre-created tapPASS (from source + non-boot runs). The socket path must be < 108 bytes; a stale socket file makes firecracker exit
E6os.Executable() after a rename-over returns the path; re-exec runs the NEW binary; /proc/self/exe runs the OLD inodePASS, both confirmed. If the file was deleted rather than replaced, exec by path gives ENOENT
E7The repo builds for linux/darwin/windowslinux PASS. darwin FAIL: hostos has no darwin twins, which blocks 23 packages. windows FAIL: nodevm/disk.go has no build tag, which blocks 10 packages

E0 Host facts (e0.txt)

Identity and init

  • uid 1000; groups wheel, ollama, keyd, docker. Not in the kvm group.
  • systemd 259.9 (GetUnitByPIDFD is present, which needs 253 or later).

cgroups

  • cgroup2fs. Root controllers: cpuset cpu io memory hugetlb pids rdma misc dmem.
  • user@1000.service has controllers cpu io memory pids, and the same list in subtree_control. app.slice has the same.
  • Fedora adds 00-uresourced.conf: Delegate=cpu io memory. Stock systemd delegates only memory+pids, so the prod host may lack cpu/io. Check it there; I did not touch it.
  • No cpuset is delegated.

Linger and buses

  • Linger is no (State=active, 3 sessions). Prod needs a lingering dedicated account. I did not enable linger here.
  • $XDG_RUNTIME_DIR/systemd/private exists (srwx------). The session bus /run/user/1000/bus exists.

KVM and tun

  • /dev/kvm is crw-rw-rw- root kvm, so it is world-RW on this Fedora host. Do not assume that on prod.
  • /dev/net/tun is 0666.

OOM handling

  • systemd-oomd is active. User manager app.slice has ManagedOOMMemoryPressure=kill (80%/20s), and oomctl lists app.slice as monitored.
  • DefaultTimeoutStopUSec is 45s in the user manager.

Binaries

  • pasta is at /usr/bin/pasta (passt-0^20261002), label pasta_exec_t, and runs in domain pasta_t.
  • firecracker v1.13.1 (static PIE, sha256 prefix 7d3924e0...) is at:
    • /var/tmp/m3live-k8s-tree/.../hono-firecracker-standalone/firecracker
    • /var/tmp/socialweb-k8s-tree/...
    • /opt/socialweb-computer-bidder/org-root/hono-compute-provider/hono-firecracker-standalone/firecracker
  • It is not on PATH and not in ~/.cache.

Kernel and security

  • There is no kernel.unprivileged_userns_clone or apparmor_restrict_unprivileged_userns; this is a Fedora kernel with AppArmor off.
  • user.max_user_namespaces=100000. SELinux is Enforcing. LSMs: lockdown, capability, yama, selinux, bpf, landlock, ipe, ima, evm.
  • /tmp is mounted nosuid.

Go modules

  • go-systemd v22.7.0 was in the module cache. godbus/dbus v5.1.0 was not, so it was fetched from proxy.golang.org into the scratch module only.

E1 Survival (e1.sh, e1-session.txt, e1-private.txt)

Setup

  • The bidder stand-in is bin/bidder, which uses go-systemd StartTransientUnitContext.
  • It is started with: systemd-run --user --unit=swproof-bidder-<mode> -p Type=exec bin/bidder -mode <mode> -unit swproof-vm-1-<mode>.service.
  • The bidder starts swproof-vm-1-* (/usr/bin/sleep infinity). It also spawns two plain children: setsid+Pdeathsig=SIGKILL, and setsid only.

Session-bus run (23:14:29Z)

StartTransientUnit swproof-vm-1-session.service result="done" err=<nil> took=6.2ms
vm baseline: active running bff842fd272a4aaf822673f6aadd6b8e 48315 success
vm cgroup: .../app.slice/swproof-vm-1-session.service  bidder cgroup: .../app.slice/swproof-bidder-session.service
step1 systemctl --user restart -> vm unchanged  STEP1 PASS
  (restarted bidder: StartTransientUnit ... err=Unit swproof-vm-1-session.service was already loaded or has a fragment file.)
  plain children of first bidder: 48326 DEAD, 48327 DEAD
step2 kill -9 MainPID -> vm unchanged  STEP2 PASS ; bidder: failed failed ... signal ; children DEAD, DEAD
step3 systemctl --user stop -> vm unchanged STEP3 PASS ; children DEAD, DEAD

Private-socket run (-mode private)

  • It connects with sd.NewConnection(func(){ dbus.Dial("unix:path=$XDG_RUNTIME_DIR/systemd/private"); Auth(EXTERNAL uid) }), with no Hello.
  • STEP1, STEP2 and STEP3 all PASS. JobRemoved signals arrive on the private connection (result="done").
  • go-systemd has no constructor for the user private socket:
    • NewUserConnectionContext goes to the session bus.
    • NewSystemdConnectionContext is hardwired to /run/systemd/private.
  • So the private path needs the 5-line custom dialer in gosrc/busdial/busdial.go.

Verdict. PASS.

  • Transient units are siblings in app.slice, so the bidder unit's KillMode=control-group cannot reach them.
  • The negative case is also proven: plain children die on restart, on kill -9 (systemd reaps the failed unit's cgroup) and on stop, with or without Pdeathsig.
  • A restarted bidder must adopt, not re-create. StartTransientUnit on an existing name fails with "already loaded".

E2 Unit properties (e2.sh, e2.txt, e2m.txt, e2-flag.txt, e2f.txt)

Readback (startvm -unit swproof-vm-2.service, 23:15:24Z):

Restart=no KillMode=control-group OOMPolicy=stop MemoryMax=268435456 CPUQuotaPerSecUSec=200ms
TasksMax=64 ManagedOOMPreference=avoid CollectMode=inactive-or-failed AllowedCPUs=1 IOWeight=100(!)
kernel: memory.max=268435456  cpu.max="20000 100000"  pids.max=64  io.weight="default 100"
cpuset.cpus: No such file  (cgroup.controllers: cpu io memory pids)
xattr user.oomd_avoid="1" set on unit cgroup
taskset of MainPID: 0-15   <- AllowedCPUs accepted, silently NOT enforced

Enforcement

  • pids: TasksMax=4 gives pids.current=4 pids.max=4 and /bin/sh: fork: retry: Resource temporarily unavailable. ENFORCED.
  • cpu: CPUQuota=20% on a busy loop for 3s gives usage_usec 616757, nr_throttled 30, throttled_usec 2378345. ENFORCED.
  • memory with MemorySwapMax=0 (touching 512M under a 64M MemoryMax) gives Result=oom-kill ExecMainCode=2 ExecMainStatus=9. Journal: "The kernel OOM killer killed some processes in this unit". ENFORCED; OOMPolicy=stop works.
  • memory with swap left at its default: 256M touched under a 64M MemoryMax gives active success memory.current=66990080 memory.swap.current=210948096. The workload was swapped, not killed.
  • io: IOWeight is written to io.weight, but every disk uses scheduler none, so it has no effect.
  • IOWeight quirk: ManagedOOMPreference=avoid or omit together with IOWeight=50 reads back IOWeight=100, in either order. ManagedOOMPreference=none keeps 50. Reproduced with plain systemd-run --user. This is a systemd 259.9 quirk.

Restart=no plus a killed main process (e2-flag.txt)

PropExecStart(argv,true):  ExecStart ignore_errors=yes -> after kill -9: not-found inactive success 0 0  (GC'd, failure invisible)
PropExecStart(argv,false): ExecStart ignore_errors=no  -> after kill -9: loaded failed signal 2 9
  • go-systemd's parameter name uncleanIsFailure is inverted. It is sent as the D-Bus ignore_failure boolean.
  • With CollectMode=inactive (the default) and correct flags, a failed unit stays loaded with Result=signal|oom-kill|exit-code, ExecMainCode and ExecMainStatus until reset-failed.
  • With CollectMode=inactive-or-failed, the unit disappears immediately (not-found). Its Result can then only be had from a JobRemoved/PropertiesChanged subscription or from the journal.

daemon-reload (e2f.txt)

  • A transient unit keeps the same MainPID and InvocationID across systemctl --user daemon-reload.
  • Its definition lives in /run/user/1000/systemd/transient/ on tmpfs, so it is gone after a reboot.

E3 Identity (e3.sh, e3.txt)

Method GetUnitByPIDFD (h) -> (o, s, ay) is present. Over both the private socket and the session bus (unixfd=true):

pidfd_open ok; stat starttime(field22)=277273 (CLK_TCK=100); pidfd_send_signal(0)=ok after stat read
GetUnitByPIDFD: unit=swproof-vm-3id.service invocation=1aba3eac...   (identical to systemctl InvocationID)
GetUnitByPID(69044): same unit, same InvocationID
verdict Alive(same unit+invocation, pidfd-pinned)

Each failure mode maps to a verdict:

SituationObservedCorrect mapping
ticks mismatch (simulated PID reuse)start ticks mismatch -> Lost(pid reused)Lost
bus socket missingconnect: no such file or directory in 33usUnknown
stale socket fileconnect: connection refused in 40usUnknown
bus accepts but never answersctx 2s timeout -> use of closed network connectionUnknown. Must pass a ctx, or godbus Auth blocks forever
held pidfd, process exitedpoll readable=true; pidfd_send_signal: ESRCH; /proc/<pid>/stat: ENOENTLost
GetUnitByPIDFD with an exited pidfdorg.freedesktop.DBus.Error.UnixProcessIdUnknownLost (my classifier said Unknown; fix the mapping)
GetUnitByPID / GetUnit after stopNoUnitForPID / NoSuchUnitLost
fresh pidfd_open of the old pidESRCHLost

PID-reuse reasoning.

  1. Order the check as pidfd_open(pid), then read /proc/pid/stat start ticks, then pidfd_send_signal(fd, 0).
  2. If the signal call succeeds, the stat read came from the process the pidfd pins. A recycled pid would give ESRCH on the pinned fd, or a ticks mismatch.
  3. From then on, GetUnitByPIDFD(fd) is race-free. systemd resolves the pidfd, not the number.
  4. InvocationID distinguishes two runs of the same unit name.
  5. Only transport errors (dial/auth/timeout) are Unknown. Everything the kernel or systemd answers definitively is Alive or Lost.

Socket path length. AF_UNIX paths are limited to 107 bytes. The first E3 run failed with connect: invalid argument on the 135-byte scratch path, and Python failed with AF_UNIX path too long. Any socket path the design builds under a long workdir hits this. See also E5c.

E4 pasta + tap + execve (e4.sh, e4.txt, e4b.sh, e4b.txt, e4-avc.txt)

Direct exec from pasta fails under SELinux enforcing

pasta --config-net -q -f ... -- ./bin/tapexec tap0 ...  ->  "Failed to start command or shell: Permission denied" (execve EACCES)
AVC: { execute } pasta_t -> user_tmp_t ; { execute_no_trans } pasta_t -> bin_t (copy chcon'd to bin_t) ;
     { nosuid_transition } pasta_t -> unconfined_t   (/tmp is nosuid)
  • With the shell trampoline it works: pasta ... -- /bin/sh -c 'exec "$0" tap0 /bin/sh -c "$1"' ./bin/tapexec '<inner>'. This is the exact shape of nodevm.PastaArgv.
  • After sh, the context is unconfined_t.

Interactive run (23:20:25Z)

tapexec: pid=1 ppid=0 Uid: 0 0 0 0 | CapEff: 000001ffffffffff ; uid_map="0 1000 1"
tapexec: created persistent tap0, up; execve [/bin/sh -c ip -br link show tap0; ...; exec tapexec attach tap0]
tap0  DOWN  <NO-CARRIER,BROADCAST,MULTICAST,UP>      (exists after exec)
attach: pid=1 opened existing tap0 with IFF_TAP|IFF_NO_PI|IFF_VNET_HDR fd=4 (as firecracker does)
tree:  70962 pasta.avx2  \_ 70965 tapexec(attach)   -- one child, no grandchildren, no Go "launcher" left
ip -d link (nsenter): tun type tap pi off vnet_hdr on persist on ; state UP after attach
fds of final process: only its own /dev/net/tun (creator's tun fd was O_CLOEXEC and closed)
  • CAP_NET_ADMIN in pasta's userns (root mapped to 1000) is enough for TUNSETIFF, TUNSETPERSIST and SIOCSIFFLAGS.
  • /dev/net/tun is accessible.
  • pasta also creates a PID namespace: the child is NSpid 71299 1, so the VMM is PID 1 there.
  • kill -TERM pasta gives pasta gone and child STILL ALIVE. kill -9 pasta gives the same: the child is orphaned and keeps its netns, but now has no uplink.
  • kill -9 of the child makes pasta exit with rc=137.

Inside a user unit swproof-vm-3.service (e4b.txt; Restart=no, KillMode=control-group, MemorySwapMax=0):

  • E4b-1: The cgroup holds only pasta (MainPID) and tapexec attach (the VMM stand-in). Running ident on the child pid gives GetUnitByPIDFD -> swproof-vm-3.service, with the same InvocationID, so Alive.
  • E4b-2: Bidder unit restart, kill -9 and stop: PASS, PASS, PASS. The VMM stand-in stays alive.
  • E4b-3: kill -9 of pasta (MainPID) gives unit failed signal 2 9, and systemd killed the VMM stand-in (control-group).
  • E4b-4: kill -9 of the VMM gives pasta exit 137, and the unit becomes failed exit-code 1 137.
  • E4b-5, systemctl stop:
    • With a Go stand-in (SigCgt has SIGTERM) it took 0.03s.
    • With sleep infinity (SigCgt=0, the same as firecracker for SIGTERM; see E5) it hit the client timeout 15 with the VMM still alive. That is the PID-ns init ignoring SIGTERM, so stop only completes at TimeoutStopSec (45s) with SIGKILL.

E5 Firecracker --config-file / API (e5.sh, e5.txt, sources in linux-proofs/fcsrc/)

Sources: firecracker v1.13.1. Files: src/firecracker/src/main.rs, src/firecracker/src/api_server_adapter.rs, src/vmm/src/signal_handler.rs, src/vmm/src/resources.rs, src/vmm/src/devices/virtio/net/tap.rs, src/utils/src/validators.rs, docs/getting-started.md, docs/api_requests/actions.md, docs/logger.md, docs/metrics.md, src/firecracker/swagger/firecracker.yaml, all at https://github.com/firecracker-microvm/firecracker/blob/v1.13.1/....

Does --config-file boot immediately? Yes.

  • run_with_api binds the API socket first ("Listening on API socket"), then spawns thread fc_api, then for Some(json) calls build_microvm_from_json.
  • That calls vmm::builder::build_and_boot_microvm ("Successfully started microvm that was configured from one single json").
  • It then enters run_microvm, which keeps serving the API until shutdown.
  • No InstanceStart is needed. The API stays up for GET / (InstanceInfo {id,state,vmm_version,app_name}), PUT /actions (SendCtrlAltDel on x86_64 only; the guest needs i8042/atkbd; it returns 400 pre-boot), and post-boot PATCHes.

Non-boot run (23:22:53Z, pre-boot API only)

GET /                -> {"id":"swproof-e5","state":"Not started","vmm_version":"1.13.1","app_name":"Firecracker"}
GET /machine-config  -> {"vcpu_count":1,"mem_size_mib":128,...}
PUT /actions SendCtrlAltDel -> 400 "not supported before starting the microVM"
SigCgt=0x41801449 -> HUP,ILL,BUS,SEGV,PIPE,XCPU,XFSZ,SYS ; **no SIGTERM handler** (matches register_signal_handlers())

Parse failure with --config-file. A config file naming a nonexistent kernel prints "Listening on API socket" and then BuildFromJson(ParseFromJson(BootSource(InvalidKernelPath))), rc=1. This proves the socket is bound before the build.

  • The JSON keys (VmmConfig, kebab-case) are: boot-source, drives, machine-config, network-interfaces, logger, metrics, mmds-config, vsock, balloon, entropy, cpu-config.
  • Logger and metrics can be given in the file or as --log-path/--metrics-path, and can be configured only once.

--no-api requires --config-file (MissingArgument("config-file"), exit_code 153). There is then no socket at all: no GET /, no CtrlAltDel, no live inspection. Do not use it.

--id must be 1-64 chars of [A-Za-z0-9-] (Unicode alphanumeric). _ is rejected (InvalidChar('_',3)) and 65 chars are rejected. The id is echoed in GET / and in every log line ([swproof-e5:fc_api]). It is a cheap identity cross-check that needs no bus.

host_dev_name.

  • Tap::open_named does TUNSETIFF with IFF_TAP|IFF_NO_PI|IFF_VNET_HDR on the existing name, so it attaches to a pre-created persistent tap. E4's attach stand-in reproduces exactly this.
  • The name must be 15 bytes or fewer (16 is rejected per the tap.rs test).

Socket path rules.

  • An api-sock path longer than 107 bytes fails with "path must be shorter than SUN_LEN", rc=1.
  • If the api-sock path already exists (stale file), firecracker fails with FailedToBindSocket, rc=1. The launcher must unlink it first, as RunVMM does today.

Timestamps. Log timestamps are local time with no offset (2026-10-11T16:22:53... while UTC was 23:22).

E6 os.Executable after rename-over (e6.sh, e6.txt)

[start] build=OLD os.Executable="./e6/nodeboot" inode=21002
(mv nodeboot.new nodeboot)   readlink /proc/PID/exe -> "./e6/nodeboot (deleted)"
[after-rename] os.Executable="./e6/nodeboot"   (Go strips " (deleted)")
exec(os.Executable()) -> [reexec-via-path] build=NEW inode=21010
exec("/proc/self/exe") -> [reexec-via-procself] build=OLD inode=21018 (still "(deleted)")
rm (no replacement): exec(os.Executable()) -> "no such file or directory"; exec(/proc/self/exe) -> OLD runs

Implication for cmd/socialweb-nodeboot/main.go:306.

  • The self path is handed to pasta and resolved later through sh -c 'exec "$0" ...'.
  • If an upgrade renames over the binary between the parent's start and pasta's exec, the namespace verb runs the new build, which is version skew with the parent.
  • If the binary was deleted, it fails with ENOENT.
  • /proc/self/exe cannot be used across pasta, because inside pasta's child it names sh/pasta.
  • Use immutable versioned install paths (.../releases/<version>/socialweb-nodeboot, with a symlink swap for "current"), and resolve EvalSymlinks(os.Executable()) once.

E7 Build-tag sanity (e7.txt, e7b.txt, e7c.txt)

Commands: GOOS=<os> GOARCH=<arch> go build -mod=readonly ./... and -gcflags=all=-e. HEAD is f0460fd. No files were written; go.mod and go.sum hashes are identical before and after.

linux/amd64: rc=0.

darwin/arm64: rc=1.

  • internal/hostos has only _linux/_windows twins. On darwin the only file is hostos.go.
  • Undefined symbols:
    • hostos.DieWithParent (processrunner)
    • hostos.Lock/Unlock/LockExclusive/LockShared (portallocator, subnetallocator)
    • hostos.LocalPortRange and probeListen (portallocator has only probe_linux.go/probe_windows.go)
  • Blocked packages (23): cmd/{cloud-controller-manager, clusterprovider, computer-kcp-check-host, nodeboot, nodeimage, nodeinit, tenantcredential, vault} and internal/{clusterprovider, clusterstack, esovault, liveports, materialiser, nodefirecracker, nodemarket, noderecord, nodevm, portallocator, portscan, processrunner, procgroup, subnetallocator, vaultcert}.

windows/amd64: rc=1.

  • internal/nodevm/disk.go has no build tag but uses unix.IoctlFileClone, unix.Seek, SEEK_DATA, SEEK_HOLE and ENXIO.
  • Blocked packages (10): cmd/{clusterprovider, computer-kcp-check-host, nodeboot, nodeimage, nodeinit, tenantcredential} and internal/{clusterprovider, nodefirecracker, nodemarket, nodevm}.

Design consequences for plans 0006/0007

Confirmed (the plan can state these as proven on Linux/systemd 259)

  1. One VM per transient service in the user manager. It is created by StartTransientUnit from the bidder and survives the bidder's restart, kill -9 and stop.
    • Plain children (Pdeathsig or setsid) cannot survive; units are the only mechanism.
    • A bidder that restarts must adopt existing socialweb-vm-* units (ListUnitsByPatterns), never re-create them. Re-creating fails with "already loaded".
  2. Connect over $XDG_RUNTIME_DIR/systemd/private with a custom go-systemd dialer: EXTERNAL auth, no Hello.
    • It works for StartTransientUnit, JobRemoved, GetUnitByPIDFD (fd passing) and properties.
    • It needs no dbus-broker user instance. It needs XDG_RUNTIME_DIR set in the bidder unit.
    • Always pass a context with a deadline.
  3. Adoption record: {unit, InvocationID, pasta MainPID + start ticks, VMM pid + start ticks, firecracker --id}.
    • Check order: pidfd_open, then stat ticks, then pidfd_send_signal 0, then GetUnitByPIDFD (unit+InvocationID match).
    • Map these to Lost: ESRCH, ticks mismatch, UnixProcessIdUnknown, NoUnitForPID, NoSuchUnit.
    • Map only dial/auth/timeout errors to Unknown.
    • GetUnitByPIDFD works on the VMM child as well as on MainPID.
  4. pasta -> /bin/sh -c 'exec "$0" verb "$1"' -> verb creates the tap and execs firecracker. It leaves exactly pasta + VMM in the cgroup.
    • The tap persists across exec and firecracker's TUNSETIFF attach works.
    • Keep the shell trampoline and say why: SELinux pasta_t may not exec our binaries directly (AVCs recorded).

Changes and additions the plan must make

  1. Stop semantics. The VMM is PID 1 of pasta's PID namespace and firecracker has no SIGTERM handler, so the default systemctl stop hangs until TimeoutStopSec.
    • The plan must specify graceful stop first: PUT /actions SendCtrlAltDel (x86_64, guest needs i8042), then wait for exit.
    • After that, either KillSignal=SIGKILL, or a short TimeoutStopSec= (e.g. 10-15s) plus FinalKillSignal. ExecStop= is not needed.
    • On aarch64 there is no CtrlAltDel. Use an in-guest agent shutdown or go straight to SIGKILL.
  2. Use PropExecStart(argv, false). go-systemd's bool means ignore failure. With true, VM crashes and OOM kills report success, and the unit vanishes.
    • Keep CollectMode=inactive (the default) so Result, ExecMainCode and ExecMainStatus can be read. The bidder records them and then calls ResetFailedUnit.
    • Alternatively, use inactive-or-failed only together with a PropertiesChanged subscription.
  3. Resource properties.
    • MemoryMax = guest mem + VMM overhead, plus MemorySwapMax=0. Otherwise the guest silently swaps.
    • OOMPolicy=stop, TasksMax, CPUQuotaPerSecUSec and Restart=no are enforced.
    • Drop AllowedCPUs/cpuset: it is not delegated, so it is a silent no-op.
    • Drop IOWeight: it has no effect without BFQ or io.cost, and systemd 259.9 resets it when ManagedOOMPreference=avoid.
    • Prod's Delegate= must be verified to include cpu. Stock systemd delegates only memory+pids; Fedora adds cpu/io via uresourced. Otherwise CPUQuota is accepted but not enforced.
  4. systemd-oomd. User app.slice is ManagedOOMMemoryPressure=kill on Fedora.
    • Set ManagedOOMPreference=avoid on VM units (the xattr user.oomd_avoid=1 was observed), or put VMs in a dedicated user slice (e.g. Slice=socialweb-vms.slice) with explicit pressure policy.
    • Whether oomd honors avoid on user-owned cgroups was not stress-tested.
  5. Linger is mandatory. This host has Linger=no. Without linger, the user manager and every VM unit stop at the last logout.
    • Transient units live on tmpfs and do not survive a reboot. After a reboot, adoption must yield Lost for everything, and the plan should say so.
  6. Socket paths must be 107 bytes or fewer. Put the firecracker API socket at $XDG_RUNTIME_DIR/swvm/<short-id>.sock (or a short workdir path) and unlink it before exec. Firecracker refuses a stale file (FailedToBindSocket). Tap names must be 15 bytes or fewer.
  7. Firecracker invocation. Use --config-file cfg.json --api-sock <short> --id <vm-name, [A-Za-z0-9-]{1,64}>.
  • The guest boots immediately. The socket is bound before the build.
  • GET / gives {id,state,vmm_version} for live verification. Cross-check id against the unit record.
  • Do not use --no-api.
  • The existing RunVMM (API-configure, then WriteInstance, then Wait, with a Go parent left in the tree) is replaced by execve. The instance record is then written by the bidder from GetUnitByPIDFD and pidfd data, not by the verb.
  • Firecracker log timestamps are local time without an offset. Run the unit with Environment=TZ=UTC or normalize when parsing.
  1. Binary path. os.Executable() returns the path, not the inode. Rename-over upgrades make the namespace verb run a newer build than its parent, and a deletion gives ENOENT.
  • Install to immutable versioned paths, and put the resolved versioned path into ExecStart.
  • Do not rely on /proc/self/exe across pasta.
  1. Cross-OS build is broken today.
  • darwin needs hostos twins: lock_darwin.go, sysproc_darwin.go, portrange_darwin.go, proc_darwin.go, caps_darwin.go, or unix-tagged shared files. It also needs portallocator/probe_darwin.go.
  • windows needs internal/nodevm/disk.go split or tagged (disk_linux.go + stub).
  • Plans that claim the macOS/Windows bidders build from this tree must include these.

Disproven or corrected statements

  • "Killing pasta tears down the namespace and the VMM." False for a bare process: the VMM is orphaned and keeps running. It is true only inside a unit with KillMode=control-group, where MainPID exit triggers a cgroup kill.
  • "uncleanIsFailure=true makes crashes visible." It is inverted.
  • "systemctl stop stops the VM." It only does so after TimeoutStopSec, unless the stop path is designed (item 5).
  • "The pasta command can be our binary directly." It cannot under SELinux enforcing; the shell trampoline is required.

Not proven here (left to RFP-flow live tests): a real firecracker guest boot from --config-file through the full requester -> RFP -> bid -> accept -> cloud-init path; guest reachability through the relay; CtrlAltDel shutdown timing; oomd behavior under real pressure.

E8 (live, RFP flow, 2026-10-12): guest reachability and CtrlAltDel stop timing

Run: alice-request.sh (requester) -> socialweb-computer-bidder@bobvmbuilder.bsky.social on hat-8, firecracker provider (runner-container era), guest fc-1379be663dd9-6e867147, node image sha256-5b823ce0...

Defect found first (run at 2026-10-11T23:53Z, guest fc-1379be663dd9-762b64d5): the guest's on-network report carried its bare hostname, and the requester polled wss://fc-1379be663dd9-762b64d5/... until its 300 s timeout.

  • oidc-issuer-hono's reporter waited 60 s for /run/guest-fqdn, then fell back to hostname.
  • The iroh listener that writes /run/guest-fqdn was enabled last in runcmd with a blocking systemctl enable --now, after a blocking systemctl enable --now ssh whose start job never reported Started in the first 60 s. Its unit was also ordered After=ssh.service.
  • The requester classified any non-IP, non-iroh string as a relay FQDN.

Fixes, uncommitted on 2026-10-12:

  • atproto-market: a single-label name is kind hostname and is skipped like an IP.
  • cloud-init-common iroh module: After=network-online.target only; iroh is queued --no-block before ssh; ssh is started --no-block; /run/guest-fqdn is written atomically.
  • oidc-issuer-hono reporter: no hostname fallback; waits 600 s; After=iroh.service; logs to journal+console; dumps jobs and unit status when it gives up.

Re-run at 2026-10-12T00:05Z: ticket reported 6.5 s after boot, vm_ssh_ready about 20 s after the request; ssh.service Started at 4.9 s.

CtrlAltDel, timed on that guest through its API socket:

  • PUT /actions SendCtrlAltDel received at 00:06:21.6827 (firecracker log).
  • The guest ran systemd shutdown to reboot.target ("reboot: Restarting system" at guest uptime 65.7 s).
  • "Firecracker exiting successfully. exit_code=0" at 00:06:21.8913.
  • Total 0.21 s for an idle guest. The guest kernel (host Fedora 7.2.8 config) has CONFIG_SERIO_I8042=y and CONFIG_KEYBOARD_ATKBD=y, and ctrl-alt-del.target -> reboot.target, which firecracker treats as VM exit.
  • Grace window still to be measured with a workload (k3s agent + pods).

Defects in today's provider seen during the test:

  • After firecracker exited, the nodeboot parents stayed alive and the runner container stayed Up. The bidder never noticed the VM died, and the requester's ssh session hung until it was closed by hand (then vm.delete 200).
  • Destroyed guests' work dirs are never removed: 32 dirs, 63 GB on disk (1.6 TB apparent, sparse rootfs).

Plans 0007 M2-VM-RUNTIME / M2-CONTROLLER (unit failure -> Lost; work dir removed before the finalizer) address both.