~ knotless Search/
↑↓ move↵ open⇧↵ all results
0
0
main
84 lines · 4.1 KB Raw History

Acceptance 0007: bidder-side VM termination (ops)

Proves a bidder operator can end a VM whose requester cannot release it (tab or app gone, cascade gap, stray debug VM) and that the delete leaves nothing behind: the contract is released, vm.terminated is pinned in the bidder repo, and every atirc token leased to the contract dies.

  • Procedure: bidder.md, Terminate a VM from the bidder
  • Used on prod: R2 orphan (hat-7, 2026-10-30), iOS cross-restart retry VM (hat-7, 2026-10-31 02:07Z), stray debug VM (hat-7, 2026-10-31 18:11Z)
  • Last pass: 2026-10-31 (hat-7)
sequenceDiagram
  participant Op as operator (bidder host)
  participant K as bidder kcp
  participant P as bidder repo
  participant Rq as requester (maybe gone)
  participant A as atirc
  Op->>K: delete ComputeVM fc-<did12>-<hash> --wait=false
  K->>P: termination StateSync + vm.terminated (reason deleted_on_request)
  K->>K: stop VM process, unit, work dir; release contract
  K-->>Rq: vm.terminated event (404 = requester gone, fine)
  A->>P: lease check sees vm.terminated -> token dead (latched)
  K->>K: drop finalizer, ComputeVM gone

Preconditions

  • Authority: the owner asked for it, or the VM is matched to a contract you are ending (stray test / orphan). Never an owner-held VM unless the owner says so
  • Shell on the bidder host with sudo; announce on atirc: <nick>: deleting <vm> on <host> out of band (contract <name>)
  • Know the requester DID and the contract (kb get computevms ... -o wide, watcher timeline)

Steps

K=$(sudo -n -u swc-bidder sh -c 'ls /var/lib/socialweb-computer/swc-bidder/handles/*/kcp/admin.kubeconfig | head -1')
kb() { sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG="$K" kubectl "$@"; }
kb get computevms.socialweb.computer -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,CREATED:.metadata.creationTimestamp
# name = fc-<first 12 chars of requester DID after did:plc:>-<hash>; match DID prefix AND creation time
kb delete computevms.socialweb.computer <name> --wait=false
kb get computevms.socialweb.computer -w      # until it is gone

Then pin vm.terminated (getRecord on the bidder repo, see 0008) and ask atirc-dev to confirm the leased token is dead.

Expected observations

Bidder journal, in order (hat-7 example, 2026-10-31 22:51Z):

LineTime
termination StateSync termination-<hash>22:51:38.49
Running -> Releasing
released22:51:39.75 (+1.3 s)
released: the backend is gone and the finalizer is dropped22:51:39.81
a released bid is marked terminated
vm.terminated record pinned, reason deleted_on_request
  • Requester side: vm.terminated delivery may 404 (requester gone): record in the bidder repo still stands
  • atirc: next use of any token leased on the contract -> 401 lease is not alive, latched (no revival if the record later disappears)

Pass criteria

  • ComputeVM gone within seconds; contract released
  • vm.terminated exists in the bidder repo with reason deleted_on_request
  • Host clean: kb get computevms,firecrackervms "No resources found", pgrep -c -x firecracker 0, sudo ls -A <handle>/vms | wc -l 0
  • Leased tokens dead (atirc-dev check, or a probe with the token gets 401)
  • Unleased (plain) tokens of the contract revoked manually (atirc-dev, by subhandle)

Cleanup

  • Post the timeline and the pinned record on atirc (split lines < 300 chars)
  • Note the root cause of the orphan (why the requester did not release) as an open item; out-of-band delete is a mitigation, not the fix

Known pitfalls

SymptomCause / fix
deleted the wrong VMmatched by DID prefix only; always match DID prefix + creation time + contract
vms/ count 0 while a VM runsdir is 0700; read with sudo
token still answerstoken was plain (pre-lease or test mint); revoke it by subhandle
ComputeVM Lost after uninstall --forceexpected: forced uninstall does not run the finalizer path; do not use it to end one VM
requester resumes later and re-releasesharmless: release is idempotent; the contract is already Released