1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
# Acceptance 0007: bidder-side VM termination (ops)
Proves a bidder operator can end a VM whose requester cannot release it (tab or app
gone, cascade gap, stray debug VM) and that the delete leaves nothing behind: the
contract is released, `vm.terminated` is pinned in the bidder repo, and every atirc
token leased to the contract dies.
- Procedure: [bidder.md, Terminate a VM from the bidder](../bidder.md#terminate-a-vm-from-the-bidder)
- Used on prod: R2 orphan (hat-7, 2026-10-30), iOS cross-restart retry VM (hat-7, 2026-10-31 02:07Z), stray debug VM (hat-7, 2026-10-31 18:11Z)
- Last pass: 2026-10-31 (hat-7)
```mermaid
sequenceDiagram
participant Op as operator (bidder host)
participant K as bidder kcp
participant P as bidder repo
participant Rq as requester (maybe gone)
participant A as atirc
Op->>K: delete ComputeVM fc-<did12>-<hash> --wait=false
K->>P: termination StateSync + vm.terminated (reason deleted_on_request)
K->>K: stop VM process, unit, work dir; release contract
K-->>Rq: vm.terminated event (404 = requester gone, fine)
A->>P: lease check sees vm.terminated -> token dead (latched)
K->>K: drop finalizer, ComputeVM gone
```
## Preconditions
- Authority: the owner asked for it, or the VM is matched to a contract you are ending (stray test / orphan). Never an owner-held VM unless the owner says so
- Shell on the bidder host with sudo; announce on atirc: `<nick>: deleting <vm> on <host> out of band (contract <name>)`
- Know the requester DID and the contract (`kb get computevms ... -o wide`, watcher timeline)
## Steps
```bash
K=$(sudo -n -u swc-bidder sh -c 'ls /var/lib/socialweb-computer/swc-bidder/handles/*/kcp/admin.kubeconfig | head -1')
kb() { sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG="$K" kubectl "$@"; }
kb get computevms.socialweb.computer -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,CREATED:.metadata.creationTimestamp
# name = fc-<first 12 chars of requester DID after did:plc:>-<hash>; match DID prefix AND creation time
kb delete computevms.socialweb.computer <name> --wait=false
kb get computevms.socialweb.computer -w # until it is gone
```
Then pin `vm.terminated` (getRecord on the bidder repo, see [0008](0008-host-observer.md)) and ask atirc-dev
to confirm the leased token is dead.
## Expected observations
Bidder journal, in order (hat-7 example, 2026-10-31 22:51Z):
| Line | Time |
| --- | --- |
| termination StateSync `termination-<hash>` | 22:51:38.49 |
| `Running -> Releasing` | |
| `released` | 22:51:39.75 (+1.3 s) |
| `released: the backend is gone and the finalizer is dropped` | 22:51:39.81 |
| `a released bid is marked terminated` | |
| `vm.terminated` record pinned, reason `deleted_on_request` | |
- Requester side: `vm.terminated` delivery may 404 (requester gone): record in the bidder repo still stands
- atirc: next use of any token leased on the contract -> 401 `lease is not alive`, latched (no revival if the record later disappears)
## Pass criteria
- ComputeVM gone within seconds; contract released
- `vm.terminated` exists in the bidder repo with reason `deleted_on_request`
- Host clean: `kb get computevms,firecrackervms` "No resources found", `pgrep -c -x firecracker` 0, `sudo ls -A <handle>/vms | wc -l` 0
- Leased tokens dead (atirc-dev check, or a probe with the token gets 401)
- Unleased (plain) tokens of the contract revoked manually (atirc-dev, by subhandle)
## Cleanup
- Post the timeline and the pinned record on atirc (split lines < 300 chars)
- Note the root cause of the orphan (why the requester did not release) as an open item; out-of-band delete is a mitigation, not the fix
## Known pitfalls
| Symptom | Cause / fix |
| --- | --- |
| deleted the wrong VM | matched by DID prefix only; always match DID prefix + creation time + contract |
| `vms/` count 0 while a VM runs | dir is 0700; read with sudo |
| token still answers | token was plain (pre-lease or test mint); revoke it by subhandle |
| ComputeVM `Lost` after `uninstall --force` | expected: forced uninstall does not run the finalizer path; do not use it to end one VM |
| requester resumes later and re-releases | harmless: release is idempotent; the contract is already Released |
