0
0
Acceptance runbooks
Runnable end-to-end checks against prod. Each doc: purpose, preconditions, steps, expected observations, pass criteria, cleanup, pitfalls. They change prod state (VMs, contracts, tokens): run only with the lead's go, announce on atirc first, release everything you rent.
Each runbook is also a Rust system test, tests/system/tests/acc_NNNN_*.rs (../../tests/testkit/README.md);
cargo run -p testkit --bin last-pass writes a passing run into the last pass column.
| id | name | what it proves | hosts | how long | last pass |
|---|---|---|---|---|---|
| 0001 | k8s: account modes, pause, one VM per host | localRepo + oauthQr bidders, per-component suspend, tenant k3s 0 -> 3 -> 0 nodes, one VM per host, typed settings, ACC34 | gateway, local + a temporary oauthQr bidder (hat-8), hat-9, hat-7 | ~45 min | CLI v0.0.0-rc167 fc4c18d9, bidders rc167 fc4c18d9, gateway rc167, 2026-11-14 (0001:run vs prod, 735 s, bob); CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0001:run vs prod, 1486 s, bob); CLI v0.0.0-rc166 a529874b, bidders 23b9c7d9, gateway rc166, 2026-11-14 (0001:run vs prod, 703 s, bob); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0001:run vs prod, 708 s, bob); section 9 rollout 20 (rc165-pre1 038bd32e), 2026-11-13 (local, hat-9, hat-7, alice, no bidders list: restarted ssh 5.7 / 7.1 / 2.0 s, 0 startup-wait timeouts, releases leave 0 records); FAIL e190a3b1 (rollout 19), 2026-11-12: sections 1-8 pass (alice + bob oauthQr, no bidders list; 1 -> 2 -> 3 -> 2 -> 1 -> 0 on local, hat-7, hat-9, A2); section 9 ACC34 restarted ssh ~64 s on all 4 bidders (cdd93727 endpoint-slice wait times out 1 min on every core start); e496cbff, 2026-11-11 (all nine sections; alice + bob oauthQr; 4 bidders, ACC34 on all 4, no bidders list); c770a887, 2026-11-10 (alice + bob; 4 nodes on 4 bidders, ACC34 on all 4); gateway re-check 5e6c4e3d, 2026-11-11 (sections 3-6, 0 -> 3 -> 0 one step at a time on hat-7, hat-8, hat-9) |
| 0002 | ssh gateway, per-user OAuth | ssh <handle>@socialweb.computer atirc agent run <name> rents as the user, agent joins, revoke before release on disconnect / cap, restart survival | gateway (ssh gw), any bidder | ~3 min + session | CLI v0.0.0-rc167 fc4c18d9, bidders rc167 fc4c18d9, gateway rc167, 2026-11-14 (0002:run vs prod, 605 s, bob); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0002:run vs prod, 613 s, bob); rollout 19 e190a3b1, 2026-11-12 (alice, no bidders list: transient + held atirc agent run, agent answered, revoke before Released both, key-claim refusal); 2026-11-10 transient command, alice (5b8682fd); rc158 2026-11-01 agent run |
| 0003 | CLI ssh over WebRTC (ACC34) | CLI requester rents per host over the bidder's offering endpoints, ssh over WebRTC fresh + restarted kcp, no plaintext signaling | local, hat-9, hat-7 | ~3 min / host | CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0003:run vs prod, 1198 s, alice); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0003:run vs prod, 128 s, bob); rollout 20 (rc165-pre1 038bd32e), 2026-11-13 (local, hat-9, hat-7, alice, no bidders list: fresh 4.0 / 2.6 / 9.5 s, restarted 6.7 / 5.2 / 5.5 s, releases leave 0 records); FAIL rollout 19 e190a3b1, 2026-11-12 (local, hat-9, hat-7, alice, no bidders list: all pass except restarted-kcp ssh 65-68 s, cdd93727 endpoint wait); 2026-11-10 hat-9 + hat-7 (69aa8b0f), alice |
| 0004 | atirc agent in a VM (plan 0035, p2p) | no mint; iroh key + agent config (harness, model) in the bundle; the agent a co-located did:plc of the in-VM R2 (ident.link pair + grant agent -> R2 -> renter, peer record in the agent's repo, no lease), guest.agentStart starts atirc mesh run + the harness, messages both ways, the agent answers, release deletes the renter's grant + endorsement and tombstones the DIDs | any bidder, an owner mesh node | ~5 min | CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0004:run vs prod, 252 s, bob); CLI v0.0.0-rc99.0.20261114075518-a529874bde38 a529874b, bidders 23b9c7d9, 2026-11-14 (0004:run vs prod, 256 s, bob); rollout 19 e190a3b1, 2026-11-13 00:20-00:53Z (alice by QR, no bidders list, winner hat-7: association +97 s, online +165 s, answer <10 s, records 14/14, both DIDs tombstoned); FAIL e496cbff, 2026-11-11 (OAuth renter cannot endorse R2: putRecord needs action=update; R2 never gets its association ref without r2-atirc.env; 24 min release); 2026-11-10 (alice, rollout 5, no workarounds, prompt answered) |
| 0005 | guest requester R2, association (plan 0036) | contract-scoped grant + link.ident.endorsement (rkey = R2 DID, after the nonce check) lets R2 rent a nested VM, grandchild agent, outer release cascades, all tokens revoked, records deleted; co-located accounts on the guest requester's PDS, computer.socialweb.peer records without lease, the identlink walk (agent-of, maxDepth), PLC tombstone on release | 2 bidders, atirc, relay.mini-cloud-0002, plc.directory | ~10 min | CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0005:r2fail vs prod, 268 s, alice); CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0005:b vs prod, 148 s, bob); CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0005:run vs prod, 226 s, alice); CLI v0.0.0-rc99.0.20261114075518-a529874bde38 a529874b, bidders 23b9c7d9, 2026-11-14 (0005:r2fail vs prod, 271 s, bob); CLI v0.0.0-rc99.0.20261114075518-a529874bde38 a529874b, bidders 23b9c7d9, 2026-11-14 (0005:run vs prod, 232 s, alice); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0005:b vs prod, 137 s, alice); rollout 20 038bd32e, 2026-11-13 (alice, no bidders list: A cascade, B mesh1, step 5 out-of-band loss; all DIDs plc_tombstone, records gone); f482083c, 2026-11-11 (A, B); FAIL e496cbff, 2026-11-11 (association ref not mirrored; R2 --agent refused: R2 handle does not resolve) |
| 0006 | rc rollout checks + compat smoke | every component on rcN, Ready / SignedIn / 0 ERROR / host empty, unchanged page + CLI still work | all | ~5-15 min | CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0006:compat vs prod, 118 s, bob); CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0006:run vs prod, 2 s, bob); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0006:compat vs prod, 90 s, bob); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0006:run vs prod, 4 s, alice); rollout 20 (rc165-pre1 038bd32e, 24a8b530) every component, 2026-11-13 (0 ERROR, 0 startup-wait timeouts, NRestarts 0; compat smoke rollout 19 CLI, discovery, alice: the old CLI still leaves its release records); rollout 19 (e190a3b1, 66f3bd4d) every component, 2026-11-12 (gateway ERRORs only the pre-existing testaccount-alice-atirc-operator; compat smoke rollout 18 CLI + prod page files, discovery, alice; the CLI release leaves its release market.event + vm.delete); rollout 8 (5e6c4e3d) 2026-11-10 |
| 0007 | bidder-side VM termination (ops) | operator delete ends an orphan cleanly: released, vm.terminated pinned, leased tokens dead, host empty | one bidder | ~1 min | CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0007:run vs prod, 82 s, bob); CLI (devel) a529874b, bidders a529874b (dev build), 2026-11-14 (0007:restart vs staging, 99 s, alice); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0007:run vs prod, 91 s, bob); rollout 19 e190a3b1, 2026-11-12 (hat-8, alice, no bidders list: delete + expiry); e496cbff, 2026-11-11 (hat-8, delete + expiry) |
| 0008 | host observer verification | per-host watcher protocol: CONFIRMED after rollout, VM timeline, vm.terminated pin, orphan check | each bidder | continuous | CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0008:run vs prod, 89 s, alice); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0008:run vs prod, 92 s, bob); rollout 19 e190a3b1, 2026-11-12 (hat-7, hat-8, hat-9 CONFIRMED; hat-8 timelines + pins; 0/0/0); rollout 5 438ca0fa, 2026-11-10 (hat-7, hat-9) |
| 0009 | browser requester page | A1-A3, A6, A7: unit, OAuth sign-in, VM lifecycle over the bidder's WebRTC endpoint (no https), compat, /chat | browser, bidders | ~5 min / test | CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0009:run vs prod, 142 s, bob); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0009:run vs prod, 157 s, bob); rollout 20 (rc165-pre1 038bd32e), 2026-11-13 (A1-A3, A6 on hat-7, hat-9, local, A7; alice, no bidders list; hat-9 needed mdns allowed); 2026-11-10 prod page vs hat-9 (438ca0fa), alice |
| 0011 | atirc | repo + deploy gates, token lifecycle gates for the minting paths (ssh gateway, r2), token-less mesh gates for VM agents, pipes, /chat, lease death | irc.fedfork.com, an owner mesh node | ~10 min | CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0011:token vs prod, 10 s, alice); CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0011:run vs prod, 1 s, alice); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0011:token vs prod, 10 s, bob); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0011:run vs prod, 2 s, alice); atirc e962ee8 + kcp rollout 20 038bd32e, 2026-11-13 (sections 1-4: repo gates, deploy gates over HTTP, ssh gateway mint/JOIN/answer/revoke 1.26 s before termination, /chat signed in + Origin lock, lease death after its 60 s liveness window; not run: prod pipes, revocations.ndjson); 9e83813, 2026-11-11 (sections 1, 2; section 4 FAIL with kcp e496cbff, see 0004); 2026-11-01 (atirc v0.1.0-5a8dfa2); mesh 2026-11-10 (atirc 2898af3) |
| 0012 | Infra quickstart | kubectl apply of every examples/infra file: VM ssh, tenant k8s node + pod, Deno sandbox stdout, agent VM refused before its RFP locally (.local handle; join on prod, 0013), release | own market network (+ prod for the agent join) | ~15 min | CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0012:run vs prod, 86 s, alice); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0012:run vs prod, 100 s, alice); rollout 20 release binary (rc165-pre1 038bd32e, 24a8b530) on own network a12r, 2026-11-13 (steps 1-9; tenant node slow: 4 m 40 s behind the guest OIDC issuer); 2026-11-10 eb018e0e; main ec97f78e local, 2026-11-01 |
| 0013 | Mixed prod on a user-configured bidder | run requester / install requester with a real account, mixed-prod.yaml pinned to one bidder under the default policies and both vouches: VM ssh, Deno stdout, agent join, restart keeps Held, delete revokes before vm.terminated; the gateway part waits on plan 0039 C | a user-configured bidder, irc.fedfork.com | ~20 min | CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0013:c vs prod, 290 s, bob); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0013:run vs prod, 269 s, bob); rollout 20 (rc165-pre1 038bd32e), 2026-11-13, bob, no pin: A, B, C (A6 re-run after a kit-file key fault); rollout 19 e190a3b1 (C), 2026-11-12; rc159 (A, C) and rc160 CLI (B), 2026-11-01 |
| 0014 | accept fallback, committed capacity | a bidder refuses an accept (a rental with no bid waits for capacity: giveup, wait) that would overcommit it (409 CapacityExceeded, no receipt); the requester deletes it, denies that bidder and takes the next bid on the same RFP, else a new RFP without the refusers; concurrent rentals never overcommit | own market network (staging; SKIP on prod) | ~4 min / entry | CLI (devel) a529874b, bidders a529874b (dev build), 2026-11-14 (0014:backtoback vs staging, 354 s, alice); CLI (devel) a529874b, bidders a529874b (dev build), 2026-11-14 (0014:exhausted vs staging, 109 s, bob); CLI (devel) a529874b, bidders a529874b (dev build), 2026-11-14 (0014:concurrent vs staging, 133 s, alice); CLI (devel) a529874b, bidders a529874b (dev build), 2026-11-14 (0014:run vs staging, 164 s, bob); capt dev network, hat-8, bcf42c0d, 2026-11-13: run (a rental refused by its first two picks, Held on the third), concurrent (4 on 2 bidders, 1 refusal, 16/16 vCPUs), exhausted |
| 0015 | VM memory headroom | a guest at ~95% of its RAM writing with fsync for 5 min keeps its VM: Held, no vm.terminated, no reboot, ComputeVM Running, no host oomKilled | any bidder (prod and staging) | ~7 min | CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0015:run vs prod, 385 s, bob); CLI (devel) a529874b, bidders a529874b (dev build), 2026-11-14 (0015:pressure vs staging, 817 s, alice); CLI (devel) a529874b, bidders a529874b (dev build), 2026-11-14 (0015:run vs staging, 395 s, alice); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0015:run vs prod, 403 s, alice); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0015:run vs prod, 416 s, bob); capt dev network, hat-8, 22c89875, 2026-11-13 (proc backend: the guest side only; prod waits on the gate) |
| 0016 | requester-chosen VM image | rent with an image {url, sha256}: the bidder verifies kernel + rootfs, boots them, the guest reports the marker; a wrong sha256 is refused before boot | one bidder with images | ~5-10 min (first fetch of a 4.8 GB rootfs) | CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0016:badsha vs prod, 3617 s, alice); CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0016:run vs prod, 128 s, alice); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0016:run vs prod, 126 s, bob); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0016:badsha vs prod, 261 s, bob) |
| 0017 | release of an invocation whose owner holds an interactive ssh session | devsup restart after a SIGKILLed dev down (devsup, host only); a second run request release settles within 60 s while the owner holds a pty ssh session, the owner exits only after Released; with the owner SIGKILLed after the ask the release client takes the invocation over | any bidder (prod and staging) | ~2 min each | CLI v0.0.0-rc99.0.20261114134524-fc4c18d95fa7 fc4c18d9, bidders 4d54b2b4, 2026-11-14 (0017:killedlate vs prod, 106 s, alice); CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0017:half vs prod, 53 s, bob); CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0017:killed vs prod, 86 s, bob); CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0017:run vs prod, 87 s, bob); CLI v0.0.0-rc99.0.20261114093436-166aa9055209 166aa905, bidders 23b9c7d9, 2026-11-14 (0017:killedlate vs prod, 99 s, bob); CLI (devel) a529874b, bidders a529874b (dev build), 2026-11-14 (0017:killed vs staging, 105 s, bob); CLI (devel) a529874b, bidders a529874b (dev build), 2026-11-14 (0017:run vs staging, 92 s, alice); CLI v0.0.0-rc99.0.20261114075518-a529874bde38 a529874b, bidders 23b9c7d9, 2026-11-14 (0017:killed vs prod, 87 s, bob); CLI v0.0.0-rc99.0.20261114075518-a529874bde38 a529874b, bidders 23b9c7d9, 2026-11-14 (0017:run vs prod, 90 s, alice); capt dev network, hat-8, 716f4ef4 + 0017 tests, 2026-11-14 (run: old and fix pass; killed: old fails, fix passes; killedlate and half: old fails, fix passes) |
| 0018 | the shipped binary never uses a tampered downloads tree | dev verify-assets on a tampered asset cache (a download; a download and its sidecar) heals it from the embedded blob; a copy of the binary with one digit of its embedded build record flipped refuses | the test host (staging only) | ~1 min | CLI of this commit |
0011 is a copy; the source is the atirc repo (git@tangled.org:did:plc:vtad4w7ujsvz7t6o6iuvmhjq,
docs/acceptance/0001-atirc.md @ 2f9e8a2).
Shared rules
- Announce before, post done after:
<nick> -> all: <test> now; hold VM tests/... done; tests may resume - One driver per account at a time (alice, bob); owner-held VMs are never touched
- Policies:
bidder-tangled-vouchon every prod bidder,requester-tangled-vouchon requesters; neveropenunless the owner sets it - Evidence comes from three sides: requester log, host watcher (0008), atirc-dev (
revocations.ndjson); plus requester-repo getRecord at computed keys (RecordNotFound after release or termination) - Token order: revoke strictly before
vm.terminated; the lease kills it anyway once termination exists - No secrets anywhere: no tokens, passwords, cookies, session secrets or creds file contents in any log, terminal or post; never paste a sign-in URL anywhere (the CLI's own log prints the short-lived QR sign-in URL by design); creds files stay mode 0600 and are read only inside the one signing command
- A workaround is reported as one; the run is re-done without it before the item counts as passed
Which runbook for which change
| Change | Run |
|---|---|
| any rc to prod | 0006, 0008 (watchers), then the runbooks of the changed area |
| bidder market / policy code | 0003 on all hosts, 0006 compat, 0001 before a milestone |
| ssh gateway / per-user requester | 0002 (side port first, then 22) |
| requester mint / revoke / lease, agent peer records | 0004, 0005, 0011 section 3, 0009 A3 |
| offering endpoints (iroh / webrtc / https), signed market.event, requester record deletion | 0003 on every host (hat-9 is iroh and webrtc only), 0005, 0013 |
| association / guest requester | 0005 |
| page build | requester repo scripts/deploy-page.sh, then 0009 A1-A3 on prod |
| Infra / EphemeralInfra, examples, quickstart-local / quickstart-prod | 0012 |
| run requester / install requester, agent identity, bidder compute kinds or operator sign-in | 0013 |
| bidder capacity, accept refusal, requester fallback | 0014 (staging) |
| VM cgroup limits, QEMU drive options, guest memory | 0015 |
run request owner / release / takeover (request.go), devsup restart after a killed stop | 0017 |
| OAuth role scope | re-sign every OAuth identity first (0001 section 9 note, 0006), then 0003 hat-7 |