~ knotless Search/
↑↓ move↵ open⇧↵ all results

socialweb.computer/socialweb-computer-kcp

socialweb.computer: a compute market on atproto (kcp)

1 collaborator
0
0
60 lines · 18.9 KB Raw History
# Acceptance runbooks

Runnable end-to-end checks against prod. Each doc: purpose, preconditions, steps,
expected observations, pass criteria, cleanup, pitfalls. They change prod state
(VMs, contracts, tokens): run only with the lead's go, announce on atirc first,
release everything you rent.

Each runbook is also a Rust system test, `tests/system/tests/acc_NNNN_*.rs` ([../../tests/testkit/README.md](../../tests/testkit/README.md));
`cargo run -p testkit --bin last-pass` writes a passing run into the last pass column.

| id | name | what it proves | hosts | how long | last pass |
| --- | --- | --- | --- | --- | --- |
| [0001](0001-auth-modes.md) | k8s: account modes, pause, one VM per host | localRepo + oauthQr bidders, per-component suspend, tenant k3s 0 -> 3 -> 0 nodes, one VM per host, typed settings, ACC34 | gateway, local + a temporary oauthQr bidder (hat-8), hat-9, hat-7 | ~45 min | CLI v0.0.0-rc167 fc4c18d9, bidders rc167 fc4c18d9, gateway rc167, 2026-11-14 (`0001:run` vs prod, 735 s, bob); CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (`0001:run` vs prod, 1486 s, bob); CLI v0.0.0-rc166 a529874b, bidders 23b9c7d9, gateway rc166, 2026-11-14 (`0001:run` vs prod, 703 s, bob); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (`0001:run` vs prod, 708 s, bob); section 9 rollout 20 (rc165-pre1 038bd32e), 2026-11-13 (local, hat-9, hat-7, alice, no bidders list: restarted ssh 5.7 / 7.1 / 2.0 s, 0 startup-wait timeouts, releases leave 0 records); FAIL e190a3b1 (rollout 19), 2026-11-12: sections 1-8 pass (alice + bob oauthQr, no bidders list; 1 -> 2 -> 3 -> 2 -> 1 -> 0 on local, hat-7, hat-9, A2); section 9 ACC34 restarted ssh ~64 s on all 4 bidders (cdd93727 endpoint-slice wait times out 1 min on every core start); e496cbff, 2026-11-11 (all nine sections; alice + bob oauthQr; 4 bidders, ACC34 on all 4, no bidders list); c770a887, 2026-11-10 (alice + bob; 4 nodes on 4 bidders, ACC34 on all 4); gateway re-check 5e6c4e3d, 2026-11-11 (sections 3-6, 0 -> 3 -> 0 one step at a time on hat-7, hat-8, hat-9) |
| [0002](0002-ssh-gateway.md) | ssh gateway, per-user OAuth | `ssh <handle>@socialweb.computer atirc agent run <name>` rents as the user, agent joins, revoke before release on disconnect / cap, restart survival | gateway (ssh gw), any bidder | ~3 min + session | CLI v0.0.0-rc167 fc4c18d9, bidders rc167 fc4c18d9, gateway rc167, 2026-11-14 (`0002:run` vs prod, 605 s, bob); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (`0002:run` vs prod, 613 s, bob); rollout 19 e190a3b1, 2026-11-12 (alice, no bidders list: transient + held `atirc agent run`, agent answered, revoke before Released both, key-claim refusal); 2026-11-10 transient command, alice (5b8682fd); rc158 2026-11-01 agent run |
| [0003](0003-cli-webrtc-ssh.md) | CLI ssh over WebRTC (ACC34) | CLI requester rents per host over the bidder's offering endpoints, ssh over WebRTC fresh + restarted kcp, no plaintext signaling | local, hat-9, hat-7 | ~3 min / host | CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (`0003:run` vs prod, 1198 s, alice); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (`0003:run` vs prod, 128 s, bob); rollout 20 (rc165-pre1 038bd32e), 2026-11-13 (local, hat-9, hat-7, alice, no bidders list: fresh 4.0 / 2.6 / 9.5 s, restarted 6.7 / 5.2 / 5.5 s, releases leave 0 records); FAIL rollout 19 e190a3b1, 2026-11-12 (local, hat-9, hat-7, alice, no bidders list: all pass except restarted-kcp ssh 65-68 s, cdd93727 endpoint wait); 2026-11-10 hat-9 + hat-7 (69aa8b0f), alice |
| [0004](0004-vm-atirc-agent.md) | atirc agent in a VM (plan 0035, p2p) | no mint; iroh key + agent config (harness, model) in the bundle; the agent a co-located did:plc of the in-VM R2 (ident.link pair + grant agent -> R2 -> renter, peer record in the agent's repo, no `lease`), `guest.agentStart` starts `atirc mesh run` + the harness, messages both ways, the agent answers, release deletes the renter's grant + endorsement and tombstones the DIDs | any bidder, an owner mesh node | ~5 min | CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (`0004:run` vs prod, 252 s, bob); CLI v0.0.0-rc99.0.20261114075518-a529874bde38 a529874b, bidders 23b9c7d9, 2026-11-14 (`0004:run` vs prod, 256 s, bob); rollout 19 e190a3b1, 2026-11-13 00:20-00:53Z (alice by QR, no bidders list, winner hat-7: association +97 s, online +165 s, answer <10 s, records 14/14, both DIDs tombstoned); FAIL e496cbff, 2026-11-11 (OAuth renter cannot endorse R2: putRecord needs `action=update`; R2 never gets its association ref without `r2-atirc.env`; 24 min release); 2026-11-10 (alice, rollout 5, no workarounds, prompt answered) |
| [0005](0005-guest-requester-association.md) | guest requester R2, association (plan 0036) | contract-scoped grant + `link.ident.endorsement` (rkey = R2 DID, after the nonce check) lets R2 rent a nested VM, grandchild agent, outer release cascades, all tokens revoked, records deleted; co-located accounts on the guest requester's PDS, `computer.socialweb.peer` records without `lease`, the identlink walk (agent-of, `maxDepth`), PLC tombstone on release | 2 bidders, atirc, relay.mini-cloud-0002, plc.directory | ~10 min | CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (`0005:r2fail` vs prod, 268 s, alice); CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (`0005:b` vs prod, 148 s, bob); CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (`0005:run` vs prod, 226 s, alice); CLI v0.0.0-rc99.0.20261114075518-a529874bde38 a529874b, bidders 23b9c7d9, 2026-11-14 (`0005:r2fail` vs prod, 271 s, bob); CLI v0.0.0-rc99.0.20261114075518-a529874bde38 a529874b, bidders 23b9c7d9, 2026-11-14 (`0005:run` vs prod, 232 s, alice); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (`0005:b` vs prod, 137 s, alice); rollout 20 038bd32e, 2026-11-13 (alice, no bidders list: A cascade, B mesh1, step 5 out-of-band loss; all DIDs `plc_tombstone`, records gone); f482083c, 2026-11-11 (A, B); FAIL e496cbff, 2026-11-11 (association ref not mirrored; R2 `--agent` refused: R2 handle does not resolve) |
| [0006](0006-rollout-compat.md) | rc rollout checks + compat smoke | every component on rcN, Ready / SignedIn / 0 ERROR / host empty, unchanged page + CLI still work | all | ~5-15 min | CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (`0006:compat` vs prod, 118 s, bob); CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (`0006:run` vs prod, 2 s, bob); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (`0006:compat` vs prod, 90 s, bob); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (`0006:run` vs prod, 4 s, alice); rollout 20 (rc165-pre1 038bd32e, 24a8b530) every component, 2026-11-13 (0 ERROR, 0 startup-wait timeouts, NRestarts 0; compat smoke rollout 19 CLI, discovery, alice: the old CLI still leaves its release records); rollout 19 (e190a3b1, 66f3bd4d) every component, 2026-11-12 (gateway ERRORs only the pre-existing `testaccount-alice-atirc-operator`; compat smoke rollout 18 CLI + prod page files, discovery, alice; the CLI release leaves its release `market.event` + `vm.delete`); rollout 8 (5e6c4e3d) 2026-11-10 |
| [0007](0007-bidder-termination.md) | bidder-side VM termination (ops) | operator delete ends an orphan cleanly: released, `vm.terminated` pinned, leased tokens dead, host empty | one bidder | ~1 min | CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (`0007:run` vs prod, 82 s, bob); CLI (devel) a529874b, bidders a529874b (dev build), 2026-11-14 (`0007:restart` vs staging, 99 s, alice); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (`0007:run` vs prod, 91 s, bob); rollout 19 e190a3b1, 2026-11-12 (hat-8, alice, no bidders list: delete + expiry); e496cbff, 2026-11-11 (hat-8, delete + expiry) |
| [0008](0008-host-observer.md) | host observer verification | per-host watcher protocol: CONFIRMED after rollout, VM timeline, `vm.terminated` pin, orphan check | each bidder | continuous | CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (`0008:run` vs prod, 89 s, alice); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (`0008:run` vs prod, 92 s, bob); rollout 19 e190a3b1, 2026-11-12 (hat-7, hat-8, hat-9 CONFIRMED; hat-8 timelines + pins; 0/0/0); rollout 5 438ca0fa, 2026-11-10 (hat-7, hat-9) |
| [0009](0009-browser-page.md) | browser requester page | A1-A3, A6, A7: unit, OAuth sign-in, VM lifecycle over the bidder's WebRTC endpoint (no https), compat, `/chat` | browser, bidders | ~5 min / test | CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (`0009:run` vs prod, 142 s, bob); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (`0009:run` vs prod, 157 s, bob); rollout 20 (rc165-pre1 038bd32e), 2026-11-13 (A1-A3, A6 on hat-7, hat-9, local, A7; alice, no bidders list; hat-9 needed mdns allowed); 2026-11-10 prod page vs hat-9 (438ca0fa), alice |
| [0011](0011-atirc.md) | atirc | repo + deploy gates, token lifecycle gates for the minting paths (ssh gateway, r2), token-less mesh gates for VM agents, pipes, `/chat`, lease death | irc.fedfork.com, an owner mesh node | ~10 min | CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (`0011:token` vs prod, 10 s, alice); CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (`0011:run` vs prod, 1 s, alice); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (`0011:token` vs prod, 10 s, bob); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (`0011:run` vs prod, 2 s, alice); atirc e962ee8 + kcp rollout 20 038bd32e, 2026-11-13 (sections 1-4: repo gates, deploy gates over HTTP, ssh gateway mint/JOIN/answer/revoke 1.26 s before termination, `/chat` signed in + Origin lock, lease death after its 60 s liveness window; not run: prod pipes, revocations.ndjson); 9e83813, 2026-11-11 (sections 1, 2; section 4 FAIL with kcp e496cbff, see 0004); 2026-11-01 (atirc v0.1.0-5a8dfa2); mesh 2026-11-10 (atirc 2898af3) |
| [0012](0012-infra-quickstart.md) | Infra quickstart | kubectl apply of every `examples/infra` file: VM ssh, tenant k8s node + pod, Deno sandbox stdout, agent VM refused before its RFP locally (`.local` handle; join on prod, 0013), release | own market network (+ prod for the agent join) | ~15 min | CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (`0012:run` vs prod, 86 s, alice); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (`0012:run` vs prod, 100 s, alice); rollout 20 release binary (rc165-pre1 038bd32e, 24a8b530) on own network `a12r`, 2026-11-13 (steps 1-9; tenant node slow: 4 m 40 s behind the guest OIDC issuer); 2026-11-10 eb018e0e; main ec97f78e local, 2026-11-01 |
| [0013](0013-mixed-prod.md) | Mixed prod on a user-configured bidder | `run requester` / `install requester` with a real account, `mixed-prod.yaml` pinned to one bidder under the default policies and both vouches: VM ssh, Deno stdout, agent join, restart keeps Held, delete revokes before `vm.terminated`; the gateway part waits on plan 0039 C | a user-configured bidder, irc.fedfork.com | ~20 min | CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (`0013:c` vs prod, 290 s, bob); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (`0013:run` vs prod, 269 s, bob); rollout 20 (rc165-pre1 038bd32e), 2026-11-13, bob, no pin: A, B, C (A6 re-run after a kit-file key fault); rollout 19 e190a3b1 (C), 2026-11-12; rc159 (A, C) and rc160 CLI (B), 2026-11-01 |
| [0014](0014-accept-fallback.md) | accept fallback, committed capacity | a bidder refuses an accept (a rental with no bid waits for capacity: giveup, wait) that would overcommit it (409 `CapacityExceeded`, no receipt); the requester deletes it, denies that bidder and takes the next bid on the same RFP, else a new RFP without the refusers; concurrent rentals never overcommit | own market network (staging; SKIP on prod) | ~4 min / entry | CLI (devel) a529874b, bidders a529874b (dev build), 2026-11-14 (`0014:backtoback` vs staging, 354 s, alice); CLI (devel) a529874b, bidders a529874b (dev build), 2026-11-14 (`0014:exhausted` vs staging, 109 s, bob); CLI (devel) a529874b, bidders a529874b (dev build), 2026-11-14 (`0014:concurrent` vs staging, 133 s, alice); CLI (devel) a529874b, bidders a529874b (dev build), 2026-11-14 (`0014:run` vs staging, 164 s, bob); capt dev network, hat-8, bcf42c0d, 2026-11-13: run (a rental refused by its first two picks, Held on the third), concurrent (4 on 2 bidders, 1 refusal, 16/16 vCPUs), exhausted |
| [0015](0015-vm-memory.md) | VM memory headroom | a guest at ~95% of its RAM writing with fsync for 5 min keeps its VM: Held, no vm.terminated, no reboot, ComputeVM Running, no host oomKilled | any bidder (prod and staging) | ~7 min | CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (`0015:run` vs prod, 385 s, bob); CLI (devel) a529874b, bidders a529874b (dev build), 2026-11-14 (`0015:pressure` vs staging, 817 s, alice); CLI (devel) a529874b, bidders a529874b (dev build), 2026-11-14 (`0015:run` vs staging, 395 s, alice); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (`0015:run` vs prod, 403 s, alice); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (`0015:run` vs prod, 416 s, bob); capt dev network, hat-8, 22c89875, 2026-11-13 (proc backend: the guest side only; prod waits on the gate) |
| [0016](0016-vm-image.md) | requester-chosen VM image | rent with an image `{url, sha256}`: the bidder verifies kernel + rootfs, boots them, the guest reports the marker; a wrong sha256 is refused before boot | one bidder with `images` | ~5-10 min (first fetch of a 4.8 GB rootfs) | CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (`0016:badsha` vs prod, 3617 s, alice); CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (`0016:run` vs prod, 128 s, alice); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (`0016:run` vs prod, 126 s, bob); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (`0016:badsha` vs prod, 261 s, bob) |
| [0017](0017-owner-release.md) | release of an invocation whose owner holds an interactive ssh session | devsup restart after a SIGKILLed dev down (devsup, host only); a second `run request release` settles within 60 s while the owner holds a pty ssh session, the owner exits only after Released; with the owner SIGKILLed after the ask the release client takes the invocation over | any bidder (prod and staging) | ~2 min each | CLI v0.0.0-rc99.0.20261114134524-fc4c18d95fa7 fc4c18d9, bidders 4d54b2b4, 2026-11-14 (`0017:killedlate` vs prod, 106 s, alice); CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (`0017:half` vs prod, 53 s, bob); CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (`0017:killed` vs prod, 86 s, bob); CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (`0017:run` vs prod, 87 s, bob); CLI v0.0.0-rc99.0.20261114093436-166aa9055209 166aa905, bidders 23b9c7d9, 2026-11-14 (`0017:killedlate` vs prod, 99 s, bob); CLI (devel) a529874b, bidders a529874b (dev build), 2026-11-14 (`0017:killed` vs staging, 105 s, bob); CLI (devel) a529874b, bidders a529874b (dev build), 2026-11-14 (`0017:run` vs staging, 92 s, alice); CLI v0.0.0-rc99.0.20261114075518-a529874bde38 a529874b, bidders 23b9c7d9, 2026-11-14 (`0017:killed` vs prod, 87 s, bob); CLI v0.0.0-rc99.0.20261114075518-a529874bde38 a529874b, bidders 23b9c7d9, 2026-11-14 (`0017:run` vs prod, 90 s, alice); capt dev network, hat-8, 716f4ef4 + 0017 tests, 2026-11-14 (run: old and fix pass; killed: old fails, fix passes; killedlate and half: old fails, fix passes) |
| [0018](0018-downloads-integrity.md) | the shipped binary never uses a tampered downloads tree | `dev verify-assets` on a tampered asset cache (a download; a download and its sidecar) heals it from the embedded blob; a copy of the binary with one digit of its embedded build record flipped refuses | the test host (staging only) | ~1 min | CLI of this commit |

0011 is a copy; the source is the atirc repo (`git@tangled.org:did:plc:vtad4w7ujsvz7t6o6iuvmhjq`,
`docs/acceptance/0001-atirc.md` @ `2f9e8a2`).

## Shared rules

- Announce before, post done after: `<nick> -> all: <test> now; hold VM tests` / `... done; tests may resume`
- One driver per account at a time (alice, bob); owner-held VMs are never touched
- Policies: `bidder-tangled-vouch` on every prod bidder, `requester-tangled-vouch` on requesters; never `open` unless the owner sets it
- Evidence comes from three sides: requester log, host watcher ([0008](0008-host-observer.md)), atirc-dev (`revocations.ndjson`); plus requester-repo getRecord at computed keys (RecordNotFound after release or termination)
- Token order: revoke strictly before `vm.terminated`; the lease kills it anyway once termination exists
- No secrets anywhere: no tokens, passwords, cookies, session secrets or creds file contents in any log, terminal or post; never paste a sign-in URL anywhere (the CLI's own log prints the short-lived QR sign-in URL by design); creds files stay mode 0600 and are read only inside the one signing command
- A workaround is reported as one; the run is re-done without it before the item counts as passed

## Which runbook for which change

| Change | Run |
| --- | --- |
| any rc to prod | 0006, 0008 (watchers), then the runbooks of the changed area |
| bidder market / policy code | 0003 on all hosts, 0006 compat, 0001 before a milestone |
| ssh gateway / per-user requester | 0002 (side port first, then 22) |
| requester mint / revoke / lease, agent peer records | 0004, 0005, 0011 section 3, 0009 A3 |
| offering endpoints (iroh / webrtc / https), signed market.event, requester record deletion | 0003 on every host (hat-9 is iroh and webrtc only), 0005, 0013 |
| association / guest requester | 0005 |
| page build | requester repo `scripts/deploy-page.sh`, then 0009 A1-A3 on prod |
| Infra / EphemeralInfra, examples, quickstart-local / quickstart-prod | 0012 |
| run requester / install requester, agent identity, bidder compute kinds or operator sign-in | 0013 |
| bidder capacity, accept refusal, requester fallback | 0014 (staging) |
| VM cgroup limits, QEMU drive options, guest memory | 0015 |
| `run request` owner / release / takeover (request.go), devsup restart after a killed stop | 0017 |
| OAuth role scope | re-sign every OAuth identity first (0001 section 9 note, 0006), then 0003 hat-7 |