~ knotless Search/
↑↓ move↵ open⇧↵ all results

socialweb.computer/socialweb-computer-kcp

socialweb.computer: a compute market on atproto (kcp)

1 collaborator
0
0
README.md

Acceptance runbooks

Runnable end-to-end checks against prod. Each doc: purpose, preconditions, steps, expected observations, pass criteria, cleanup, pitfalls. They change prod state (VMs, contracts, tokens): run only with the lead's go, announce on atirc first, release everything you rent.

Each runbook is also a Rust system test, tests/system/tests/acc_NNNN_*.rs (../../tests/testkit/README.md); cargo run -p testkit --bin last-pass writes a passing run into the last pass column.

idnamewhat it proveshostshow longlast pass
0001k8s: account modes, pause, one VM per hostlocalRepo + oauthQr bidders, per-component suspend, tenant k3s 0 -> 3 -> 0 nodes, one VM per host, typed settings, ACC34gateway, local + a temporary oauthQr bidder (hat-8), hat-9, hat-7~45 minCLI v0.0.0-rc167 fc4c18d9, bidders rc167 fc4c18d9, gateway rc167, 2026-11-14 (0001:run vs prod, 735 s, bob); CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0001:run vs prod, 1486 s, bob); CLI v0.0.0-rc166 a529874b, bidders 23b9c7d9, gateway rc166, 2026-11-14 (0001:run vs prod, 703 s, bob); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0001:run vs prod, 708 s, bob); section 9 rollout 20 (rc165-pre1 038bd32e), 2026-11-13 (local, hat-9, hat-7, alice, no bidders list: restarted ssh 5.7 / 7.1 / 2.0 s, 0 startup-wait timeouts, releases leave 0 records); FAIL e190a3b1 (rollout 19), 2026-11-12: sections 1-8 pass (alice + bob oauthQr, no bidders list; 1 -> 2 -> 3 -> 2 -> 1 -> 0 on local, hat-7, hat-9, A2); section 9 ACC34 restarted ssh ~64 s on all 4 bidders (cdd93727 endpoint-slice wait times out 1 min on every core start); e496cbff, 2026-11-11 (all nine sections; alice + bob oauthQr; 4 bidders, ACC34 on all 4, no bidders list); c770a887, 2026-11-10 (alice + bob; 4 nodes on 4 bidders, ACC34 on all 4); gateway re-check 5e6c4e3d, 2026-11-11 (sections 3-6, 0 -> 3 -> 0 one step at a time on hat-7, hat-8, hat-9)
0002ssh gateway, per-user OAuthssh <handle>@socialweb.computer atirc agent run <name> rents as the user, agent joins, revoke before release on disconnect / cap, restart survivalgateway (ssh gw), any bidder~3 min + sessionCLI v0.0.0-rc167 fc4c18d9, bidders rc167 fc4c18d9, gateway rc167, 2026-11-14 (0002:run vs prod, 605 s, bob); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0002:run vs prod, 613 s, bob); rollout 19 e190a3b1, 2026-11-12 (alice, no bidders list: transient + held atirc agent run, agent answered, revoke before Released both, key-claim refusal); 2026-11-10 transient command, alice (5b8682fd); rc158 2026-11-01 agent run
0003CLI ssh over WebRTC (ACC34)CLI requester rents per host over the bidder's offering endpoints, ssh over WebRTC fresh + restarted kcp, no plaintext signalinglocal, hat-9, hat-7~3 min / hostCLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0003:run vs prod, 1198 s, alice); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0003:run vs prod, 128 s, bob); rollout 20 (rc165-pre1 038bd32e), 2026-11-13 (local, hat-9, hat-7, alice, no bidders list: fresh 4.0 / 2.6 / 9.5 s, restarted 6.7 / 5.2 / 5.5 s, releases leave 0 records); FAIL rollout 19 e190a3b1, 2026-11-12 (local, hat-9, hat-7, alice, no bidders list: all pass except restarted-kcp ssh 65-68 s, cdd93727 endpoint wait); 2026-11-10 hat-9 + hat-7 (69aa8b0f), alice
0004atirc agent in a VM (plan 0035, p2p)no mint; iroh key + agent config (harness, model) in the bundle; the agent a co-located did:plc of the in-VM R2 (ident.link pair + grant agent -> R2 -> renter, peer record in the agent's repo, no lease), guest.agentStart starts atirc mesh run + the harness, messages both ways, the agent answers, release deletes the renter's grant + endorsement and tombstones the DIDsany bidder, an owner mesh node~5 minCLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0004:run vs prod, 252 s, bob); CLI v0.0.0-rc99.0.20261114075518-a529874bde38 a529874b, bidders 23b9c7d9, 2026-11-14 (0004:run vs prod, 256 s, bob); rollout 19 e190a3b1, 2026-11-13 00:20-00:53Z (alice by QR, no bidders list, winner hat-7: association +97 s, online +165 s, answer <10 s, records 14/14, both DIDs tombstoned); FAIL e496cbff, 2026-11-11 (OAuth renter cannot endorse R2: putRecord needs action=update; R2 never gets its association ref without r2-atirc.env; 24 min release); 2026-11-10 (alice, rollout 5, no workarounds, prompt answered)
0005guest requester R2, association (plan 0036)contract-scoped grant + link.ident.endorsement (rkey = R2 DID, after the nonce check) lets R2 rent a nested VM, grandchild agent, outer release cascades, all tokens revoked, records deleted; co-located accounts on the guest requester's PDS, computer.socialweb.peer records without lease, the identlink walk (agent-of, maxDepth), PLC tombstone on release2 bidders, atirc, relay.mini-cloud-0002, plc.directory~10 minCLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0005:r2fail vs prod, 268 s, alice); CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0005:b vs prod, 148 s, bob); CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0005:run vs prod, 226 s, alice); CLI v0.0.0-rc99.0.20261114075518-a529874bde38 a529874b, bidders 23b9c7d9, 2026-11-14 (0005:r2fail vs prod, 271 s, bob); CLI v0.0.0-rc99.0.20261114075518-a529874bde38 a529874b, bidders 23b9c7d9, 2026-11-14 (0005:run vs prod, 232 s, alice); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0005:b vs prod, 137 s, alice); rollout 20 038bd32e, 2026-11-13 (alice, no bidders list: A cascade, B mesh1, step 5 out-of-band loss; all DIDs plc_tombstone, records gone); f482083c, 2026-11-11 (A, B); FAIL e496cbff, 2026-11-11 (association ref not mirrored; R2 --agent refused: R2 handle does not resolve)
0006rc rollout checks + compat smokeevery component on rcN, Ready / SignedIn / 0 ERROR / host empty, unchanged page + CLI still workall~5-15 minCLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0006:compat vs prod, 118 s, bob); CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0006:run vs prod, 2 s, bob); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0006:compat vs prod, 90 s, bob); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0006:run vs prod, 4 s, alice); rollout 20 (rc165-pre1 038bd32e, 24a8b530) every component, 2026-11-13 (0 ERROR, 0 startup-wait timeouts, NRestarts 0; compat smoke rollout 19 CLI, discovery, alice: the old CLI still leaves its release records); rollout 19 (e190a3b1, 66f3bd4d) every component, 2026-11-12 (gateway ERRORs only the pre-existing testaccount-alice-atirc-operator; compat smoke rollout 18 CLI + prod page files, discovery, alice; the CLI release leaves its release market.event + vm.delete); rollout 8 (5e6c4e3d) 2026-11-10
0007bidder-side VM termination (ops)operator delete ends an orphan cleanly: released, vm.terminated pinned, leased tokens dead, host emptyone bidder~1 minCLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0007:run vs prod, 82 s, bob); CLI (devel) a529874b, bidders a529874b (dev build), 2026-11-14 (0007:restart vs staging, 99 s, alice); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0007:run vs prod, 91 s, bob); rollout 19 e190a3b1, 2026-11-12 (hat-8, alice, no bidders list: delete + expiry); e496cbff, 2026-11-11 (hat-8, delete + expiry)
0008host observer verificationper-host watcher protocol: CONFIRMED after rollout, VM timeline, vm.terminated pin, orphan checkeach biddercontinuousCLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0008:run vs prod, 89 s, alice); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0008:run vs prod, 92 s, bob); rollout 19 e190a3b1, 2026-11-12 (hat-7, hat-8, hat-9 CONFIRMED; hat-8 timelines + pins; 0/0/0); rollout 5 438ca0fa, 2026-11-10 (hat-7, hat-9)
0009browser requester pageA1-A3, A6, A7: unit, OAuth sign-in, VM lifecycle over the bidder's WebRTC endpoint (no https), compat, /chatbrowser, bidders~5 min / testCLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0009:run vs prod, 142 s, bob); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0009:run vs prod, 157 s, bob); rollout 20 (rc165-pre1 038bd32e), 2026-11-13 (A1-A3, A6 on hat-7, hat-9, local, A7; alice, no bidders list; hat-9 needed mdns allowed); 2026-11-10 prod page vs hat-9 (438ca0fa), alice
0011atircrepo + deploy gates, token lifecycle gates for the minting paths (ssh gateway, r2), token-less mesh gates for VM agents, pipes, /chat, lease deathirc.fedfork.com, an owner mesh node~10 minCLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0011:token vs prod, 10 s, alice); CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0011:run vs prod, 1 s, alice); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0011:token vs prod, 10 s, bob); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0011:run vs prod, 2 s, alice); atirc e962ee8 + kcp rollout 20 038bd32e, 2026-11-13 (sections 1-4: repo gates, deploy gates over HTTP, ssh gateway mint/JOIN/answer/revoke 1.26 s before termination, /chat signed in + Origin lock, lease death after its 60 s liveness window; not run: prod pipes, revocations.ndjson); 9e83813, 2026-11-11 (sections 1, 2; section 4 FAIL with kcp e496cbff, see 0004); 2026-11-01 (atirc v0.1.0-5a8dfa2); mesh 2026-11-10 (atirc 2898af3)
0012Infra quickstartkubectl apply of every examples/infra file: VM ssh, tenant k8s node + pod, Deno sandbox stdout, agent VM refused before its RFP locally (.local handle; join on prod, 0013), releaseown market network (+ prod for the agent join)~15 minCLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0012:run vs prod, 86 s, alice); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0012:run vs prod, 100 s, alice); rollout 20 release binary (rc165-pre1 038bd32e, 24a8b530) on own network a12r, 2026-11-13 (steps 1-9; tenant node slow: 4 m 40 s behind the guest OIDC issuer); 2026-11-10 eb018e0e; main ec97f78e local, 2026-11-01
0013Mixed prod on a user-configured bidderrun requester / install requester with a real account, mixed-prod.yaml pinned to one bidder under the default policies and both vouches: VM ssh, Deno stdout, agent join, restart keeps Held, delete revokes before vm.terminated; the gateway part waits on plan 0039 Ca user-configured bidder, irc.fedfork.com~20 minCLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0013:c vs prod, 290 s, bob); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0013:run vs prod, 269 s, bob); rollout 20 (rc165-pre1 038bd32e), 2026-11-13, bob, no pin: A, B, C (A6 re-run after a kit-file key fault); rollout 19 e190a3b1 (C), 2026-11-12; rc159 (A, C) and rc160 CLI (B), 2026-11-01
0014accept fallback, committed capacitya bidder refuses an accept (a rental with no bid waits for capacity: giveup, wait) that would overcommit it (409 CapacityExceeded, no receipt); the requester deletes it, denies that bidder and takes the next bid on the same RFP, else a new RFP without the refusers; concurrent rentals never overcommitown market network (staging; SKIP on prod)~4 min / entryCLI (devel) a529874b, bidders a529874b (dev build), 2026-11-14 (0014:backtoback vs staging, 354 s, alice); CLI (devel) a529874b, bidders a529874b (dev build), 2026-11-14 (0014:exhausted vs staging, 109 s, bob); CLI (devel) a529874b, bidders a529874b (dev build), 2026-11-14 (0014:concurrent vs staging, 133 s, alice); CLI (devel) a529874b, bidders a529874b (dev build), 2026-11-14 (0014:run vs staging, 164 s, bob); capt dev network, hat-8, bcf42c0d, 2026-11-13: run (a rental refused by its first two picks, Held on the third), concurrent (4 on 2 bidders, 1 refusal, 16/16 vCPUs), exhausted
0015VM memory headrooma guest at ~95% of its RAM writing with fsync for 5 min keeps its VM: Held, no vm.terminated, no reboot, ComputeVM Running, no host oomKilledany bidder (prod and staging)~7 minCLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0015:run vs prod, 385 s, bob); CLI (devel) a529874b, bidders a529874b (dev build), 2026-11-14 (0015:pressure vs staging, 817 s, alice); CLI (devel) a529874b, bidders a529874b (dev build), 2026-11-14 (0015:run vs staging, 395 s, alice); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0015:run vs prod, 403 s, alice); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0015:run vs prod, 416 s, bob); capt dev network, hat-8, 22c89875, 2026-11-13 (proc backend: the guest side only; prod waits on the gate)
0016requester-chosen VM imagerent with an image {url, sha256}: the bidder verifies kernel + rootfs, boots them, the guest reports the marker; a wrong sha256 is refused before bootone bidder with images~5-10 min (first fetch of a 4.8 GB rootfs)CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0016:badsha vs prod, 3617 s, alice); CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0016:run vs prod, 128 s, alice); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0016:run vs prod, 126 s, bob); CLI v0.0.0-rc99.0.20261114031527-2898680b24ce 2898680b, bidders 2898680b, 2026-11-14 (0016:badsha vs prod, 261 s, bob)
0017release of an invocation whose owner holds an interactive ssh sessiondevsup restart after a SIGKILLed dev down (devsup, host only); a second run request release settles within 60 s while the owner holds a pty ssh session, the owner exits only after Released; with the owner SIGKILLed after the ask the release client takes the invocation overany bidder (prod and staging)~2 min eachCLI v0.0.0-rc99.0.20261114134524-fc4c18d95fa7 fc4c18d9, bidders 4d54b2b4, 2026-11-14 (0017:killedlate vs prod, 106 s, alice); CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0017:half vs prod, 53 s, bob); CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0017:killed vs prod, 86 s, bob); CLI v0.0.0-rc99.0.20261114122108-4d54b2b459d6 4d54b2b4, bidders 4d54b2b4, 2026-11-14 (0017:run vs prod, 87 s, bob); CLI v0.0.0-rc99.0.20261114093436-166aa9055209 166aa905, bidders 23b9c7d9, 2026-11-14 (0017:killedlate vs prod, 99 s, bob); CLI (devel) a529874b, bidders a529874b (dev build), 2026-11-14 (0017:killed vs staging, 105 s, bob); CLI (devel) a529874b, bidders a529874b (dev build), 2026-11-14 (0017:run vs staging, 92 s, alice); CLI v0.0.0-rc99.0.20261114075518-a529874bde38 a529874b, bidders 23b9c7d9, 2026-11-14 (0017:killed vs prod, 87 s, bob); CLI v0.0.0-rc99.0.20261114075518-a529874bde38 a529874b, bidders 23b9c7d9, 2026-11-14 (0017:run vs prod, 90 s, alice); capt dev network, hat-8, 716f4ef4 + 0017 tests, 2026-11-14 (run: old and fix pass; killed: old fails, fix passes; killedlate and half: old fails, fix passes)
0018the shipped binary never uses a tampered downloads treedev verify-assets on a tampered asset cache (a download; a download and its sidecar) heals it from the embedded blob; a copy of the binary with one digit of its embedded build record flipped refusesthe test host (staging only)~1 minCLI of this commit

0011 is a copy; the source is the atirc repo (git@tangled.org:did:plc:vtad4w7ujsvz7t6o6iuvmhjq, docs/acceptance/0001-atirc.md @ 2f9e8a2).

Shared rules

  • Announce before, post done after: <nick> -> all: <test> now; hold VM tests / ... done; tests may resume
  • One driver per account at a time (alice, bob); owner-held VMs are never touched
  • Policies: bidder-tangled-vouch on every prod bidder, requester-tangled-vouch on requesters; never open unless the owner sets it
  • Evidence comes from three sides: requester log, host watcher (0008), atirc-dev (revocations.ndjson); plus requester-repo getRecord at computed keys (RecordNotFound after release or termination)
  • Token order: revoke strictly before vm.terminated; the lease kills it anyway once termination exists
  • No secrets anywhere: no tokens, passwords, cookies, session secrets or creds file contents in any log, terminal or post; never paste a sign-in URL anywhere (the CLI's own log prints the short-lived QR sign-in URL by design); creds files stay mode 0600 and are read only inside the one signing command
  • A workaround is reported as one; the run is re-done without it before the item counts as passed

Which runbook for which change

ChangeRun
any rc to prod0006, 0008 (watchers), then the runbooks of the changed area
bidder market / policy code0003 on all hosts, 0006 compat, 0001 before a milestone
ssh gateway / per-user requester0002 (side port first, then 22)
requester mint / revoke / lease, agent peer records0004, 0005, 0011 section 3, 0009 A3
offering endpoints (iroh / webrtc / https), signed market.event, requester record deletion0003 on every host (hat-9 is iroh and webrtc only), 0005, 0013
association / guest requester0005
page buildrequester repo scripts/deploy-page.sh, then 0009 A1-A3 on prod
Infra / EphemeralInfra, examples, quickstart-local / quickstart-prod0012
run requester / install requester, agent identity, bidder compute kinds or operator sign-in0013
bidder capacity, accept refusal, requester fallback0014 (staging)
VM cgroup limits, QEMU drive options, guest memory0015
run request owner / release / takeover (request.go), devsup restart after a killed stop0017
OAuth role scopere-sign every OAuth identity first (0001 section 9 note, 0006), then 0003 hat-7