Acceptance 0006: rc rollout checks and compat smoke

Proves a release candidate is live and healthy on every prod component and that the unchanged clients (prod page, iOS app, CLI of the previous rc) still work against it. Run after every rollout; it is the gate before "tests may resume".

• Components: gateway (socialweb.computer), ssh gateway, bidders local (hat-8) / hat-9 / hat-7, page (shell.socialweb.computer)
• Build: local tag v0.0.0-rcN off local main, local build with pin check (no GitHub push, no CI deploy key; installs run from the lead host over ssh)
• Last pass: rc154, 2026-11-01 02:22-02:25Z (all components, 0 restarts); page compat vs rc153 01:17Z

flowchart LR
  ann[announce on atirc<br/>hold VM / phone tests] --> meta{scope change?}
  meta -- yes --> qr[deploy client metadata<br/>wait ~5 min] --> canary
  meta -- no --> canary[canary: local bidder<br/>+ CLI trust run]
  canary -- fail --> rb[rollback slot<br/>halt rc]
  canary -- pass --> h9[hat-9] --> h7[hat-7<br/>re-sign bob if scope] --> gw[gateway + ssh gateway]
  gw --> checks[per-host checks<br/>watchers post] --> compat[compat smoke<br/>page + CLI] --> done[announce done<br/>tests may resume]

Preconditions

• Release built from a local tag; sha256sum of the binary recorded; independent review verdict "no blocking" for every change in it
• Peers told: linux-kcp -> all, wasm: rcN rollout now (gateway, ssh gateway, then bidders); hold VM/phone tests until done. Wait for any in-flight phone / VM run to release first
• Owner-held VMs listed (never restarted under, never released)
• Role-scope change? then qr / client metadata deployed first, metadata cache waited out (~5 min), drives-oauth lined up to re-sign oauthQr identities (hat-7 bob, gateway alice)
• Floor rule: never roll a bidder below rc139 (offering/self writer)

Steps

1. Canary: upgrade the local bidder only (others stay on the old rc), CLI trust run (alice, bidder-tangled-vouch): Held, Released rc=0. Pause the other bidders if the canary must land locally (see 0003)
2. Upgrade hat-9, then hat-7 (upgrade --binary <staged> --sha256 <sha>, then install bidder --handle <handle>; bidder.md)
3. Gateway + ssh gateway (deploy.md)
4. Per-host checks below (host watchers post one CONFIRMED line + one host line each)
5. Compat smoke (below)
6. Announce: rcN done: <components>. tests may resume.

Per-bidder check (run as root on the host, or read the watcher post):

STATE=/var/lib/socialweb-computer/swc-bidder
K=$(sudo -n -u swc-bidder sh -c "ls $STATE/handles/*/kcp/admin.kubeconfig | head -1")
kb() { sudo -n -u swc-bidder env HOME=$STATE KUBECONFIG="$K" kubectl "$@"; }
sudo "$STATE/bin/current/socialweb-computer" version --json | jq -r '.version,.commit'
kb get marketidentities.socialweb.computer bidder -o jsonpath='{.status.phase} {.status.conditions[?(@.type=="SignedIn")].status} {.status.pendingAction}{"\n"}'
kb get bidders.socialweb.computer bidder -o jsonpath='{.spec.policy}{"\n"}'
U='socialweb-computer-bidder@<handle>.service'
sudo journalctl _SYSTEMD_USER_UNIT="$U" --since "<restart time>" | command grep -c '"level":"ERROR"'
sudo journalctl _SYSTEMD_USER_UNIT="$U" --since "<restart time>" | command grep -c 'unsupported type'
kb get computevms.socialweb.computer,firecrackervms.socialweb.computer --no-headers; pgrep -c -x firecracker

Compat smoke (unchanged clients vs new servers):

• Page: one short prod run as hat on a fresh profile (page-check.ts, see 0009 A6): Request VM -> Held -> 60 s -> reload -> Release. Announce "no parallel hat runs"
• CLI: run request --hold --exec "id" (previous rc CLI if the RFP / bid shape changed), Held, Released
• atirc-dev confirms revoke via=account before release in revocations.ndjson

Expected observations

Timings: one bidder restart ~2 min (VMs survive the upgrade); rc154 whole rollout 02:22:05 -> 02:24:37 (no scope change). Watcher CONFIRMED lines land within ~30 s of the restart.

Pass criteria

• Every row above passes on every component
• Compat smoke passes on the unchanged prod page (no page rebuild needed) and the CLI
• No in-flight peer run was broken by the rollout

Cleanup / rollback

• Bidder: rollback swaps bin/current / bin/previous and restarts (bidder.md)
• Page: redeploy the previous page-rcN tag
• ssh gateway on 22: re-enable the legacy unit only as last resort (kept for rollback)
• Release compat smoke VMs; hosts empty

Known pitfalls