~ knotless Search/
↑↓ move↵ open⇧↵ all results

socialweb.computer/socialweb-computer-kcp

socialweb.computer: a compute market on atproto (kcp)

1 collaborator
0
0
657 lines · 53.7 KB Raw History
# CI by hand and debugging a CI run

Run a CI stage by hand, hold a failed run, and fix it in place. Plan: `docs/plans/0040-release-pipeline.md`.

## Rules

- Every workflow step is a script; a hand run calls the same scripts.
- Before a hand run, the market holds none of our rentals (`scripts/ci-hand-market.sh list`).
- The hand requester rents the staging VMs through its associated host requester (prod alice, setup below); the entries
  run as the staging network's own driver accounts alice1 to alice4 (tests/compositions/staging/README.md, Accounts).
- Prod gets code only through CI promote (plan 0040): no hand roll.
- Every change under the workflows' `paths:` is run by hand, off CI, before its pull is queued (plan 0040 decision 51):
  a hand cycle is "One cycle: release, then test" below on the branch's commit (published to `local/<sha>/` only, no
  channel moves), the whole list of `tests/entries.txt` on a fresh staging network. The pull says what was run and
  seen. The bar is the checklist under "Mistakes that cost a cycle".
- A failed run is held (frozen) and fixed in place; its finding goes into `docs/plans/0040-findings.md` (the plan's notes file).

## Prerequisites

- A Linux host with the kcp toolchain: `scripts/build-socialweb-computer.sh` and `tests/publish-test-host.sh` work there.
- `kubectl`, `jq`, `curl`, `git`, `deno`, `node` 22 or newer (else the watch capture fetches one), ssh to the bidder hosts.
- sudo on each bidder host (for `scripts/ci-hand-market.sh`).
- The creds files: the requester account's (`~/.aliceoa_bsky_social_source` shape, used once, at setup step 2) and the
  bucket key (`~/.spaces_swc_kcp_source`: `SPACES_ACCESS_KEY_ID`, `SPACES_SECRET_ACCESS_KEY`, `SPACES_BUCKET`,
  `SPACES_REGION`). Owner-only, never printed.
- A worktree at the branch commit under test (`CI_HAND_WORKTREE`). Nothing else runs in it during a cycle: every
  `ci-release.sh` step reads `git rev-parse HEAD`.

## The hand runner's env file (not in git)

`~/.config/socialweb-ci-hand/env`, mode 0600, paths and public values only. Each runner writes its own.

```sh
CI_ACCOUNT=alice                                  # an account of tests/targets/prod.json (CI: r2)
CI_ACCOUNT_DID=<its DID>
CI_ACCOUNT_CREDS=$HOME/.aliceoa_bsky_social_source
CI_ACCOUNT_PREFIX=ALICE                           # the creds file's variable prefix
CI_IDENTITY_NAME=<name>                           # the associated host requester identity (setup)
CI_IDENTITY_DIR=$HOME/.swc/<name>                 # short: its unix sockets must fit 107 bytes
CI_IDENTITY_DID=<written by setup step 1>
CI_HOME=$HOME/.tmp/ci-hand                        # what CI puts in /home/r2, /usr/local/bin, /work/runs, /root
CI_BUCKET_CREDS=$HOME/.spaces_swc_kcp_source
CI_PLATFORMS=linux/amd64                          # release by hand builds amd64 only; CI builds both
CI_BIDDER_HOSTS="hat-7 hat-8 hat-9 hat-10"        # every prod bidder host (scripts/ci-hand-market.sh)
CI_HAND_WORKTREE=<path of the worktree>
CI_FREEZE_TTL=7200                                # a failed run holds its VMs for the debugging (end it yourself)
```

Every block below starts with `set -a; . ~/.config/socialweb-ci-hand/env; set +a; export TMPDIR=$HOME/.tmp/ci-hand-tmp; cd "$CI_HAND_WORKTREE"`.

## Setup, once per runner

```sh
B=/var/tmp/ci-hand-bin; scripts/build-socialweb-computer.sh --dev --publish $B   # a binary with plan 0046

# 1. the associated host requester: a localRepo identity (its own did:plc and PDS), agent-of the account
$B/bin/socialweb-computer run requester --name stg --state-dir "$CI_IDENTITY_DIR" --identity "$CI_IDENTITY_NAME" \
  --associate-with "$CI_ACCOUNT_DID"                                                 # foreground
jq -r .did "$CI_IDENTITY_DIR/requesters/stg/ready.json"                              # -> CI_IDENTITY_DID in the env file

# 2. the account writes the standing association (market.association + link.ident.endorsement): its one sign-in
deno run -A --no-lock scripts/associate-bidder.ts --creds "$CI_ACCOUNT_CREDS" --account "$CI_ACCOUNT_PREFIX" \
  --agent-did "$CI_IDENTITY_DID"
$B/bin/socialweb-computer internal association "$CI_IDENTITY_DID" | jq -c '{roots, errors}'   # roots [CI_ACCOUNT_DID]

# 3. stop it (SIGINT to the socialweb-computer process itself; it keeps its state)
kill -INT "$(pgrep -f "socialweb-computer run requester --name stg --state-dir $CI_IDENTITY_DIR")"
```

The staging composition names its requester `stg`, and `CI_IDENTITY_DIR` is its state dir from then on (run
one `stg` requester at a time on a host).

## Workaround: seed the asset cache from a hand runner 

A cold CI `release` fetches every pinned upstream file (44+ debs from Launchpad and apt-archive.postgresql.org per
platform); one 503 fails the build. Its restore (`ASSET_CACHE_URL=<bucket>/asset-cache`) was empty: main's `release`
fills it only after a build passes. Seed it once from the hand runner's blob cache, which the hand cycles filled:

```sh
c=~/.tmp/socialweb-computer-build/cache/blobs
sort -u $c/.new | while read -r e; do k="${e##*/}"; [ -f "$c/$k.tar.zst" ] && stat -c %s "$c/$k.tar.zst"; done \
  | awk '{s+=$1; n++} END {printf "%d blobs %.1f GiB\n", n, s/1073741824}'           # 2026-11-16: 535 blobs 8.2 GiB
(set -a; . "$CI_BUCKET_CREDS"; set +a; scripts/asset-cache-upload.sh $c) > ~/.tmp/asset-cache-seed.log 2>&1
grep -c uploaded ~/.tmp/asset-cache-seed.log                                         # 594 files (34 already there)
```

Content-addressed: a key already in the bucket is not sent again. Not needed again once main's `release` passes (its
`asset-cache` step uploads what is new).

## CI's cycle, in series

A push to main that changes a path of the workflows' `paths:` (the same list in all three: `deploy/check.sh`) files all
three checks at once, as queued; each runs once the one before it passed (owner, 2026-11-19: series, less compute):

| Workflow | Waits for | Rents at once (4 vCPU / 8G VMs unless said) | Takes |
| --- | --- | --- | --- |
| `release` | nothing | the build VM, 12 vCPU / 16G | about 22 min |
| `test` | `after: [release]` | the CI VM, then the staging network: atproto, gateway, 2 nested-virt bidders (5 VMs, 20 vCPU / 40G; the entries' guests run inside the bidder VMs) | about 2 h 40 min: 9 min stand-up, then 47 entries |
| `promote` | `after: [release, test]` | the release VM, then the prod gate's rentals: two drivers (bob, alice), at most two entries at once, at most 6 VMs of their weights (2 vCPU / 4G each by default); 0001 alone: one k8s node per bidder | about 100 min: deploy and heartbeats, then 31 entries (about 4840 s scheduled from entries.txt, 5300 to 5800 s expected; three drivers measured 3142 to 4237 s) |

So about 4 h 15 min a cycle. A VM commits its memory plus a tenth (at least 512M): 8.8 GiB for 8G, 4.5 GiB for 4G. The
four prod bidders' offerings (2026-11-19): hat-7 12 cores / 31 GiB, hat-8 16 / 46 GiB (about 33 GiB free: hat-8 builds),
hat-9 8 / 31 GiB, hat-10 4 / 15 GiB (about 7 GiB free: it runs a GitLab, so no 8G VM and one 4G one); 40 cores, 80 vCPUs
at the bidders' 2x, about 100 GiB free. `test` needs 44 GiB of 8G VMs (nine fit: three each on hat-7, 8, 9); `promote`
about 45 GiB. A push to main while a cycle runs starts the next `release` at once and its `test` when this one's ends,
so that `test` and this `promote` rent together (about 89 GiB): the series holds for one commit only (the runner's
`superseded: finish` turn is per workflow name), so land one code change per cycle; closing it needs a runner turn group
over test and promote (not built).

### After a red test

The failed `test` holds its network 30 min for `attach` (above), then releases it. When its check ends, `promote` is filed
`failure` without running ("not run: test, which it runs after, ended failure"; a cancelled `test`: `cancelled`):
nothing reached prod (knotless `crates/knotless-run/src/lib.rs` `after`, `not_after`).

1. Find the cause (the check's log and `runs/` artifact; AGENTS.md "Debug a failed CI run").
2. A fix in code: land it; its push runs the whole cycle. Nothing to ask.
3. No code change (an outside failure, root-caused): ask `test` again, at main's head (a rerun names a ref, not a commit:
   when main moved on, the newer commit's own cycle covers it; ask nothing). A rerun of `test` does not start `promote`
   (an asked run is one workflow), so ask it once the new `test` check reads `running`:

```sh
U=knotless://socialweb.computer/socialweb-computer-kcp
knotless run rerun $U --ref refs/heads/main --workflow test
knotless run list $U --ref refs/heads/main --workflow test --commit <sha>    # wait for: running
knotless run rerun $U --ref refs/heads/main --workflow promote                # waits for that test; filed failure again if it is red
```

Asked before the new `test` check is filed, `promote` reads the old red one and is filed `failure` again: ask it again.
`promote.yml`'s first step, a plain one (`scripts/promote.sh test`: `scripts/ci-check-wait.sh test <sha>`), also requires
the commit's `test` to have run green before release-workload does anything (it moves channel rc before the release
step's command runs), so a `promote` asked at a commit whose `test` is red, missing or path-skipped (a docs-only commit:
`success` with a `skipped` field) ends there: rc never moves, and the stop finds no rc of this run.

A rerun of `test` at a commit channel rc already points at (after a green promote moved rc to it) fails `0007:upgrade`:
its from binary is the previous rc (`SWC_COMPAT_REQUESTER`, `scripts/release-prev-rc.sh` reads channel rc), which is
then the binary under test, and the entry ends `unmet: the from binary (...) is the binary under test: channel rc
already points at this commit ...`, a FAIL. A normal cycle runs `test` before promote moves rc, so it never meets this;
by hand, give that rerun an earlier release (`sha:<its commit>`) as the input's `compat`, or as `SWC_UPGRADE_FROM` for
the entry run alone (step 5b). `0007:fence` reads neither: its from binary is its own (`release:v0.0.0-rc186`, the last
release without decision 54's rule), whatever channel rc points at.

## One cycle: release, then test

```sh
# 0. the market is clear (list; `delete` deletes every ComputeVM on every bidder's kcp)
scripts/ci-hand-market.sh list

# 1. release, by hand, for HEAD (main's commit): the steps of release.yml
scripts/swc-heavy.sh scripts/ci-release.sh checks
scripts/swc-heavy.sh scripts/ci-release.sh binaries
( set -a; . "$CI_BUCKET_CREDS"; set +a
  scripts/swc-heavy.sh scripts/ci-release.sh test-host     # builds and publishes tests/<tree>/, meshd/<tree>/ when absent
  scripts/swc-heavy.sh scripts/ci-release.sh page          # 0009's page payload (stage reads it)
  scripts/ci-release.sh guest-image                         # builds only when guest-images/<target>/<pin>/ is not published
  scripts/ci-release.sh stage
  scripts/ci-release.sh publish                             # local/<sha>/ only (main/<sha>/ is main's CI's)
  scripts/ci-release.sh verify )

# 2. test, by hand: the steps of test.yml (the account through its associated requester)
export CI_IDENTITY_NAME CI_IDENTITY_DIR
scripts/ci-test.sh host
scripts/ci-test.sh binary                                   # requester spec sha:<HEAD>
scripts/ci-test.sh staging 2>&1 | tee "$CI_HOME/staging-$(git rev-parse --short HEAD).log"   # not under swc-heavy
```

CI's input is the default (`input_default`: one staging network, two bidders, every entry of `tests/entries.txt`,
`deadlineMin` 250). A single entry is for the held network (below). There is no subset form: in the multi layout the
gate starts only when `suite` is `true`; a list (`"suite": ["0005"]`) stands the network up, holds it 65 s, takes it down
and exits 0 with zero entries run (2026-11-20; to be made a failure after plan 0040's rolls).

CI, for a commit:

```sh
U=knotless://socialweb.computer/socialweb-computer-kcp
scripts/ci-staging-total.sh <sha>                 # test: <status> N of 47 PASS, then gate: staging: N of 47 PASS
knotless run list $U --workflow test --commit <sha> --log
knotless run attach $U --workflow test --commit <sha> --key ~/.tmp/ci-attach/attach_key.pub > attach.json   # its frozen run
# a branch: an asked run is one workflow, so ask each; test once release's new check reads running (after: waits for it then;
# asked before, test may read release's older check, or none, and not wait)
knotless run rerun $U --ref refs/heads/<branch> --workflow release
knotless run rerun $U --ref refs/heads/<branch> --workflow test
```

`scripts/ci-staging-total.sh <sha> [--wait <secs>]` exits 0 only when `test` ended success with 47 of 47; 3 while it is
queued, running or has no check yet; 1 otherwise.

### Shares by hand

`tests/gate.sh` can deal the entries into n shares (`GATE_SHARD=i/n`; `scripts/ci-test.sh` `CI_SHARD=i/n` sets the
staging input's `shard`): CI runs n = 1 (owner, 2026-11-19: one network, less compute). The shares are disjoint, none
is empty and their union is the list, for any n up to the entry count (`scripts/check-shards.sh`, in `deploy/check.sh`).
The rule (`gate_ids`): the run-alone entries (`EXCLUSIVE`) first, then the rest, each class longest first by the seconds
column of `tests/entries.txt`, each to the share with the least seconds so far; a share keeps gate order. No entry needs
another on its network (plan 0040, Finding "Gate shards"). One hand identity is one requester, so one staging run at a
time: shares by hand run one after another.

```sh
for i in 1 2 3 4; do echo "$i/4: $(GATE_SHARD=$i/4 tests/gate.sh --list staging | tr '\n' ' ')"; done   # nothing started
CI_SHARD=2/4 CI_TEST_INPUT='{"layout": "multi", "provision": "infra", "deadlineMin": 150, "suite": true, "bidders": 2, "gateJobs": 4, "freezeTtl": 5}' \
  scripts/ci-test.sh staging 2>&1 | tee "$CI_HOME/staging-2of4.log"                 # gate: staging: K of M PASS (shard 2/4)
```

Shares in CI again would be one workflow per share (a workflow file has no fan-out; one CI VM is one guest requester):
the generator `scripts/ci-test-workflows.sh` and its template, removed by the commit that put CI in series (its parent
has them), and `promote.yml` after every share.

A fix: commit it on the branch, move `CI_HAND_WORKTREE` to the new branch commit, and run the cycle again. Land (`git push
swc HEAD:refs/heads/<b>`, `knotless pull create`, `knotless pull merge <pull> --queue`) only once a cycle is green.

Differences from CI, and only these:

| | CI | by hand |
|---|---|---|
| account | r2, the CI VM's guest requester (root runs it with `runuser`) | `CI_ACCOUNT` through its associated host requester: no sign-in per run |
| paths | /home/r2, /usr/local/bin, /work/runs, /root | `CI_HOME` |
| artifacts | `KNOTLESS_ARTIFACTS_URL` | the bucket |
| publish | main/<sha>/ and main/latest/ | local/<sha>/ only (requester spec `local:<sha>`) |
| platforms | linux/amd64, linux/arm64 | `CI_PLATFORMS` |
| freeze TTL | 1800 s on main | `CI_FREEZE_TTL=7200`, ended by hand (SIGTERM) when debugging is done |

## Promote, by hand: the dry run on a VM with the release image

`promote.yml` cannot be rehearsed by an asked run (release-workload writes the rc's records before its command runs).
`scripts/promote.sh --dry-run` needs no secret, deploys nothing, moves no channel and runs no gate. It runs
`scripts/ci-prod-target.sh --dry-run` (the prod target with the paths of its secret files, none written; the pinned node,
Chrome and deno when the page is there), fetches and checks the binary, reads each rc operator's heartbeat once, reads the
previous rc (`scripts/release-prev-rc.sh`, 0006:compat's requester), prints the droplet deploys and the gate command,
runs the test host's `--dry-run` of every prod composition on that target (the target, the composition and the requester
binary load; no account is touched), and prints the bucket copy a green promote ends with (`main/<sha>/` to
`releases/<RELEASE_STABLE_VERSION>/`, plan 0040 step 15) with what both hold now (public reads; nothing is copied). On the release binding's image and size
(socialweb.computer's `com.fedfork.exec.binding/release`: `ci-toolchain` 0049d853, 4 vCPUs, 8 GiB), as the account:

```sh
S=$(git rev-parse HEAD) M=<the main commit whose release published this tests/ and rust/ tree>
B=https://socialweb-computer-kcp.sfo3.digitaloceanspaces.com I=0049d853de17ff3bedfc3c9931b99906fd46c6bc108c1c8be2118d68d27f4dec
d=$HOME/.tmp/local-src/$S; mkdir -p $d; git archive HEAD | gzip > $d/src.tar.gz; cp scripts/promote-rehearse.sh $d/
(set -a; . "$CI_BUCKET_CREDS"; set +a; deno run -A --no-lock scripts/publish-spaces.ts --dir $d --prefix local/$S/src)
RU=$B/main/$M/binaries/socialweb-computer-linux-amd64
RS=$(curl -fsS $B/main/$M/binaries/SHA256SUMS | awk '$2 ~ /linux-amd64$/ {print $1}')
( . "$CI_ACCOUNT_CREDS"; p=${CI_ACCOUNT_PREFIX}_ATPROTO_PASSWORD; export ATPROTO_PASSWORD=${!p}
  socialweb-computer run request --state-dir ~/.tmp/rehearse-state --name promote-rehearse --image "$B/vm-images/ci-toolchain/$I@$I" \
    --cpus 4 --mem 8G --expires-in 1h --atproto-identity-mode pdsAccount --atproto-handle <handle> --atproto-pds-url https://bsky.social \
    --exec -- bash -c "curl -fsSL $B/local/$S/src/promote-rehearse.sh -o /var/tmp/r.sh && \
      sudo -n bash /var/tmp/r.sh $B/local/$S/src/src.tar.gz $(sha256sum < $d/src.tar.gz | cut -c1-64) $M $RU $RS" ) 2>&1 | tee ~/.tmp/rehearse.log
```

Pass: `==> test at <commit> ...: <status> now` and the `promote.sh test` that promote.yml runs first, one `==> would
write` line per secret file (two creds: alice, bob; the ops read secret, the
agent config), `==> page ... tools ...`, `==> prod target` with its paths (four bidders), `==> compat requester sha:<commit>`,
`==> would run:` the deploy and the ssh restart, one `==>` line per rc operator (four), the gate command
(`./gate.sh /home/r2/target-prod.json --account bob,alice` with `SWC_COMPAT_REQUESTER` and `GATE_VMS=6`), one `dry run <runbook>:
composition ok` line per prod runbook (17), the `copy-spaces.ts` command and its check, `==> rehearsal done`. 2026-11-17 on main de0b91ee the older dry run (no
target, no tools, no test host) passed; this one has not run on a VM yet.

The prod target's bidders: `scripts/ci-prod-target.sh` gives each bidder `ops: {url}` (its https market endpoint) when
its offering has one, else `ops: {iroh}` (its iroh endpoint ticket: every prod bidder today); it exits 1 for a bidder with
neither (`no https or iroh market endpoint in its offering`). Check on hat-8 with an empty HOME, no secret:

```sh
H=$(mktemp -d ~/.tmp/fakehome-ops.XXXX); env -u TMPDIR HOME=$H CI_HOME=$H/ci bash scripts/ci-prod-target.sh --dry-run $H/t.json
jq -c '[.bidders[] | {name, ops: (.ops | keys)}]' $H/t.json
```

Reach of one bidder's market mux over its iroh endpoint, by hand (needs dumbpipe on PATH; `/healthz` 200 on every
build, an ops route 404 on a build without them or without the secret configured, 401 on a wrong bearer):

```sh
t=$(jq -r '.bidders[0].ops.iroh' $H/t.json)
printf 'GET /healthz HTTP/1.1\r\nhost: guest\r\nconnection: close\r\n\r\n' | IROH_SECRET=$(openssl rand -hex 32) timeout 60 dumbpipe connect $t | head -1
```

The same without a VM, on hat-8, as the caller (no tools unless `$CI_HOME/page` holds the page; the test host's dry run
only when `tests/target/release/swc-test-host` is there):

```sh
CI_HOME=$HOME/.tmp/promote-dry RELEASE_VERSION=<the rc> RELEASE_STABLE_VERSION=<its vT> RELEASE_URL=$RU RELEASE_SHA256=$RS \
  bash scripts/promote.sh --dry-run
```

Without `$CI_HOME/requester.spec` (`scripts/ci-test.sh binary` writes it, `sha:<commit>`) the target keeps prod.json's
`release`, the version channel stable points at: before the first stable that fails the test host's dry run (it names
the fix); write `sha:<M>` there first.

The real promote's gate is the hat-8 gate's command below, as the unix user r2 on the release VM (`promote.sh prepare`
makes it; no guest requester runs there), on the target
`scripts/ci-prod-target.sh` writes from the secrets `.knotless/workflows/promote.yml` lists. Open (plan 0040 step 11):
the release VM does not reach the bidder hosts (`ssh hat-N`: each bidder's kcp and its host's journal), so the entries
that read them fail there until the owner decides how.

Promote runs after the staging gate (`after: [release, test]`; on every push to main with their paths, plan 0040 step 15;
in series since 2026-11-19): prod gets the build only once staging passed it. Its first step, a plain one before the
release step (`scripts/promote.sh test`), reads the same commit's `test` check again and goes on only when it ran green
(red, cancelled, path-skipped or none at that commit, as an asked promote may find: exit 1 before release-workload moves
channel rc, so nothing moves). After a green prod gate `promote.sh` copies `main/<sha>/` to
`releases/<vT>/` in the bucket (`scripts/copy-spaces.ts`, the bucket key `SPACES_*` sealed to main; vT is
release-workload's `RELEASE_STABLE_VERSION`) and checks the copy from the public URLs as release's `verify` does (the two
SHA256SUMS equal, `RELEASE_SHA256` in it, every file read without a credential and hashed); a failure there is a red
promote too. Exit 0: release-workload writes version vT on the rc's manifest and moves channel stable to it. The wait
is `scripts/ci-check-wait.sh`, which reads the runner's public check record (no credential, no knotless CLI):

```sh
bash scripts/ci-check-wait.sh --once test <commit>      # the status now
bash scripts/ci-check-wait.sh test <commit>             # waits while queued or running; exit 0 only on success
```

A red, cancelled or timed-out promote rolls back at once (plan 0040 step 12). Red: release-workload moves channel rc
back and runs `scripts/promote.sh rollback` in the step itself. Cancelled or timed out: the runner drops the step, so
`promote.yml`'s `stop: release-rollback` does it (the same module, from the records: the newest rc is this commit's,
not stable, and its `previous` names where rc goes back to); after a green or an already rolled back promote the stop
moves nothing. `promote.sh rollback` first ends the promote still running on the VM (`$CI_HOME/promote.pid`: SIGTERM to
its process tree, a test host then releases its rentals), then the droplets, then every rc operator's heartbeat on the
previous binary. The module's half is rehearsed in the requester repo (fail, cancel, timeout, each then the stop):

```sh
cargo wasm -p release-workload -p cicd-workload -p process-mcp && node --test modules/release-workload/test.mjs
```

Debug a rolled-back promote from its `gate-runs` artifact and its log; nothing is held.

### The release binding

socialweb.computer's `com.fedfork.exec.binding/release` (only promote uses it) asks for no guest requester: its inputs
are `{mem, cpus, image}`, no `association`. The prod gate signs in as bob and alice from their creds files
(`ci-prod-target.sh`); nothing on the release VM runs as R2. With `association: true` the runner associates a guest
requester beside the step (knotless `crates/knotless-run/src/composition.rs`, `computer.socialweb.vm.associate`, 200
polls of 10 s from the rent) and, when that gives up, stops the step mid-gate (`==> stopped: the guest requester was
not associated: ... (gave up)`, then a misleading `==> killed after 14400s`). `promote.sh prepare`, promote's first
command, exits 1 at once on such a VM (it has the association cloud-init's `/etc/systemd/system/socialweb-r2.service`).
Read it (public), and drop `association` as the account (its creds file `~/.socialweb_computer_source`, as
`publish-chain.sh` in the requester repo uses it; a swapped write, the record read first is the undo):

```sh
D=did:plc:gmlkrhimoueejcertevr4hl3 P=https://brittlegill.us-west.host.bsky.network
mkdir -p ~/.tmp/release-binding
curl -fsS "$P/xrpc/com.atproto.repo.getRecord?repo=$D&collection=com.fedfork.exec.binding&rkey=release" > ~/.tmp/release-binding/before.json
jq -c .value.inputs ~/.tmp/release-binding/before.json
tok=$( (set -a; . ~/.socialweb_computer_source; jq -n --arg i "$D" '{identifier: $i, password: env.SOCIALWEB_COMPUTER_ATPROTO_PASSWORD}') |
  curl -fsS -H 'content-type: application/json' -d @- "$P/xrpc/com.atproto.server.createSession" | jq -r .accessJwt)
jq -c --arg d "$D" '{repo: $d, collection: "com.fedfork.exec.binding", rkey: "release", swapRecord: .cid, record: (.value | del(.inputs.association))}' \
  ~/.tmp/release-binding/before.json | curl -fsS -H "authorization: Bearer $tok" -H 'content-type: application/json' -d @- "$P/xrpc/com.atproto.repo.putRecord"
curl -fsS "$P/xrpc/com.atproto.repo.getRecord?repo=$D&collection=com.fedfork.exec.binding&rkey=release" | jq -c .value.inputs
```

Undo: the same putRecord with `record: .value` of `before.json` (and `swapRecord` the new CID).

## The hat-8 gate

```sh
cd tests && SWC_COMPAT_REQUESTER=sha:<previous rc commit> GATE_VMS=6 ./gate.sh $PWD/targets/prod.json --account bob,alice
```

The prod gate's drivers are bob and alice (owner, 2026-11-19: "should just be alice and bob now"; john, a third driver
from 2026-11-18, plan 0040 decision 39, is out of the target), two entries at a time; `GATE_VMS=6` is two slots times the
heaviest entry weight (3: 0003, 0012), so the accounts, not the VM cap, bound it: at most 6 VMs at once, about 27 GiB, which
the four prod bidders hold beside the release VM ("CI's cycle, in series"). Each account's creds file
exports `<P>_ATPROTO_HANDLE` and `<P>_ATPROTO_PASSWORD` (P its `oauthAccount`: ALICE, BOB), one value per line
written as is (the test host and `scripts/oauth-signin.ts` strip only a quote pair around the whole value, never shell
escapes). socialweb.computer is not a driver: CI rents its
own VMs as that DID while promote runs, so 0002's and 0006's `assert.noneSince` scans of the driver's repo would count
them as leftovers (plan 0040 findings, 2026-11-18).

As bob on hat-8; the target's `requester` set to `sha:<the rc's commit>` (plan 0040 step 14). No landing on main while it
runs.

The static test host needs `SWC_REPO=<the checkout>` in its environment, as `promote.sh` sets it; without it the run
ends at once.

This gate runs its requester on the host. `promote` runs it inside a VM rented on a prod bidder, so a hat-8 pass says
nothing about guest-side rules or WebRTC from a guest to a guest on the same host: for those, rent a VM on prod with
`--guest-tools socialweb-computer` and run the entry from inside it ("Mistakes that cost a cycle", 5).

## Iterate on a held staging network

Do not stand staging up per fix (about 8 min each). One network per build, every entry on it (owner, 2026-11-15):

1. Run CI's entry list (the default input; `suite.sh` goes on past a failed entry, one `RESULT` line each).
2. The run freezes when any entry failed; a run whose suite started can be frozen by hand (`pkill -USR1 -f
   '^target/release/swc-test-host compositions/staging'`: the suite already running on the VM finishes).
3. On the held network, each failed entry: root cause, then a fix, patched in place when it can be (a `tests/` change: put
   the suite again; a Go change: swap the binary) and the entry rerun alone. Each goes into Findings.
4. Only when every failed entry has a root cause and a fix: end the run, commit the fixes on the branch, next build, next
   full run; land once one is green.

The commands:

```sh
# 1. hold it: a failed entry freezes the run (the default), every VM held for CI_FREEZE_TTL; iterating, keep it long
CI_FREEZE_TTL=7200 scripts/ci-test.sh staging ...            # a passing run: pkill -USR1 -f '^target/release/swc-test-host compositions/staging'
# 2. reach it: the held contracts (einfra-stg-<n>-atproto, -bidder-1 to -bidder-<bidders>, -gateway) and ssh to each
BIDDERS=$(seq -f bidder-%g 1 2)                              # the input's `bidders` (default input: 2)
K=$CI_IDENTITY_DIR/requesters/stg/kubeconfig WS=$(jq -r .workspace "$CI_IDENTITY_DIR/requesters/stg/ready.json")
S=$(sed -n 's/^ *server: //p' "$K" | head -1)
kubectl --kubeconfig "$K" --server="${S%/clusters/*}/clusters/$WS" get computecontracts
SWC=~/.tmp/swc-system-tests/bin/main-$(git rev-parse HEAD)/socialweb-computer-linux-amd64
on() { $SWC run requester ssh stg --state-dir "$CI_IDENTITY_DIR" --contract "einfra-stg-<n>-$1" --exec -- bash -c "$2"; }
# 3. rerun ONE entry: step 5 (hand.sh). suite.sh takes only a run name and runs the WHOLE gate (it removes gate/ first)
# 4. a tests/ change (a module, a composition, target.sh, suite.sh): rebuild the wasm, put the suite the way the module does
(cd tests/modules && cargo wasm)
tar -czf - --exclude=did-key-associator/dist --exclude=did-key-associator/dist-backend tests/gate.sh tests/entries.txt tests/compositions \
  tests/targets did-key-associator typescript-helpers/lib/oauth-scope typescript-helpers/lib/cli-args-env typescript-helpers/lib/logger \
  typescript-helpers/lib/serve tests/modules/*/module.json tests/modules/target/wasm32-unknown-unknown/release/*.wasm deploy/vm-images \
  examples scripts | on atproto 'd=/home/stg/stg; cat > $d/suite.tgz; rm -rf $d/suite; mkdir -p $d/suite; tar -xzf $d/suite.tgz -C $d/suite; chown -R stg:stg $d/suite'
# 4b. 0009's page (plan 0040 step 8f): the suite carries it when the host that puts it has SWC_PAGE_REPO, a built checkout
#     of the requester page repo (tests/testkit/README.md, pageRepo); by hand, the same files beside the suite, then target.sh
#     (page.sh: pinned node and Chrome into /home/stg/stg/page/tools, the target's pageRepo). A full run: SWC_PAGE_REPO=<checkout> scripts/ci-test.sh staging
R=~/.tmp/swc-system-tests/page/atlibs-socialweb-computer-requester; p=$(mktemp -d); mkdir -p $p/page/web $p/page/scripts
cp -rL $R/web/webrtc $p/page/web/; cp $R/scripts/page-round.mjs $R/scripts/browser.mjs $p/page/scripts/; git -C $R rev-parse HEAD > $p/page/COMMIT
tar -czf - -C $p page | on atproto 'd=/home/stg/stg/suite; rm -rf $d/page; tar -xzf - -C $d; chown -R stg:stg $d/page'
on atproto 'runuser -u stg -- env HOME=/home/stg SWC_REPO=/home/stg/stg/suite /home/stg/stg/suite/tests/compositions/staging/page.sh'
on atproto 'D=/home/stg/stg; jq --arg p $D/page ".pageRepo = \$p" $D/target-staging.json > $D/t.json && mv $D/t.json $D/target-staging.json && chown stg:stg $D/target-staging.json'
# 5. a Go change: a RELEASE build of the fix's commit (a dev build's requester never comes Ready), one
#    platform, about 10 s warm; to the bucket, then onto the VM by URL (ssh streaming is slow) and in by mv (running
#    processes keep the old file). Only the atproto VM's binary: it is the requester under test and writes guests' user_data
C=$(git rev-parse HEAD); scripts/swc-heavy.sh scripts/build-socialweb-computer.sh --commit $C --platform linux/amd64
P=~/.tmp/ci-hand/pub-$C; mkdir -p $P; cp ~/.tmp/socialweb-computer-build/release/$C/socialweb-computer-linux-amd64 $P/socialweb-computer
(cd $P && sha256sum socialweb-computer > SHA256SUMS); T=local/$C/fix
(set -a; . ~/.spaces_swc_kcp_source; set +a; deno run -A --no-lock scripts/publish-spaces.ts --dir $P --prefix $T)
on atproto "B=/home/stg/stg/socialweb-computer-linux-amd64; curl -fsSo \$B.new https://socialweb-computer-kcp.sfo3.digitaloceanspaces.com/$T/socialweb-computer \
  && echo '$(cut -d' ' -f1 $P/SHA256SUMS)  '\$B.new | sha256sum -c --quiet && chmod 0755 \$B.new && chown stg:stg \$B.new && mv \$B.new \$B"
# 5b. rerun one entry frozen on failure (suite.sh cleans up a failed entry: its guests are gone), its own log file
on atproto 'D=/home/stg/stg; cd $D/suite/tests && (setsid nohup runuser -u stg -- env HOME=/home/stg XDG_RUNTIME_DIR=/run/user/$(id -u stg) \
  SWC_READINGS=$D/readings SWC_REPO=$D/suite SWC_TARGET=$D/target-staging.json $D/swc-test-host compositions/0005 --account alice1 \
  --entry run --on-fail freeze --freeze-ttl 3600 > $D/hand-0005.log 2>&1 < /dev/null &)'
# 5c. inside the entry's guests while it runs or is frozen: the entry's control socket (its run dir) has the tools of
#     `attach` (state, failure, resources, kube_get, ssh_exec ...). ssh_exec runs a command in a run request's VM (the
#     name from `resources`) over its iroh ssh as the guest user (sudo -n for root); "connection lost" after the output is
#     its teardown, not a failure. The command goes in a file: no nested quoting
cat > ~/.tmp/sx.py <<'PY'
import socket,json,sys
s=socket.socket(socket.AF_UNIX); s.connect(sys.argv[1]); f=s.makefile("rw")
tool,args=sys.argv[2],json.loads(sys.argv[3]) if len(sys.argv)>3 else {}
if tool=="ssh_exec": args["exec"]=open(args.pop("file")).read()
for m in [{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}},{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":tool,"arguments":args}}]:
    f.write(json.dumps(m)+"\n"); f.flush(); r=json.loads(f.readline())
    if m["id"]==2:
        t=r["result"]["content"][0]["text"]
        try: print("\n".join(l for l in json.loads(t).get("output",t).splitlines() if not l.startswith('{"time') and "connection lost" not in l and "tracing-subscriber" not in l))
        except Exception: print(t)
PY
on atproto "echo $(base64 -w0 ~/.tmp/sx.py) | base64 -d > /tmp/sx.py"
R=$(on atproto 'ls -dt /home/stg/.tmp/swc-system-tests/runs/0005-composition-* | head -1')/control.sock
on atproto "runuser -u stg -- python3 /tmp/sx.py $R state"                                  # phase, marks
on atproto "runuser -u stg -- python3 /tmp/sx.py $R resources"                              # invocation names
printf '%s\n' 'sudo -n sh -c "ps -eo args | grep [r]un.request; tail -c 3000 /home/r2/.local/state/socialweb-computer/requests/r2-st/dev/r2-st/logs/core.log"' > ~/.tmp/g.sh
on atproto "echo $(base64 -w0 ~/.tmp/g.sh) | base64 -d > /tmp/g.sh; runuser -u stg -- python3 /tmp/sx.py $R ssh_exec '{\"name\":\"<invocation>\",\"file\":\"/tmp/g.sh\"}'"
#     its requester (the outer `run request`): /home/stg/stg/s/<invocation>-request.log while it runs, under the run's
#     logs/ after; R2 inside the guest: /home/r2/.local/state/socialweb-computer/requests/r2-st/dev/r2-st/{logs,kcp}
#     end it: python3 /tmp/sx.py $R cleanup
# 5d. a Go change to the network itself: swap the binary (5), then restart its core on it (the unit's path is the same,
#     the process is new: check its start time), then what new VMs get from user_data, by hand on the held ones
on atproto 'runuser -u stg -- env HOME=/home/stg XDG_RUNTIME_DIR=/run/user/$(id -u stg) /home/stg/stg/socialweb-computer-linux-amd64 \
  dev restart --instance stg --state-dir /home/stg/stg/s core --binary /home/stg/stg/socialweb-computer-linux-amd64'
on atproto 'cd /home/stg/stg/suite/tests && runuser -u stg -- compositions/staging/target.sh hand > /home/stg/stg/target-staging.json'
#     the bidder VMs keep the old network value (net.json): their name overrides dial the old front ("networkdial: dialing
#     10.44.0.1:<old port> ... connection refused"). Give each the new one and restart its core unit (the bidder is
#     installed: account swc-bidder, unit socialweb-computer-bidder@<handle>; kcp and VM units stay)
NET=$(on atproto 'cd /home/stg; runuser -u stg -- env HOME=/home/stg XDG_RUNTIME_DIR=/run/user/$(id -u stg) /home/stg/stg/socialweb-computer-linux-amd64 \
  dev status --instance stg --state-dir /home/stg/stg/s | sed -n "/^{/,\$p" | jq -c .network.status | base64 -w0')
for v in $BIDDERS; do on $v "echo $NET | base64 -d > /etc/socialweb-computer/net.json; chmod 0644 /etc/socialweb-computer/net.json"
  on $v 'A=swc-bidder; runuser -u $A -- env XDG_RUNTIME_DIR=/run/user/$(id -u $A) systemctl --user restart "socialweb-computer-bidder@*"'; done
#     a new binary on a bidder VM (the URL and sha256 of step 5): `upgrade` as swc-bidder switches bin/current and
#     restarts the core unit (`install bidder` with another binary than bin/current refuses and names this command)
for v in $BIDDERS; do on $v "B=/var/tmp/swc-fix; curl -fsSo \$B https://socialweb-computer-kcp.sfo3.digitaloceanspaces.com/$T/socialweb-computer && chmod 0755 \$B \
  && A=swc-bidder ST=/var/lib/socialweb-computer/swc-bidder && runuser -u \$A -- env XDG_RUNTIME_DIR=/run/user/\$(id -u \$A) \
  \$ST/bin/current/socialweb-computer upgrade --binary \$B --sha256 $(cut -d' ' -f1 $P/SHA256SUMS) --state-dir \$ST"; done
on bidder-1 'A=swc-bidder; runuser -u $A -- env XDG_RUNTIME_DIR=/run/user/$(id -u $A) systemctl --user status "socialweb-computer-bidder@*" --no-pager | head -5'
#     from nothing (a new DID: trust.sh vouch and bidders.json again; the stand-up's own command, tests/compositions/staging/README.md step 4)
on bidder-1 '/var/lib/socialweb-computer/swc-bidder/bin/current/socialweb-computer uninstall bidder --purge'
# 5e. past the freeze TTL (it cannot be extended; its thaw runs the cleanup): pause the driver, the VMs stay up to their
#     EphemeralInfra ttl; resume it to end the run (its cleanup then runs at once)
kill -STOP $(pgrep -f '^target/release/swc-test-host compositions/staging')     # later: kill -CONT <pid>
# 6. done: SIGTERM the driver; its cleanup releases every VM (then scripts/ci-hand-market.sh list)
```

Bidder logs on a bidder VM (installed, plan 0040 step 8b): `journalctl _UID=$(id -u swc-bidder) -o cat --no-pager` as root (the
unit alone: `_SYSTEMD_USER_UNIT=socialweb-computer-bidder@<handle>.service`); its kcp: `/var/lib/socialweb-computer/swc-bidder/handles/*/kcp/admin.kubeconfig`;
its VM units: `runuser -u swc-bidder -- env XDG_RUNTIME_DIR=/run/user/$(id -u swc-bidder) systemctl --user list-units "socialweb-vm-*"`; an entry's
requester on the atproto VM: `/home/stg/stg/s/dev/stg/apps/dev/stg-<request name>/logs/core.log`.

## Mistakes that cost a cycle

The lead's own, on plan 0040 (2026-11-20). Each has the rule it left; plan 0040 Findings has the runs.

| # | What I did | What it cost | Rule |
|---|---|---|---|
| 1 | Proved `0007:upgrade` on a network where I had put the binaries on the bidders by hand | In the full gate the test host copied 4 x 590 MB over the overlay: the entry timed out at 60 min and the load failed `0009:run` | A proof on a hand-prepared network does not count for anything a fresh network does differently. Prove on a fresh network, nothing pre-staged |
| 2 | Accepted the capacity redesign on six reviews and the entries written for it; nobody ran `0014:abandon` | A product regression (500 for an accept the bidder cannot read yet) reached the final hand cycle; one more 2 h cycle | A change is proven by every entry of each family it touches, not a sample. A review is not a run |
| 3 | Let the secrets work change a shared test-host call (`repo.listCreated`) and proved it only on staging | `0006:compat` failed on prod as alice (her repo is large; staging repos are new); one more 2 h cycle | A change to a shared host call is run by every caller, against the largest real repo it will read (alice on prod, read-only) |
| 4 | Asked for a seven-entry staging run without checking a subset path exists; did not read the run's end | The run exited 0 with zero entries; 40 min before I saw it | Read the gate summary line and count the RESULT lines of every run. `exit 0` is not a result. If the count is not the list's, it failed |
| 5 | Ran the hand prod entries from hat-8 itself and called three WebRTC entries "first proven by the roll" | Promote 93fd8c11 red (28 of 31): its requester is inside a rented guest, booted by the old bidder; a 4 h roll and a rollback | Rehearse in promote's topology: the requester inside a VM rented on a prod bidder, the target on the same host. Anything "only the roll can prove" gets a rehearsal designed first |
| 6 | Shipped a guest-side rule (decision 54) and expected promote's own VM to have it | Same red promote | What a release changes in guest rules, guest agent or vmexec is not on guests booted before the roll. Ask of every such change: who boots the guest that needs it |
| 7 | Trusted agents' watchers and my own (a Monitor that expired, a watcher on the wrong file, a tail opened before the file existed) | Failures and ended runs sat unread four times, 27 to 30 min each | After arming a watcher, check it against the real file once. Read result files myself every 15 to 20 min. A watcher must fire on every end state, not only on success |
| 8 | Told an agent to stop new proofs, and another to do a subset, to save time | The proofs cut were the ones that mattered | Never trade a proof for time. If a full gate is the only path that exists, run the full gate |
| 9 | Let a fix change the tree under test after its "final" cycle, three times | Each needed a further cycle | Land only a tree that passed whole. A fix after the cycle means another cycle: plan for it, do not argue it away |
| 10 | Changed a host's firewall with a command I had not read back | It errored on the zone name and changed nothing; caught because I printed before and after | Print the state before and after every change on a host |
| 11 | Accepted a test entry whose precondition was "the previous release lacks the feature" (`0007:fence` took its from binary from the previous rc) | The entry failed for ever the moment its own roll went green; caught before the next roll, one more fix | Ask of every new entry: does it still pass after it lands, on the roll after, and after a rollback |
| 12 | Added a pinned-binary bump (dumbpipe) to a fix stack late, proven by pins and a file diff, without switching a running bidder between the two builds | The full hand cycle found both bidders deaf over iroh after the switch, in both directions: on prod it would have been an outage at the roll and at the rollback | A change of a pinned binary or of anything a core stores across restarts is proven by `0007:upgrade` alone first, both directions, before it joins any stack |
| 13 | Kept adding fixes to the stack as I found intermittents | Five hand cycles and no second roll in a day | Freeze the stack once it holds what failed; anything new that is not a regression of the stack goes to its own roll |
| 14 | Rolled after checking the four bidder hosts but not the runner | `test` red in 80 s: the runner container held 173 zombie ssh, 199 of its 256 processes | Before a roll read the runner container's `pids.current` on the gateway host; over 100: `systemctl restart knotless-fork-runner.service` (no run active) |
| 15 | Asked for a rerun of a red `test` before pulling its artifacts | The rerun pruned them; the bidder's side of the failure was never read | Pull a failed run's artifacts (`knotless run artifacts`, a fork-built CLI) before any rerun |
| 16 | Waited on a CI `test` by its state word only | Three hours on a run that was 6 PASS of 20 | At the first doubt read the run from inside: `knotless run attach`, then `swc-test-host attach --ci` (state, the gate dir's RESULT lines); a pass ends near 2 h |
| 17 | Took seven hand cycles as a rehearsal of CI | CI's placement (its own VM on the hosts) and its runner were never exercised by hand | Before the roll of a batch, one hand network in CI's layout with the overlay measured (pair matrix); save a meshd log of a healthy network |
| 18 | Named a cause (VM placement) from one run's layout | Two green rolls had the same layout | Compare with the runs that passed before naming a cause |

Before a pull is queued, each of these is a yes:

- Every entry of every family the change touches ran, on a fresh network, nothing pre-staged.
- The gate summary line and the RESULT count were read from the log, by the lead.
- Shared test-host or module changes ran against prod's largest repo, read-only.
- Anything that differs on prod (hosts, routers, repo sizes, a requester inside a guest, guests booted by the old
  bidder) has a rehearsal that has that difference, or is written in the pull as unproven with the reason.
- The tree in the pull is the tree that passed.
- The runner container's process count was read (mistake 14); a healthy network's overlay (meshd) log is saved.

## Do not waste time

Learned on plan 0040's hand cycles (2026-11-17/18); each line cost time once.

Two loops, never mixed:

| Loop | For | Takes |
|---|---|---|
| survey: the whole gate on a network | the number (N of 47), entries that break each other | 2 h 17 min of gate on two guest slots (hand cycle 3, three gate jobs), after 11 to 25 min of stand-up |
| fix: one entry on a held network | proving one fix | minutes |

1. A survey freezes nothing: `"freezeTtl": 5` in the input, with `"bidders": 2` (the network's bidder VMs `bidder-1`
   and `bidder-2`, bob2 and bob3, one guest slot each; the gate's VM cap `gateVms` follows it) and `"gateJobs": 4` (four
   entries at once, one per driver account). All are in the default input (`scripts/ci-test.sh` `input_default`, CI's
   too), so a survey needs no `CI_TEST_INPUT` (`"deadlineMin": 250`: every entry on the one network). The suite's own default of 300 s per failed entry made the first survey 8004 s: 14
   failures were 70 min of waiting. A network is 4 VMs of 4 vCPU / 8G on the prod bidders (16 vCPU / 32G); with
   `"bidders": 4`, 6 VMs. The network of before the bidders input:

   ```sh
   export CI_TEST_INPUT='{"layout": "multi", "provision": "infra", "deadlineMin": 290, "suite": true, "bidders": 2, "gateJobs": 3, "freezeTtl": 5}'
   ```

   ```sh
   scripts/ci-test.sh staging > "$CI_HOME/staging-$(git rev-parse --short HEAD).log" 2>&1 &
   ```
2. Keep the network. A failed suite holds the outer run for `CI_FREEZE_TTL` (7200 s): prove every fix on it before it
   goes, and run the gate again on it instead of a new stand-up (no release build, no 25 min):

   ```sh
   on atproto 'runuser -u stg -- env HOME=/home/stg XDG_RUNTIME_DIR=/run/user/$(id -u stg) GATE_FREEZE_TTL=5 \
     /home/stg/stg/suite/tests/compositions/staging/suite.sh <run> > /home/stg/suite2.log 2>&1; tail -3 /home/stg/suite2.log'
   ```
3. One entry by hand (debug, never a gate result). Two at once need two accounts (one driver per account):

   ```sh
   on atproto 'cat > /home/stg/stg/hand.sh <<"EOF"
   #!/bin/sh
   D=/home/stg/stg; cd $D/suite/tests
   exec runuser -u stg -- env HOME=/home/stg XDG_RUNTIME_DIR=/run/user/$(id -u stg) SWC_READINGS=$D/readings SWC_REPO=$D/suite \
     SWC_TARGET=$D/target-staging.json $D/swc-test-host compositions/$1 --account $3 --entry $2 --on-fail cleanup > $D/$4 2>&1
   EOF
   chmod +x /home/stg/stg/hand.sh'
   on atproto '(setsid nohup /home/stg/stg/hand.sh 0017 run alice1 hand-0017.log < /dev/null > /dev/null 2>&1 &)'
   on atproto 'sed -n "/^{/,\$p" /home/stg/stg/hand-0017.log | jq -c "{pass, error}"'
   ```
4. Ship a fix in under a minute: a Go change is a release build of its commit (about 8 s warm) through `local/<sha>/fix`
   (step 5 of "Iterate on a held staging network"); a test module is one crate and one file:

   ```sh
   (cd tests/modules && cargo build -q --release --target wasm32-unknown-unknown -p acc-0013)
   on atproto 'T=/home/stg/stg/suite/tests/modules/target/wasm32-unknown-unknown/release/acc_0013.wasm; cat > $T.new &&
     chown stg:stg $T.new && mv $T.new $T' < tests/modules/target/wasm32-unknown-unknown/release/acc_0013.wasm
   ```
5. The gate's RESULT lines are in the run's ticks, not in the staging log. Read only a run dir made after the cycle
   began (an older one shows last cycle's results as new):

   ```sh
   touch "$CI_HOME/cycle.start"    # before the cycle
   R=$(find "$CI_HOME/runs" -maxdepth 1 -name 'staging-composition-*' -newer "$CI_HOME/cycle.start" | head -1)
   { zstd -dc $R/ticks.ndjson.zst 2>/dev/null; cat $R/ticks.ndjson 2>/dev/null; } | tail -c 400000 | sed 's/\\n/\n/g' |
     grep -o 'RESULT [0-9]\{4\}:[a-z0-9]* [A-Z]* [0-9]*[^"]\{0,120\}\|gate: staging: [0-9]* of [0-9]* PASS' | sort -u
   ```
6. A fix is proven when the symptom is seen gone on the held network, not when it builds. A fix landed "proof pending"
   can do nothing: the first releaseOk fix patched a branch the releases never take (96 of 96 summaries still false).
7. A failed entry's logs go with its cleanup: read them while it is frozen, or rerun it frozen
   (`--on-fail freeze --freeze-ttl 2400`). A component's own log (k3s, postgres, openbao) is under the requester's or
   cluster's state dir, not in the entry's log.
8. What is deployed is read from the running system, never from a local checkout: `docker image inspect` on the gateway
   for the runner, a bidder's heartbeat `version.sha256`, `socialweb-computer version` on the host.
9. An entry that never ran needs things nobody installed (0017: socat) and may be wrong itself (0013:b gave its password
   after the install that waits for it). Read its module and host calls before its first run.
10. A failed entry can leave an install behind (`swc-requester`, `swc-st-qr` under /var/lib/socialweb-computer): purge it
    before the rerun (`<state>/bin/current/socialweb-computer uninstall <role> --account <a> --purge`).
11. Never `pkill -f` and the program's name in one command string (local, ssh or `--exec`): the shell matches itself and
    dies silently, and the later steps never run. Kill by pid in a command of its own, then check the next step happened.
12. Landing: one batch per step when CI is idle. Each push to main runs release, test and promote in series (about 4 h
    15 min); a push during a cycle runs its test beside this promote ("CI's cycle, in series"), and stacked build VMs
    (12 vCPU each) leave the next one without capacity.
13. Independent work runs in parallel: one agent per layer on its own branch and worktree, read-only agents for open
    root causes; the lead keeps the held network and merges.

## Timing and watching

Times (hat-8, warm caches, 2026-11-15) and the limits that can hold a run:

| Part | Takes | Limit that holds it |
|---|---|---|
| release by hand (binaries, test-host when published, stage, publish, verify) | 3 to 5 min | none |
| `checks` | ~15 min | none |
| `ci-test.sh host`, `binary` | ~1 min | none |
| staging stand-up (4 VMs rented, installed, joined, probed) | 8 to 25 min | a full market waits up to 30 min per contract (`DefaultCapacityWait`) |
| one entry on staging | prod gate rc168: 0003 2 min, 0005 4 min, 0007 1.5 min, 0012 2 min, 0013 4 min | the entry's own deadline, 40 to 60 min (`tests/modules/acc-*`) |
| the whole gate on staging (43 entries; 2 accounts then, 4 now: alice1 to alice4) | 2 h 17 min with `freezeTtl` 5, `gateJobs` 3 and two guest slots (hand cycle 3, 34 of 43); 133 min with 14 failures frozen 300 s each (2 accounts); 152 min estimated with two bidders (plan 0040, Finding "Gate shards") | `deadlineMin` 250 (CI; test.yml's step 300) |
| the prod gate (31 entries; the seconds below are of the 30 before 0007:webrtc (70 s); bob, alice, `GATE_VMS=6`) | about 4840 s scheduled (the entries' seconds: 0001:run 932 alone, then 7817 s on two slots), 5300 to 5800 s expected at the 1.1x to 1.2x three drivers ran over their schedule (3142 to 4237 s, promotes of 2026-11-18/19); the one two-driver run, by hand before the ops routes, 5471 s with 5 FAILs | promote.yml's step 240 |
| RISK: CI's staging step | 250 min of `deadlineMin` (stand-up and gate) for about 160 min estimated; the shares ran about 1.3x their estimates and home hosts stand up slower, so about 30 min of margin: past it the suite ends at its deadline and the entries not run FAIL | 300, the runner's cap: `--step-timeout 18000` in knotless-fork-infra `host/systemd/knotless-fork-runner.service` (a push run's step never goes past it; an asked run's may); raising it is that flag and a runner deploy, then test.yml's timeout and `deadlineMin`; the other levers are the 30 min freeze (kept) and four staging bidders (about 27 min faster, 2 VMs more) |
| a failed run | | the freeze, `CI_FREEZE_TTL` (by hand: 7200; CI main: 1800) |

Watch every run, and act on a stall instead of waiting for a deadline:

- Background the step with its output in a log (`... > "$CI_HOME/<step>.log" 2>&1 &`).
- Watch the log for the end and every failure: `STEPOK`, `STEPFAIL`, `done pass=`, `FROZEN`, `cleanup:`, `error`, `RESULT`.
- Also watch for a stall: a heartbeat every 5 min with the last line that is not a poll (`timer poll-`, `result ...:get`,
  `suite-poll`, `guest_iroh`). An entry past 3x its prod time is stuck: read its log tail (the suite poll records it in
  the ticks: `swc-test-host replay "$R"`, or `$R/logs` after the failure), then SIGTERM the driver rather than wait for
  the deadline.
- An entry that repeats the same failure (one `requester_association_failed` error every 30 s) has failed: do not wait out
  its deadline (owner, 2026-11-15; cycle 13 lost 8 min of 0005's 60 before this). Every entry fails fast on a repeated
  identical error itself (`steps::market_event`: a retried `requester_association_failed` with the same error 5 times over
  3 min fails it; a refusal is left to the entry). By hand, end the entry and
  let the suite go on to the next one; the network stays up and is held at the suite's end:

  ```sh
  on atproto 'pgrep -af swc-test-host | grep -v attach'   # the running entry: swc-test-host compositions/<n> ... --on-fail cleanup
  on atproto 'kill -TERM <pid>'                            # it releases its own rentals, RESULT <n> 1, the suite takes the next entry
  ```
- An agent hand runner: one watch per run that exits when the run ends, re-armed when the watch tool's time limit ends
  it; one success-or-failure watch never sees a hang.

```sh
L="$CI_HOME/staging.log"
tail -n +1 -F "$L" | grep --line-buffered -E 'STEPOK|STEPFAIL|done pass=|FROZEN|cleanup:|RESULT|error:' &   # outcomes
while sleep 300; do                                                                                 # stall heartbeat
  echo "$(date -u +%H:%MZ) last: $(grep -vE 'timer poll-|:get|suite-poll|guest_iroh|message req-' "$L" | tail -1 | cut -c1-160)"
done
```

## Debug

```sh
R=$(ls -td "$CI_HOME"/runs/staging-composition-* | head -1)   # the run dump; its work dir: ~/.tmp/swc-system-tests/work/<same>
jq . "$R/failure.json"                                         # the error, with the failed entry's log tail
ls "$R/logs"                                                   # per VM: its journal and logs (taken before the freeze)
cd tests && target/release/swc-test-host attach "$R"           # a frozen run (MCP): state, ticks, failure, kube_get, ssh_exec
target/release/swc-test-host replay "$R"                       # offline: the module's decisions, the first diverging tick
#    an entry on the atproto VM: its readings and log are under /home/stg (ssh_exec, or
#    socialweb-computer run requester ssh stg --state-dir "$CI_IDENTITY_DIR" --contract <cc> --exec -- <command>)
#    the market during the run: load $CI_HOME/runs/watch.ndjson in https://watch.socialweb.computer/
```

End a frozen run with SIGTERM to its driver (`pkill -TERM -f "swc-test-host compositions/staging"`): it releases
everything and exits. If its cleanup reports errors (`cleanupErrors` in `$R/run.json`), run
`scripts/ci-hand-market.sh list`, then `delete`.