~ knotless Search/
↑↓ move↵ open⇧↵ all results

socialweb.computer/socialweb-computer-kcp

socialweb.computer: a compute market on atproto (kcp)

1 collaborator
0
0
657 lines · 53.7 KB Raw History

CI by hand and debugging a CI run

Run a CI stage by hand, hold a failed run, and fix it in place. Plan: docs/plans/0040-release-pipeline.md.

Rules

  • Every workflow step is a script; a hand run calls the same scripts.
  • Before a hand run, the market holds none of our rentals (scripts/ci-hand-market.sh list).
  • The hand requester rents the staging VMs through its associated host requester (prod alice, setup below); the entries run as the staging network's own driver accounts alice1 to alice4 (tests/compositions/staging/README.md, Accounts).
  • Prod gets code only through CI promote (plan 0040): no hand roll.
  • Every change under the workflows' paths: is run by hand, off CI, before its pull is queued (plan 0040 decision 51): a hand cycle is "One cycle: release, then test" below on the branch's commit (published to local/<sha>/ only, no channel moves), the whole list of tests/entries.txt on a fresh staging network. The pull says what was run and seen. The bar is the checklist under "Mistakes that cost a cycle".
  • A failed run is held (frozen) and fixed in place; its finding goes into docs/plans/0040-findings.md (the plan's notes file).

Prerequisites

  • A Linux host with the kcp toolchain: scripts/build-socialweb-computer.sh and tests/publish-test-host.sh work there.
  • kubectl, jq, curl, git, deno, node 22 or newer (else the watch capture fetches one), ssh to the bidder hosts.
  • sudo on each bidder host (for scripts/ci-hand-market.sh).
  • The creds files: the requester account's (~/.aliceoa_bsky_social_source shape, used once, at setup step 2) and the bucket key (~/.spaces_swc_kcp_source: SPACES_ACCESS_KEY_ID, SPACES_SECRET_ACCESS_KEY, SPACES_BUCKET, SPACES_REGION). Owner-only, never printed.
  • A worktree at the branch commit under test (CI_HAND_WORKTREE). Nothing else runs in it during a cycle: every ci-release.sh step reads git rev-parse HEAD.

The hand runner's env file (not in git)

~/.config/socialweb-ci-hand/env, mode 0600, paths and public values only. Each runner writes its own.

CI_ACCOUNT=alice                                  # an account of tests/targets/prod.json (CI: r2)
CI_ACCOUNT_DID=<its DID>
CI_ACCOUNT_CREDS=$HOME/.aliceoa_bsky_social_source
CI_ACCOUNT_PREFIX=ALICE                           # the creds file's variable prefix
CI_IDENTITY_NAME=<name>                           # the associated host requester identity (setup)
CI_IDENTITY_DIR=$HOME/.swc/<name>                 # short: its unix sockets must fit 107 bytes
CI_IDENTITY_DID=<written by setup step 1>
CI_HOME=$HOME/.tmp/ci-hand                        # what CI puts in /home/r2, /usr/local/bin, /work/runs, /root
CI_BUCKET_CREDS=$HOME/.spaces_swc_kcp_source
CI_PLATFORMS=linux/amd64                          # release by hand builds amd64 only; CI builds both
CI_BIDDER_HOSTS="hat-7 hat-8 hat-9 hat-10"        # every prod bidder host (scripts/ci-hand-market.sh)
CI_HAND_WORKTREE=<path of the worktree>
CI_FREEZE_TTL=7200                                # a failed run holds its VMs for the debugging (end it yourself)

Every block below starts with set -a; . ~/.config/socialweb-ci-hand/env; set +a; export TMPDIR=$HOME/.tmp/ci-hand-tmp; cd "$CI_HAND_WORKTREE".

Setup, once per runner

B=/var/tmp/ci-hand-bin; scripts/build-socialweb-computer.sh --dev --publish $B   # a binary with plan 0046

# 1. the associated host requester: a localRepo identity (its own did:plc and PDS), agent-of the account
$B/bin/socialweb-computer run requester --name stg --state-dir "$CI_IDENTITY_DIR" --identity "$CI_IDENTITY_NAME" \
  --associate-with "$CI_ACCOUNT_DID"                                                 # foreground
jq -r .did "$CI_IDENTITY_DIR/requesters/stg/ready.json"                              # -> CI_IDENTITY_DID in the env file

# 2. the account writes the standing association (market.association + link.ident.endorsement): its one sign-in
deno run -A --no-lock scripts/associate-bidder.ts --creds "$CI_ACCOUNT_CREDS" --account "$CI_ACCOUNT_PREFIX" \
  --agent-did "$CI_IDENTITY_DID"
$B/bin/socialweb-computer internal association "$CI_IDENTITY_DID" | jq -c '{roots, errors}'   # roots [CI_ACCOUNT_DID]

# 3. stop it (SIGINT to the socialweb-computer process itself; it keeps its state)
kill -INT "$(pgrep -f "socialweb-computer run requester --name stg --state-dir $CI_IDENTITY_DIR")"

The staging composition names its requester stg, and CI_IDENTITY_DIR is its state dir from then on (run one stg requester at a time on a host).

Workaround: seed the asset cache from a hand runner

A cold CI release fetches every pinned upstream file (44+ debs from Launchpad and apt-archive.postgresql.org per platform); one 503 fails the build. Its restore (ASSET_CACHE_URL=<bucket>/asset-cache) was empty: main's release fills it only after a build passes. Seed it once from the hand runner's blob cache, which the hand cycles filled:

c=~/.tmp/socialweb-computer-build/cache/blobs
sort -u $c/.new | while read -r e; do k="${e##*/}"; [ -f "$c/$k.tar.zst" ] && stat -c %s "$c/$k.tar.zst"; done \
  | awk '{s+=$1; n++} END {printf "%d blobs %.1f GiB\n", n, s/1073741824}'           # 2026-11-16: 535 blobs 8.2 GiB
(set -a; . "$CI_BUCKET_CREDS"; set +a; scripts/asset-cache-upload.sh $c) > ~/.tmp/asset-cache-seed.log 2>&1
grep -c uploaded ~/.tmp/asset-cache-seed.log                                         # 594 files (34 already there)

Content-addressed: a key already in the bucket is not sent again. Not needed again once main's release passes (its asset-cache step uploads what is new).

CI's cycle, in series

A push to main that changes a path of the workflows' paths: (the same list in all three: deploy/check.sh) files all three checks at once, as queued; each runs once the one before it passed (owner, 2026-11-19: series, less compute):

WorkflowWaits forRents at once (4 vCPU / 8G VMs unless said)Takes
releasenothingthe build VM, 12 vCPU / 16Gabout 22 min
testafter: [release]the CI VM, then the staging network: atproto, gateway, 2 nested-virt bidders (5 VMs, 20 vCPU / 40G; the entries' guests run inside the bidder VMs)about 2 h 40 min: 9 min stand-up, then 47 entries
promoteafter: [release, test]the release VM, then the prod gate's rentals: two drivers (bob, alice), at most two entries at once, at most 6 VMs of their weights (2 vCPU / 4G each by default); 0001 alone: one k8s node per bidderabout 100 min: deploy and heartbeats, then 31 entries (about 4840 s scheduled from entries.txt, 5300 to 5800 s expected; three drivers measured 3142 to 4237 s)

So about 4 h 15 min a cycle. A VM commits its memory plus a tenth (at least 512M): 8.8 GiB for 8G, 4.5 GiB for 4G. The four prod bidders' offerings (2026-11-19): hat-7 12 cores / 31 GiB, hat-8 16 / 46 GiB (about 33 GiB free: hat-8 builds), hat-9 8 / 31 GiB, hat-10 4 / 15 GiB (about 7 GiB free: it runs a GitLab, so no 8G VM and one 4G one); 40 cores, 80 vCPUs at the bidders' 2x, about 100 GiB free. test needs 44 GiB of 8G VMs (nine fit: three each on hat-7, 8, 9); promote about 45 GiB. A push to main while a cycle runs starts the next release at once and its test when this one's ends, so that test and this promote rent together (about 89 GiB): the series holds for one commit only (the runner's superseded: finish turn is per workflow name), so land one code change per cycle; closing it needs a runner turn group over test and promote (not built).

After a red test

The failed test holds its network 30 min for attach (above), then releases it. When its check ends, promote is filed failure without running ("not run: test, which it runs after, ended failure"; a cancelled test: cancelled): nothing reached prod (knotless crates/knotless-run/src/lib.rs after, not_after).

  1. Find the cause (the check's log and runs/ artifact; AGENTS.md "Debug a failed CI run").
  2. A fix in code: land it; its push runs the whole cycle. Nothing to ask.
  3. No code change (an outside failure, root-caused): ask test again, at main's head (a rerun names a ref, not a commit: when main moved on, the newer commit's own cycle covers it; ask nothing). A rerun of test does not start promote (an asked run is one workflow), so ask it once the new test check reads running:
U=knotless://socialweb.computer/socialweb-computer-kcp
knotless run rerun $U --ref refs/heads/main --workflow test
knotless run list $U --ref refs/heads/main --workflow test --commit <sha>    # wait for: running
knotless run rerun $U --ref refs/heads/main --workflow promote                # waits for that test; filed failure again if it is red

Asked before the new test check is filed, promote reads the old red one and is filed failure again: ask it again. promote.yml's first step, a plain one (scripts/promote.sh test: scripts/ci-check-wait.sh test <sha>), also requires the commit's test to have run green before release-workload does anything (it moves channel rc before the release step's command runs), so a promote asked at a commit whose test is red, missing or path-skipped (a docs-only commit: success with a skipped field) ends there: rc never moves, and the stop finds no rc of this run.

A rerun of test at a commit channel rc already points at (after a green promote moved rc to it) fails 0007:upgrade: its from binary is the previous rc (SWC_COMPAT_REQUESTER, scripts/release-prev-rc.sh reads channel rc), which is then the binary under test, and the entry ends unmet: the from binary (...) is the binary under test: channel rc already points at this commit ..., a FAIL. A normal cycle runs test before promote moves rc, so it never meets this; by hand, give that rerun an earlier release (sha:<its commit>) as the input's compat, or as SWC_UPGRADE_FROM for the entry run alone (step 5b). 0007:fence reads neither: its from binary is its own (release:v0.0.0-rc186, the last release without decision 54's rule), whatever channel rc points at.

One cycle: release, then test

# 0. the market is clear (list; `delete` deletes every ComputeVM on every bidder's kcp)
scripts/ci-hand-market.sh list

# 1. release, by hand, for HEAD (main's commit): the steps of release.yml
scripts/swc-heavy.sh scripts/ci-release.sh checks
scripts/swc-heavy.sh scripts/ci-release.sh binaries
( set -a; . "$CI_BUCKET_CREDS"; set +a
  scripts/swc-heavy.sh scripts/ci-release.sh test-host     # builds and publishes tests/<tree>/, meshd/<tree>/ when absent
  scripts/swc-heavy.sh scripts/ci-release.sh page          # 0009's page payload (stage reads it)
  scripts/ci-release.sh guest-image                         # builds only when guest-images/<target>/<pin>/ is not published
  scripts/ci-release.sh stage
  scripts/ci-release.sh publish                             # local/<sha>/ only (main/<sha>/ is main's CI's)
  scripts/ci-release.sh verify )

# 2. test, by hand: the steps of test.yml (the account through its associated requester)
export CI_IDENTITY_NAME CI_IDENTITY_DIR
scripts/ci-test.sh host
scripts/ci-test.sh binary                                   # requester spec sha:<HEAD>
scripts/ci-test.sh staging 2>&1 | tee "$CI_HOME/staging-$(git rev-parse --short HEAD).log"   # not under swc-heavy

CI's input is the default (input_default: one staging network, two bidders, every entry of tests/entries.txt, deadlineMin 250). A single entry is for the held network (below). There is no subset form: in the multi layout the gate starts only when suite is true; a list ("suite": ["0005"]) stands the network up, holds it 65 s, takes it down and exits 0 with zero entries run (2026-11-20; to be made a failure after plan 0040's rolls).

CI, for a commit:

U=knotless://socialweb.computer/socialweb-computer-kcp
scripts/ci-staging-total.sh <sha>                 # test: <status> N of 47 PASS, then gate: staging: N of 47 PASS
knotless run list $U --workflow test --commit <sha> --log
knotless run attach $U --workflow test --commit <sha> --key ~/.tmp/ci-attach/attach_key.pub > attach.json   # its frozen run
# a branch: an asked run is one workflow, so ask each; test once release's new check reads running (after: waits for it then;
# asked before, test may read release's older check, or none, and not wait)
knotless run rerun $U --ref refs/heads/<branch> --workflow release
knotless run rerun $U --ref refs/heads/<branch> --workflow test

scripts/ci-staging-total.sh <sha> [--wait <secs>] exits 0 only when test ended success with 47 of 47; 3 while it is queued, running or has no check yet; 1 otherwise.

Shares by hand

tests/gate.sh can deal the entries into n shares (GATE_SHARD=i/n; scripts/ci-test.sh CI_SHARD=i/n sets the staging input's shard): CI runs n = 1 (owner, 2026-11-19: one network, less compute). The shares are disjoint, none is empty and their union is the list, for any n up to the entry count (scripts/check-shards.sh, in deploy/check.sh). The rule (gate_ids): the run-alone entries (EXCLUSIVE) first, then the rest, each class longest first by the seconds column of tests/entries.txt, each to the share with the least seconds so far; a share keeps gate order. No entry needs another on its network (plan 0040, Finding "Gate shards"). One hand identity is one requester, so one staging run at a time: shares by hand run one after another.

for i in 1 2 3 4; do echo "$i/4: $(GATE_SHARD=$i/4 tests/gate.sh --list staging | tr '\n' ' ')"; done   # nothing started
CI_SHARD=2/4 CI_TEST_INPUT='{"layout": "multi", "provision": "infra", "deadlineMin": 150, "suite": true, "bidders": 2, "gateJobs": 4, "freezeTtl": 5}' \
  scripts/ci-test.sh staging 2>&1 | tee "$CI_HOME/staging-2of4.log"                 # gate: staging: K of M PASS (shard 2/4)

Shares in CI again would be one workflow per share (a workflow file has no fan-out; one CI VM is one guest requester): the generator scripts/ci-test-workflows.sh and its template, removed by the commit that put CI in series (its parent has them), and promote.yml after every share.

A fix: commit it on the branch, move CI_HAND_WORKTREE to the new branch commit, and run the cycle again. Land (git push swc HEAD:refs/heads/<b>, knotless pull create, knotless pull merge <pull> --queue) only once a cycle is green.

Differences from CI, and only these:

CIby hand
accountr2, the CI VM's guest requester (root runs it with runuser)CI_ACCOUNT through its associated host requester: no sign-in per run
paths/home/r2, /usr/local/bin, /work/runs, /rootCI_HOME
artifactsKNOTLESS_ARTIFACTS_URLthe bucket
publishmain/<sha>/ and main/latest/local/<sha>/ only (requester spec local:<sha>)
platformslinux/amd64, linux/arm64CI_PLATFORMS
freeze TTL1800 s on mainCI_FREEZE_TTL=7200, ended by hand (SIGTERM) when debugging is done

Promote, by hand: the dry run on a VM with the release image

promote.yml cannot be rehearsed by an asked run (release-workload writes the rc's records before its command runs). scripts/promote.sh --dry-run needs no secret, deploys nothing, moves no channel and runs no gate. It runs scripts/ci-prod-target.sh --dry-run (the prod target with the paths of its secret files, none written; the pinned node, Chrome and deno when the page is there), fetches and checks the binary, reads each rc operator's heartbeat once, reads the previous rc (scripts/release-prev-rc.sh, 0006:compat's requester), prints the droplet deploys and the gate command, runs the test host's --dry-run of every prod composition on that target (the target, the composition and the requester binary load; no account is touched), and prints the bucket copy a green promote ends with (main/<sha>/ to releases/<RELEASE_STABLE_VERSION>/, plan 0040 step 15) with what both hold now (public reads; nothing is copied). On the release binding's image and size (socialweb.computer's com.fedfork.exec.binding/release: ci-toolchain 0049d853, 4 vCPUs, 8 GiB), as the account:

S=$(git rev-parse HEAD) M=<the main commit whose release published this tests/ and rust/ tree>
B=https://socialweb-computer-kcp.sfo3.digitaloceanspaces.com I=0049d853de17ff3bedfc3c9931b99906fd46c6bc108c1c8be2118d68d27f4dec
d=$HOME/.tmp/local-src/$S; mkdir -p $d; git archive HEAD | gzip > $d/src.tar.gz; cp scripts/promote-rehearse.sh $d/
(set -a; . "$CI_BUCKET_CREDS"; set +a; deno run -A --no-lock scripts/publish-spaces.ts --dir $d --prefix local/$S/src)
RU=$B/main/$M/binaries/socialweb-computer-linux-amd64
RS=$(curl -fsS $B/main/$M/binaries/SHA256SUMS | awk '$2 ~ /linux-amd64$/ {print $1}')
( . "$CI_ACCOUNT_CREDS"; p=${CI_ACCOUNT_PREFIX}_ATPROTO_PASSWORD; export ATPROTO_PASSWORD=${!p}
  socialweb-computer run request --state-dir ~/.tmp/rehearse-state --name promote-rehearse --image "$B/vm-images/ci-toolchain/$I@$I" \
    --cpus 4 --mem 8G --expires-in 1h --atproto-identity-mode pdsAccount --atproto-handle <handle> --atproto-pds-url https://bsky.social \
    --exec -- bash -c "curl -fsSL $B/local/$S/src/promote-rehearse.sh -o /var/tmp/r.sh && \
      sudo -n bash /var/tmp/r.sh $B/local/$S/src/src.tar.gz $(sha256sum < $d/src.tar.gz | cut -c1-64) $M $RU $RS" ) 2>&1 | tee ~/.tmp/rehearse.log

Pass: ==> test at <commit> ...: <status> now and the promote.sh test that promote.yml runs first, one ==> would write line per secret file (two creds: alice, bob; the ops read secret, the agent config), ==> page ... tools ..., ==> prod target with its paths (four bidders), ==> compat requester sha:<commit>, ==> would run: the deploy and the ssh restart, one ==> line per rc operator (four), the gate command (./gate.sh /home/r2/target-prod.json --account bob,alice with SWC_COMPAT_REQUESTER and GATE_VMS=6), one dry run <runbook>: composition ok line per prod runbook (17), the copy-spaces.ts command and its check, ==> rehearsal done. 2026-11-17 on main de0b91ee the older dry run (no target, no tools, no test host) passed; this one has not run on a VM yet.

The prod target's bidders: scripts/ci-prod-target.sh gives each bidder ops: {url} (its https market endpoint) when its offering has one, else ops: {iroh} (its iroh endpoint ticket: every prod bidder today); it exits 1 for a bidder with neither (no https or iroh market endpoint in its offering). Check on hat-8 with an empty HOME, no secret:

H=$(mktemp -d ~/.tmp/fakehome-ops.XXXX); env -u TMPDIR HOME=$H CI_HOME=$H/ci bash scripts/ci-prod-target.sh --dry-run $H/t.json
jq -c '[.bidders[] | {name, ops: (.ops | keys)}]' $H/t.json

Reach of one bidder's market mux over its iroh endpoint, by hand (needs dumbpipe on PATH; /healthz 200 on every build, an ops route 404 on a build without them or without the secret configured, 401 on a wrong bearer):

t=$(jq -r '.bidders[0].ops.iroh' $H/t.json)
printf 'GET /healthz HTTP/1.1\r\nhost: guest\r\nconnection: close\r\n\r\n' | IROH_SECRET=$(openssl rand -hex 32) timeout 60 dumbpipe connect $t | head -1

The same without a VM, on hat-8, as the caller (no tools unless $CI_HOME/page holds the page; the test host's dry run only when tests/target/release/swc-test-host is there):

CI_HOME=$HOME/.tmp/promote-dry RELEASE_VERSION=<the rc> RELEASE_STABLE_VERSION=<its vT> RELEASE_URL=$RU RELEASE_SHA256=$RS \
  bash scripts/promote.sh --dry-run

Without $CI_HOME/requester.spec (scripts/ci-test.sh binary writes it, sha:<commit>) the target keeps prod.json's release, the version channel stable points at: before the first stable that fails the test host's dry run (it names the fix); write sha:<M> there first.

The real promote's gate is the hat-8 gate's command below, as the unix user r2 on the release VM (promote.sh prepare makes it; no guest requester runs there), on the target scripts/ci-prod-target.sh writes from the secrets .knotless/workflows/promote.yml lists. Open (plan 0040 step 11): the release VM does not reach the bidder hosts (ssh hat-N: each bidder's kcp and its host's journal), so the entries that read them fail there until the owner decides how.

Promote runs after the staging gate (after: [release, test]; on every push to main with their paths, plan 0040 step 15; in series since 2026-11-19): prod gets the build only once staging passed it. Its first step, a plain one before the release step (scripts/promote.sh test), reads the same commit's test check again and goes on only when it ran green (red, cancelled, path-skipped or none at that commit, as an asked promote may find: exit 1 before release-workload moves channel rc, so nothing moves). After a green prod gate promote.sh copies main/<sha>/ to releases/<vT>/ in the bucket (scripts/copy-spaces.ts, the bucket key SPACES_* sealed to main; vT is release-workload's RELEASE_STABLE_VERSION) and checks the copy from the public URLs as release's verify does (the two SHA256SUMS equal, RELEASE_SHA256 in it, every file read without a credential and hashed); a failure there is a red promote too. Exit 0: release-workload writes version vT on the rc's manifest and moves channel stable to it. The wait is scripts/ci-check-wait.sh, which reads the runner's public check record (no credential, no knotless CLI):

bash scripts/ci-check-wait.sh --once test <commit>      # the status now
bash scripts/ci-check-wait.sh test <commit>             # waits while queued or running; exit 0 only on success

A red, cancelled or timed-out promote rolls back at once (plan 0040 step 12). Red: release-workload moves channel rc back and runs scripts/promote.sh rollback in the step itself. Cancelled or timed out: the runner drops the step, so promote.yml's stop: release-rollback does it (the same module, from the records: the newest rc is this commit's, not stable, and its previous names where rc goes back to); after a green or an already rolled back promote the stop moves nothing. promote.sh rollback first ends the promote still running on the VM ($CI_HOME/promote.pid: SIGTERM to its process tree, a test host then releases its rentals), then the droplets, then every rc operator's heartbeat on the previous binary. The module's half is rehearsed in the requester repo (fail, cancel, timeout, each then the stop):

cargo wasm -p release-workload -p cicd-workload -p process-mcp && node --test modules/release-workload/test.mjs

Debug a rolled-back promote from its gate-runs artifact and its log; nothing is held.

The release binding

socialweb.computer's com.fedfork.exec.binding/release (only promote uses it) asks for no guest requester: its inputs are {mem, cpus, image}, no association. The prod gate signs in as bob and alice from their creds files (ci-prod-target.sh); nothing on the release VM runs as R2. With association: true the runner associates a guest requester beside the step (knotless crates/knotless-run/src/composition.rs, computer.socialweb.vm.associate, 200 polls of 10 s from the rent) and, when that gives up, stops the step mid-gate (==> stopped: the guest requester was not associated: ... (gave up), then a misleading ==> killed after 14400s). promote.sh prepare, promote's first command, exits 1 at once on such a VM (it has the association cloud-init's /etc/systemd/system/socialweb-r2.service). Read it (public), and drop association as the account (its creds file ~/.socialweb_computer_source, as publish-chain.sh in the requester repo uses it; a swapped write, the record read first is the undo):

D=did:plc:gmlkrhimoueejcertevr4hl3 P=https://brittlegill.us-west.host.bsky.network
mkdir -p ~/.tmp/release-binding
curl -fsS "$P/xrpc/com.atproto.repo.getRecord?repo=$D&collection=com.fedfork.exec.binding&rkey=release" > ~/.tmp/release-binding/before.json
jq -c .value.inputs ~/.tmp/release-binding/before.json
tok=$( (set -a; . ~/.socialweb_computer_source; jq -n --arg i "$D" '{identifier: $i, password: env.SOCIALWEB_COMPUTER_ATPROTO_PASSWORD}') |
  curl -fsS -H 'content-type: application/json' -d @- "$P/xrpc/com.atproto.server.createSession" | jq -r .accessJwt)
jq -c --arg d "$D" '{repo: $d, collection: "com.fedfork.exec.binding", rkey: "release", swapRecord: .cid, record: (.value | del(.inputs.association))}' \
  ~/.tmp/release-binding/before.json | curl -fsS -H "authorization: Bearer $tok" -H 'content-type: application/json' -d @- "$P/xrpc/com.atproto.repo.putRecord"
curl -fsS "$P/xrpc/com.atproto.repo.getRecord?repo=$D&collection=com.fedfork.exec.binding&rkey=release" | jq -c .value.inputs

Undo: the same putRecord with record: .value of before.json (and swapRecord the new CID).

The hat-8 gate

cd tests && SWC_COMPAT_REQUESTER=sha:<previous rc commit> GATE_VMS=6 ./gate.sh $PWD/targets/prod.json --account bob,alice

The prod gate's drivers are bob and alice (owner, 2026-11-19: "should just be alice and bob now"; john, a third driver from 2026-11-18, plan 0040 decision 39, is out of the target), two entries at a time; GATE_VMS=6 is two slots times the heaviest entry weight (3: 0003, 0012), so the accounts, not the VM cap, bound it: at most 6 VMs at once, about 27 GiB, which the four prod bidders hold beside the release VM ("CI's cycle, in series"). Each account's creds file exports <P>_ATPROTO_HANDLE and <P>_ATPROTO_PASSWORD (P its oauthAccount: ALICE, BOB), one value per line written as is (the test host and scripts/oauth-signin.ts strip only a quote pair around the whole value, never shell escapes). socialweb.computer is not a driver: CI rents its own VMs as that DID while promote runs, so 0002's and 0006's assert.noneSince scans of the driver's repo would count them as leftovers (plan 0040 findings, 2026-11-18).

As bob on hat-8; the target's requester set to sha:<the rc's commit> (plan 0040 step 14). No landing on main while it runs.

The static test host needs SWC_REPO=<the checkout> in its environment, as promote.sh sets it; without it the run ends at once.

This gate runs its requester on the host. promote runs it inside a VM rented on a prod bidder, so a hat-8 pass says nothing about guest-side rules or WebRTC from a guest to a guest on the same host: for those, rent a VM on prod with --guest-tools socialweb-computer and run the entry from inside it ("Mistakes that cost a cycle", 5).

Iterate on a held staging network

Do not stand staging up per fix (about 8 min each). One network per build, every entry on it (owner, 2026-11-15):

  1. Run CI's entry list (the default input; suite.sh goes on past a failed entry, one RESULT line each).
  2. The run freezes when any entry failed; a run whose suite started can be frozen by hand (pkill -USR1 -f '^target/release/swc-test-host compositions/staging': the suite already running on the VM finishes).
  3. On the held network, each failed entry: root cause, then a fix, patched in place when it can be (a tests/ change: put the suite again; a Go change: swap the binary) and the entry rerun alone. Each goes into Findings.
  4. Only when every failed entry has a root cause and a fix: end the run, commit the fixes on the branch, next build, next full run; land once one is green.

The commands:

# 1. hold it: a failed entry freezes the run (the default), every VM held for CI_FREEZE_TTL; iterating, keep it long
CI_FREEZE_TTL=7200 scripts/ci-test.sh staging ...            # a passing run: pkill -USR1 -f '^target/release/swc-test-host compositions/staging'
# 2. reach it: the held contracts (einfra-stg-<n>-atproto, -bidder-1 to -bidder-<bidders>, -gateway) and ssh to each
BIDDERS=$(seq -f bidder-%g 1 2)                              # the input's `bidders` (default input: 2)
K=$CI_IDENTITY_DIR/requesters/stg/kubeconfig WS=$(jq -r .workspace "$CI_IDENTITY_DIR/requesters/stg/ready.json")
S=$(sed -n 's/^ *server: //p' "$K" | head -1)
kubectl --kubeconfig "$K" --server="${S%/clusters/*}/clusters/$WS" get computecontracts
SWC=~/.tmp/swc-system-tests/bin/main-$(git rev-parse HEAD)/socialweb-computer-linux-amd64
on() { $SWC run requester ssh stg --state-dir "$CI_IDENTITY_DIR" --contract "einfra-stg-<n>-$1" --exec -- bash -c "$2"; }
# 3. rerun ONE entry: step 5 (hand.sh). suite.sh takes only a run name and runs the WHOLE gate (it removes gate/ first)
# 4. a tests/ change (a module, a composition, target.sh, suite.sh): rebuild the wasm, put the suite the way the module does
(cd tests/modules && cargo wasm)
tar -czf - --exclude=did-key-associator/dist --exclude=did-key-associator/dist-backend tests/gate.sh tests/entries.txt tests/compositions \
  tests/targets did-key-associator typescript-helpers/lib/oauth-scope typescript-helpers/lib/cli-args-env typescript-helpers/lib/logger \
  typescript-helpers/lib/serve tests/modules/*/module.json tests/modules/target/wasm32-unknown-unknown/release/*.wasm deploy/vm-images \
  examples scripts | on atproto 'd=/home/stg/stg; cat > $d/suite.tgz; rm -rf $d/suite; mkdir -p $d/suite; tar -xzf $d/suite.tgz -C $d/suite; chown -R stg:stg $d/suite'
# 4b. 0009's page (plan 0040 step 8f): the suite carries it when the host that puts it has SWC_PAGE_REPO, a built checkout
#     of the requester page repo (tests/testkit/README.md, pageRepo); by hand, the same files beside the suite, then target.sh
#     (page.sh: pinned node and Chrome into /home/stg/stg/page/tools, the target's pageRepo). A full run: SWC_PAGE_REPO=<checkout> scripts/ci-test.sh staging
R=~/.tmp/swc-system-tests/page/atlibs-socialweb-computer-requester; p=$(mktemp -d); mkdir -p $p/page/web $p/page/scripts
cp -rL $R/web/webrtc $p/page/web/; cp $R/scripts/page-round.mjs $R/scripts/browser.mjs $p/page/scripts/; git -C $R rev-parse HEAD > $p/page/COMMIT
tar -czf - -C $p page | on atproto 'd=/home/stg/stg/suite; rm -rf $d/page; tar -xzf - -C $d; chown -R stg:stg $d/page'
on atproto 'runuser -u stg -- env HOME=/home/stg SWC_REPO=/home/stg/stg/suite /home/stg/stg/suite/tests/compositions/staging/page.sh'
on atproto 'D=/home/stg/stg; jq --arg p $D/page ".pageRepo = \$p" $D/target-staging.json > $D/t.json && mv $D/t.json $D/target-staging.json && chown stg:stg $D/target-staging.json'
# 5. a Go change: a RELEASE build of the fix's commit (a dev build's requester never comes Ready), one
#    platform, about 10 s warm; to the bucket, then onto the VM by URL (ssh streaming is slow) and in by mv (running
#    processes keep the old file). Only the atproto VM's binary: it is the requester under test and writes guests' user_data
C=$(git rev-parse HEAD); scripts/swc-heavy.sh scripts/build-socialweb-computer.sh --commit $C --platform linux/amd64
P=~/.tmp/ci-hand/pub-$C; mkdir -p $P; cp ~/.tmp/socialweb-computer-build/release/$C/socialweb-computer-linux-amd64 $P/socialweb-computer
(cd $P && sha256sum socialweb-computer > SHA256SUMS); T=local/$C/fix
(set -a; . ~/.spaces_swc_kcp_source; set +a; deno run -A --no-lock scripts/publish-spaces.ts --dir $P --prefix $T)
on atproto "B=/home/stg/stg/socialweb-computer-linux-amd64; curl -fsSo \$B.new https://socialweb-computer-kcp.sfo3.digitaloceanspaces.com/$T/socialweb-computer \
  && echo '$(cut -d' ' -f1 $P/SHA256SUMS)  '\$B.new | sha256sum -c --quiet && chmod 0755 \$B.new && chown stg:stg \$B.new && mv \$B.new \$B"
# 5b. rerun one entry frozen on failure (suite.sh cleans up a failed entry: its guests are gone), its own log file
on atproto 'D=/home/stg/stg; cd $D/suite/tests && (setsid nohup runuser -u stg -- env HOME=/home/stg XDG_RUNTIME_DIR=/run/user/$(id -u stg) \
  SWC_READINGS=$D/readings SWC_REPO=$D/suite SWC_TARGET=$D/target-staging.json $D/swc-test-host compositions/0005 --account alice1 \
  --entry run --on-fail freeze --freeze-ttl 3600 > $D/hand-0005.log 2>&1 < /dev/null &)'
# 5c. inside the entry's guests while it runs or is frozen: the entry's control socket (its run dir) has the tools of
#     `attach` (state, failure, resources, kube_get, ssh_exec ...). ssh_exec runs a command in a run request's VM (the
#     name from `resources`) over its iroh ssh as the guest user (sudo -n for root); "connection lost" after the output is
#     its teardown, not a failure. The command goes in a file: no nested quoting
cat > ~/.tmp/sx.py <<'PY'
import socket,json,sys
s=socket.socket(socket.AF_UNIX); s.connect(sys.argv[1]); f=s.makefile("rw")
tool,args=sys.argv[2],json.loads(sys.argv[3]) if len(sys.argv)>3 else {}
if tool=="ssh_exec": args["exec"]=open(args.pop("file")).read()
for m in [{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}},{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":tool,"arguments":args}}]:
    f.write(json.dumps(m)+"\n"); f.flush(); r=json.loads(f.readline())
    if m["id"]==2:
        t=r["result"]["content"][0]["text"]
        try: print("\n".join(l for l in json.loads(t).get("output",t).splitlines() if not l.startswith('{"time') and "connection lost" not in l and "tracing-subscriber" not in l))
        except Exception: print(t)
PY
on atproto "echo $(base64 -w0 ~/.tmp/sx.py) | base64 -d > /tmp/sx.py"
R=$(on atproto 'ls -dt /home/stg/.tmp/swc-system-tests/runs/0005-composition-* | head -1')/control.sock
on atproto "runuser -u stg -- python3 /tmp/sx.py $R state"                                  # phase, marks
on atproto "runuser -u stg -- python3 /tmp/sx.py $R resources"                              # invocation names
printf '%s\n' 'sudo -n sh -c "ps -eo args | grep [r]un.request; tail -c 3000 /home/r2/.local/state/socialweb-computer/requests/r2-st/dev/r2-st/logs/core.log"' > ~/.tmp/g.sh
on atproto "echo $(base64 -w0 ~/.tmp/g.sh) | base64 -d > /tmp/g.sh; runuser -u stg -- python3 /tmp/sx.py $R ssh_exec '{\"name\":\"<invocation>\",\"file\":\"/tmp/g.sh\"}'"
#     its requester (the outer `run request`): /home/stg/stg/s/<invocation>-request.log while it runs, under the run's
#     logs/ after; R2 inside the guest: /home/r2/.local/state/socialweb-computer/requests/r2-st/dev/r2-st/{logs,kcp}
#     end it: python3 /tmp/sx.py $R cleanup
# 5d. a Go change to the network itself: swap the binary (5), then restart its core on it (the unit's path is the same,
#     the process is new: check its start time), then what new VMs get from user_data, by hand on the held ones
on atproto 'runuser -u stg -- env HOME=/home/stg XDG_RUNTIME_DIR=/run/user/$(id -u stg) /home/stg/stg/socialweb-computer-linux-amd64 \
  dev restart --instance stg --state-dir /home/stg/stg/s core --binary /home/stg/stg/socialweb-computer-linux-amd64'
on atproto 'cd /home/stg/stg/suite/tests && runuser -u stg -- compositions/staging/target.sh hand > /home/stg/stg/target-staging.json'
#     the bidder VMs keep the old network value (net.json): their name overrides dial the old front ("networkdial: dialing
#     10.44.0.1:<old port> ... connection refused"). Give each the new one and restart its core unit (the bidder is
#     installed: account swc-bidder, unit socialweb-computer-bidder@<handle>; kcp and VM units stay)
NET=$(on atproto 'cd /home/stg; runuser -u stg -- env HOME=/home/stg XDG_RUNTIME_DIR=/run/user/$(id -u stg) /home/stg/stg/socialweb-computer-linux-amd64 \
  dev status --instance stg --state-dir /home/stg/stg/s | sed -n "/^{/,\$p" | jq -c .network.status | base64 -w0')
for v in $BIDDERS; do on $v "echo $NET | base64 -d > /etc/socialweb-computer/net.json; chmod 0644 /etc/socialweb-computer/net.json"
  on $v 'A=swc-bidder; runuser -u $A -- env XDG_RUNTIME_DIR=/run/user/$(id -u $A) systemctl --user restart "socialweb-computer-bidder@*"'; done
#     a new binary on a bidder VM (the URL and sha256 of step 5): `upgrade` as swc-bidder switches bin/current and
#     restarts the core unit (`install bidder` with another binary than bin/current refuses and names this command)
for v in $BIDDERS; do on $v "B=/var/tmp/swc-fix; curl -fsSo \$B https://socialweb-computer-kcp.sfo3.digitaloceanspaces.com/$T/socialweb-computer && chmod 0755 \$B \
  && A=swc-bidder ST=/var/lib/socialweb-computer/swc-bidder && runuser -u \$A -- env XDG_RUNTIME_DIR=/run/user/\$(id -u \$A) \
  \$ST/bin/current/socialweb-computer upgrade --binary \$B --sha256 $(cut -d' ' -f1 $P/SHA256SUMS) --state-dir \$ST"; done
on bidder-1 'A=swc-bidder; runuser -u $A -- env XDG_RUNTIME_DIR=/run/user/$(id -u $A) systemctl --user status "socialweb-computer-bidder@*" --no-pager | head -5'
#     from nothing (a new DID: trust.sh vouch and bidders.json again; the stand-up's own command, tests/compositions/staging/README.md step 4)
on bidder-1 '/var/lib/socialweb-computer/swc-bidder/bin/current/socialweb-computer uninstall bidder --purge'
# 5e. past the freeze TTL (it cannot be extended; its thaw runs the cleanup): pause the driver, the VMs stay up to their
#     EphemeralInfra ttl; resume it to end the run (its cleanup then runs at once)
kill -STOP $(pgrep -f '^target/release/swc-test-host compositions/staging')     # later: kill -CONT <pid>
# 6. done: SIGTERM the driver; its cleanup releases every VM (then scripts/ci-hand-market.sh list)

Bidder logs on a bidder VM (installed, plan 0040 step 8b): journalctl _UID=$(id -u swc-bidder) -o cat --no-pager as root (the unit alone: _SYSTEMD_USER_UNIT=socialweb-computer-bidder@<handle>.service); its kcp: /var/lib/socialweb-computer/swc-bidder/handles/*/kcp/admin.kubeconfig; its VM units: runuser -u swc-bidder -- env XDG_RUNTIME_DIR=/run/user/$(id -u swc-bidder) systemctl --user list-units "socialweb-vm-*"; an entry's requester on the atproto VM: /home/stg/stg/s/dev/stg/apps/dev/stg-<request name>/logs/core.log.

Mistakes that cost a cycle

The lead's own, on plan 0040 (2026-11-20). Each has the rule it left; plan 0040 Findings has the runs.

#What I didWhat it costRule
1Proved 0007:upgrade on a network where I had put the binaries on the bidders by handIn the full gate the test host copied 4 x 590 MB over the overlay: the entry timed out at 60 min and the load failed 0009:runA proof on a hand-prepared network does not count for anything a fresh network does differently. Prove on a fresh network, nothing pre-staged
2Accepted the capacity redesign on six reviews and the entries written for it; nobody ran 0014:abandonA product regression (500 for an accept the bidder cannot read yet) reached the final hand cycle; one more 2 h cycleA change is proven by every entry of each family it touches, not a sample. A review is not a run
3Let the secrets work change a shared test-host call (repo.listCreated) and proved it only on staging0006:compat failed on prod as alice (her repo is large; staging repos are new); one more 2 h cycleA change to a shared host call is run by every caller, against the largest real repo it will read (alice on prod, read-only)
4Asked for a seven-entry staging run without checking a subset path exists; did not read the run's endThe run exited 0 with zero entries; 40 min before I saw itRead the gate summary line and count the RESULT lines of every run. exit 0 is not a result. If the count is not the list's, it failed
5Ran the hand prod entries from hat-8 itself and called three WebRTC entries "first proven by the roll"Promote 93fd8c11 red (28 of 31): its requester is inside a rented guest, booted by the old bidder; a 4 h roll and a rollbackRehearse in promote's topology: the requester inside a VM rented on a prod bidder, the target on the same host. Anything "only the roll can prove" gets a rehearsal designed first
6Shipped a guest-side rule (decision 54) and expected promote's own VM to have itSame red promoteWhat a release changes in guest rules, guest agent or vmexec is not on guests booted before the roll. Ask of every such change: who boots the guest that needs it
7Trusted agents' watchers and my own (a Monitor that expired, a watcher on the wrong file, a tail opened before the file existed)Failures and ended runs sat unread four times, 27 to 30 min eachAfter arming a watcher, check it against the real file once. Read result files myself every 15 to 20 min. A watcher must fire on every end state, not only on success
8Told an agent to stop new proofs, and another to do a subset, to save timeThe proofs cut were the ones that matteredNever trade a proof for time. If a full gate is the only path that exists, run the full gate
9Let a fix change the tree under test after its "final" cycle, three timesEach needed a further cycleLand only a tree that passed whole. A fix after the cycle means another cycle: plan for it, do not argue it away
10Changed a host's firewall with a command I had not read backIt errored on the zone name and changed nothing; caught because I printed before and afterPrint the state before and after every change on a host
11Accepted a test entry whose precondition was "the previous release lacks the feature" (0007:fence took its from binary from the previous rc)The entry failed for ever the moment its own roll went green; caught before the next roll, one more fixAsk of every new entry: does it still pass after it lands, on the roll after, and after a rollback
12Added a pinned-binary bump (dumbpipe) to a fix stack late, proven by pins and a file diff, without switching a running bidder between the two buildsThe full hand cycle found both bidders deaf over iroh after the switch, in both directions: on prod it would have been an outage at the roll and at the rollbackA change of a pinned binary or of anything a core stores across restarts is proven by 0007:upgrade alone first, both directions, before it joins any stack
13Kept adding fixes to the stack as I found intermittentsFive hand cycles and no second roll in a dayFreeze the stack once it holds what failed; anything new that is not a regression of the stack goes to its own roll
14Rolled after checking the four bidder hosts but not the runnertest red in 80 s: the runner container held 173 zombie ssh, 199 of its 256 processesBefore a roll read the runner container's pids.current on the gateway host; over 100: systemctl restart knotless-fork-runner.service (no run active)
15Asked for a rerun of a red test before pulling its artifactsThe rerun pruned them; the bidder's side of the failure was never readPull a failed run's artifacts (knotless run artifacts, a fork-built CLI) before any rerun
16Waited on a CI test by its state word onlyThree hours on a run that was 6 PASS of 20At the first doubt read the run from inside: knotless run attach, then swc-test-host attach --ci (state, the gate dir's RESULT lines); a pass ends near 2 h
17Took seven hand cycles as a rehearsal of CICI's placement (its own VM on the hosts) and its runner were never exercised by handBefore the roll of a batch, one hand network in CI's layout with the overlay measured (pair matrix); save a meshd log of a healthy network
18Named a cause (VM placement) from one run's layoutTwo green rolls had the same layoutCompare with the runs that passed before naming a cause

Before a pull is queued, each of these is a yes:

  • Every entry of every family the change touches ran, on a fresh network, nothing pre-staged.
  • The gate summary line and the RESULT count were read from the log, by the lead.
  • Shared test-host or module changes ran against prod's largest repo, read-only.
  • Anything that differs on prod (hosts, routers, repo sizes, a requester inside a guest, guests booted by the old bidder) has a rehearsal that has that difference, or is written in the pull as unproven with the reason.
  • The tree in the pull is the tree that passed.
  • The runner container's process count was read (mistake 14); a healthy network's overlay (meshd) log is saved.

Do not waste time

Learned on plan 0040's hand cycles (2026-11-17/18); each line cost time once.

Two loops, never mixed:

LoopForTakes
survey: the whole gate on a networkthe number (N of 47), entries that break each other2 h 17 min of gate on two guest slots (hand cycle 3, three gate jobs), after 11 to 25 min of stand-up
fix: one entry on a held networkproving one fixminutes
  1. A survey freezes nothing: "freezeTtl": 5 in the input, with "bidders": 2 (the network's bidder VMs bidder-1 and bidder-2, bob2 and bob3, one guest slot each; the gate's VM cap gateVms follows it) and "gateJobs": 4 (four entries at once, one per driver account). All are in the default input (scripts/ci-test.sh input_default, CI's too), so a survey needs no CI_TEST_INPUT ("deadlineMin": 250: every entry on the one network). The suite's own default of 300 s per failed entry made the first survey 8004 s: 14 failures were 70 min of waiting. A network is 4 VMs of 4 vCPU / 8G on the prod bidders (16 vCPU / 32G); with "bidders": 4, 6 VMs. The network of before the bidders input:

    export CI_TEST_INPUT='{"layout": "multi", "provision": "infra", "deadlineMin": 290, "suite": true, "bidders": 2, "gateJobs": 3, "freezeTtl": 5}'
    
    scripts/ci-test.sh staging > "$CI_HOME/staging-$(git rev-parse --short HEAD).log" 2>&1 &
    
  2. Keep the network. A failed suite holds the outer run for CI_FREEZE_TTL (7200 s): prove every fix on it before it goes, and run the gate again on it instead of a new stand-up (no release build, no 25 min):

    on atproto 'runuser -u stg -- env HOME=/home/stg XDG_RUNTIME_DIR=/run/user/$(id -u stg) GATE_FREEZE_TTL=5 \
      /home/stg/stg/suite/tests/compositions/staging/suite.sh <run> > /home/stg/suite2.log 2>&1; tail -3 /home/stg/suite2.log'
    
  3. One entry by hand (debug, never a gate result). Two at once need two accounts (one driver per account):

    on atproto 'cat > /home/stg/stg/hand.sh <<"EOF"
    #!/bin/sh
    D=/home/stg/stg; cd $D/suite/tests
    exec runuser -u stg -- env HOME=/home/stg XDG_RUNTIME_DIR=/run/user/$(id -u stg) SWC_READINGS=$D/readings SWC_REPO=$D/suite \
      SWC_TARGET=$D/target-staging.json $D/swc-test-host compositions/$1 --account $3 --entry $2 --on-fail cleanup > $D/$4 2>&1
    EOF
    chmod +x /home/stg/stg/hand.sh'
    on atproto '(setsid nohup /home/stg/stg/hand.sh 0017 run alice1 hand-0017.log < /dev/null > /dev/null 2>&1 &)'
    on atproto 'sed -n "/^{/,\$p" /home/stg/stg/hand-0017.log | jq -c "{pass, error}"'
    
  4. Ship a fix in under a minute: a Go change is a release build of its commit (about 8 s warm) through local/<sha>/fix (step 5 of "Iterate on a held staging network"); a test module is one crate and one file:

    (cd tests/modules && cargo build -q --release --target wasm32-unknown-unknown -p acc-0013)
    on atproto 'T=/home/stg/stg/suite/tests/modules/target/wasm32-unknown-unknown/release/acc_0013.wasm; cat > $T.new &&
      chown stg:stg $T.new && mv $T.new $T' < tests/modules/target/wasm32-unknown-unknown/release/acc_0013.wasm
    
  5. The gate's RESULT lines are in the run's ticks, not in the staging log. Read only a run dir made after the cycle began (an older one shows last cycle's results as new):

    touch "$CI_HOME/cycle.start"    # before the cycle
    R=$(find "$CI_HOME/runs" -maxdepth 1 -name 'staging-composition-*' -newer "$CI_HOME/cycle.start" | head -1)
    { zstd -dc $R/ticks.ndjson.zst 2>/dev/null; cat $R/ticks.ndjson 2>/dev/null; } | tail -c 400000 | sed 's/\\n/\n/g' |
      grep -o 'RESULT [0-9]\{4\}:[a-z0-9]* [A-Z]* [0-9]*[^"]\{0,120\}\|gate: staging: [0-9]* of [0-9]* PASS' | sort -u
    
  6. A fix is proven when the symptom is seen gone on the held network, not when it builds. A fix landed "proof pending" can do nothing: the first releaseOk fix patched a branch the releases never take (96 of 96 summaries still false).

  7. A failed entry's logs go with its cleanup: read them while it is frozen, or rerun it frozen (--on-fail freeze --freeze-ttl 2400). A component's own log (k3s, postgres, openbao) is under the requester's or cluster's state dir, not in the entry's log.

  8. What is deployed is read from the running system, never from a local checkout: docker image inspect on the gateway for the runner, a bidder's heartbeat version.sha256, socialweb-computer version on the host.

  9. An entry that never ran needs things nobody installed (0017: socat) and may be wrong itself (0013:b gave its password after the install that waits for it). Read its module and host calls before its first run.

  10. A failed entry can leave an install behind (swc-requester, swc-st-qr under /var/lib/socialweb-computer): purge it before the rerun (<state>/bin/current/socialweb-computer uninstall <role> --account <a> --purge).

  11. Never pkill -f and the program's name in one command string (local, ssh or --exec): the shell matches itself and dies silently, and the later steps never run. Kill by pid in a command of its own, then check the next step happened.

  12. Landing: one batch per step when CI is idle. Each push to main runs release, test and promote in series (about 4 h 15 min); a push during a cycle runs its test beside this promote ("CI's cycle, in series"), and stacked build VMs (12 vCPU each) leave the next one without capacity.

  13. Independent work runs in parallel: one agent per layer on its own branch and worktree, read-only agents for open root causes; the lead keeps the held network and merges.

Timing and watching

Times (hat-8, warm caches, 2026-11-15) and the limits that can hold a run:

PartTakesLimit that holds it
release by hand (binaries, test-host when published, stage, publish, verify)3 to 5 minnone
checks~15 minnone
ci-test.sh host, binary~1 minnone
staging stand-up (4 VMs rented, installed, joined, probed)8 to 25 mina full market waits up to 30 min per contract (DefaultCapacityWait)
one entry on stagingprod gate rc168: 0003 2 min, 0005 4 min, 0007 1.5 min, 0012 2 min, 0013 4 minthe entry's own deadline, 40 to 60 min (tests/modules/acc-*)
the whole gate on staging (43 entries; 2 accounts then, 4 now: alice1 to alice4)2 h 17 min with freezeTtl 5, gateJobs 3 and two guest slots (hand cycle 3, 34 of 43); 133 min with 14 failures frozen 300 s each (2 accounts); 152 min estimated with two bidders (plan 0040, Finding "Gate shards")deadlineMin 250 (CI; test.yml's step 300)
the prod gate (31 entries; the seconds below are of the 30 before 0007:webrtc (70 s); bob, alice, GATE_VMS=6)about 4840 s scheduled (the entries' seconds: 0001:run 932 alone, then 7817 s on two slots), 5300 to 5800 s expected at the 1.1x to 1.2x three drivers ran over their schedule (3142 to 4237 s, promotes of 2026-11-18/19); the one two-driver run, by hand before the ops routes, 5471 s with 5 FAILspromote.yml's step 240
RISK: CI's staging step250 min of deadlineMin (stand-up and gate) for about 160 min estimated; the shares ran about 1.3x their estimates and home hosts stand up slower, so about 30 min of margin: past it the suite ends at its deadline and the entries not run FAIL300, the runner's cap: --step-timeout 18000 in knotless-fork-infra host/systemd/knotless-fork-runner.service (a push run's step never goes past it; an asked run's may); raising it is that flag and a runner deploy, then test.yml's timeout and deadlineMin; the other levers are the 30 min freeze (kept) and four staging bidders (about 27 min faster, 2 VMs more)
a failed runthe freeze, CI_FREEZE_TTL (by hand: 7200; CI main: 1800)

Watch every run, and act on a stall instead of waiting for a deadline:

  • Background the step with its output in a log (... > "$CI_HOME/<step>.log" 2>&1 &).

  • Watch the log for the end and every failure: STEPOK, STEPFAIL, done pass=, FROZEN, cleanup:, error, RESULT.

  • Also watch for a stall: a heartbeat every 5 min with the last line that is not a poll (timer poll-, result ...:get, suite-poll, guest_iroh). An entry past 3x its prod time is stuck: read its log tail (the suite poll records it in the ticks: swc-test-host replay "$R", or $R/logs after the failure), then SIGTERM the driver rather than wait for the deadline.

  • An entry that repeats the same failure (one requester_association_failed error every 30 s) has failed: do not wait out its deadline (owner, 2026-11-15; cycle 13 lost 8 min of 0005's 60 before this). Every entry fails fast on a repeated identical error itself (steps::market_event: a retried requester_association_failed with the same error 5 times over 3 min fails it; a refusal is left to the entry). By hand, end the entry and let the suite go on to the next one; the network stays up and is held at the suite's end:

    on atproto 'pgrep -af swc-test-host | grep -v attach'   # the running entry: swc-test-host compositions/<n> ... --on-fail cleanup
    on atproto 'kill -TERM <pid>'                            # it releases its own rentals, RESULT <n> 1, the suite takes the next entry
    
  • An agent hand runner: one watch per run that exits when the run ends, re-armed when the watch tool's time limit ends it; one success-or-failure watch never sees a hang.

L="$CI_HOME/staging.log"
tail -n +1 -F "$L" | grep --line-buffered -E 'STEPOK|STEPFAIL|done pass=|FROZEN|cleanup:|RESULT|error:' &   # outcomes
while sleep 300; do                                                                                 # stall heartbeat
  echo "$(date -u +%H:%MZ) last: $(grep -vE 'timer poll-|:get|suite-poll|guest_iroh|message req-' "$L" | tail -1 | cut -c1-160)"
done

Debug

R=$(ls -td "$CI_HOME"/runs/staging-composition-* | head -1)   # the run dump; its work dir: ~/.tmp/swc-system-tests/work/<same>
jq . "$R/failure.json"                                         # the error, with the failed entry's log tail
ls "$R/logs"                                                   # per VM: its journal and logs (taken before the freeze)
cd tests && target/release/swc-test-host attach "$R"           # a frozen run (MCP): state, ticks, failure, kube_get, ssh_exec
target/release/swc-test-host replay "$R"                       # offline: the module's decisions, the first diverging tick
#    an entry on the atproto VM: its readings and log are under /home/stg (ssh_exec, or
#    socialweb-computer run requester ssh stg --state-dir "$CI_IDENTITY_DIR" --contract <cc> --exec -- <command>)
#    the market during the run: load $CI_HOME/runs/watch.ndjson in https://watch.socialweb.computer/

End a frozen run with SIGTERM to its driver (pkill -TERM -f "swc-test-host compositions/staging"): it releases everything and exits. If its cleanup reports errors (cleanupErrors in $R/run.json), run scripts/ci-hand-market.sh list, then delete.