0
0
# testkit
The harness of the system tests (`tests/system`). Tests run against a target: staging (stood up fresh on rented VMs)
or prod (after a redeploy). No unit tests.
## Run
```sh
cd tests
SWC_TARGET=$PWD/targets/prod.json SWC_DRY_RUN=1 cargo test # preflight only: target, creds files, requester binary
SWC_TARGET=$PWD/targets/prod.json cargo test --test acc_0004_vm_atirc_agent -- --nocapture
cargo run -p testkit --bin last-pass -- readings/0004-*.json # passing runs onto docs/acceptance/README.md
```
| env | what |
| --- | --- |
| `SWC_TARGET` | the target.json (required) |
| `SWC_DRY_RUN=1` | stop after the preflight; touches no account |
| `SWC_SIGNIN=oauth-qr` | orchestrators sign in by OAuth QR (`scripts/oauth-signin.ts`, headless Chrome); default the app password |
| `SWC_READINGS` | readings dir (default `tests/readings`) |
## target.json
`targets/prod.json` is prod. Fields: `plc`, `pds` (where app-password sessions sign in), `relay`, `jetstream`, `atirc`,
`gateway` (`url`, `aud`, `ssh`, optional `sshPort`, optional `sshWeb` (the ssh gateway's web origin; absent: `https://ssh.<ssh>`), optional `kcp`, optional `log` {`host`, `path`}: the ssh gateway's core log over an ssh alias, 0002), `bidders[]` (`name`, `did`, `operator`, optional `kcp`), `accounts`
(`handle`, `did`, `creds`, `oauthAccount`, `throwaway`), `roles` (role -> a key of `accounts`), `agentConfig`, `pageRepo`, `requester`.
- `pageRepo` (0009): a built checkout of the requester page repo (its README's Build: `cargo wasm` there and in the
atlibs checkout at its pin, `cargo wasm -p iroh-user-data`); unset, 0009 is skipped. Prod's:
```sh
mkdir -p ~/.tmp/swc-system-tests/page && cd ~/.tmp/swc-system-tests/page
git clone -q https://knotless.dev/johnandersen777.bsky.social/atlibs-socialweb-computer-requester
git clone -q https://knotless.dev/johnandersen777.bsky.social/atlibs && git -C atlibs checkout -q e40f5d6
(cd atlibs && cargo wasm -p iroh-user-data) && (cd atlibs-socialweb-computer-requester && cargo wasm)
```
Staging: `tests/compositions/staging/target.sh` writes it (`/home/stg/stg/page`) when the suite tarball carries the
page: the host that puts the suite has `SWC_PAGE_REPO`, such a built checkout; `tests/compositions/staging/page.sh`
adds the pinned node and Chrome (`page-tools.json`) under its `tools/`, which `page.round` uses when they are there.
For a target other than prod `page.round` gives the script the target's `pds`, `plc` and `relay` (`PAGE_PDS`,
`PAGE_PLC`, `PAGE_RELAY`) and, with an `xrpcFront`, the network CA: `NODE_EXTRA_CA_CERTS` for node, and for Chrome
(which takes no CA file) `--ignore-certificate-errors-spki-list` of the leaf the front presents, verified against that
CA by `openssl s_client` first. The network's names resolve by the VM's own resolver (the network DNS).
- `requester`: the orchestrator binary: `release` (the version channel stable points at: the public records of
did:plc:gmlkrhimoueejcertevr4hl3, `software.release.channel/socialweb-computer:stable` -> its version -> its manifest -> the
artifact's url, its sha256 from the artifact's cid; no stable yet: it fails and says to name one), `release:<version>` (that
version's record, an rc or a stable, the same way), `sha:<commit>` (CI's `main/<sha>`),
`oci:<host/name>@sha256:<manifest digest>` (a CI build on ttl.sh, pulled with `oras`, which checks it against the digest), `path:<file>` (a local
build). Downloads are checked against `SHA256SUMS` (a release: the cid's sha256), cached in `~/.tmp/swc-system-tests/bin`.
- `roles`: what an account does in the tests -> its key in `accounts`. A test asks by role (`run(.., &[role::REQUESTER], ..)`,
`ctx.role(role::REQUESTER)`), never by an account's name, so one test runs on every target. A role the target has no
account for is left out, and a test that asks for it fails with `the target <name> names no <role> account` (it is not
skipped). A role naming a key that is not in `accounts` fails the load. `Target::account(name)` is for a caller that
was given a name (`swc-test-host --account`).
| role | what it does | tests | prod | staging (`compositions/staging/accounts.sh`) |
| --- | --- | --- | --- | --- |
| `requester` | the acceptance requester: rents, signs in at the gateway and the ssh gateway, deletes its own records; `swc-test-host`'s account when no `--account` is given | 0001 auth modes, 0002 to 0009, 0012 | alice | alice1 |
| `secondRequester` | another requester account, vouched as the first is: 0013's long-lived requester under its own account lock | 0013 | bob | alice2 |
| `bidderOperator` | the account the target's bidders are agents of (`bidders[].operator`): 0001 section 2's temporary bidder signs in as it | 0001 bidder pause `s2` | bob | none: the bidders are installed with association Skip |
| `atircOwner` | an account that is its own atirc root: mints a token (`atirc token`) and reads its owner channel | 0011 `s3` | bob | alice3 |
- `accounts[].creds`: a shell file exporting `*ATPROTO_HANDLE` and `*ATPROTO_PASSWORD`, mode 0600 (refused otherwise).
Read in process, never printed; a log that holds a password fails the test. Staging accounts are throwaway
accounts on the staging PDS (`throwaway: true`), tombstoned with the staging PLC at teardown.
## Pieces
| module | what |
| --- | --- |
| `lib.rs` | `run(runbook, test, roles, body)`: per-account file locks (one driver per account), owner-only scratch dir, cleanups that always run (LIFO, also on failure or panic), readings JSON |
| `requester.rs` | the orchestrator: a Go requester (`run requester` / `run request`) signed in as an account; tests drive it through its kcp and CLI, never a market of their own |
| `kcp.rs` | `Kcp::connect(KcpEndpoint)`: a kube-rs client of any kcp (dynamic objects, server-side apply, waits) |
| `tenant.rs` | a tenant k3s (typed k8s-openapi): deployments, Ready pods per node, nodes, logs |
| `gateway.rs` | the gateway's cluster XRPC with the account's service auth |
| `atproto.rs` | atlibs-host: app-password sessions saved per account (no createSession per run), getRecord at computed keys (3 reads), PLC doc / `log/last` tombstones, rotation keys |
| `atirc.rs` | the owner mesh node: an `AtircNode` in the orchestrator's kcp, `atirc mesh pub/peers/sub` |
| `bidder.rs` | a bidder's own kcp: host-empty checks, pause / resume, fault injection |
## Reaching a kcp
kcp everywhere: a test talks to each kcp directly (the requester's, the gateway's, each bidder's), not only to public
endpoints, so it can inject faults (delete an in-flight VM object, pause a component, kill a guest) and assert the
recovery the runbooks describe (0007, 0006, 0001 section 7).
`KcpEndpoint { kubeconfig, workspace, via }`: the kubeconfig's credentials (client certificate, CA) over a transport.
| `via.kind` | transport |
| --- | --- |
| `direct` | the kubeconfig's server URL |
| `overlay` | the same port on a meshd overlay name (its DNS), TLS name kept |
| `ssh` | an `ssh -L` forward through `host` (an ssh alias) to the kubeconfig's host:port: a bidder's front on its loopback |
| `iroh` | atlibs-iroh: its ticket check, then its raw form `dumbpipe connect <ticket>` as the stream, one per TCP connection (`kcp::stdio_bridge`), TLS name kept |
| `webRtc` | `proxy`: a command piping stdin/stdout through a datachannel, one run per TCP connection (`kcp::stdio_bridge`); `Requester::webrtc_proxy(contract, tag)` builds it from the orchestrator's `run requester proxy` (the Go requester's WebRTC offerer, signalled through its kcp) for a kcp a guest tunnels as `tag` |
## What it reuses
| piece | from | for |
| --- | --- | --- |
| app-password sessions, service auth, PDS / PLC reads | atlibs-host v0.5.1 (`6e113df`, the rev atlibs-iroh pins: one copy) | `atproto.rs`, `gateway.rs` |
| iroh tickets, the raw `dumbpipe connect` stream | atlibs-iroh `e1cdf12` | `via iroh` |
| ident.link walks (verified `sync.getRecord`), the market's computed rkeys (`termination_rkey`, `release_rkey`) | identlink (atlibs-identlink `aa7cacc`, `native`) | `ctx.walk`, vm.terminated pins, the release's vm.delete |
| WebRTC offerer | the Go requester (`run requester proxy`) | `via webRtc` |
| the requester, its kcp, AtircNode, OpenBao | the release under test | the orchestrator |
Gaps, to close in atlibs rather than here:
- atlibs-iroh has no in-process QUIC: it shells out to dumbpipe (as js/index.js does). An in-process stream needs the
`iroh` crate's endpoint dialing the dumbpipe ALPN in atlibs-iroh, returning an `AsyncRead + AsyncWrite`; kube-rs then
takes it as a connector.
- atlibs-webrtc's Rust side is the answerer (`webrtc-peer`, str0m); the offerer is JavaScript (`WebrtcHost`,
`recordSignals`). A native offerer needs, in that crate: str0m as offerer, the sealed offer record at
`computer.socialweb.ingress.offer/<DID>.<tid>` and the answer read at its computed key (`recordSignals`), and a
datachannel as `AsyncRead + AsyncWrite`.
Prod bidder kcps (`targets/prod.json`): owner-only copies of each bidder's admin kubeconfig under
`~/.config/swc-system-tests/kcp/<bidder>.kubeconfig` on the runner (refused unless 0600). Fault injection on prod
touches only VMs and contracts the test itself created (`BidderKcp::own_vm` matches the test's own receipt).
A requester elsewhere is driven the same way: the ssh gateway's per-user requester is a kcp (its endpoint in the
target); the browser requester (shell.socialweb.computer) is the atlibs requester wasm module in the page, driven by
headless Chrome and its WebRTC session.
## Rules
- Integration only: a test proves what a runbook says, on a real target. No unit tests, no mocks.
- Discovery: requests name no bidders list.
- Cleanups release VMs, delete records and check tombstones even when the test failed.
- One file per runbook in `tests/system/tests/acc_NNNN_*.rs`; each test names the runbook section it implements.
## Tests as atlibs compositions
Owner-approved (2026-11-12): each runbook becomes a composition on atlibs' step loop (events `start`, `result`,
`message`/`closed`, `timer`; outputs `calls`, `send`, `cancel`, `timers`, `state`, `done`). A runbook's Rust test here
stays until its composition passes on staging and prod.
A module whose XRPC needs a scope (0001's gateway calls) names it in its module.json `scope`, the scope of the
client a published module would have; without one the host refuses those calls.
| composition | entries (`--entry`) | live (prod rc165, bob, `gate.sh` 2026-11-13) |
| --- | --- | --- |
| 0001 | `run` (sections 3-6), `s2` (staging: a temporary oauthQr bidder) | `run`: pass (atlibs-kube 8a71567) |
| 0002 | `run` | pass |
| 0003 (0001 section 9) | `run` (`--per-host`: each host in turn) | pass (discovery) |
| 0004 | `run` | pass |
| 0005 | `run` (A), `b`, `r2fail` (a failed R2 leaves no live DID at PLC: known gap, fails until rc166) | `run`, `b`: pass |
| 0006 | `run` (bidders), `compat` (`SWC_COMPAT_REQUESTER`) | pass (compat: rc164 CLI) |
| 0007, 0008, 0009, 0012 | `run` | pass |
| 0011 | `run` (gates), `token` | pass |
| 0013 | `run` (A), `c`, `b` (staging) | `run`, `c`: pass |
Follow-ups: the CI freeze on the knotless orchestrator VM (a frozen run's check names its attach
route); `freeze::attach` ends when stdin closes (with the next atlibs-host change).
```sh
cd tests/modules && cargo wasm && cd .. # the modules (wasm32)
cd compositions/0004 && deno run -A --no-lock compose.ts > composition.json && cd ../..
SWC_TARGET=$PWD/targets/prod.json cargo run -p swc-test-host -- compositions/0004 --dry-run
SWC_TARGET=$PWD/targets/prod.json cargo run -p swc-test-host -- compositions/0004 --account alice # no --account: the target's requester role
SWC_TARGET=$PWD/targets/prod.json cargo run -p swc-test-host -- compositions/0004 --instance assert --entry tombstoned --input '{"dids":["did:plc:lpfuqerea3deuoyrn7ojser4"]}'
```
| piece | where | what |
| --- | --- | --- |
| `com.fedfork.exec.kube.{get,list,apply,patch,delete}`, `kube.watch` (subscription) | atlibs-kube (knotless) | any kcp or cluster the host names; writes only to what the run created, or what the host's guard allows; `cleanup()` deletes them and restores patched specs |
| `com.fedfork.exec.ssh.{exec,session}` | atlibs-ssh (knotless) | the system ssh to destinations the host names (`-F /dev/null`, `SetEnv`) |
| `computer.socialweb.test.{requester,request,account,atirc}.*` | `tests/host` (`swc-test-host`) | the requester under test as the account: `run requester` (its kcp becomes cluster `requester:<name>`), `run request` (its log events a subscription), the owner's mesh node; released and stopped after the run |
| `computer.socialweb.test.bidder.health`, `computer.socialweb.test.gateway.log` | `tests/host` | a bidder host's health without sudo (its `run bidder` journal: ERROR lines since start and in the first 10 min, automatic restarts, VM processes), over the bidder's ssh alias or here; the ssh gateway log lines of a DID (target `gateway.log`). A bidder with `ops {url}` (https) or `ops {iroh}` (its iroh market endpoint ticket, fetched over atlibs-iroh's raw `dumbpipe connect`) and no `kcp` in the target (and `gateway.ops`; the secret in the target's `opsSecretFile`) is read through its read-only routes `com.fedfork.exec.ops.*` instead (`tests/host/src/ops.rs`): these calls, `bidder.logCount` / `bidder.logLines`, `vm.memory` without setHigh, and kube `get` / `list` of ComputeVM, FirecrackerVM, MarketIdentity, Bidder and MarketBidder on `bidder:<name>`; every other kube verb there is refused, and the module sees `write: false` on that bidder |
| `com.fedfork.exec.identlink.{walk,plc}` | `tests/host` (to move into atlibs-host) | identlink's verified walk, PLC log/last and documents of the target's PLC |
| `assert` | `tests/modules/assert` | records gone (x3), tombstones, an agent VM's chain, no market or compute record created since a time (`none_since`, every listRecords page, each record as {uri, createdAt} from the host's `computer.socialweb.test.repo.listCreated`: full values pass the 1 MiB module input; the accepts a bidder refused, by the requester's log, reported as `declined`, not left: the requester keeps them, docs/acceptance/0014); each asks again every 10 s for up to `within` s (180) |
| `acc-0004` | `tests/modules/acc-0004`, `tests/compositions/0004` | the runbook as a state machine |
### Freeze, inspect, replay
A failure (pass false, a run error, the module's deadline) freezes the run when `--on-fail freeze` (the default
outside CI; `CI` set: cleanup): no more events reach the module, every VM, requester and kube object stays, and the
run dir holds the dump. `--freeze-at <step|phase>` (repeatable) is a breakpoint; `kill -USR1 <pid>` freezes a live
run. A freeze ends at `continue` / `cleanup` or its TTL (`--freeze-ttl`, default 7200 s), then cleanup runs.
A run has two dirs: its dump (`$SWC_RUNS/<run>`, default `~/.tmp/swc-system-tests/runs/<run>`: the files below, no
secret; a CI workflow sets `SWC_RUNS=runs` and declares `artifacts: [runs/]`) and its work dir
(`~/.tmp/swc-system-tests/work/<run>`: requester state, kubeconfigs, keys, the store; never uploaded).
| file in the dump | what |
| --- | --- |
| `ticks.ndjson.zst` | every tick of every instance (atlibs-host `ticks`): seq, times, what woke it (a call's NSID and latency), the event, the output, the state's hash and diff, full snapshots; secret-named keys redacted; zstd, rotated past 64 MiB |
| `state.json`, `failure.json`, `freeze.ndjson` | the root's state; the failure with the last ticks; freezes and thaws |
| `resources.json` | requesters (state dir, pid, cluster), invocations (receipt, winner, logs, ssh command), kube objects, clusters (kubeconfig paths), temporary keys, observer nicks: paths only, no secret |
| `run.json`, `control.sock` | what ran; the control socket (MCP, 0600) while the run lives |
```sh
swc-test-host attach <run dir> # stdio MCP: state, ticks, state_at, failure, freeze, step, continue, inject_event,
# resources, kube_get, kube_list, kube_logs, ssh_exec, requester_ssh,
# record_get, run_assert, cleanup, thaw, abort (ends with stdin)
swc-test-host replay <run dir> [--to <seq>] [--modules <dir>] # the whole composition again from its ticks, no network:
# {ticks, same, firstDiverge}; --to prints every run's state there; --modules: other wasm
swc-test-host cleanup <run dir> # from resources.json, when the host died
```
Checked 2026-11-13 (signed out): the 0004 composition's `assert` instance, 7 ticks, replayed 7/7, `--to 3` gave its
state, a changed `assert` (one more result field) diverged at seq 7 (the tick of its result), attach answered `ticks`
and `state_at` while frozen at step 2 and ended with its stdin; the 0011 gates composition, 5 ticks, replayed 5/5.
Size: 1.4 KB for the 7-tick run, 27 KB for 0011's (HTTP bodies); a 0004 run (some 500 ticks: the invocation's log
events, mesh lines, calls) about 0.5 MB, estimated.
The host passes the guard: a test deletes only VMs of its own receipts, patches a bidder root only to have it restored.
Budgets: calls, events and wall time unlimited per run (each module bounds itself with a deadline timer).