# testkit The harness of the system tests (`tests/system`). Tests run against a target: staging (stood up fresh on rented VMs) or prod (after a redeploy). No unit tests. ## Run ```sh cd tests SWC_TARGET=$PWD/targets/prod.json SWC_DRY_RUN=1 cargo test # preflight only: target, creds files, requester binary SWC_TARGET=$PWD/targets/prod.json cargo test --test acc_0004_vm_atirc_agent -- --nocapture cargo run -p testkit --bin last-pass -- readings/0004-*.json # passing runs onto docs/acceptance/README.md ``` | env | what | | --- | --- | | `SWC_TARGET` | the target.json (required) | | `SWC_DRY_RUN=1` | stop after the preflight; touches no account | | `SWC_SIGNIN=oauth-qr` | orchestrators sign in by OAuth QR (`scripts/oauth-signin.ts`, headless Chrome); default the app password | | `SWC_READINGS` | readings dir (default `tests/readings`) | ## target.json `targets/prod.json` is prod. Fields: `plc`, `pds` (where app-password sessions sign in), `relay`, `jetstream`, `atirc`, `gateway` (`url`, `aud`, `ssh`, optional `sshPort`, optional `sshWeb` (the ssh gateway's web origin; absent: `https://ssh.`), optional `kcp`, optional `log` {`host`, `path`}: the ssh gateway's core log over an ssh alias, 0002), `bidders[]` (`name`, `did`, `operator`, optional `kcp`), `accounts` (`handle`, `did`, `creds`, `oauthAccount`, `throwaway`), `roles` (role -> a key of `accounts`), `agentConfig`, `pageRepo`, `requester`. - `pageRepo` (0009): a built checkout of the requester page repo (its README's Build: `cargo wasm` there and in the atlibs checkout at its pin, `cargo wasm -p iroh-user-data`); unset, 0009 is skipped. Prod's: ```sh mkdir -p ~/.tmp/swc-system-tests/page && cd ~/.tmp/swc-system-tests/page git clone -q https://knotless.dev/johnandersen777.bsky.social/atlibs-socialweb-computer-requester git clone -q https://knotless.dev/johnandersen777.bsky.social/atlibs && git -C atlibs checkout -q e40f5d6 (cd atlibs && cargo wasm -p iroh-user-data) && (cd atlibs-socialweb-computer-requester && cargo wasm) ``` Staging: `tests/compositions/staging/target.sh` writes it (`/home/stg/stg/page`) when the suite tarball carries the page: the host that puts the suite has `SWC_PAGE_REPO`, such a built checkout; `tests/compositions/staging/page.sh` adds the pinned node and Chrome (`page-tools.json`) under its `tools/`, which `page.round` uses when they are there. For a target other than prod `page.round` gives the script the target's `pds`, `plc` and `relay` (`PAGE_PDS`, `PAGE_PLC`, `PAGE_RELAY`) and, with an `xrpcFront`, the network CA: `NODE_EXTRA_CA_CERTS` for node, and for Chrome (which takes no CA file) `--ignore-certificate-errors-spki-list` of the leaf the front presents, verified against that CA by `openssl s_client` first. The network's names resolve by the VM's own resolver (the network DNS). - `requester`: the orchestrator binary: `release` (the version channel stable points at: the public records of did:plc:gmlkrhimoueejcertevr4hl3, `software.release.channel/socialweb-computer:stable` -> its version -> its manifest -> the artifact's url, its sha256 from the artifact's cid; no stable yet: it fails and says to name one), `release:` (that version's record, an rc or a stable, the same way), `sha:` (CI's `main/`), `oci:@sha256:` (a CI build on ttl.sh, pulled with `oras`, which checks it against the digest), `path:` (a local build). Downloads are checked against `SHA256SUMS` (a release: the cid's sha256), cached in `~/.tmp/swc-system-tests/bin`. - `roles`: what an account does in the tests -> its key in `accounts`. A test asks by role (`run(.., &[role::REQUESTER], ..)`, `ctx.role(role::REQUESTER)`), never by an account's name, so one test runs on every target. A role the target has no account for is left out, and a test that asks for it fails with `the target names no account` (it is not skipped). A role naming a key that is not in `accounts` fails the load. `Target::account(name)` is for a caller that was given a name (`swc-test-host --account`). | role | what it does | tests | prod | staging (`compositions/staging/accounts.sh`) | | --- | --- | --- | --- | --- | | `requester` | the acceptance requester: rents, signs in at the gateway and the ssh gateway, deletes its own records; `swc-test-host`'s account when no `--account` is given | 0001 auth modes, 0002 to 0009, 0012 | alice | alice1 | | `secondRequester` | another requester account, vouched as the first is: 0013's long-lived requester under its own account lock | 0013 | bob | alice2 | | `bidderOperator` | the account the target's bidders are agents of (`bidders[].operator`): 0001 section 2's temporary bidder signs in as it | 0001 bidder pause `s2` | bob | none: the bidders are installed with association Skip | | `atircOwner` | an account that is its own atirc root: mints a token (`atirc token`) and reads its owner channel | 0011 `s3` | bob | alice3 | - `accounts[].creds`: a shell file exporting `*ATPROTO_HANDLE` and `*ATPROTO_PASSWORD`, mode 0600 (refused otherwise). Read in process, never printed; a log that holds a password fails the test. Staging accounts are throwaway accounts on the staging PDS (`throwaway: true`), tombstoned with the staging PLC at teardown. ## Pieces | module | what | | --- | --- | | `lib.rs` | `run(runbook, test, roles, body)`: per-account file locks (one driver per account), owner-only scratch dir, cleanups that always run (LIFO, also on failure or panic), readings JSON | | `requester.rs` | the orchestrator: a Go requester (`run requester` / `run request`) signed in as an account; tests drive it through its kcp and CLI, never a market of their own | | `kcp.rs` | `Kcp::connect(KcpEndpoint)`: a kube-rs client of any kcp (dynamic objects, server-side apply, waits) | | `tenant.rs` | a tenant k3s (typed k8s-openapi): deployments, Ready pods per node, nodes, logs | | `gateway.rs` | the gateway's cluster XRPC with the account's service auth | | `atproto.rs` | atlibs-host: app-password sessions saved per account (no createSession per run), getRecord at computed keys (3 reads), PLC doc / `log/last` tombstones, rotation keys | | `atirc.rs` | the owner mesh node: an `AtircNode` in the orchestrator's kcp, `atirc mesh pub/peers/sub` | | `bidder.rs` | a bidder's own kcp: host-empty checks, pause / resume, fault injection | ## Reaching a kcp kcp everywhere: a test talks to each kcp directly (the requester's, the gateway's, each bidder's), not only to public endpoints, so it can inject faults (delete an in-flight VM object, pause a component, kill a guest) and assert the recovery the runbooks describe (0007, 0006, 0001 section 7). `KcpEndpoint { kubeconfig, workspace, via }`: the kubeconfig's credentials (client certificate, CA) over a transport. | `via.kind` | transport | | --- | --- | | `direct` | the kubeconfig's server URL | | `overlay` | the same port on a meshd overlay name (its DNS), TLS name kept | | `ssh` | an `ssh -L` forward through `host` (an ssh alias) to the kubeconfig's host:port: a bidder's front on its loopback | | `iroh` | atlibs-iroh: its ticket check, then its raw form `dumbpipe connect ` as the stream, one per TCP connection (`kcp::stdio_bridge`), TLS name kept | | `webRtc` | `proxy`: a command piping stdin/stdout through a datachannel, one run per TCP connection (`kcp::stdio_bridge`); `Requester::webrtc_proxy(contract, tag)` builds it from the orchestrator's `run requester proxy` (the Go requester's WebRTC offerer, signalled through its kcp) for a kcp a guest tunnels as `tag` | ## What it reuses | piece | from | for | | --- | --- | --- | | app-password sessions, service auth, PDS / PLC reads | atlibs-host v0.5.1 (`6e113df`, the rev atlibs-iroh pins: one copy) | `atproto.rs`, `gateway.rs` | | iroh tickets, the raw `dumbpipe connect` stream | atlibs-iroh `e1cdf12` | `via iroh` | | ident.link walks (verified `sync.getRecord`), the market's computed rkeys (`termination_rkey`, `release_rkey`) | identlink (atlibs-identlink `aa7cacc`, `native`) | `ctx.walk`, vm.terminated pins, the release's vm.delete | | WebRTC offerer | the Go requester (`run requester proxy`) | `via webRtc` | | the requester, its kcp, AtircNode, OpenBao | the release under test | the orchestrator | Gaps, to close in atlibs rather than here: - atlibs-iroh has no in-process QUIC: it shells out to dumbpipe (as js/index.js does). An in-process stream needs the `iroh` crate's endpoint dialing the dumbpipe ALPN in atlibs-iroh, returning an `AsyncRead + AsyncWrite`; kube-rs then takes it as a connector. - atlibs-webrtc's Rust side is the answerer (`webrtc-peer`, str0m); the offerer is JavaScript (`WebrtcHost`, `recordSignals`). A native offerer needs, in that crate: str0m as offerer, the sealed offer record at `computer.socialweb.ingress.offer/.` and the answer read at its computed key (`recordSignals`), and a datachannel as `AsyncRead + AsyncWrite`. Prod bidder kcps (`targets/prod.json`): owner-only copies of each bidder's admin kubeconfig under `~/.config/swc-system-tests/kcp/.kubeconfig` on the runner (refused unless 0600). Fault injection on prod touches only VMs and contracts the test itself created (`BidderKcp::own_vm` matches the test's own receipt). A requester elsewhere is driven the same way: the ssh gateway's per-user requester is a kcp (its endpoint in the target); the browser requester (shell.socialweb.computer) is the atlibs requester wasm module in the page, driven by headless Chrome and its WebRTC session. ## Rules - Integration only: a test proves what a runbook says, on a real target. No unit tests, no mocks. - Discovery: requests name no bidders list. - Cleanups release VMs, delete records and check tombstones even when the test failed. - One file per runbook in `tests/system/tests/acc_NNNN_*.rs`; each test names the runbook section it implements. ## Tests as atlibs compositions Owner-approved (2026-11-12): each runbook becomes a composition on atlibs' step loop (events `start`, `result`, `message`/`closed`, `timer`; outputs `calls`, `send`, `cancel`, `timers`, `state`, `done`). A runbook's Rust test here stays until its composition passes on staging and prod. A module whose XRPC needs a scope (0001's gateway calls) names it in its module.json `scope`, the scope of the client a published module would have; without one the host refuses those calls. | composition | entries (`--entry`) | live (prod rc165, bob, `gate.sh` 2026-11-13) | | --- | --- | --- | | 0001 | `run` (sections 3-6), `s2` (staging: a temporary oauthQr bidder) | `run`: pass (atlibs-kube 8a71567) | | 0002 | `run` | pass | | 0003 (0001 section 9) | `run` (`--per-host`: each host in turn) | pass (discovery) | | 0004 | `run` | pass | | 0005 | `run` (A), `b`, `r2fail` (a failed R2 leaves no live DID at PLC: known gap, fails until rc166) | `run`, `b`: pass | | 0006 | `run` (bidders), `compat` (`SWC_COMPAT_REQUESTER`) | pass (compat: rc164 CLI) | | 0007, 0008, 0009, 0012 | `run` | pass | | 0011 | `run` (gates), `token` | pass | | 0013 | `run` (A), `c`, `b` (staging) | `run`, `c`: pass | Follow-ups: the CI freeze on the knotless orchestrator VM (a frozen run's check names its attach route); `freeze::attach` ends when stdin closes (with the next atlibs-host change). ```sh cd tests/modules && cargo wasm && cd .. # the modules (wasm32) cd compositions/0004 && deno run -A --no-lock compose.ts > composition.json && cd ../.. SWC_TARGET=$PWD/targets/prod.json cargo run -p swc-test-host -- compositions/0004 --dry-run SWC_TARGET=$PWD/targets/prod.json cargo run -p swc-test-host -- compositions/0004 --account alice # no --account: the target's requester role SWC_TARGET=$PWD/targets/prod.json cargo run -p swc-test-host -- compositions/0004 --instance assert --entry tombstoned --input '{"dids":["did:plc:lpfuqerea3deuoyrn7ojser4"]}' ``` | piece | where | what | | --- | --- | --- | | `com.fedfork.exec.kube.{get,list,apply,patch,delete}`, `kube.watch` (subscription) | atlibs-kube (knotless) | any kcp or cluster the host names; writes only to what the run created, or what the host's guard allows; `cleanup()` deletes them and restores patched specs | | `com.fedfork.exec.ssh.{exec,session}` | atlibs-ssh (knotless) | the system ssh to destinations the host names (`-F /dev/null`, `SetEnv`) | | `computer.socialweb.test.{requester,request,account,atirc}.*` | `tests/host` (`swc-test-host`) | the requester under test as the account: `run requester` (its kcp becomes cluster `requester:`), `run request` (its log events a subscription), the owner's mesh node; released and stopped after the run | | `computer.socialweb.test.bidder.health`, `computer.socialweb.test.gateway.log` | `tests/host` | a bidder host's health without sudo (its `run bidder` journal: ERROR lines since start and in the first 10 min, automatic restarts, VM processes), over the bidder's ssh alias or here; the ssh gateway log lines of a DID (target `gateway.log`). A bidder with `ops {url}` (https) or `ops {iroh}` (its iroh market endpoint ticket, fetched over atlibs-iroh's raw `dumbpipe connect`) and no `kcp` in the target (and `gateway.ops`; the secret in the target's `opsSecretFile`) is read through its read-only routes `com.fedfork.exec.ops.*` instead (`tests/host/src/ops.rs`): these calls, `bidder.logCount` / `bidder.logLines`, `vm.memory` without setHigh, and kube `get` / `list` of ComputeVM, FirecrackerVM, MarketIdentity, Bidder and MarketBidder on `bidder:`; every other kube verb there is refused, and the module sees `write: false` on that bidder | | `com.fedfork.exec.identlink.{walk,plc}` | `tests/host` (to move into atlibs-host) | identlink's verified walk, PLC log/last and documents of the target's PLC | | `assert` | `tests/modules/assert` | records gone (x3), tombstones, an agent VM's chain, no market or compute record created since a time (`none_since`, every listRecords page, each record as {uri, createdAt} from the host's `computer.socialweb.test.repo.listCreated`: full values pass the 1 MiB module input; the accepts a bidder refused, by the requester's log, reported as `declined`, not left: the requester keeps them, docs/acceptance/0014); each asks again every 10 s for up to `within` s (180) | | `acc-0004` | `tests/modules/acc-0004`, `tests/compositions/0004` | the runbook as a state machine | ### Freeze, inspect, replay A failure (pass false, a run error, the module's deadline) freezes the run when `--on-fail freeze` (the default outside CI; `CI` set: cleanup): no more events reach the module, every VM, requester and kube object stays, and the run dir holds the dump. `--freeze-at ` (repeatable) is a breakpoint; `kill -USR1 ` freezes a live run. A freeze ends at `continue` / `cleanup` or its TTL (`--freeze-ttl`, default 7200 s), then cleanup runs. A run has two dirs: its dump (`$SWC_RUNS/`, default `~/.tmp/swc-system-tests/runs/`: the files below, no secret; a CI workflow sets `SWC_RUNS=runs` and declares `artifacts: [runs/]`) and its work dir (`~/.tmp/swc-system-tests/work/`: requester state, kubeconfigs, keys, the store; never uploaded). | file in the dump | what | | --- | --- | | `ticks.ndjson.zst` | every tick of every instance (atlibs-host `ticks`): seq, times, what woke it (a call's NSID and latency), the event, the output, the state's hash and diff, full snapshots; secret-named keys redacted; zstd, rotated past 64 MiB | | `state.json`, `failure.json`, `freeze.ndjson` | the root's state; the failure with the last ticks; freezes and thaws | | `resources.json` | requesters (state dir, pid, cluster), invocations (receipt, winner, logs, ssh command), kube objects, clusters (kubeconfig paths), temporary keys, observer nicks: paths only, no secret | | `run.json`, `control.sock` | what ran; the control socket (MCP, 0600) while the run lives | ```sh swc-test-host attach # stdio MCP: state, ticks, state_at, failure, freeze, step, continue, inject_event, # resources, kube_get, kube_list, kube_logs, ssh_exec, requester_ssh, # record_get, run_assert, cleanup, thaw, abort (ends with stdin) swc-test-host replay [--to ] [--modules ] # the whole composition again from its ticks, no network: # {ticks, same, firstDiverge}; --to prints every run's state there; --modules: other wasm swc-test-host cleanup # from resources.json, when the host died ``` Checked 2026-11-13 (signed out): the 0004 composition's `assert` instance, 7 ticks, replayed 7/7, `--to 3` gave its state, a changed `assert` (one more result field) diverged at seq 7 (the tick of its result), attach answered `ticks` and `state_at` while frozen at step 2 and ended with its stdin; the 0011 gates composition, 5 ticks, replayed 5/5. Size: 1.4 KB for the 7-tick run, 27 KB for 0011's (HTTP bodies); a 0004 run (some 500 ticks: the invocation's log events, mesh lines, calls) about 0.5 MB, estimated. The host passes the guard: a test deletes only VMs of its own receipts, patches a bidder root only to have it restored. Budgets: calls, events and wall time unlimited per run (each module bounds itself with a deadline timer).