testkit
The harness of the system tests (tests/system). Tests run against a target: staging (stood up fresh on rented VMs)
or prod (after a redeploy). No unit tests.
Run
cd tests
SWC_TARGET=$PWD/targets/prod.json SWC_DRY_RUN=1 cargo test # preflight only: target, creds files, requester binary
SWC_TARGET=$PWD/targets/prod.json cargo test --test acc_0004_vm_atirc_agent -- --nocapture
cargo run -p testkit --bin last-pass -- readings/0004-*.json # passing runs onto docs/acceptance/README.md
| env | what |
|---|---|
SWC_TARGET | the target.json (required) |
SWC_DRY_RUN=1 | stop after the preflight; touches no account |
SWC_SIGNIN=oauth-qr | orchestrators sign in by OAuth QR (scripts/oauth-signin.ts, headless Chrome); default the app password |
SWC_READINGS | readings dir (default tests/readings) |
target.json
targets/prod.json is prod. Fields: plc, pds (where app-password sessions sign in), relay, jetstream, atirc,
gateway (url, aud, ssh, optional sshPort, optional sshWeb (the ssh gateway's web origin; absent: https://ssh.<ssh>), optional kcp, optional log {host, path}: the ssh gateway's core log over an ssh alias, 0002), bidders[] (name, did, operator, optional kcp), accounts
(handle, did, creds, oauthAccount, throwaway), roles (role -> a key of accounts), agentConfig, pageRepo, requester.
-
pageRepo(0009): a built checkout of the requester page repo (its README's Build:cargo wasmthere and in the atlibs checkout at its pin,cargo wasm -p iroh-user-data); unset, 0009 is skipped. Prod's:mkdir -p ~/.tmp/swc-system-tests/page && cd ~/.tmp/swc-system-tests/page git clone -q https://knotless.dev/johnandersen777.bsky.social/atlibs-socialweb-computer-requester git clone -q https://knotless.dev/johnandersen777.bsky.social/atlibs && git -C atlibs checkout -q e40f5d6 (cd atlibs && cargo wasm -p iroh-user-data) && (cd atlibs-socialweb-computer-requester && cargo wasm)Staging:
tests/compositions/staging/target.shwrites it (/home/stg/stg/page) when the suite tarball carries the page: the host that puts the suite hasSWC_PAGE_REPO, such a built checkout;tests/compositions/staging/page.shadds the pinned node and Chrome (page-tools.json) under itstools/, whichpage.rounduses when they are there. For a target other than prodpage.roundgives the script the target'spds,plcandrelay(PAGE_PDS,PAGE_PLC,PAGE_RELAY) and, with anxrpcFront, the network CA:NODE_EXTRA_CA_CERTSfor node, and for Chrome (which takes no CA file)--ignore-certificate-errors-spki-listof the leaf the front presents, verified against that CA byopenssl s_clientfirst. The network's names resolve by the VM's own resolver (the network DNS). -
requester: the orchestrator binary:release(the version channel stable points at: the public records of did:plc:gmlkrhimoueejcertevr4hl3,software.release.channel/socialweb-computer:stable-> its version -> its manifest -> the artifact's url, its sha256 from the artifact's cid; no stable yet: it fails and says to name one),release:<version>(that version's record, an rc or a stable, the same way),sha:<commit>(CI'smain/<sha>),oci:<host/name>@sha256:<manifest digest>(a CI build on ttl.sh, pulled withoras, which checks it against the digest),path:<file>(a local build). Downloads are checked againstSHA256SUMS(a release: the cid's sha256), cached in~/.tmp/swc-system-tests/bin. -
roles: what an account does in the tests -> its key inaccounts. A test asks by role (run(.., &[role::REQUESTER], ..),ctx.role(role::REQUESTER)), never by an account's name, so one test runs on every target. A role the target has no account for is left out, and a test that asks for it fails withthe target <name> names no <role> account(it is not skipped). A role naming a key that is not inaccountsfails the load.Target::account(name)is for a caller that was given a name (swc-test-host --account).role what it does tests prod staging ( compositions/staging/accounts.sh)requesterthe acceptance requester: rents, signs in at the gateway and the ssh gateway, deletes its own records; swc-test-host's account when no--accountis given0001 auth modes, 0002 to 0009, 0012 alice alice1 secondRequesteranother requester account, vouched as the first is: 0013's long-lived requester under its own account lock 0013 bob alice2 bidderOperatorthe account the target's bidders are agents of ( bidders[].operator): 0001 section 2's temporary bidder signs in as it0001 bidder pause s2bob none: the bidders are installed with association Skip atircOwneran account that is its own atirc root: mints a token ( atirc token) and reads its owner channel0011 s3bob alice3 -
accounts[].creds: a shell file exporting*ATPROTO_HANDLEand*ATPROTO_PASSWORD, mode 0600 (refused otherwise). Read in process, never printed; a log that holds a password fails the test. Staging accounts are throwaway accounts on the staging PDS (throwaway: true), tombstoned with the staging PLC at teardown.
Pieces
| module | what |
|---|---|
lib.rs | run(runbook, test, roles, body): per-account file locks (one driver per account), owner-only scratch dir, cleanups that always run (LIFO, also on failure or panic), readings JSON |
requester.rs | the orchestrator: a Go requester (run requester / run request) signed in as an account; tests drive it through its kcp and CLI, never a market of their own |
kcp.rs | Kcp::connect(KcpEndpoint): a kube-rs client of any kcp (dynamic objects, server-side apply, waits) |
tenant.rs | a tenant k3s (typed k8s-openapi): deployments, Ready pods per node, nodes, logs |
gateway.rs | the gateway's cluster XRPC with the account's service auth |
atproto.rs | atlibs-host: app-password sessions saved per account (no createSession per run), getRecord at computed keys (3 reads), PLC doc / log/last tombstones, rotation keys |
atirc.rs | the owner mesh node: an AtircNode in the orchestrator's kcp, atirc mesh pub/peers/sub |
bidder.rs | a bidder's own kcp: host-empty checks, pause / resume, fault injection |
Reaching a kcp
kcp everywhere: a test talks to each kcp directly (the requester's, the gateway's, each bidder's), not only to public endpoints, so it can inject faults (delete an in-flight VM object, pause a component, kill a guest) and assert the recovery the runbooks describe (0007, 0006, 0001 section 7).
KcpEndpoint { kubeconfig, workspace, via }: the kubeconfig's credentials (client certificate, CA) over a transport.
via.kind | transport |
|---|---|
direct | the kubeconfig's server URL |
overlay | the same port on a meshd overlay name (its DNS), TLS name kept |
ssh | an ssh -L forward through host (an ssh alias) to the kubeconfig's host:port: a bidder's front on its loopback |
iroh | atlibs-iroh: its ticket check, then its raw form dumbpipe connect <ticket> as the stream, one per TCP connection (kcp::stdio_bridge), TLS name kept |
webRtc | proxy: a command piping stdin/stdout through a datachannel, one run per TCP connection (kcp::stdio_bridge); Requester::webrtc_proxy(contract, tag) builds it from the orchestrator's run requester proxy (the Go requester's WebRTC offerer, signalled through its kcp) for a kcp a guest tunnels as tag |
What it reuses
| piece | from | for |
|---|---|---|
| app-password sessions, service auth, PDS / PLC reads | atlibs-host v0.5.1 (6e113df, the rev atlibs-iroh pins: one copy) | atproto.rs, gateway.rs |
iroh tickets, the raw dumbpipe connect stream | atlibs-iroh e1cdf12 | via iroh |
ident.link walks (verified sync.getRecord), the market's computed rkeys (termination_rkey, release_rkey) | identlink (atlibs-identlink aa7cacc, native) | ctx.walk, vm.terminated pins, the release's vm.delete |
| WebRTC offerer | the Go requester (run requester proxy) | via webRtc |
| the requester, its kcp, AtircNode, OpenBao | the release under test | the orchestrator |
Gaps, to close in atlibs rather than here:
- atlibs-iroh has no in-process QUIC: it shells out to dumbpipe (as js/index.js does). An in-process stream needs the
irohcrate's endpoint dialing the dumbpipe ALPN in atlibs-iroh, returning anAsyncRead + AsyncWrite; kube-rs then takes it as a connector. - atlibs-webrtc's Rust side is the answerer (
webrtc-peer, str0m); the offerer is JavaScript (WebrtcHost,recordSignals). A native offerer needs, in that crate: str0m as offerer, the sealed offer record atcomputer.socialweb.ingress.offer/<DID>.<tid>and the answer read at its computed key (recordSignals), and a datachannel asAsyncRead + AsyncWrite.
Prod bidder kcps (targets/prod.json): owner-only copies of each bidder's admin kubeconfig under
~/.config/swc-system-tests/kcp/<bidder>.kubeconfig on the runner (refused unless 0600). Fault injection on prod
touches only VMs and contracts the test itself created (BidderKcp::own_vm matches the test's own receipt).
A requester elsewhere is driven the same way: the ssh gateway's per-user requester is a kcp (its endpoint in the target); the browser requester (shell.socialweb.computer) is the atlibs requester wasm module in the page, driven by headless Chrome and its WebRTC session.
Rules
- Integration only: a test proves what a runbook says, on a real target. No unit tests, no mocks.
- Discovery: requests name no bidders list.
- Cleanups release VMs, delete records and check tombstones even when the test failed.
- One file per runbook in
tests/system/tests/acc_NNNN_*.rs; each test names the runbook section it implements.
Tests as atlibs compositions
Owner-approved (2026-11-12): each runbook becomes a composition on atlibs' step loop (events start, result,
message/closed, timer; outputs calls, send, cancel, timers, state, done). A runbook's Rust test here
stays until its composition passes on staging and prod.
A module whose XRPC needs a scope (0001's gateway calls) names it in its module.json scope, the scope of the
client a published module would have; without one the host refuses those calls.
| composition | entries (--entry) | live (prod rc165, bob, gate.sh 2026-11-13) |
|---|---|---|
| 0001 | run (sections 3-6), s2 (staging: a temporary oauthQr bidder) | run: pass (atlibs-kube 8a71567) |
| 0002 | run | pass |
| 0003 (0001 section 9) | run (--per-host: each host in turn) | pass (discovery) |
| 0004 | run | pass |
| 0005 | run (A), b, r2fail (a failed R2 leaves no live DID at PLC: known gap, fails until rc166) | run, b: pass |
| 0006 | run (bidders), compat (SWC_COMPAT_REQUESTER) | pass (compat: rc164 CLI) |
| 0007, 0008, 0009, 0012 | run | pass |
| 0011 | run (gates), token | pass |
| 0013 | run (A), c, b (staging) | run, c: pass |
Follow-ups: the CI freeze on the knotless orchestrator VM (a frozen run's check names its attach
route); freeze::attach ends when stdin closes (with the next atlibs-host change).
cd tests/modules && cargo wasm && cd .. # the modules (wasm32)
cd compositions/0004 && deno run -A --no-lock compose.ts > composition.json && cd ../..
SWC_TARGET=$PWD/targets/prod.json cargo run -p swc-test-host -- compositions/0004 --dry-run
SWC_TARGET=$PWD/targets/prod.json cargo run -p swc-test-host -- compositions/0004 --account alice # no --account: the target's requester role
SWC_TARGET=$PWD/targets/prod.json cargo run -p swc-test-host -- compositions/0004 --instance assert --entry tombstoned --input '{"dids":["did:plc:lpfuqerea3deuoyrn7ojser4"]}'
| piece | where | what |
|---|---|---|
com.fedfork.exec.kube.{get,list,apply,patch,delete}, kube.watch (subscription) | atlibs-kube (knotless) | any kcp or cluster the host names; writes only to what the run created, or what the host's guard allows; cleanup() deletes them and restores patched specs |
com.fedfork.exec.ssh.{exec,session} | atlibs-ssh (knotless) | the system ssh to destinations the host names (-F /dev/null, SetEnv) |
computer.socialweb.test.{requester,request,account,atirc}.* | tests/host (swc-test-host) | the requester under test as the account: run requester (its kcp becomes cluster requester:<name>), run request (its log events a subscription), the owner's mesh node; released and stopped after the run |
computer.socialweb.test.bidder.health, computer.socialweb.test.gateway.log | tests/host | a bidder host's health without sudo (its run bidder journal: ERROR lines since start and in the first 10 min, automatic restarts, VM processes), over the bidder's ssh alias or here; the ssh gateway log lines of a DID (target gateway.log). A bidder with ops {url} (https) or ops {iroh} (its iroh market endpoint ticket, fetched over atlibs-iroh's raw dumbpipe connect) and no kcp in the target (and gateway.ops; the secret in the target's opsSecretFile) is read through its read-only routes com.fedfork.exec.ops.* instead (tests/host/src/ops.rs): these calls, bidder.logCount / bidder.logLines, vm.memory without setHigh, and kube get / list of ComputeVM, FirecrackerVM, MarketIdentity, Bidder and MarketBidder on bidder:<name>; every other kube verb there is refused, and the module sees write: false on that bidder |
com.fedfork.exec.identlink.{walk,plc} | tests/host (to move into atlibs-host) | identlink's verified walk, PLC log/last and documents of the target's PLC |
assert | tests/modules/assert | records gone (x3), tombstones, an agent VM's chain, no market or compute record created since a time (none_since, every listRecords page, each record as {uri, createdAt} from the host's computer.socialweb.test.repo.listCreated: full values pass the 1 MiB module input; the accepts a bidder refused, by the requester's log, reported as declined, not left: the requester keeps them, docs/acceptance/0014); each asks again every 10 s for up to within s (180) |
acc-0004 | tests/modules/acc-0004, tests/compositions/0004 | the runbook as a state machine |
Freeze, inspect, replay
A failure (pass false, a run error, the module's deadline) freezes the run when --on-fail freeze (the default
outside CI; CI set: cleanup): no more events reach the module, every VM, requester and kube object stays, and the
run dir holds the dump. --freeze-at <step|phase> (repeatable) is a breakpoint; kill -USR1 <pid> freezes a live
run. A freeze ends at continue / cleanup or its TTL (--freeze-ttl, default 7200 s), then cleanup runs.
A run has two dirs: its dump ($SWC_RUNS/<run>, default ~/.tmp/swc-system-tests/runs/<run>: the files below, no
secret; a CI workflow sets SWC_RUNS=runs and declares artifacts: [runs/]) and its work dir
(~/.tmp/swc-system-tests/work/<run>: requester state, kubeconfigs, keys, the store; never uploaded).
| file in the dump | what |
|---|---|
ticks.ndjson.zst | every tick of every instance (atlibs-host ticks): seq, times, what woke it (a call's NSID and latency), the event, the output, the state's hash and diff, full snapshots; secret-named keys redacted; zstd, rotated past 64 MiB |
state.json, failure.json, freeze.ndjson | the root's state; the failure with the last ticks; freezes and thaws |
resources.json | requesters (state dir, pid, cluster), invocations (receipt, winner, logs, ssh command), kube objects, clusters (kubeconfig paths), temporary keys, observer nicks: paths only, no secret |
run.json, control.sock | what ran; the control socket (MCP, 0600) while the run lives |
swc-test-host attach <run dir> # stdio MCP: state, ticks, state_at, failure, freeze, step, continue, inject_event,
# resources, kube_get, kube_list, kube_logs, ssh_exec, requester_ssh,
# record_get, run_assert, cleanup, thaw, abort (ends with stdin)
swc-test-host replay <run dir> [--to <seq>] [--modules <dir>] # the whole composition again from its ticks, no network:
# {ticks, same, firstDiverge}; --to prints every run's state there; --modules: other wasm
swc-test-host cleanup <run dir> # from resources.json, when the host died
Checked 2026-11-13 (signed out): the 0004 composition's assert instance, 7 ticks, replayed 7/7, --to 3 gave its
state, a changed assert (one more result field) diverged at seq 7 (the tick of its result), attach answered ticks
and state_at while frozen at step 2 and ended with its stdin; the 0011 gates composition, 5 ticks, replayed 5/5.
Size: 1.4 KB for the 7-tick run, 27 KB for 0011's (HTTP bodies); a 0004 run (some 500 ticks: the invocation's log
events, mesh lines, calls) about 0.5 MB, estimated.
The host passes the guard: a test deletes only VMs of its own receipts, patches a bidder root only to have it restored. Budgets: calls, events and wall time unlimited per run (each module bounds itself with a deadline timer).