~ knotless Search/
↑↓ move↵ open⇧↵ all results

socialweb.computer/socialweb-computer-kcp

socialweb.computer: a compute market on atproto (kcp)

1 collaborator
0
0
README.md

testkit

The harness of the system tests (tests/system). Tests run against a target: staging (stood up fresh on rented VMs) or prod (after a redeploy). No unit tests.

Run

cd tests
SWC_TARGET=$PWD/targets/prod.json SWC_DRY_RUN=1 cargo test            # preflight only: target, creds files, requester binary
SWC_TARGET=$PWD/targets/prod.json cargo test --test acc_0004_vm_atirc_agent -- --nocapture
cargo run -p testkit --bin last-pass -- readings/0004-*.json          # passing runs onto docs/acceptance/README.md
envwhat
SWC_TARGETthe target.json (required)
SWC_DRY_RUN=1stop after the preflight; touches no account
SWC_SIGNIN=oauth-qrorchestrators sign in by OAuth QR (scripts/oauth-signin.ts, headless Chrome); default the app password
SWC_READINGSreadings dir (default tests/readings)

target.json

targets/prod.json is prod. Fields: plc, pds (where app-password sessions sign in), relay, jetstream, atirc, gateway (url, aud, ssh, optional sshPort, optional sshWeb (the ssh gateway's web origin; absent: https://ssh.<ssh>), optional kcp, optional log {host, path}: the ssh gateway's core log over an ssh alias, 0002), bidders[] (name, did, operator, optional kcp), accounts (handle, did, creds, oauthAccount, throwaway), roles (role -> a key of accounts), agentConfig, pageRepo, requester.

  • pageRepo (0009): a built checkout of the requester page repo (its README's Build: cargo wasm there and in the atlibs checkout at its pin, cargo wasm -p iroh-user-data); unset, 0009 is skipped. Prod's:

    mkdir -p ~/.tmp/swc-system-tests/page && cd ~/.tmp/swc-system-tests/page
    git clone -q https://knotless.dev/johnandersen777.bsky.social/atlibs-socialweb-computer-requester
    git clone -q https://knotless.dev/johnandersen777.bsky.social/atlibs && git -C atlibs checkout -q e40f5d6
    (cd atlibs && cargo wasm -p iroh-user-data) && (cd atlibs-socialweb-computer-requester && cargo wasm)
    

    Staging: tests/compositions/staging/target.sh writes it (/home/stg/stg/page) when the suite tarball carries the page: the host that puts the suite has SWC_PAGE_REPO, such a built checkout; tests/compositions/staging/page.sh adds the pinned node and Chrome (page-tools.json) under its tools/, which page.round uses when they are there. For a target other than prod page.round gives the script the target's pds, plc and relay (PAGE_PDS, PAGE_PLC, PAGE_RELAY) and, with an xrpcFront, the network CA: NODE_EXTRA_CA_CERTS for node, and for Chrome (which takes no CA file) --ignore-certificate-errors-spki-list of the leaf the front presents, verified against that CA by openssl s_client first. The network's names resolve by the VM's own resolver (the network DNS).

  • requester: the orchestrator binary: release (the version channel stable points at: the public records of did:plc:gmlkrhimoueejcertevr4hl3, software.release.channel/socialweb-computer:stable -> its version -> its manifest -> the artifact's url, its sha256 from the artifact's cid; no stable yet: it fails and says to name one), release:<version> (that version's record, an rc or a stable, the same way), sha:<commit> (CI's main/<sha>), oci:<host/name>@sha256:<manifest digest> (a CI build on ttl.sh, pulled with oras, which checks it against the digest), path:<file> (a local build). Downloads are checked against SHA256SUMS (a release: the cid's sha256), cached in ~/.tmp/swc-system-tests/bin.

  • roles: what an account does in the tests -> its key in accounts. A test asks by role (run(.., &[role::REQUESTER], ..), ctx.role(role::REQUESTER)), never by an account's name, so one test runs on every target. A role the target has no account for is left out, and a test that asks for it fails with the target <name> names no <role> account (it is not skipped). A role naming a key that is not in accounts fails the load. Target::account(name) is for a caller that was given a name (swc-test-host --account).

    rolewhat it doestestsprodstaging (compositions/staging/accounts.sh)
    requesterthe acceptance requester: rents, signs in at the gateway and the ssh gateway, deletes its own records; swc-test-host's account when no --account is given0001 auth modes, 0002 to 0009, 0012alicealice1
    secondRequesteranother requester account, vouched as the first is: 0013's long-lived requester under its own account lock0013bobalice2
    bidderOperatorthe account the target's bidders are agents of (bidders[].operator): 0001 section 2's temporary bidder signs in as it0001 bidder pause s2bobnone: the bidders are installed with association Skip
    atircOwneran account that is its own atirc root: mints a token (atirc token) and reads its owner channel0011 s3bobalice3
  • accounts[].creds: a shell file exporting *ATPROTO_HANDLE and *ATPROTO_PASSWORD, mode 0600 (refused otherwise). Read in process, never printed; a log that holds a password fails the test. Staging accounts are throwaway accounts on the staging PDS (throwaway: true), tombstoned with the staging PLC at teardown.

Pieces

modulewhat
lib.rsrun(runbook, test, roles, body): per-account file locks (one driver per account), owner-only scratch dir, cleanups that always run (LIFO, also on failure or panic), readings JSON
requester.rsthe orchestrator: a Go requester (run requester / run request) signed in as an account; tests drive it through its kcp and CLI, never a market of their own
kcp.rsKcp::connect(KcpEndpoint): a kube-rs client of any kcp (dynamic objects, server-side apply, waits)
tenant.rsa tenant k3s (typed k8s-openapi): deployments, Ready pods per node, nodes, logs
gateway.rsthe gateway's cluster XRPC with the account's service auth
atproto.rsatlibs-host: app-password sessions saved per account (no createSession per run), getRecord at computed keys (3 reads), PLC doc / log/last tombstones, rotation keys
atirc.rsthe owner mesh node: an AtircNode in the orchestrator's kcp, atirc mesh pub/peers/sub
bidder.rsa bidder's own kcp: host-empty checks, pause / resume, fault injection

Reaching a kcp

kcp everywhere: a test talks to each kcp directly (the requester's, the gateway's, each bidder's), not only to public endpoints, so it can inject faults (delete an in-flight VM object, pause a component, kill a guest) and assert the recovery the runbooks describe (0007, 0006, 0001 section 7).

KcpEndpoint { kubeconfig, workspace, via }: the kubeconfig's credentials (client certificate, CA) over a transport.

via.kindtransport
directthe kubeconfig's server URL
overlaythe same port on a meshd overlay name (its DNS), TLS name kept
sshan ssh -L forward through host (an ssh alias) to the kubeconfig's host:port: a bidder's front on its loopback
irohatlibs-iroh: its ticket check, then its raw form dumbpipe connect <ticket> as the stream, one per TCP connection (kcp::stdio_bridge), TLS name kept
webRtcproxy: a command piping stdin/stdout through a datachannel, one run per TCP connection (kcp::stdio_bridge); Requester::webrtc_proxy(contract, tag) builds it from the orchestrator's run requester proxy (the Go requester's WebRTC offerer, signalled through its kcp) for a kcp a guest tunnels as tag

What it reuses

piecefromfor
app-password sessions, service auth, PDS / PLC readsatlibs-host v0.5.1 (6e113df, the rev atlibs-iroh pins: one copy)atproto.rs, gateway.rs
iroh tickets, the raw dumbpipe connect streamatlibs-iroh e1cdf12via iroh
ident.link walks (verified sync.getRecord), the market's computed rkeys (termination_rkey, release_rkey)identlink (atlibs-identlink aa7cacc, native)ctx.walk, vm.terminated pins, the release's vm.delete
WebRTC offererthe Go requester (run requester proxy)via webRtc
the requester, its kcp, AtircNode, OpenBaothe release under testthe orchestrator

Gaps, to close in atlibs rather than here:

  • atlibs-iroh has no in-process QUIC: it shells out to dumbpipe (as js/index.js does). An in-process stream needs the iroh crate's endpoint dialing the dumbpipe ALPN in atlibs-iroh, returning an AsyncRead + AsyncWrite; kube-rs then takes it as a connector.
  • atlibs-webrtc's Rust side is the answerer (webrtc-peer, str0m); the offerer is JavaScript (WebrtcHost, recordSignals). A native offerer needs, in that crate: str0m as offerer, the sealed offer record at computer.socialweb.ingress.offer/<DID>.<tid> and the answer read at its computed key (recordSignals), and a datachannel as AsyncRead + AsyncWrite.

Prod bidder kcps (targets/prod.json): owner-only copies of each bidder's admin kubeconfig under ~/.config/swc-system-tests/kcp/<bidder>.kubeconfig on the runner (refused unless 0600). Fault injection on prod touches only VMs and contracts the test itself created (BidderKcp::own_vm matches the test's own receipt).

A requester elsewhere is driven the same way: the ssh gateway's per-user requester is a kcp (its endpoint in the target); the browser requester (shell.socialweb.computer) is the atlibs requester wasm module in the page, driven by headless Chrome and its WebRTC session.

Rules

  • Integration only: a test proves what a runbook says, on a real target. No unit tests, no mocks.
  • Discovery: requests name no bidders list.
  • Cleanups release VMs, delete records and check tombstones even when the test failed.
  • One file per runbook in tests/system/tests/acc_NNNN_*.rs; each test names the runbook section it implements.

Tests as atlibs compositions

Owner-approved (2026-11-12): each runbook becomes a composition on atlibs' step loop (events start, result, message/closed, timer; outputs calls, send, cancel, timers, state, done). A runbook's Rust test here stays until its composition passes on staging and prod. A module whose XRPC needs a scope (0001's gateway calls) names it in its module.json scope, the scope of the client a published module would have; without one the host refuses those calls.

compositionentries (--entry)live (prod rc165, bob, gate.sh 2026-11-13)
0001run (sections 3-6), s2 (staging: a temporary oauthQr bidder)run: pass (atlibs-kube 8a71567)
0002runpass
0003 (0001 section 9)run (--per-host: each host in turn)pass (discovery)
0004runpass
0005run (A), b, r2fail (a failed R2 leaves no live DID at PLC: known gap, fails until rc166)run, b: pass
0006run (bidders), compat (SWC_COMPAT_REQUESTER)pass (compat: rc164 CLI)
0007, 0008, 0009, 0012runpass
0011run (gates), tokenpass
0013run (A), c, b (staging)run, c: pass

Follow-ups: the CI freeze on the knotless orchestrator VM (a frozen run's check names its attach route); freeze::attach ends when stdin closes (with the next atlibs-host change).

cd tests/modules && cargo wasm && cd ..                                     # the modules (wasm32)
cd compositions/0004 && deno run -A --no-lock compose.ts > composition.json && cd ../..
SWC_TARGET=$PWD/targets/prod.json cargo run -p swc-test-host -- compositions/0004 --dry-run
SWC_TARGET=$PWD/targets/prod.json cargo run -p swc-test-host -- compositions/0004 --account alice   # no --account: the target's requester role
SWC_TARGET=$PWD/targets/prod.json cargo run -p swc-test-host -- compositions/0004 --instance assert --entry tombstoned --input '{"dids":["did:plc:lpfuqerea3deuoyrn7ojser4"]}'
piecewherewhat
com.fedfork.exec.kube.{get,list,apply,patch,delete}, kube.watch (subscription)atlibs-kube (knotless)any kcp or cluster the host names; writes only to what the run created, or what the host's guard allows; cleanup() deletes them and restores patched specs
com.fedfork.exec.ssh.{exec,session}atlibs-ssh (knotless)the system ssh to destinations the host names (-F /dev/null, SetEnv)
computer.socialweb.test.{requester,request,account,atirc}.*tests/host (swc-test-host)the requester under test as the account: run requester (its kcp becomes cluster requester:<name>), run request (its log events a subscription), the owner's mesh node; released and stopped after the run
computer.socialweb.test.bidder.health, computer.socialweb.test.gateway.logtests/hosta bidder host's health without sudo (its run bidder journal: ERROR lines since start and in the first 10 min, automatic restarts, VM processes), over the bidder's ssh alias or here; the ssh gateway log lines of a DID (target gateway.log). A bidder with ops {url} (https) or ops {iroh} (its iroh market endpoint ticket, fetched over atlibs-iroh's raw dumbpipe connect) and no kcp in the target (and gateway.ops; the secret in the target's opsSecretFile) is read through its read-only routes com.fedfork.exec.ops.* instead (tests/host/src/ops.rs): these calls, bidder.logCount / bidder.logLines, vm.memory without setHigh, and kube get / list of ComputeVM, FirecrackerVM, MarketIdentity, Bidder and MarketBidder on bidder:<name>; every other kube verb there is refused, and the module sees write: false on that bidder
com.fedfork.exec.identlink.{walk,plc}tests/host (to move into atlibs-host)identlink's verified walk, PLC log/last and documents of the target's PLC
asserttests/modules/assertrecords gone (x3), tombstones, an agent VM's chain, no market or compute record created since a time (none_since, every listRecords page, each record as {uri, createdAt} from the host's computer.socialweb.test.repo.listCreated: full values pass the 1 MiB module input; the accepts a bidder refused, by the requester's log, reported as declined, not left: the requester keeps them, docs/acceptance/0014); each asks again every 10 s for up to within s (180)
acc-0004tests/modules/acc-0004, tests/compositions/0004the runbook as a state machine

Freeze, inspect, replay

A failure (pass false, a run error, the module's deadline) freezes the run when --on-fail freeze (the default outside CI; CI set: cleanup): no more events reach the module, every VM, requester and kube object stays, and the run dir holds the dump. --freeze-at <step|phase> (repeatable) is a breakpoint; kill -USR1 <pid> freezes a live run. A freeze ends at continue / cleanup or its TTL (--freeze-ttl, default 7200 s), then cleanup runs.

A run has two dirs: its dump ($SWC_RUNS/<run>, default ~/.tmp/swc-system-tests/runs/<run>: the files below, no secret; a CI workflow sets SWC_RUNS=runs and declares artifacts: [runs/]) and its work dir (~/.tmp/swc-system-tests/work/<run>: requester state, kubeconfigs, keys, the store; never uploaded).

file in the dumpwhat
ticks.ndjson.zstevery tick of every instance (atlibs-host ticks): seq, times, what woke it (a call's NSID and latency), the event, the output, the state's hash and diff, full snapshots; secret-named keys redacted; zstd, rotated past 64 MiB
state.json, failure.json, freeze.ndjsonthe root's state; the failure with the last ticks; freezes and thaws
resources.jsonrequesters (state dir, pid, cluster), invocations (receipt, winner, logs, ssh command), kube objects, clusters (kubeconfig paths), temporary keys, observer nicks: paths only, no secret
run.json, control.sockwhat ran; the control socket (MCP, 0600) while the run lives
swc-test-host attach <run dir>     # stdio MCP: state, ticks, state_at, failure, freeze, step, continue, inject_event,
                                   # resources, kube_get, kube_list, kube_logs, ssh_exec, requester_ssh,
                                   # record_get, run_assert, cleanup, thaw, abort (ends with stdin)
swc-test-host replay <run dir> [--to <seq>] [--modules <dir>]   # the whole composition again from its ticks, no network:
                                   # {ticks, same, firstDiverge}; --to prints every run's state there; --modules: other wasm
swc-test-host cleanup <run dir>    # from resources.json, when the host died

Checked 2026-11-13 (signed out): the 0004 composition's assert instance, 7 ticks, replayed 7/7, --to 3 gave its state, a changed assert (one more result field) diverged at seq 7 (the tick of its result), attach answered ticks and state_at while frozen at step 2 and ended with its stdin; the 0011 gates composition, 5 ticks, replayed 5/5. Size: 1.4 KB for the 7-tick run, 27 KB for 0011's (HTTP bodies); a 0004 run (some 500 ticks: the invocation's log events, mesh lines, calls) about 0.5 MB, estimated.

The host passes the guard: a test deletes only VMs of its own receipts, patches a bidder root only to have it restored. Budgets: calls, events and wall time unlimited per run (each module bounds itself with a deadline timer).