Staging
A self-contained socialweb.computer on one rented VM, for one run of the suite, then torn down. The smallest step of
the staging plan (CI VM, associated requester R2 renting staging VMs, dev up --parent onto them, one meshd overlay).
The VM runs socialweb-computer dev up --profile market-network: its own PLC, relay, PDSes (alice.test,
bob.test), 1 bidder (a nested VM; input bidders), the gateway, vouches between them, a per-network CA in network.caBundle.
Every service is on the VM's loopback, so anything that uses it runs on the VM (run requester --network stg).
Run
run request --hold --nested-virt --cpus 4 --mem 8G --expires-in 2has the target's account (inputcpus,mem).- As root on the VM: the account
stg(groupkvm, one subordinate id range above the image's,uidmap, linger), and the binary under test in/home/stg/stg:<bucket>/{releases/<tag>|main/<commit>}/binaries/socialweb-computer-linux-amd64, checked againstSHA256SUMS.devrefuses root. - As
stg, detached from the ssh session and polled (/home/stg/up.rc):dev up --instance stg --state-dir /home/stg/stg/s --profile market-network --bidders 1 --with-gateway --compute-kinds vm,deno --host-backend proc(inputbidders, default 1). dev status --instance stg --state-dir /home/stg/stg/s.- Always:
dev down(detached, polled;--keep-statewhen the run failed); the host copies a failed run's logs into the dump'slogs/(redacted), then releases the VM.
Input fault: "status" makes step 4 fail (dev status of an instance the VM does not have): the proof of a failed
run's logs/ (only the run's own VM is touched).
The target's requester picks the binary: release:<version> (releases/<version>/: a stable version, promote's copy of its
main/<commit>/), sha:<commit> (CI's main/<commit>) or local:<commit> (a hand build
published to local/<commit>/binaries/ with scripts/publish-spaces.ts); a bare release is refused.
oci:<host/name>@sha256:<manifest digest> is a CI build pushed to https://ttl.sh (an anonymous, ephemeral OCI registry, the tag is the
TTL) by scripts/oci-push.sh with oras (pinned in scripts/ci-tools.json, baked into the CI image): a branch's own build, no bucket key, so a
fork's run does it too. Each VM installs the pinned oras and pulls by the manifest digest, which oras checks; a tag is never trusted.
path:<file> puts the host's SWC_BINARY (that file) on the VM with request.put (slow: the fallback when ttl.sh is down).
cd tests/modules && cargo wasm && cd ..
cd compositions/staging && deno run -A --no-lock compose.ts > composition.json && cd ../..
SWC_TARGET=$PWD/targets/prod.json cargo run -p swc-test-host -- compositions/staging --dry-run
Live (rents one VM; requester set to a published build):
jq '.requester = "sha:<commit>"' targets/prod.json > ~/.tmp/target-staging.json
SWC_TARGET=~/.tmp/target-staging.json cargo run -p swc-test-host -- compositions/staging
Multi-VM (--input '{"layout": "multi"}')
Small VMs on one meshd overlay, joined by the atproto VM's network value: the atproto VM (10.44.0.1), bidders bidder VMs
(input bidders, default 4: bidder-<i> at 10.44.0.<i+1>) and the gateway VM (the next address: 10.44.0.6; with
"bidders": 2, 10.44.0.4). Each bidder VM is 4 vCPU / 8G with nested virt, rented from the prod bidders, and holds one
default guest: bidders is the network's guest slots, and the gate's VM cap (gateVms) follows it. The default network
is 6 VMs, 24 vCPU / 48G; "bidders": 2 (4 VMs, 16 vCPU / 32G) is the constrained run.
| VM | size | runs |
|---|---|---|
| atproto | 2/4 | dev up --profile market-network --bidders 0 --network-bind 10.44.0.1 (PLC, relay, PDSes) |
bidder-1 to bidder-<bidders> | 4/8, nested virt | install bidder --market-network-file --bidder-spec-file --vmm qemu --compute-kinds vm,deno as root (the default policy), as prod's |
| gateway | 4/8 | a PLC probe, then the network's gateway: serve gateway kcp --gateway-profile market-network (gateway.sh) |
- Rent them all (
--nested-virtonly for the bidders), install each (above), putsocialweb-meshd(request.putwhatmeshd, the host'sSWC_MESHD: a static x86_64 build ofrust/build.sh). socialweb-meshd identityon each, one roster file of every VM's endpoint,socialweb-meshd run --rosteras root.- atproto:
dev up --network-bind: the PLC, relay, PDSes, ingress and TLS front listen on 10.44.0.1, andstatus.networknames that address.dev status | jq .network.statusis the network file. - Bidders, installed as prod's are (docs/bidder.md "Installed"; plan 0040 step 8b, decision 9). As root on each bidder VM:
/etc/socialweb-computer/net.json(the network file) and/etc/socialweb-computer/bidder.json(the spec file{"options":[{"name":"vouch","value":"<the DIDS of trust.sh accounts: alice1 to alice5>"}]}: whatrun bidder --bidder-vouchsets), both 0644 (the account cannot read/home/stg), then, detached and polled (/home/stg/bidder.rc,bidder.log), the binary under test:install bidder --handle bob<n+1>-<run>.bob.test --market-network-file /etc/socialweb-computer/net.json --bidder-spec-file /etc/socialweb-computer/bidder.json --vmm qemu --compute-kinds vm,deno. That is the accountswc-bidder, its user unitsocialweb-computer-bidder@<handle>, state/var/lib/socialweb-computer/swc-bidder(kcp admin kubeconfighandles/<sha256(handle)[:10]>/kcp/admin.kubeconfig), host backend systemd-user: every VM is a unit of that account's user manager with its own cgroup (0015:pressure), the log is the journal (journalctl _UID=$(id -u swc-bidder)). Their MarketBidderspec.identity.association: Skip(no operator; a merge patch on the bidder kcp's admin token); a bidder forwards its127.0.0.1:<port>to each remote network host, so its guests'relay.localhostpath (192.0.2.1-> the host's loopback) reaches the atproto VM. Beforetrust.sh accounts,atirc.sh up <url>starts the network's own atirc (plan 0040 step 8c) at the URL dev up published asstatus.network.atircUrl(--network-atirc-url);target.shwrites it as the target'satirc. Thenqr.sh up https://qr.stg.test:8446starts the network's own QR session-transfer service (plan 0040 step 8g, 0001:s2) at the origin dev up published asstatus.network.qrOrigin(--network-qr-origin;net.jsoncarries it, so every--market-network-filebidder's oauthQr identity shows<origin>/#oauth=..., takes<origin>/oauth-client-metadata.jsonas its client id and polls with the audiencedid:web:qr.stg.test, the host without the port). It is the suite'sdid-key-associator/(the tarball carries it and the fourtypescript-helpers/libpackages it imports): the site built on the VM by the pinned deno (tools.sh deno) withDKA_ORIGIN,DKA_PLC_URL(the PLC's ingress name) andDKA_PDS_URL(the bidder PDS's public URL, its/.well-known/oauth-protected-resourceresource), the backend from source on127.0.0.1:18110(SERVICE_DID=did:web:qr.stg.test,PLC_URL,NETWORK_ORIGINSthe bidder PDS's public URL,DENO_CERTthe network CA, state/home/stg/stg/qr),qr-front.pyas its edge (TLS with the run's leaf, the site,/xrpc/*to the backend withX-Qr-Client-Ip: prod's Caddyfile). Its failure does not fail the stand-up (qr.sh failedin the step's output): 0001:s2 fails on it. The backend's checks are prod's (INSECURE_LOOPBACK_IDENTITYunset);NETWORK_ORIGINSnames the one origin it may reach on a private address besides the PLC (an https port-443 origin, exact match). The network's PDSes fetch the page's client metadata (<origin>/oauth-client-metadata.json, a name under the network's domain) with their egress client and the network CA; aclient_idoutside the domain keeps the public-only fetch.tools.sh <tool>...installs the pinned node, Chrome and deno (page-tools.json) under/home/stg/stg/page/tools;target.shnames that directory as the target'stools, and the test host runsscripts/oauth-signin.ts(oauth.signin,page.signin) with its deno and Chrome, Chrome accepting the verified leaf keys of the network's names (--ignore-certificate-errors-spki-list). trust.sh accounts(the accounts of "Accounts" below) before the bidders join (they vouch them:--bidder-vouch),trust.sh vouchafter (each account vouches every bidder: the vouches prod has; the network writes them only between its own apps), then twoprobe.shrentals as alice1 and alice2 with the default policies, each on a nested bidder (plan 0047: neveropen). Between the bidders' join andtrust.sh vouch,gateway.sh up https://gateway.stg.test:8444 stg@<the gateway VM's address>(plan 0040 step 8d) starts the gateway on the gateway VM as its own kcp, over ssh as stg (the suite's key is authorized there as on the bidders): its spec is the network value ofnet.json,bidderDidsofbidders.json,webAppFronton the overlay, the optionsgateway-audience(did:web:gateway.stg.test%3A8444: the did:web of the URL's origin, which a module's XRPC call mints its token for) andpublic-origin(the URL), andoperatorthe DID of the account alice5 ofaccounts.json. Its requester is prod's shape (docs/acceptance/0001-auth-modes.md "Association"): localRepo, its own Pds published through the network's ingress and its own did:plc at the network's PLC, the agent of its operator: the gateway addshttps://ident.link/agent-of/<alice5>to its DID and, once the DID exists,gateway.shhastrust.sh associate <DID>write alice5's grant and endorsement (prod:scripts/associate-bidder.tsfrom alice); the root is Ready when the walk reaches alice5. The bidders vouch alice5 at their start andtrust.sh vouchhas it vouch them, as prod's bob and alice. The URL is a name of the zone on a port of the network's address: socat on the atproto VM terminates TLS with the run's leaf and forwards to the web app front; the tenant kubeconfigs' server (the client certificate front,https://127.0.0.1:16444) is forwarded from the atproto VM's loopback, TLS untouched.gateway.jsonis the target'sgateway.{url, aud, requesterDid, kcp}(the admin kubeconfig, reached by ssh over the overlay). Thengateway.sh ssh-up https://ssh-web.stg.test:8445 ssh-gateway.stg.test:2222 stg@<the gateway VM's address>(step 8e) runs the ssh gateway there in prod's shape: standalonerun gateway ssh --instance sg --ssh-gateway-spec-filewith the network value, the optionpublic-origin(the web URL),ssh.publicListenandssh.webListenon the overlay andssh.operatorDidthe account bob1 ofaccounts.json, whichtrust.sh vouchhas vouch alice1 to alice4 (the sign-in'sbidder-tangled-vouch). The web origin is forwarded as the gateway's URL is, the sshd as plain TCP from port 2222 of the network's address;gateway.jsongainsssh,sshPort,sshWeb(the target's web origin field; absent:https://ssh.<ssh>) andlog(stg@<the gateway VM's address>,/home/stg/stg/gw/sg/dev/sg/logs/core.log).dev downon atproto; the host releases the VMs.
"provision": "infra" rents them through one EphemeralInfra stg-<run> of a run requester (requester.start
stg), contracts atproto, bidder-1 to bidder-<bidders>, gateway (vmName stg-<layer>, per-VM size and nested virt,
ttlSeconds the deadline, its record computer.socialweb.ephemeralInfra/stg-<run>); the module's ssh and put go
through run requester ssh --contract einfra-stg-<run>-<layer>, and the teardown deletes the EphemeralInfra (its
finalizer releases every contract).
A failed contract (a refused accept, a guest off network) is requested again as <layer>-r<n> (twice at most) with
denyBidderDids naming the bidder that failed it. As the guest requester R2 (an account with guestRequester), the run
requester is run requester --guest-requester (one R2 DID for all four contracts); it needs its own
XDG_RUNTIME_DIR (loginctl enable-linger r2, then XDG_RUNTIME_DIR=/run/user/$(id -u r2)).
SWC_MESHD=<static socialweb-meshd x86_64> SWC_TARGET=~/.tmp/target-staging.json \
cargo run -p swc-test-host -- compositions/staging --input '{"layout": "multi"}'
Suite on the multi-VM layout
With "suite": ["0003", "0007:restart", ...] the multi layout goes on after the two run requests: the host puts swc-test-host
(SWC_TEST_HOST_STATIC) and the suite tarball on the atproto VM (request.put, one at a time) and runs suite.sh there as stg, detached
and polled, as the single-VM layout does: target.sh reads dev status of the network (stg), whose bidders are on other VMs and
so are not MarketBidder apps of it: the module writes them to bidders.json (name, DID, the kcp admin kubeconfig fetched
over the overlay, via: ssh root@10.44.0.<n>) and target.sh adds them to the target. The suite's ssh key (the atproto VM's
stg) is authorized for root on every bidder VM, as it is for the atproto VM's own root (rootHosts): an installed
bidder's units and journal belong to the system account swc-bidder (no login shell), so the test host reads its journal
(bidder.logCount, bidder.logLines, bidder.health), restarts its core unit in place (bidder.restart: systemctl --user restart socialweb-computer-bidder@<handle> as swc-bidder, what upgrade does; kcp and VM units stay), upgrades it to another
release binary (bidder.upgrade, 0007:upgrade: upgrade --binary --sha256 --state-dir as swc-bidder) and reads and sets
a VM unit's memory properties (vm.memory, on the winner's VM) as root there. The binary under test of
every VM may be path:<file> (SWC_BINARY); install refuses a --dev build.
On a held network, no new stand-up (docs/ci-debug.md, "Iterate on a held staging network", step 5d): a new network file is
/etc/socialweb-computer/net.json replaced and the core unit restarted; a new binary is upgrade --binary --sha256 --state-dir as swc-bidder (it restarts the core unit; install bidder refuses a binary other than bin/current); from
nothing is uninstall bidder --purge, then step 4's install bidder (a new DID: trust.sh vouch and bidders.json again).
Accounts
Every account of the staging network is alice<n> (the requester side) or bob<n> (the bidder side), as on prod, where
alice is the acceptance requester and bob operates the bidders. The names and roles are defined once, in accounts.sh
(STG_DRIVERS, STG_GATEWAY, STG_OPERATOR), which trust.sh, target.sh, suite.sh, gateway.sh and probe.sh
source; the bidder handles are handle_of of tests/modules/staging. All are on the network's bidder PDS (bob.test).
Prod's target (tests/targets/prod.json: alice, bob) is not this list.
| name | handle | role | vouches | vouched by |
|---|---|---|---|---|
| alice1, alice2, alice3, alice4 | <name>-<run><time>.bob.test | the gate's drivers, one gate slot each (suite.sh); the target's accounts, and its roles (alice1 requester, alice2 secondRequester, alice3 atircOwner); the stand-up probes are alice1 and alice2 | every bidder (and its operator, when it has one) | bob2, bob3 (--bidder-vouch), bob1 |
| alice5 | alice5-<run><time>.bob.test | the operator of the gateway's requester (gateway.sh up: spec.operator; the requester is localRepo with its own did:plc, alice5's agent by trust.sh associate); not in the target | every bidder | bob2, bob3 (--bidder-vouch) |
| bob1 | bob1-<run><time>.bob.test | the ssh gateway's operator (gateway.sh ssh-up, ssh.operatorDid); not in the target, not in DIDS | alice1 to alice4 | nobody |
bob2 to bob<bidders+1> (default bob2 to bob5) | bob<n>-<tag>.bob.test | the bidders of the VMs bidder-1 to bidder-<bidders>, bob<i+1> on bidder-<i> (install bidder --handle); the target's bidders[].name | alice1 to alice5 | alice1 to alice5 (trust.sh vouch) |
bob<bidders+2>t<tag> (default bob6t<tag>; two bidders: bob4t<tag>) | bob<bidders+2>t<tag>-<bob2's tag>.bob.test | the bidder 0020 installs itself, the number after the target's bidders', named after bob2 | nobody | nobody |
The oauth-signin prefix and the creds variable prefix of an account are its name in upper case (ALICE1:
/home/stg/creds-alice1 holds ALICE1_ATPROTO_HANDLE, ALICE1_ATPROTO_PASSWORD). The single-VM layout has no trust.sh:
target.sh makes alice1 to alice4 there, and its bidder is the network's own app.
The target's roles (STG_ROLES of accounts.sh, written by target.sh) say which driver plays which role of the Rust
system tests (tests/testkit/README.md, roles): requester alice1, secondRequester alice2, atircOwner alice3.
There is no bidderOperator: a staging bidder has no operator account, so a test that asks for one fails with the target staging names no bidderOperator account. swc-test-host runs as the requester role (alice1) when no
--account is given.
Later
- Multi-VM: layers placed through another kcp
(
dev up --parent), the roster as acomputer.socialweb.mesh.rosterrecord with TXT and wildcard names.