# Staging A self-contained socialweb.computer on one rented VM, for one run of the suite, then torn down. The smallest step of the staging plan (CI VM, associated requester R2 renting staging VMs, `dev up --parent` onto them, one meshd overlay). The VM runs `socialweb-computer dev up --profile market-network`: its own PLC, relay, PDSes (`alice.test`, `bob.test`), 1 bidder (a nested VM; input `bidders`), the gateway, vouches between them, a per-network CA in `network.caBundle`. Every service is on the VM's loopback, so anything that uses it runs on the VM (`run requester --network stg`). ## Run 1. `run request --hold --nested-virt --cpus 4 --mem 8G --expires-in 2h` as the target's account (input `cpus`, `mem`). 2. As root on the VM: the account `stg` (group `kvm`, one subordinate id range above the image's, `uidmap`, linger), and the binary under test in `/home/stg/stg`: `/{releases/|main/}/binaries/socialweb-computer-linux-amd64`, checked against `SHA256SUMS`. `dev` refuses root. 3. As `stg`, detached from the ssh session and polled (`/home/stg/up.rc`): `dev up --instance stg --state-dir /home/stg/stg/s --profile market-network --bidders 1 --with-gateway --compute-kinds vm,deno --host-backend proc` (input `bidders`, default 1). 4. `dev status --instance stg --state-dir /home/stg/stg/s`. 5. Always: `dev down` (detached, polled; `--keep-state` when the run failed); the host copies a failed run's logs into the dump's `logs/` (redacted), then releases the VM. Input `fault: "status"` makes step 4 fail (`dev status` of an instance the VM does not have): the proof of a failed run's `logs/` (only the run's own VM is touched). The target's `requester` picks the binary: `release:` (`releases//`: a stable version, promote's copy of its `main//`), `sha:` (CI's `main/`) or `local:` (a hand build published to `local//binaries/` with `scripts/publish-spaces.ts`); a bare `release` is refused. `oci:@sha256:` is a CI build pushed to https://ttl.sh (an anonymous, ephemeral OCI registry, the tag is the TTL) by `scripts/oci-push.sh` with oras (pinned in `scripts/ci-tools.json`, baked into the CI image): a branch's own build, no bucket key, so a fork's run does it too. Each VM installs the pinned oras and pulls by the manifest digest, which oras checks; a tag is never trusted. `path:` puts the host's `SWC_BINARY` (that file) on the VM with `request.put` (slow: the fallback when ttl.sh is down). ```sh cd tests/modules && cargo wasm && cd .. cd compositions/staging && deno run -A --no-lock compose.ts > composition.json && cd ../.. SWC_TARGET=$PWD/targets/prod.json cargo run -p swc-test-host -- compositions/staging --dry-run ``` Live (rents one VM; `requester` set to a published build): ```sh jq '.requester = "sha:"' targets/prod.json > ~/.tmp/target-staging.json SWC_TARGET=~/.tmp/target-staging.json cargo run -p swc-test-host -- compositions/staging ``` ## Multi-VM (`--input '{"layout": "multi"}'`) Small VMs on one meshd overlay, joined by the atproto VM's network value: the atproto VM (10.44.0.1), `bidders` bidder VMs (input `bidders`, default 4: `bidder-` at 10.44.0.) and the gateway VM (the next address: 10.44.0.6; with `"bidders": 2`, 10.44.0.4). Each bidder VM is 4 vCPU / 8G with nested virt, rented from the prod bidders, and holds one default guest: `bidders` is the network's guest slots, and the gate's VM cap (`gateVms`) follows it. The default network is 6 VMs, 24 vCPU / 48G; `"bidders": 2` (4 VMs, 16 vCPU / 32G) is the constrained run. | VM | size | runs | | --- | --- | --- | | atproto | 2/4 | `dev up --profile market-network --bidders 0 --network-bind 10.44.0.1` (PLC, relay, PDSes) | | bidder-1 to bidder-`` | 4/8, nested virt | `install bidder --market-network-file --bidder-spec-file --vmm qemu --compute-kinds vm,deno` as root (the default policy), as prod's | | gateway | 4/8 | a PLC probe, then the network's gateway: `serve gateway kcp --gateway-profile market-network` (`gateway.sh`) | 1. Rent them all (`--nested-virt` only for the bidders), install each (above), put `socialweb-meshd` (`request.put` what `meshd`, the host's `SWC_MESHD`: a static x86_64 build of `rust/build.sh`). 2. `socialweb-meshd identity` on each, one roster file of every VM's endpoint, `socialweb-meshd run --roster` as root. 3. atproto: `dev up --network-bind`: the PLC, relay, PDSes, ingress and TLS front listen on 10.44.0.1, and `status.network` names that address. `dev status | jq .network.status` is the network file. 4. Bidders, installed as prod's are (docs/bidder.md "Installed"; plan 0040 step 8b, decision 9). As root on each bidder VM: `/etc/socialweb-computer/net.json` (the network file) and `/etc/socialweb-computer/bidder.json` (the spec file `{"options":[{"name":"vouch","value":""}]}`: what `run bidder --bidder-vouch` sets), both 0644 (the account cannot read `/home/stg`), then, detached and polled (`/home/stg/bidder.rc`, `bidder.log`), the binary under test: `install bidder --handle bob-.bob.test --market-network-file /etc/socialweb-computer/net.json --bidder-spec-file /etc/socialweb-computer/bidder.json --vmm qemu --compute-kinds vm,deno`. That is the account `swc-bidder`, its user unit `socialweb-computer-bidder@`, state `/var/lib/socialweb-computer/swc-bidder` (kcp admin kubeconfig `handles//kcp/admin.kubeconfig`), host backend systemd-user: every VM is a unit of that account's user manager with its own cgroup (0015:pressure), the log is the journal (`journalctl _UID=$(id -u swc-bidder)`). Their MarketBidder `spec.identity.association: Skip` (no operator; a merge patch on the bidder kcp's admin token); a bidder forwards its `127.0.0.1:` to each remote network host, so its guests' `relay.localhost` path (`192.0.2.1` -> the host's loopback) reaches the atproto VM. Before `trust.sh accounts`, `atirc.sh up ` starts the network's own atirc (plan 0040 step 8c) at the URL dev up published as `status.network.atircUrl` (`--network-atirc-url`); `target.sh` writes it as the target's `atirc`. Then `qr.sh up https://qr.stg.test:8446` starts the network's own QR session-transfer service (plan 0040 step 8g, 0001:s2) at the origin dev up published as `status.network.qrOrigin` (`--network-qr-origin`; `net.json` carries it, so every `--market-network-file` bidder's oauthQr identity shows `/#oauth=...`, takes `/oauth-client-metadata.json` as its client id and polls with the audience `did:web:qr.stg.test`, the host without the port). It is the suite's `did-key-associator/` (the tarball carries it and the four `typescript-helpers/lib` packages it imports): the site built on the VM by the pinned deno (`tools.sh deno`) with `DKA_ORIGIN`, `DKA_PLC_URL` (the PLC's ingress name) and `DKA_PDS_URL` (the bidder PDS's public URL, its `/.well-known/oauth-protected-resource` `resource`), the backend from source on `127.0.0.1:18110` (`SERVICE_DID=did:web:qr.stg.test`, `PLC_URL`, `NETWORK_ORIGINS` the bidder PDS's public URL, `DENO_CERT` the network CA, state `/home/stg/stg/qr`), `qr-front.py` as its edge (TLS with the run's leaf, the site, `/xrpc/*` to the backend with `X-Qr-Client-Ip`: prod's Caddyfile). Its failure does not fail the stand-up (`qr.sh failed` in the step's output): 0001:s2 fails on it. The backend's checks are prod's (`INSECURE_LOOPBACK_IDENTITY` unset); `NETWORK_ORIGINS` names the one origin it may reach on a private address besides the PLC (an https port-443 origin, exact match). The network's PDSes fetch the page's client metadata (`/oauth-client-metadata.json`, a name under the network's domain) with their egress client and the network CA; a `client_id` outside the domain keeps the public-only fetch. `tools.sh ...` installs the pinned node, Chrome and deno (`page-tools.json`) under `/home/stg/stg/page/tools`; `target.sh` names that directory as the target's `tools`, and the test host runs `scripts/oauth-signin.ts` (`oauth.signin`, `page.signin`) with its deno and Chrome, Chrome accepting the verified leaf keys of the network's names (`--ignore-certificate-errors-spki-list`). 5. `trust.sh accounts` (the accounts of "Accounts" below) before the bidders join (they vouch them: `--bidder-vouch`), `trust.sh vouch` after (each account vouches every bidder: the vouches prod has; the network writes them only between its own apps), then two `probe.sh` rentals as alice1 and alice2 with the default policies, each on a nested bidder (plan 0047: never `open`). Between the bidders' join and `trust.sh vouch`, `gateway.sh up https://gateway.stg.test:8444 stg@` (plan 0040 step 8d) starts the gateway on the gateway VM as its own kcp, over ssh as stg (the suite's key is authorized there as on the bidders): its spec is the network value of `net.json`, `bidderDids` of `bidders.json`, `webAppFront` on the overlay, the options `gateway-audience` (`did:web:gateway.stg.test%3A8444`: the did:web of the URL's origin, which a module's XRPC call mints its token for) and `public-origin` (the URL), and `operator` the DID of the account alice5 of `accounts.json`. Its requester is prod's shape (docs/acceptance/0001-auth-modes.md "Association"): localRepo, its own Pds published through the network's ingress and its own did:plc at the network's PLC, the agent of its operator: the gateway adds `https://ident.link/agent-of/` to its DID and, once the DID exists, `gateway.sh` has `trust.sh associate ` write alice5's grant and endorsement (prod: `scripts/associate-bidder.ts` from alice); the root is Ready when the walk reaches alice5. The bidders vouch alice5 at their start and `trust.sh vouch` has it vouch them, as prod's bob and alice. The URL is a name of the zone on a port of the network's address: socat on the atproto VM terminates TLS with the run's leaf and forwards to the web app front; the tenant kubeconfigs' server (the client certificate front, `https://127.0.0.1:16444`) is forwarded from the atproto VM's loopback, TLS untouched. `gateway.json` is the target's `gateway.{url, aud, requesterDid, kcp}` (the admin kubeconfig, reached by ssh over the overlay). Then `gateway.sh ssh-up https://ssh-web.stg.test:8445 ssh-gateway.stg.test:2222 stg@` (step 8e) runs the ssh gateway there in prod's shape: standalone `run gateway ssh --instance sg --ssh-gateway-spec-file` with the network value, the option `public-origin` (the web URL), `ssh.publicListen` and `ssh.webListen` on the overlay and `ssh.operatorDid` the account bob1 of `accounts.json`, which `trust.sh vouch` has vouch alice1 to alice4 (the sign-in's `bidder-tangled-vouch`). The web origin is forwarded as the gateway's URL is, the sshd as plain TCP from port 2222 of the network's address; `gateway.json` gains `ssh`, `sshPort`, `sshWeb` (the target's web origin field; absent: `https://ssh.`) and `log` (`stg@`, `/home/stg/stg/gw/sg/dev/sg/logs/core.log`). 6. `dev down` on atproto; the host releases the VMs. `"provision": "infra"` rents them through one `EphemeralInfra` `stg-` of a run requester (`requester.start` `stg`), contracts `atproto`, `bidder-1` to `bidder-`, `gateway` (vmName `stg-`, per-VM size and nested virt, `ttlSeconds` the deadline, its record `computer.socialweb.ephemeralInfra/stg-`); the module's ssh and put go through `run requester ssh --contract einfra-stg--`, and the teardown deletes the EphemeralInfra (its finalizer releases every contract). A failed contract (a refused accept, a guest off network) is requested again as `-r` (twice at most) with `denyBidderDids` naming the bidder that failed it. As the guest requester R2 (an account with `guestRequester`), the run requester is `run requester --guest-requester` (one R2 DID for all four contracts); it needs its own `XDG_RUNTIME_DIR` (`loginctl enable-linger r2`, then `XDG_RUNTIME_DIR=/run/user/$(id -u r2)`). ```sh SWC_MESHD= SWC_TARGET=~/.tmp/target-staging.json \ cargo run -p swc-test-host -- compositions/staging --input '{"layout": "multi"}' ``` ## Suite on the multi-VM layout With `"suite": ["0003", "0007:restart", ...]` the multi layout goes on after the two `run request`s: the host puts `swc-test-host` (`SWC_TEST_HOST_STATIC`) and the suite tarball on the atproto VM (`request.put`, one at a time) and runs `suite.sh` there as `stg`, detached and polled, as the single-VM layout does: `target.sh` reads `dev status` of the network (`stg`), whose bidders are on other VMs and so are not `MarketBidder` apps of it: the module writes them to `bidders.json` (name, DID, the kcp admin kubeconfig fetched over the overlay, `via: ssh root@10.44.0.`) and `target.sh` adds them to the target. The suite's ssh key (the atproto VM's `stg`) is authorized for root on every bidder VM, as it is for the atproto VM's own root (`rootHosts`): an installed bidder's units and journal belong to the system account `swc-bidder` (no login shell), so the test host reads its journal (`bidder.logCount`, `bidder.logLines`, `bidder.health`), restarts its core unit in place (`bidder.restart`: `systemctl --user restart socialweb-computer-bidder@` as `swc-bidder`, what `upgrade` does; kcp and VM units stay), upgrades it to another release binary (`bidder.upgrade`, 0007:upgrade: `upgrade --binary --sha256 --state-dir` as `swc-bidder`) and reads and sets a VM unit's memory properties (`vm.memory`, on the winner's VM) as root there. The binary under test of every VM may be `path:` (`SWC_BINARY`); `install` refuses a `--dev` build. On a held network, no new stand-up (docs/ci-debug.md, "Iterate on a held staging network", step 5d): a new network file is `/etc/socialweb-computer/net.json` replaced and the core unit restarted; a new binary is `upgrade --binary --sha256 --state-dir` as `swc-bidder` (it restarts the core unit; `install bidder` refuses a binary other than `bin/current`); from nothing is `uninstall bidder --purge`, then step 4's `install bidder` (a new DID: `trust.sh vouch` and `bidders.json` again). ## Accounts Every account of the staging network is `alice` (the requester side) or `bob` (the bidder side), as on prod, where alice is the acceptance requester and bob operates the bidders. The names and roles are defined once, in `accounts.sh` (`STG_DRIVERS`, `STG_GATEWAY`, `STG_OPERATOR`), which `trust.sh`, `target.sh`, `suite.sh`, `gateway.sh` and `probe.sh` source; the bidder handles are `handle_of` of `tests/modules/staging`. All are on the network's bidder PDS (`bob.test`). Prod's target (`tests/targets/prod.json`: alice, bob) is not this list. | name | handle | role | vouches | vouched by | | --- | --- | --- | --- | --- | | alice1, alice2, alice3, alice4 | `-