~ knotless Search/
↑↓ move↵ open⇧↵ all results

socialweb.computer/socialweb-computer-kcp

socialweb.computer: a compute market on atproto (kcp)

1 collaborator
0
0
185 lines · 17.0 KB Raw History

Staging

A self-contained socialweb.computer on one rented VM, for one run of the suite, then torn down. The smallest step of the staging plan (CI VM, associated requester R2 renting staging VMs, dev up --parent onto them, one meshd overlay).

The VM runs socialweb-computer dev up --profile market-network: its own PLC, relay, PDSes (alice.test, bob.test), 1 bidder (a nested VM; input bidders), the gateway, vouches between them, a per-network CA in network.caBundle. Every service is on the VM's loopback, so anything that uses it runs on the VM (run requester --network stg).

Run

  1. run request --hold --nested-virt --cpus 4 --mem 8G --expires-in 2h as the target's account (input cpus, mem).
  2. As root on the VM: the account stg (group kvm, one subordinate id range above the image's, uidmap, linger), and the binary under test in /home/stg/stg: <bucket>/{releases/<tag>|main/<commit>}/binaries/socialweb-computer-linux-amd64, checked against SHA256SUMS. dev refuses root.
  3. As stg, detached from the ssh session and polled (/home/stg/up.rc): dev up --instance stg --state-dir /home/stg/stg/s --profile market-network --bidders 1 --with-gateway --compute-kinds vm,deno --host-backend proc (input bidders, default 1).
  4. dev status --instance stg --state-dir /home/stg/stg/s.
  5. Always: dev down (detached, polled; --keep-state when the run failed); the host copies a failed run's logs into the dump's logs/ (redacted), then releases the VM.

Input fault: "status" makes step 4 fail (dev status of an instance the VM does not have): the proof of a failed run's logs/ (only the run's own VM is touched).

The target's requester picks the binary: release:<version> (releases/<version>/: a stable version, promote's copy of its main/<commit>/), sha:<commit> (CI's main/<commit>) or local:<commit> (a hand build published to local/<commit>/binaries/ with scripts/publish-spaces.ts); a bare release is refused. oci:<host/name>@sha256:<manifest digest> is a CI build pushed to https://ttl.sh (an anonymous, ephemeral OCI registry, the tag is the TTL) by scripts/oci-push.sh with oras (pinned in scripts/ci-tools.json, baked into the CI image): a branch's own build, no bucket key, so a fork's run does it too. Each VM installs the pinned oras and pulls by the manifest digest, which oras checks; a tag is never trusted. path:<file> puts the host's SWC_BINARY (that file) on the VM with request.put (slow: the fallback when ttl.sh is down).

cd tests/modules && cargo wasm && cd ..
cd compositions/staging && deno run -A --no-lock compose.ts > composition.json && cd ../..
SWC_TARGET=$PWD/targets/prod.json cargo run -p swc-test-host -- compositions/staging --dry-run

Live (rents one VM; requester set to a published build):

jq '.requester = "sha:<commit>"' targets/prod.json > ~/.tmp/target-staging.json
SWC_TARGET=~/.tmp/target-staging.json cargo run -p swc-test-host -- compositions/staging

Multi-VM (--input '{"layout": "multi"}')

Small VMs on one meshd overlay, joined by the atproto VM's network value: the atproto VM (10.44.0.1), bidders bidder VMs (input bidders, default 4: bidder-<i> at 10.44.0.<i+1>) and the gateway VM (the next address: 10.44.0.6; with "bidders": 2, 10.44.0.4). Each bidder VM is 4 vCPU / 8G with nested virt, rented from the prod bidders, and holds one default guest: bidders is the network's guest slots, and the gate's VM cap (gateVms) follows it. The default network is 6 VMs, 24 vCPU / 48G; "bidders": 2 (4 VMs, 16 vCPU / 32G) is the constrained run.

VMsizeruns
atproto2/4dev up --profile market-network --bidders 0 --network-bind 10.44.0.1 (PLC, relay, PDSes)
bidder-1 to bidder-<bidders>4/8, nested virtinstall bidder --market-network-file --bidder-spec-file --vmm qemu --compute-kinds vm,deno as root (the default policy), as prod's
gateway4/8a PLC probe, then the network's gateway: serve gateway kcp --gateway-profile market-network (gateway.sh)
  1. Rent them all (--nested-virt only for the bidders), install each (above), put socialweb-meshd (request.put what meshd, the host's SWC_MESHD: a static x86_64 build of rust/build.sh).
  2. socialweb-meshd identity on each, one roster file of every VM's endpoint, socialweb-meshd run --roster as root.
  3. atproto: dev up --network-bind: the PLC, relay, PDSes, ingress and TLS front listen on 10.44.0.1, and status.network names that address. dev status | jq .network.status is the network file.
  4. Bidders, installed as prod's are (docs/bidder.md "Installed"; plan 0040 step 8b, decision 9). As root on each bidder VM: /etc/socialweb-computer/net.json (the network file) and /etc/socialweb-computer/bidder.json (the spec file {"options":[{"name":"vouch","value":"<the DIDS of trust.sh accounts: alice1 to alice5>"}]}: what run bidder --bidder-vouch sets), both 0644 (the account cannot read /home/stg), then, detached and polled (/home/stg/bidder.rc, bidder.log), the binary under test: install bidder --handle bob<n+1>-<run>.bob.test --market-network-file /etc/socialweb-computer/net.json --bidder-spec-file /etc/socialweb-computer/bidder.json --vmm qemu --compute-kinds vm,deno. That is the account swc-bidder, its user unit socialweb-computer-bidder@<handle>, state /var/lib/socialweb-computer/swc-bidder (kcp admin kubeconfig handles/<sha256(handle)[:10]>/kcp/admin.kubeconfig), host backend systemd-user: every VM is a unit of that account's user manager with its own cgroup (0015:pressure), the log is the journal (journalctl _UID=$(id -u swc-bidder)). Their MarketBidder spec.identity.association: Skip (no operator; a merge patch on the bidder kcp's admin token); a bidder forwards its 127.0.0.1:<port> to each remote network host, so its guests' relay.localhost path (192.0.2.1 -> the host's loopback) reaches the atproto VM. Before trust.sh accounts, atirc.sh up <url> starts the network's own atirc (plan 0040 step 8c) at the URL dev up published as status.network.atircUrl (--network-atirc-url); target.sh writes it as the target's atirc. Then qr.sh up https://qr.stg.test:8446 starts the network's own QR session-transfer service (plan 0040 step 8g, 0001:s2) at the origin dev up published as status.network.qrOrigin (--network-qr-origin; net.json carries it, so every --market-network-file bidder's oauthQr identity shows <origin>/#oauth=..., takes <origin>/oauth-client-metadata.json as its client id and polls with the audience did:web:qr.stg.test, the host without the port). It is the suite's did-key-associator/ (the tarball carries it and the four typescript-helpers/lib packages it imports): the site built on the VM by the pinned deno (tools.sh deno) with DKA_ORIGIN, DKA_PLC_URL (the PLC's ingress name) and DKA_PDS_URL (the bidder PDS's public URL, its /.well-known/oauth-protected-resource resource), the backend from source on 127.0.0.1:18110 (SERVICE_DID=did:web:qr.stg.test, PLC_URL, NETWORK_ORIGINS the bidder PDS's public URL, DENO_CERT the network CA, state /home/stg/stg/qr), qr-front.py as its edge (TLS with the run's leaf, the site, /xrpc/* to the backend with X-Qr-Client-Ip: prod's Caddyfile). Its failure does not fail the stand-up (qr.sh failed in the step's output): 0001:s2 fails on it. The backend's checks are prod's (INSECURE_LOOPBACK_IDENTITY unset); NETWORK_ORIGINS names the one origin it may reach on a private address besides the PLC (an https port-443 origin, exact match). The network's PDSes fetch the page's client metadata (<origin>/oauth-client-metadata.json, a name under the network's domain) with their egress client and the network CA; a client_id outside the domain keeps the public-only fetch. tools.sh <tool>... installs the pinned node, Chrome and deno (page-tools.json) under /home/stg/stg/page/tools; target.sh names that directory as the target's tools, and the test host runs scripts/oauth-signin.ts (oauth.signin, page.signin) with its deno and Chrome, Chrome accepting the verified leaf keys of the network's names (--ignore-certificate-errors-spki-list).
  5. trust.sh accounts (the accounts of "Accounts" below) before the bidders join (they vouch them: --bidder-vouch), trust.sh vouch after (each account vouches every bidder: the vouches prod has; the network writes them only between its own apps), then two probe.sh rentals as alice1 and alice2 with the default policies, each on a nested bidder (plan 0047: never open). Between the bidders' join and trust.sh vouch, gateway.sh up https://gateway.stg.test:8444 stg@<the gateway VM's address> (plan 0040 step 8d) starts the gateway on the gateway VM as its own kcp, over ssh as stg (the suite's key is authorized there as on the bidders): its spec is the network value of net.json, bidderDids of bidders.json, webAppFront on the overlay, the options gateway-audience (did:web:gateway.stg.test%3A8444: the did:web of the URL's origin, which a module's XRPC call mints its token for) and public-origin (the URL), and operator the DID of the account alice5 of accounts.json. Its requester is prod's shape (docs/acceptance/0001-auth-modes.md "Association"): localRepo, its own Pds published through the network's ingress and its own did:plc at the network's PLC, the agent of its operator: the gateway adds https://ident.link/agent-of/<alice5> to its DID and, once the DID exists, gateway.sh has trust.sh associate <DID> write alice5's grant and endorsement (prod: scripts/associate-bidder.ts from alice); the root is Ready when the walk reaches alice5. The bidders vouch alice5 at their start and trust.sh vouch has it vouch them, as prod's bob and alice. The URL is a name of the zone on a port of the network's address: socat on the atproto VM terminates TLS with the run's leaf and forwards to the web app front; the tenant kubeconfigs' server (the client certificate front, https://127.0.0.1:16444) is forwarded from the atproto VM's loopback, TLS untouched. gateway.json is the target's gateway.{url, aud, requesterDid, kcp} (the admin kubeconfig, reached by ssh over the overlay). Then gateway.sh ssh-up https://ssh-web.stg.test:8445 ssh-gateway.stg.test:2222 stg@<the gateway VM's address> (step 8e) runs the ssh gateway there in prod's shape: standalone run gateway ssh --instance sg --ssh-gateway-spec-file with the network value, the option public-origin (the web URL), ssh.publicListen and ssh.webListen on the overlay and ssh.operatorDid the account bob1 of accounts.json, which trust.sh vouch has vouch alice1 to alice4 (the sign-in's bidder-tangled-vouch). The web origin is forwarded as the gateway's URL is, the sshd as plain TCP from port 2222 of the network's address; gateway.json gains ssh, sshPort, sshWeb (the target's web origin field; absent: https://ssh.<ssh>) and log (stg@<the gateway VM's address>, /home/stg/stg/gw/sg/dev/sg/logs/core.log).
  6. dev down on atproto; the host releases the VMs.

"provision": "infra" rents them through one EphemeralInfra stg-<run> of a run requester (requester.start stg), contracts atproto, bidder-1 to bidder-<bidders>, gateway (vmName stg-<layer>, per-VM size and nested virt, ttlSeconds the deadline, its record computer.socialweb.ephemeralInfra/stg-<run>); the module's ssh and put go through run requester ssh --contract einfra-stg-<run>-<layer>, and the teardown deletes the EphemeralInfra (its finalizer releases every contract). A failed contract (a refused accept, a guest off network) is requested again as <layer>-r<n> (twice at most) with denyBidderDids naming the bidder that failed it. As the guest requester R2 (an account with guestRequester), the run requester is run requester --guest-requester (one R2 DID for all four contracts); it needs its own XDG_RUNTIME_DIR (loginctl enable-linger r2, then XDG_RUNTIME_DIR=/run/user/$(id -u r2)).

SWC_MESHD=<static socialweb-meshd x86_64> SWC_TARGET=~/.tmp/target-staging.json \
  cargo run -p swc-test-host -- compositions/staging --input '{"layout": "multi"}'

Suite on the multi-VM layout

With "suite": ["0003", "0007:restart", ...] the multi layout goes on after the two run requests: the host puts swc-test-host (SWC_TEST_HOST_STATIC) and the suite tarball on the atproto VM (request.put, one at a time) and runs suite.sh there as stg, detached and polled, as the single-VM layout does: target.sh reads dev status of the network (stg), whose bidders are on other VMs and so are not MarketBidder apps of it: the module writes them to bidders.json (name, DID, the kcp admin kubeconfig fetched over the overlay, via: ssh root@10.44.0.<n>) and target.sh adds them to the target. The suite's ssh key (the atproto VM's stg) is authorized for root on every bidder VM, as it is for the atproto VM's own root (rootHosts): an installed bidder's units and journal belong to the system account swc-bidder (no login shell), so the test host reads its journal (bidder.logCount, bidder.logLines, bidder.health), restarts its core unit in place (bidder.restart: systemctl --user restart socialweb-computer-bidder@<handle> as swc-bidder, what upgrade does; kcp and VM units stay), upgrades it to another release binary (bidder.upgrade, 0007:upgrade: upgrade --binary --sha256 --state-dir as swc-bidder) and reads and sets a VM unit's memory properties (vm.memory, on the winner's VM) as root there. The binary under test of every VM may be path:<file> (SWC_BINARY); install refuses a --dev build.

On a held network, no new stand-up (docs/ci-debug.md, "Iterate on a held staging network", step 5d): a new network file is /etc/socialweb-computer/net.json replaced and the core unit restarted; a new binary is upgrade --binary --sha256 --state-dir as swc-bidder (it restarts the core unit; install bidder refuses a binary other than bin/current); from nothing is uninstall bidder --purge, then step 4's install bidder (a new DID: trust.sh vouch and bidders.json again).

Accounts

Every account of the staging network is alice<n> (the requester side) or bob<n> (the bidder side), as on prod, where alice is the acceptance requester and bob operates the bidders. The names and roles are defined once, in accounts.sh (STG_DRIVERS, STG_GATEWAY, STG_OPERATOR), which trust.sh, target.sh, suite.sh, gateway.sh and probe.sh source; the bidder handles are handle_of of tests/modules/staging. All are on the network's bidder PDS (bob.test). Prod's target (tests/targets/prod.json: alice, bob) is not this list.

namehandlerolevouchesvouched by
alice1, alice2, alice3, alice4<name>-<run><time>.bob.testthe gate's drivers, one gate slot each (suite.sh); the target's accounts, and its roles (alice1 requester, alice2 secondRequester, alice3 atircOwner); the stand-up probes are alice1 and alice2every bidder (and its operator, when it has one)bob2, bob3 (--bidder-vouch), bob1
alice5alice5-<run><time>.bob.testthe operator of the gateway's requester (gateway.sh up: spec.operator; the requester is localRepo with its own did:plc, alice5's agent by trust.sh associate); not in the targetevery bidderbob2, bob3 (--bidder-vouch)
bob1bob1-<run><time>.bob.testthe ssh gateway's operator (gateway.sh ssh-up, ssh.operatorDid); not in the target, not in DIDSalice1 to alice4nobody
bob2 to bob<bidders+1> (default bob2 to bob5)bob<n>-<tag>.bob.testthe bidders of the VMs bidder-1 to bidder-<bidders>, bob<i+1> on bidder-<i> (install bidder --handle); the target's bidders[].namealice1 to alice5alice1 to alice5 (trust.sh vouch)
bob<bidders+2>t<tag> (default bob6t<tag>; two bidders: bob4t<tag>)bob<bidders+2>t<tag>-<bob2's tag>.bob.testthe bidder 0020 installs itself, the number after the target's bidders', named after bob2nobodynobody

The oauth-signin prefix and the creds variable prefix of an account are its name in upper case (ALICE1: /home/stg/creds-alice1 holds ALICE1_ATPROTO_HANDLE, ALICE1_ATPROTO_PASSWORD). The single-VM layout has no trust.sh: target.sh makes alice1 to alice4 there, and its bidder is the network's own app.

The target's roles (STG_ROLES of accounts.sh, written by target.sh) say which driver plays which role of the Rust system tests (tests/testkit/README.md, roles): requester alice1, secondRequester alice2, atircOwner alice3. There is no bidderOperator: a staging bidder has no operator account, so a test that asks for one fails with the target staging names no bidderOperator account. swc-test-host runs as the requester role (alice1) when no --account is given.

Later

  • Multi-VM: layers placed through another kcp (dev up --parent), the roster as a computer.socialweb.mesh.roster record with TXT and wildcard names.