johnandersen777.bsky.social/atlibs-socialweb-computer-requester

Switch repository

Loading…

# atlibs-socialweb-computer-requester

The [socialweb.computer](https://socialweb.computer) compute-market requester as an
[atlibs](knotless://johnandersen777.bsky.social/atlibs) module: it posts an RFP, takes bids, accepts
one, watches the VM come up and releases it. Its bid policies are modules too, and two pages run it in
a browser: one for compute over WebRTC, one for compute and atirc together.

Split out of atlibs with its history (`git log --follow modules/requester/src/lib.rs`).

## Layout

- `modules/requester`: the requester; `policy-composition.json` routes its policy hook to two policies
- `modules/user-data`: the VM's cloud-init (`computer.socialweb.vm.userData`), the requester's user_data hook default
- `modules/cicd-workload`: a CI job's steps on a rented VM (`computer.socialweb.cicd.workload.run`), over the process hooks
- `modules/process-mcp`: `computer.socialweb.cicd.process.*` over the guest agent's MCP shell
- `modules/vm-mcp`: VMs for a model as five tools (`vm_create`, `vm_list`, `vm_exec`, `vm_release`, `vm_forget`), over the requester and process-mcp
- `compositions/`: compositions to publish (`socialweb-computer-cicd`: VM, policies, workload; `socialweb-computer-vm-mcp`: vm-mcp over iroh), each
  written by its `<name>.compose.ts` from the published modules `atlibs.lock.json` vendors
- `modules/policy-*`: bid policies (`evaluate`), on `crates/policy`; `crates/policy/vectors.mjs`: their test vectors
- `web/webrtc`: request a VM and open a shell to it over WebRTC; `web/socialweb-computer-ui`: compute and atirc in one page
- `records/`: the requester's session record and what `atlibs publish` wrote; `lexicons/`: the market's records and the `computer.socialweb.*` contracts

## Drive it from an agent (MCP)

`module.json` types every action's fields (`type`, `required`, `description`) and marks the read-only
actions (`offerings`, `bids`, `receipt`, `guest`, `watch`: `readOnly: true`), so atlibs' MCP adapter
offers the actions as tools. To get a VM whose guest agent serves MCP (a shell: `exec`, `start`,
`read`, `write`, `kill`, `list`):

1. user_data: omit it and the requester's `computer.socialweb.vm.userData` hook (default: `modules/user-data`)
   builds it from `shell` (default true), `webrtc_ssh`, `iroh`, `ssh_key` and `tunnels` (more datachannel tunnels to local TCP ports, `[{tag, port, transport: "webrtc"}]`); ssh goes through that module's
   `computer.socialweb.vm.irohUserData` hook (default: atlibs-iroh's `iroh-user-data`).

   It writes `/etc/socialweb-guest-agent/services.json` (`{"shell":{"unix":"/run/socialweb-guest-agent/shell.sock"}}`:
   the guest agent's MCP shell, HTTP on a root-only unix socket) and `tunnels.json` (`{"tag":"shell","unix":…}`,
   plus `{"tag":"ssh","port":22}` with `webrtc_ssh`); the bidder injects the guest agent when they exist. `iroh`
   adds dumbpipe in front of sshd and root's `authorized_keys` (`ssh_key`), as for any iroh guest; over iroh the
   shell is `ssh root@guest socialweb-guest-agent relay /run/socialweb-guest-agent/shell.sock`.
2. `contract {bidders, release: false, via}`: `via: "webrtc", tag: "shell"` waits for a webrtc
   fingerprint (any tag's: the guest has one DTLS certificate for all of them), `via: "iroh"` for the ticket and host keys (no `command`). The onNetwork reports
   read are those of `tags` (default `ssh`, `shell`), of user_data's tunnels.json tags (a JSON `#cloud-config`,
   as `user-data` writes) and of `tag`. The VM is kept and the result carries `mcp`, ready for asyncai's
   `mcp_connect`, one entry per transport the guest reported (webrtc: a fingerprint for tag `shell`; iroh:
   a ticket and pinned host keys):

   ```json
   [{ "name": "vm-1a2b3c4d", "server": { "url": "webrtc:///<b64url({receipt:{uri,cid},bidder})>/shell/mcp" } },
    { "name": "vm-1a2b3c4d-iroh", "server": { "url": "iroh:///<b64url({ticket,hostKeys})>/shell/mcp" } }]
   ```

   (base64url without padding of the JSON; the host's `webrtc:` / `iroh:` fetchers, atlibs-webrtc and
   atlibs-iroh, carry the MCP Streamable HTTP requests),
   and `receipt` and `records` (what the run created and kept).
3. When done, `release {receipt}`: it signs vm.delete and the event, waits for the bidder's
   vm.terminated, then deletes the kept records, its own two and the signal records webrtc hosts
   wrote for the receipt (`cleanup: false` keeps them). The kept records are the contract's
   `records` when given; otherwise release finds them from the receipt (receipt -> accept -> rfp ->
   vm, those in your repo), so an agent that lost `records` still cleans up. A record this host's
   ledger does not hold is refused and listed in `errors`; the release still completes.

Host budgets: a contract polls the bid, receipt, guest reports and termination (`poll_ms` 3000 ×
`max_polls` 40 per wait by default), so give the run a wall budget of about 15 minutes
(`wallMs: 900000`). Cleanup deletes only what the host saw this module create: the host needs the
`managed_records` ledger, and the later `release` must run on the same host (same ledger) as the
contract to delete them.

## VMs for a model: vm-mcp

Give a model `vm-mcp`'s tools, not the requester's actions. It composes the requester (hooks
`computer.socialweb.vm.provision` and `.release`) and process-mcp (`computer.socialweb.cicd.process.exec`):

| tool | arguments | result |
|---|---|---|
| `vm_create` | `cpus?`, `mem?` (GiB), `disk?` (GiB), `bidders?`, `hold_minutes?` | `{vm, bidder, cpus, mem, disk, created, expires, shell}` |
| `vm_list` | none | `{vms: [...]}` (`expired` once `hold_minutes` is up) |
| `vm_exec` | `vm`, `command`, `stdin?`, `timeout_s?` | `{exit, stdout, stderr}` |
| `vm_release` | `vm` | `{vm, released, terminated}` |
| `vm_forget` | `vm` | `{vm, forgotten, released, terminated, note?}`: one short release try, then dropped (a bidder that is gone) |

The model names a VM by a short id (`vm-1a2b3c4d`). `vm_create` sets the compute.vm record's `expiresAt` to
now + `hold_minutes`; the bidder ends the VM then (without it, a day after the accept). Receipts and records stay in one record of the user's,
`computer.socialweb.vm.kept` at rkey `pool` (default `self`), read with getRecord, never listed. Host-set
input bounds a call: `bidders` (default hat-7, hat-9, hat-8), `via`, `pool`, `max_vms` (2), `max_cpus` (8),
`max_mem` (16), `max_disk` (100), `max_hold_minutes` (240), `wait_minutes` (5 per wait). The session needs
the requester session's optional `kept` scope group; without it `vm_create` fails before renting.

Its tools, as an MCP server lists them:

```bash
cargo wasm
node --input-type=module <<'JS'
import { readFileSync } from "node:fs";
import { actionsServer } from "../atlibs/javascript/src/mcp/index.js";
const spec = JSON.parse(readFileSync("modules/vm-mcp/module.json", "utf8"));
const server = actionsServer({ name: "vm-mcp", modules: [spec], run: async () => ({ error: "offline" }) });
const { tools } = (await server.message({ jsonrpc: "2.0", id: 1, method: "tools/list" })).result;
for (const t of tools) console.log(t.name, JSON.stringify(t.inputSchema.required ?? []));
JS
```

```output
vm_create []
vm_list []
vm_exec ["vm","command"]
vm_release ["vm"]
vm_forget ["vm"]
```

To serve it, pass `actionsServer` a `run` over the composition (each call is one `runCompose` of instance
`vm`), or serve the module alone and let its hook defaults run the requester and process-mcp (the agent
page does this, with `via: "webrtc"`). `modules/vm-mcp/test.mjs` does both over the fake market.

## Check out the siblings

This repo builds on atlibs (the harness, the JS host) and three of its split-out repos, checked out next to it
at pinned commits: atlibs-webrtc (`atlibs/webrtc/`), atlibs-iroh (iroh-user-data) and atlibs-agents (the atirc
page and module, for socialweb-computer-ui):

```bash
cd ..
for r in atlibs@88b2ba8 atlibs-webrtc@5ccfa6a atlibs-iroh@6f55b81 atlibs-agents@55a8ec5; do
  git clone -q https://knotless.dev/johnandersen777.bsky.social/${r%@*} && git -C ${r%@*} checkout -q ${r#*@}
done
cd atlibs-socialweb-computer-requester
```

The Rust side needs no checkout: `abi` is a git dependency on atlibs, pinned to 28b892e by its `rev` in `Cargo.toml`.

## Build

```bash
rustup target add wasm32-unknown-unknown
cargo wasm      # every module -> target/wasm32-unknown-unknown/release/*.wasm
```

For `web/socialweb-computer-ui`, also build atirc in the sibling: `(cd ../atlibs-agents && cargo wasm)`.

## Run a page

Each page serves on its own; its `atlibs/`, `atlibs-webrtc/` and `modules/` are symlinks into the siblings
and `target/`:

```bash
python3 -m http.server -d web/webrtc 8080     # http://127.0.0.1:8080/
```

## Test

The modules under the real host loop (`../atlibs/rust/src/harness.mjs`), and the pages' cubits with the
real wasm. The vm-mcp composition runs the published modules its pins name, vendored at those pins (a local
build hashes differently: cargo mixes the host triple into the symbol hashes, so only a build in the same
container image on the same CPU architecture reproduces a pin):

```bash
cargo wasm && (cd ../atlibs-agents && cargo wasm) && (cd ../atlibs-iroh && cargo wasm) && (cd ../atlibs && cargo host)
(cd compositions && atlibs add $(node -p "Object.values(require('./atlibs.lock.json').modules).map((m) => m.ref + '@' + m.pin).join(' ')") && git checkout atlibs.lock.json)
node --test modules/*/test.mjs web/*/*.test.mjs compositions/*.test.mjs
```

## Publish

Install the CLI, sign in, then see what would be published:

```bash
cargo install --locked --git https://knotless.dev/johnandersen777.bsky.social/atlibs --rev 88b2ba8 atlibs
atlibs login you.bsky.social
atlibs publish --dir modules --dry-run --no-build --as you.bsky.social
```

Without `--no-build`, atlibs' reproducible container build (podman, docker or Apple's `container`)
copies the workspace's tracked sources (this repo's `Cargo.toml`, `crates/`, `modules/`) and builds
every module, so `atlibs verify bobvmbuilder.bsky.social/requester` rebuilds the same wasm on arm64 (bob's
builds ran on Apple silicon; an x86_64 host builds other hashes). The bid policy composition is published
as its own record:

```bash
atlibs publish --dir modules --as you.bsky.social
atlibs publish composition modules/requester/policy-composition.json --rkey requester-policies --as you.bsky.social
```

## Compositions

Each `compositions/<name>.compose.ts` writes the `<name>.composition.json` beside it (TypeScript: a route's
hook, its target's `implements` and each instance's `config` input are checked). After a module is
republished, pin the compositions to it and regenerate:

```bash
cd compositions
atlibs add bobvmbuilder.bsky.social/vm-mcp bobvmbuilder.bsky.social/requester bobvmbuilder.bsky.social/cicd-workload \
  bobvmbuilder.bsky.social/process-mcp bobvmbuilder.bsky.social/policy-requester-tangled-vouch bobvmbuilder.bsky.social/policy-bid-payload
for n in socialweb-computer-cicd socialweb-computer-vm-mcp; do
  node $n.compose.ts > $n.composition.json && atlibs compose check $n.composition.json > /dev/null
  atlibs publish composition $n.composition.json --rkey $n --as you.bsky.social
done
```

`compositions/compose.test.mjs` fails when a composition.json is not what its compose.ts writes.

`records/wasm/` holds what the last publish wrote (bobvmbuilder.bsky.social's records); `atlibs add
bobvmbuilder.bsky.social/requester` vendors the requester and its hook default.