# Acceptance 0013: Mixed prod (VM, Deno sandbox, agent VM on a user-configured bidder) Proves [../../examples/infra/mixed-prod.yaml](../../examples/infra/mixed-prod.yaml) end to end against prod services: a long-lived requester (`run requester`, then `install requester`) with a real atproto account, kubectl apply of one EphemeralInfra pinned to a bidder the user configured, use each contract, restart the requester mid-run, delete. The default policy runs on both sides; no file and no command names `open` ([../quickstart-prod.md](../quickstart-prod.md) sections 2, 5 and 7). - Paths: A `run requester` in the foreground (OAuth QR); B `install requester` (systemd user unit of its own account); C the gateway in the requester kcp (`run requester --with-gateway`, a tenant cluster) - Plan: [0039](../plans/0039-long-lived-requester.yaml) items F-MIXED-PROD (this runbook) and ACC39-MIXED-PROD (its live pass) - Dev network, C (`0013:c`, composition, own 2-bidder network on hat-8, main aefab467 dev build + the OIDC discovery fix), 2026-11-15 00:57-01:08Z, alice: 3 passes, `pod_ready_s` 95.3 (the discovery race hit: the first fetch was rejected, the next one after the issuer was configured), 50.3, 45.1; `nodes_zero_s` 155.5, 155.5, 155.4; `seconds` 296.8, 235.4, 235.3. Before the fix, same net and binary: `pod_ready_s` 275.7 (5 min discovery cache), `seconds` 479.0 - Last pass A, B, C: 2026-11-13 03:11-04:44Z on prod rollout 20 (v0.0.0-rc165-pre1 038bd32e, binary 24a8b530 for the CLI and `install requester`), bob, no pin (the kit's `mixed-prod-nopin.yaml`; winner hat-8 every time); consent pages qr.fedfork.com for @bobvmbuilder.bsky.social (screenshots read). A: ready 42 s after start, `oauthQr`; Deployed 31 s (vm1, sbx1, agent1 Held); sbx1 `mixedprod-deno-ok sum=10`; agent1 (bundle by A4b) association +32 s after agentStart, `online and listening` +141 s, `uname -m` answered `x86_64` in 3.6 s; Ctrl-C and restart: ready 12 s, same DID, all Held, `hostname` answered 3 s; delete 10.8 s, R2 released first, `requester_association_deleted`, records 6/6, 10/10, 14/14, the 14 requester-repo URIs RecordNotFound x3 (the bidder's own `market.receipt` stays), 0 market.event / vm.delete / rfp / accept left (fix 13), R2 and agent DIDs `plc_tombstone`, vouches kept, hat-8 empty. A6 re-checked on a vm1-only EphemeralInfra (the first apply carried an old key from the kit file: `Permission denied (publickey)`, a harness fault): iroh 4.4 s, webrtc 1.4 s, `mixedprod-vm-ok x86_64`, release 10/10, RecordNotFound x3. The restart logged 10 x `Reconciler error the OpenBaoServer requester reports no address yet (phase Starting)` at one instant (first start 0 ERROR). B (swc-requester, vm1 + sbx1): install exit 0 after the sign-in; ssh iroh 3.8 s, webrtc 1.3 s; `systemctl --user restart`: kcp, kine, front and bao units keep their InvocationID, same DID, ssh 0.98 s; delete 3 s, 8/8 RecordNotFound x3; `uninstall --purge`: no account, state, linger or install record (the installed requester's `logs/` stayed empty and the journal had none of its lines, so B's ERROR count was not read). C: ready with the gateway 44 s after start, cluster c1 + kubeconfig 17 s, pod Ready 50 s after the apply (node `*-c1-node1` on hat-8), `quickstart-k8s-ok`; delete Deployment: nodes 0 in 156 s, node records 6/6, `self` and `nodes-` mirrors gone, 0 ERROR - C also reads the index: after nodes 0 `computer.socialweb.infra/self` is the account's, not the run's (it holds every deployment of the DID, live or in `recentEnded`, and is deleted only when nothing is left in it), so C checks it apart from the run's records: it is gone, or neither its `live` nor its `recentEnded` names this run's `nodes-` and it is not empty; the run's own records, the `nodes-` mirror among them, must be RecordNotFound (2026-11-18, alice: `self` held 32 `recentEnded` entries of other deployments, so the old "self gone" read failed, as on 2026-11-14); not yet run with that check; the controller reconciles the index (`infra_index_reconciled`: entries whose record ended or is missing are dropped from `live` at start and on each index write) (a 2-bidder dev net 2026-11-14 never got a Ready quickstart-k8s pod in 15 min: the k8s node did not join there) - Previous pass C: 2026-11-12 21:12-21:19Z, kcp e190a3b1 (rollout 19, rc165 candidate): 0 ERROR, the infra `self` and `nodes-` mirror gone 20 s after nodes 0 (8e849b42). Before: 2026-11-12 20:02-20:10Z, kcp ac51b91e (rollout 18): node Ready, `quickstart-k8s-ok`, the requester logs 0 ERROR (no ComponentLink errors: fixed in 363dd814); nodes 0 in 156 s; but the emptied `computer.socialweb.infra/nodes-` mirror stayed in the repo (`infra_record_replaced` with 0 nodes; deleted by hand), `self` went. Last pass: 2026-11-12 18:46-19:00Z on prod as bob, kcp cdd93727 (release 6380eee4, rollout 16), A and C, no pin (winner hat-8): agent1 online, `uname -m` answered in 2.9 s, delete in 12 s (`r2Released` 8.8 s), 45/45 RecordNotFound, `core.log` 0 ERROR; C: node Ready, `quickstart-k8s-ok`, nodes 0 in 156 s, `computer.socialweb.infra/{self,nodes-...}` gone with the cluster (none left after the stop). Open: C's requester logs `Reconciler error` `bus: no accepted ComponentLink allows root:EventStream.../watch-infra-nodes-` (and `watch-infraindex-`) `events.subscribe` x18 while the node lives. Before: 2026-11-12 17:47-18:08Z, kcp 60d0d3b7 (release 0b68ceeb, rollout 15), path A, NO `bidderDids` pin (requester discovery; winners hat-8, then hat-9): vm1 iroh + webrtc, sbx1, agent1 with its harness from the pre-written bundle (step A4b): association written 36 s after agent-start, `online` +136 s, messages both ways, `uname -m` answered in 2.9 s; delete: every contract record and agent1's grant + endorsement RecordNotFound x3, both DIDs `plc_tombstone`, hosts empty. 2026-11-11 (kcp f482083c, no pin): A, B (hat-7) and C (hat-8) pass except agent1 (no association on a declarative contract: fixed in 60d0d3b7). Earlier: 2026-11-10 as bob (bidder hat-9): A, B and C; 2026-11-01: A and C on rc159 (`ec0f4849`), B on the rc160 CLI (`840f0437`). Prod readings below ```mermaid sequenceDiagram participant U as user (kubectl, atirc) participant K as requester kcp (run requester) participant R as Requester app participant B as user-configured bidder participant I as irc.fedfork.com U->>K: apply vouch.yaml, mixed-prod.yaml K->>R: ComputeContract vm1, sbx1, agent1 (bidderDids pin) R->>B: RFP (requester-tangled-vouch attached) B->>B: bidder-tangled-vouch + attached policy: operator vouched requester? B->>R: bid R->>R: requester-tangled-vouch: requester vouched bidder? R->>B: accept B->>R: receipt, signed guest events at computed keys (onNetwork, hostKeys, vm.sandbox) R->>I: chat.atirc.mint (agent1, leased on receipt + vm.terminated) R->>K: ComputeContracts Held, EphemeralInfra Deployed U->>B: ssh vm1 (iroh, webrtc); read sbx1 stdout U->>K: agent-start ContractEvent -> guest.agentStart U->>I: JOIN of the agent nick, prompt, answer U->>K: Ctrl-C, restart: Held stays Held U->>K: delete EphemeralInfra -> revoke -> vm.delete -> vm.terminated ``` ## Preconditions | Item | Need | | --- | --- | | Bidder | a bidder the user configured: `install bidder` with the default policy (no `--bidder-policy`: `bidder-tangled-vouch`), [../bidder.md](../bidder.md); for sbx1 it must serve deno: `run bidder --compute-kinds vm,deno`, or `install bidder --compute-kinds vm,deno` (written to its `config.json`; on an already installed bidder, run the install again with the flag, which restarts the bidder unit); optional `--bidder-operator-signin ` for the operator vouch of step A4. A `prod-qr` `install bidder` lists `https` in its offering `endpoints` only with the bidder option `market-https=true` (config.json `"bidder-options": "--market-https=true"`); without it the offering lists iroh and webrtc | | Bidder DID | `MarketBidder app` `status.did` in the bidder kcp (the operator reads it); it replaces `REPLACE-BIDDER-DID` in every contract of `mixed-prod.yaml` | | Bidder association | the bidder DID is its operator's agent: `agent-of/` in its DID doc (written by the bidder) and, in the operator repo, `market.association/` then `link.ident.endorsement/` (written by the bidder with `--bidder-operator-signin`, needs the optional scope `repo:link.ident.endorsement?action=create&action=delete`, or by `deno run -A --no-lock scripts/associate-bidder.ts --creds --account --agent-did `); check: `socialweb-computer internal association ` roots [operator DID] | | Requester account | a real atproto account (handle resolves to its DID, PDS public) signed in by OAuth QR; the creds file owner-only (mode 0600), read only by `scripts/oauth-signin.ts` | | Vouches | both directions: requester -> bidder by [../../examples/infra/vouch.yaml](../../examples/infra/vouch.yaml) in the requester kcp (needs the optional scope `repo:sh.tangled.graph.vouch`) or `goat`; bidder operator -> requester by [../../examples/infra/vouch-bidder-operator.yaml](../../examples/infra/vouch-bidder-operator.yaml) in the bidder kcp (operator sign-in) or `goat` from the operator account, unless the requester account is that bidder's own user or one of the user's verified agents. Owner, 2026-11-18: "the vouch policy mode must also trust the own user and their associated DIDs/agents to N levels down". `bidder-tangled-vouch` (and `requester-tangled-vouch`) build the trust snapshot from the identlink walk of each party (`trust.BuildSnapshot`): each link verified both ways (the agent's DID doc `agent-of/`, the principal's `link.ident.endorsement/` and `com.publicdomainrelay.temp.market.association/`), N = 100 levels (`identlink.DefaultMaxDepth`), 256 lookups; a requester that shares a root with the bidder, or whose operator is on the bidder's side, is trusted (policytest cases `operated-by-counterparty`, `same-verified-operator`, `two-levels-below-counterparty`, `counterparty-two-levels-below-self(-operator)` and their forged and half-verified twins). The test runs, per bidder, the same walks of the account and of the bidder (`com.fedfork.exec.identlink.walk`) and reads the operator's vouch of the account; a bidder admits the account when that vouch exists or the walks say so | | atirc | a token of your own (`atirc token `) to `atirc sub` the owner channel; never print it | | Host | Linux, `kubectl`, `jq`, `ssh`, `dumbpipe` on PATH (the requester tries iroh first; a bidder without https, such as hat-9, is reached only over iroh or webrtc), the release or dev binary of the commit under test as `$SWC`; B: root on the requester host | | Rules | [README.md](README.md) shared rules: announce on atirc first, one driver per account, release what you rent | ## Steps The composition starts its `run requester` with `--bidder-dids ` (plan 0040 step 5): the list filters discovery for every rental, a `--with-gateway` node's too; on a market network the network sets them. `ST` is a NEW owner-only state dir for each run (`ST=~/.tmp/acc13-$(date +%s); mkdir "$ST"; chmod 700 "$ST"`): a `run requester --name mp` on an existing dir resumes that dir's requester with its old account and session (it logs `run requester resumes an existing requester`), never a new sign-in. Write each vouch at least 5 minutes before the first apply when a contract was denied before (verdicts are cached, denials too). ### A. run requester (foreground, OAuth QR) 1. Start: `$SWC run requester --name mp --state-dir "$ST" --atproto-oauth-qr > "$ST/mp.log" 2>&1 &`. Sign in from the `pendingAction.url` in the log with `scripts/oauth-signin.ts --creds --account --url ""` (read the url right before, never print or paste it). 2. Wait for `$ST/requesters/mp/ready.json` (written atomically once the root is Ready); `eval "$($SWC run requester env mp --state-dir "$ST")"` sets `KUBECONFIG` to the requester workspace; `kubectl get marketrequesters.socialweb.computer app -o jsonpath='{.status.phase} {.status.did}'` is `Ready `; the ready file's `identityMode` is `oauthQr`. 3. Requester -> bidder vouch: `sed -e "s/REPLACE-REQUESTER-REPO/repo-$(jq -r .did "$ST/requesters/mp/ready.json" | tr : -)/" -e "s/REPLACE-BIDDER-DID//g" -e "s/2026-11-01T00:00:00Z/$(date -u +%Y-%m-%dT%H:%M:%SZ)/" examples/infra/vouch.yaml | kubectl apply -f -`; AtprotoRecord `vouch-bidder` Applied with `at:///sh.tangled.graph.vouch/`. When the session lacks the vouch scope, write the same record with `goat record create` from the requester account instead. 4. Bidder operator -> requester vouch: in the bidder kcp, `vouch-bidder-operator.yaml` with `REPLACE-REQUESTER-DID` (the operator sign-in of [../bidder.md](../bidder.md)), or `goat` from the operator account; check with `goat get "at:///sh.tangled.graph.vouch/" | jq -r .kind` -> `vouch`. Skip it for a bidder whose user is the requester account or one of the account's operators (Preconditions, Vouches). When the write needs the owner's account, it is owner-only: ask the owner to write it; an agent does not. 4b. agent1's harness (optional; without it the agent joins the mesh but nothing answers): write agent1's bundle into the requester's own OpenBao before the apply, with the placeholder `/etc/atirc/atirc.env` and an atirc agent config (as [0004](0004-vm-atirc-agent.md) `--agent-config`; mode 0600 file, never printed) at `/etc/atirc/config.toml`; a bundle that is there is kept, and the requester adds the association entries itself: ```bash D=$ST/requesters/mp/dev/mp/openbaoservers/requester/openbao BAO=$(find ~/.cache/socialweb-computer/runtime -maxdepth 2 -name bao -type f | head -1) export BAO_ADDR=$(kubectl get openbaoservers.socialweb.computer requester -o jsonpath='{.status.address}') export BAO_CACERT=$D/tls/ca.crt BAO_CLIENT_CERT=$D/clients/requester/client.crt BAO_CLIENT_KEY=$D/clients/requester/client.key (umask 077; jq -n --rawfile c ~/.config/atirc/config.toml '[{path: "/etc/atirc/atirc.env", value: "", mode: "0600"}, {path: "/etc/atirc/config.toml", value: $c, mode: "0600"}]' > "$ST/agent1-bundle.json") BAO_TOKEN=$($BAO write -field=token auth/cert/login name=requester) \ $BAO kv put -mount=requester contracts/einfra-mixedprod-agent1/secrets bundle=@"$ST/agent1-bundle.json" >/dev/null rm -f "$ST/agent1-bundle.json" ``` 5. Apply: `sed -e "s|ssh-ed25519 AAAAREPLACE-WITH-YOUR-PUBLIC-KEY you@laptop|$(cat "$(jq -r .sshPublicKey "$ST/requesters/mp/ready.json")")|" -e "s/REPLACE-BIDDER-DID//" examples/infra/mixed-prod.yaml | kubectl apply -f -`; `kubectl wait ephemeralinfras.socialweb.computer/mixedprod --for=jsonpath='{.status.phase}'=Deployed --timeout=5m`. 6. vm1: `$SWC run requester ssh mp --state-dir "$ST" --contract einfra-mixedprod-vm1 --exec -- cat /root/mixedprod` prints `mixedprod-vm-ok`; the same with `--transport webrtc --exec -- uname -m` prints `x86_64`. 7. sbx1: `kubectl get computecontracts.socialweb.computer einfra-mixedprod-sbx1 -o jsonpath='{.status.contract.sandbox}'` has `stdout` `mixedprod-deno-ok sum=10 on linux/x86_64` and `exitCode` 0. 8. agent1: the requester log has `requester_agent_bundle_placeholder` (no bao step) and, after the receipt, the token-less mesh agent's `requester_agent_mesh_key_written` and `requester_association_written` (grant + endorsement at rkey = the in-VM R2 DID; the agent's peer record is R2's co-located account's, no record in the requester repo names it, [0004](0004-vm-atirc-agent.md)); then `sed -e 's/quickstart-agent-agent1-start/mixedprod-agent1-start/' -e 's/einfra-quickstart-agent-agent1/einfra-mixedprod-agent1/' examples/infra/agent-start.yaml | kubectl apply -f -` (ContractEvent `Submitted`). The owner mesh node (`atirc mesh run --peer ` as the requester account) lists the agent and sees `online: ` (the agent posts it by itself; without it the entry FAILs); pub/sub both ways and a prompt as [0004](0004-vm-atirc-agent.md) (agent1 has `sshAuthorizedKeys` for the guest side). Without step A4b the agent has no harness and never answers. 9. Restart mid-run: Ctrl-C (`kill -INT` the `run requester`): the ready file is removed, the kcp stops, the state stays; start the same command again (the same CLI options, such as `--network`; no application options are needed): same DID, ready file rewritten, every contract still Held, step 6 ssh works again, the agent still answers. 10. Delete: `kubectl delete ephemeralinfras.socialweb.computer mixedprod`: each contract releases first; agent1 `status.atircRevokes` is Done before the bidder's `vm.terminated`; the EphemeralInfra and its ComputeContracts are gone; the bidder host is empty ([0008](0008-host-observer.md) watcher: no VM unit, no sandbox); the requester log has `requester_contract_records_deleted`, and a getRecord of each contract record (accept, rfp, compute.vm, com.fedproxy.rbac; for agent1 also `market.association` and `link.ident.endorsement` at rkey = its R2 DID) at its key answers RecordNotFound (read 3 times); the two vouch records are still in the repos (`deletionPolicy: Retain`; `goat get` both). 11. Stop: Ctrl-C the `run requester`. ### B. install requester (systemd user unit) 1. `sudo $SWC install requester --name mp --dry-run`, then `sudo $SWC install requester --name mp` (account `swc-requester`, identity `--atproto-oauth-qr` by default); the sign-in URL: `$SWC run requester signin mp --state-dir /var/lib/socialweb-computer/swc-requester` (as root or the account); sign in with `scripts/oauth-signin.ts`. 2. `R=/var/lib/socialweb-computer/swc-requester/requesters/mp/ready.json`; as the account, `kubectl --kubeconfig "$(jq -r .kubeconfig "$R")"` reads `MarketRequester app` Ready. 3. Steps A3 to A8 as the account, `$SWC run requester ssh mp --state-dir /var/lib/socialweb-computer/swc-requester`. 4. Restart mid-run: `systemctl --user restart socialweb-computer-requester@mp.service` as the account (and `upgrade` to the same binary): same DID, kcp and OpenBao units adopted (same InvocationID), contracts still Held, ssh again. 5. Step A10, then `sudo $SWC uninstall requester --account swc-requester` (keeps the state dir; `--purge` removes it). ### C. Gateway in the requester kcp (Cluster and k8s-cluster.yaml) The gateway runs inside the requester's core (Gateway `spec.requesterRef`): its clusterprovider, k3s and kine are HostProcesses of that core, and every node is a contract of the same requester. Needs a subuid/subgid range for the user (`check-host` reports it). 1. `$SWC run requester --name mk --state-dir "$ST" --with-gateway --atproto-oauth-qr > "$ST/mk.log" 2>&1 &`; sign in as in A1; wait until the ready file names the gateway: `jq -r .gateway "$ST/requesters/mk/ready.json"` is not `null`. 2. `$SWC run requester cluster mk --cluster c1 --state-dir "$ST"`; then `TK=$($SWC run requester kubeconfig mk --cluster c1 --state-dir "$ST")` (retry until it prints a path: the Cluster is Ready). 3. `kubectl --kubeconfig "$TK" apply -f examples/infra/k8s-cluster.yaml`: a NodeClaim becomes an Infra in the requester kcp, its contract is bid and Held, the node VM fetches meshd by sha256 from the public bucket and joins k3s; poll until the `quickstart-k8s` pod is Ready (a `kubectl wait` before the pod exists returns no match); `kubectl logs deploy/quickstart-k8s` prints `quickstart-k8s-ok`. 4. Pin the node to a remote bidder by leaving only that bidder able to win (the run below suspended the local bidder: `kubectl patch marketbidders.socialweb.computer app --type merge -p '{"spec":{"suspend":true}}'` in its kcp, resumed after). 5. `kubectl --kubeconfig "$TK" delete deployment quickstart-k8s`: the node is reaped, its contract released, `kubectl get nodes` is empty (scale to zero); the bidder host is empty; `computer.socialweb.infra/nodes-` answers RecordNotFound; `computer.socialweb.infra/self` is gone, or names no `nodes-` of this run in `live` or `recentEnded` and is not empty. 6. Ctrl-C the `run requester`. ## Expected Observations | Event | Where | Typical | | --- | --- | --- | | ready file written, `identityMode` `oauthQr`, DID of the account | `requesters/mp/ready.json` | 11 s after the sign-in on an own network (quickstart-prod section 7) | | `vouch-bidder` Applied, `at:///sh.tangled.graph.vouch/`; operator record read back `vouch` | AtprotoRecord status, `goat get` | 1-4 s | | RFP of each contract carries `requester-tangled-vouch`; the bidder's run allows (operator -> requester present) | bidder PolicyWorkflowRun, requester log | first run after the records | | vm1 Deployed, ssh prints `mixedprod-vm-ok`, `x86_64` | EphemeralInfra `status.contracts[vm1]`, ssh | ~20 s after the apply (0012: 19 s) | | sbx1 Held with ComputeVM backend `DenoSandbox`, `sandbox.stdout` `mixedprod-deno-ok sum=10 on linux/x86_64` | contract status | ~6 s (0012) | | agent1: `requester_agent_bundle_placeholder`, `requester_service_auth` `atircSigning` `oauth`, `requester_atirc_minted`, nick JOIN after agent-start | requester log, atirc channel | Held ~20 s, JOIN within a minute of agent-start (0012 prod readings) | | restart: same DID, ready file rewritten, contracts Held | ready file, ComputeContracts | ready again in seconds (quickstart-prod section 7: 1.4 s) | | delete: `requester_atirc_revoked`, `status.atircRevokes` Done, then vm.delete, `vm.terminated`, contracts gone | requester log, bidder repo, kcp | under a minute | | contract records deleted: `requester_contract_records_deleted`, `requester_association_deleted` (agent1); accept, rfp, compute.vm, rbac, agent1's grant + endorsement RecordNotFound at their keys | requester log, getRecord in the requester repo | seconds after the release | | bidder host empty, vouch records present | 0008 watcher, `goat get` | right after the delete | ## Prod Readings (2026-11-01, rc159 `ec0f4849`, account alice) | time (Z) | path, step | reading | | --- | --- | --- | | 10:37:20 | A2 | ready file, `identityMode` `oauthQr`, alice's DID | | 10:37:55 | A5 | mixed-prod Deployed in 35 s on the local bidder (`install bidder --compute-kinds vm,deno`): vm1, sbx1, agent1 Held | | 10:38 | A6, A7 | vm1 ssh iroh and webrtc exit 0; sbx1 stdout `mixedprod-deno-ok sum=10 on linux/x86_64`, exit 0 | | 10:38:31 | A8 | placeholder bundle, mint, agent JOIN, answered `x86_64` | | 10:39 | A9 | restart: same DID, all Held, ssh again | | 10:40:16.805 | A10 | agent1 revoked, then the terminations at 10:40:17; 0 contracts in 5 s; host empty | | 11:19:31 | B3 (rc159) | `run requester ssh` as the account: did not answer within 1m (the ProxyCommand ran under the account's `nologin` shell) | | 11:46:44 | B1, B2 (rc160) | `install requester` (account `swc-requester`, `socialweb.slice`) printed the journal and `run requester signin` hints; the URL from `run requester signin mp`; Ready | | 11:47:14 | B3 | `vm.yaml` pinned to hat-7; ssh as the account: `quickstart-vm-ok`, `x86_64`, exit 0 | | 11:47:15 | B4 | `systemctl --user restart`: same DID, contract Held, ssh `quickstart-vm-ok` again | | 11:47:35 | B5 | delete: 0 contracts; `uninstall --purge` removed the account; hat-7 empty | | 11:31:52 | C1 | `--with-gateway`, ready file names the gateway | | 11:32:03 | C2 | cluster `c1` kubeconfig | | 11:33:04 | C3, C4 | node `2wjeqtgx-c1-node1` on hat-7 (remote, local bidder suspended), k3s v1.36.4+k3s1, pod Ready, `quickstart-k8s-ok` | | 11:35:41 | C5 | node reaped (hat-7 terminate 11:35:36, released 11:35:38), 0 nodes, hat-7 empty | No plaintext token, key or sign-in URL in any log (check 0 on every file). ## Prod Readings (2026-11-10, bob, bidder hat-9 on kcp 2d367fe8, hat-8 requester host) | step | result | | --- | --- | | A1-A2 | `run requester --name mp` with `--atproto-oauth-qr`, signed in by `scripts/oauth-signin.ts` (creds 0600); ready file `identityMode` `oauthQr` | | A3 | `vouch-bidder` Applied (bob -> hat-9) | | A4 | not written: hat-9's operator is bob, the requester itself | | A5-A7 | mixedprod Deployed; vm1 `mixedprod-vm-ok` over iroh, `x86_64` over webrtc; sbx1 `mixedprod-deno-ok sum=10` | | A8 | mesh key, lease and peer record written; agentStart Submitted | | A9 | Ctrl-C removes the ready file; the restart keeps the DID; 3 contracts Held; ssh again | | A10 (first run) | FAIL: records kept, `requester_record_delete_skipped` (the QR session lacked every optional delete scope: qr.fedfork.com's client metadata was behind scope.go) | | A10 (after the QR metadata fix, fresh sign-in, main ea6935f7 CLI) | `requester_contract_records_deleted` 6/6, 6/6, 10/10 and `requester_agent_peer_deleted`; no market record of the run left (the requester's badgeBlueKeys signer key stays by rule) | | B | `install requester --name mp` as swc-requester, signed in; Ready, contracts Held, ssh and deno; `systemctl --user restart` keeps the kcp and kine InvocationIDs and the DID; delete releases (on the 2d367fe8 binary the policy and vm.delete records stayed: fixed in ea6935f7); `uninstall requester --purge` leaves no account, linger or state | | C | `run requester --name mk --with-gateway`, Cluster c1, the node `*-c1-node1` from hat-9 Ready in 50 s, `quickstart-k8s-ok`; delete Deployment: nodes 0 in 160 s; the node contract's records deleted 6/6; the Infra mirror `computer.socialweb.infra/nodes-` and `self` are kept by design and were deleted by hand | ## Local Readings (2026-11-01, own market network `mxp`, main 6497fe61 dev build) `dev up --bidders 1 --compute-kinds vm,deno` (default `bidder-tangled-vouch`, network-composed vouches), then `run requester --name mp --network mxp` (localRepo, handle `root-mp.requester.local`), ready in 11.5 s. | time (Z) | step | reading | | --- | --- | --- | | 08:38 | `kubectl apply --dry-run=server`, the file as committed and filled in | `ephemeralinfra.socialweb.computer/mixedprod created (server dry run)`, exit 0; no `policy` in any contract | | 08:38:38 | apply, `bidderDids` = the network bidder | agent1 Failed at once, `refused: ... root-mp.requester.local is a .local handle ... --atproto-handle ...`, placeholder bundle written; vm1 and sbx1 Held on the pinned bidder (winner did) by 08:38:58 | | 08:38:43 | sbx1 | `sandbox` `{"exitCode":0,"stdout":"mixedprod-deno-ok sum=10 on linux/x86_64\n"}` | | 08:39:05 | `run requester ssh mp --contract einfra-mixedprod-vm1` | iroh: `mixedprod-vm-ok`, `x86_64` (0.87 s); webrtc: `x86_64` (0.68 s) | | 08:39:20 | Ctrl-C | `requester_contract_left_for_restart` vm1, sbx1; kcp stopped, state kept | | 08:39:30 | restart without `--network` | refused: "created with other options (network "mxp" ...)" (the network is part of the same command) | | 08:39:39 | restart with `--network mxp` | ready in 1.6 s, same DID; 08:39:44 vm1 and sbx1 Failed `the held contract is not resumed: the policy run for the recorded winner ... the policy engine is not running`, then `infra_ephemeral_complete`, both released, the EphemeralInfra deleted | On today's main a `run request release` of an invocation that Failed before a VM was held, or is Released, only removes it; no second release is needed. ## Pass Criteria - `kubectl apply --dry-run=server -f examples/infra/mixed-prod.yaml` passes against the requester (A and B) - no file and no command names a policy: `requester-tangled-vouch` and `bidder-tangled-vouch` run, and the contracts are bid only after both vouch records exist - every contract lands on the pinned bidder (`status.contract.winner.did` is the DID put in for `REPLACE-BIDDER-DID`) - vm1 ssh over iroh and webrtc, sbx1 stdout, agent1 JOIN, its own ` online` line (seen by the owner node within 60 s of subscribing; none is a FAIL, [0004](0004-vm-atirc-agent.md) pass criteria) and answer, all on the same requester - a Ctrl-C (A) or unit restart (B) mid-run keeps the DID and every Held contract Held - delete: every agent token revoked before `vm.terminated` (0004 pass criteria), the bidder host empty, every contract record of the requester RecordNotFound at its key, both vouch records still present - no tokens, passwords or session secrets in any log (plaintext check 0 on every log file); the short-lived QR sign-in URL is printed by design ## Cleanup - `kubectl delete ephemeralinfras.socialweb.computer mixedprod` (it ends itself at `endWhen: Complete` or the 1 h ttl) - A: Ctrl-C the `run requester`; remove `$ST` only when the DID is not needed again - B: `sudo $SWC uninstall requester --account swc-requester` (`--purge` for a throwaway account) - the vouch records stay: remove a vouch only from the account itself (`goat`), never by a kcp delete - post done on atirc; stop your `atirc sub` ## Known Pitfalls | Symptom | Cause / fix | | --- | --- | | A: FAIL `the agent never posted online` | before atirc 58f46d0 the agent's first line was the model's choice; since, `agent run` posts it itself: [0004](0004-vm-atirc-agent.md) Known pitfalls | | C: the tenant node joins 4 to 5 min after the apply (`pod_ready_s` 275.7; was 45 to 50): the node's `socialweb-node-prep` logs `HTTP 400 could not load configuration` from `auth/jwt-/login`, the cluster OpenBao `error checking oidc discovery URL: issuer did not match ... got https://127.0.0.1:0/v1/identity/oidc` | before the fix: the bidder's guest control plane fetched the OIDC discovery document 3 s before it configured the issuer and cached the OpenBao default for 5 min (`oidcDocumentRefresh`); a document that does not name the configured issuer is no longer cached (`internal/guestidentity/service.go`, bidder log `refreshing the guest OIDC document failed; no document is cached yet ... does not name the configured issuer yet`) | | C on a dev network: no node VM, ComputeContract `Requesting` / `rfp_submitted`, `requester_submit_rfp_failed` 401 from prod bidders | before 22b49fb2: the test host started `run requester` without `--network`, so it discovered the public relays and the prod bidders; `requester_bidder_discovery` must read `relay:http://127.0.0.1:` and the network's bidders | | after a restart every Held contract is Failed `the held contract is not resumed: ... the policy engine is not running` | main 6497fe61 and rc157: fixed in rc158 (resume waits for the policy engine) | | B: no sign-in URL in `journalctl --user` of the account | the user units of a system account log to the system journal, which the account cannot read: `run requester signin mp --state-dir /var/lib/socialweb-computer/swc-requester` (rc160), or as root `journalctl _UID= _SYSTEMD_USER_UNIT=socialweb-computer-requester@mp.service` | | B: `run requester ssh` as the account does not answer within 1m | rc159: ssh ran its ProxyCommand with the account's `nologin` shell; rc160 runs it under `/bin/sh` | | C: `run requester cluster` says the requester runs no gateway | rc159: the ready file was first written before the gateway was Ready; rc160 writes it once, with `.gateway` | | sbx1 `no bids received within 30s`, stays without a bid | the pinned bidder does not serve deno: `--compute-kinds vm,deno` on `run bidder`, or `install bidder --compute-kinds vm,deno` again for an installed bidder | | every contract `no bids`; bidder `policy rejected ... not vouched` | the operator -> requester record is missing (step A4); denials are cached 5 minutes | | no bids though both vouches exist; `internal association ` has no roots | the bidder is not its operator's agent: no agent-of, or the operator's endorsement is missing (`operator_endorsement_skipped`: sign the operator in again with the optional scope) | | agent1 Failed before its RFP, `refused: the atirc agent needs a requester handle that resolves to ... --atproto-handle ...` | the requester handle is `.local` (a localRepo identity, the default without an identity option) or does not resolve; use the OAuth account, or a local PDS on plc.directory with `--atproto-handle ` ([../quickstart-prod.md](../quickstart-prod.md) section 5) | | `vouch-bidder` (or the operator record) not Applied, the MarketIdentity has `pendingAction` `signInAgain` `ScopeUpgrade` | the QR page (qr.fedfork.com) granted a session without `repo:sh.tangled.graph.vouch`; the scope is optional, so the identity stays Ready: write the record with `goat`, or sign in again once the QR page asks for it | | agent1 Failed `secrets need the requester ingress (getSecrets) or secretsFallback: Sealed` | `--comms-mode Firehose` has no inbound endpoint: add `--secrets-fallback Sealed`, or keep the default comms `Both` | | a restart is refused | the application options differ from `requester.json` of the first start: give the same ones or none, or start a new name | | contracts land on another bidder | `REPLACE-BIDDER-DID` left in one contract or a wrong DID: every contract of the file carries its own `bidderDids` | | Held never reached after the accept | the winner's offering `endpoints` hold nothing the requester can use: no `dumbpipe` on PATH for iroh, and no https endpoint | | HTTP 400 on `market.event` at the bidder | the event is unsigned: the requester is an old build |