Acceptance 0013: Mixed prod (VM, Deno sandbox, agent VM on a user-configured bidder)

Proves ../../examples/infra/mixed-prod.yaml end to end against prod services: a long-lived requester (run requester, then install requester) with a real atproto account, kubectl apply of one EphemeralInfra pinned to a bidder the user configured, use each contract, restart the requester mid-run, delete. The default policy runs on both sides; no file and no command names open (../quickstart-prod.md sections 2, 5 and 7).

• Paths: A run requester in the foreground (OAuth QR); B install requester (systemd user unit of its own account); C the gateway in the requester kcp (run requester --with-gateway, a tenant cluster)
• Plan: 0039 items F-MIXED-PROD (this runbook) and ACC39-MIXED-PROD (its live pass)
• Last pass: 2026-11-01 on prod: A and C on rc159 (ec0f4849), B on the rc160 CLI (840f0437; on rc159 B failed its ssh step, Known Pitfalls). Prod readings below

sequenceDiagram
  participant U as user (kubectl, atirc)
  participant K as requester kcp (run requester)
  participant R as Requester app
  participant B as user-configured bidder
  participant I as irc.fedfork.com
  U->>K: apply vouch.yaml, mixed-prod.yaml
  K->>R: ComputeContract vm1, sbx1, agent1 (bidderDids pin)
  R->>B: RFP (requester-tangled-vouch attached)
  B->>B: bidder-tangled-vouch + attached policy: operator vouched requester?
  B->>R: bid
  R->>R: requester-tangled-vouch: requester vouched bidder?
  R->>B: accept
  B->>R: receipt, guest events (onNetwork, hostKeys, vm.sandbox)
  R->>I: chat.atirc.mint (agent1, leased on receipt + vm.terminated)
  R->>K: ComputeContracts Held, EphemeralInfra Deployed
  U->>B: ssh vm1 (iroh, webrtc); read sbx1 stdout
  U->>K: agent-start ContractEvent -> guest.agentStart
  U->>I: JOIN of the agent nick, prompt, answer
  U->>K: Ctrl-C, restart: Held stays Held
  U->>K: delete EphemeralInfra -> revoke -> vm.delete -> vm.terminated

Preconditions

Item	Need
Bidder	a bidder the user configured: install bidder with the default policy (no --bidder-policy: bidder-tangled-vouch), ../bidder.md; for sbx1 it must serve deno: run bidder --compute-kinds vm,deno, or install bidder --compute-kinds vm,deno (written to its config.json; on an already installed bidder, run the install again with the flag, which restarts the bidder unit); optional --bidder-operator-signin <operator handle> for the operator vouch of step A4
Bidder DID	MarketBidder app status.did in the bidder kcp (the operator reads it); it replaces REPLACE-BIDDER-DID in every contract of mixed-prod.yaml
Requester account	a real atproto account (handle resolves to its DID, PDS public) signed in by OAuth QR; the creds file owner-only (mode 0600), read only by scripts/oauth-signin.ts
Vouches	both directions: requester -> bidder by ../../examples/infra/vouch.yaml in the requester kcp (needs the optional scope repo:sh.tangled.graph.vouch) or goat; bidder operator -> requester by ../../examples/infra/vouch-bidder-operator.yaml in the bidder kcp (operator sign-in) or goat from the operator account
atirc	a token of your own (atirc token <nick>) to atirc sub the owner channel; never print it
Host	Linux, kubectl, jq, ssh, the release or dev binary of the commit under test as $SWC; B: root on the requester host
Rules	README.md shared rules: announce on atirc first, one driver per account, release what you rent

Steps

ST is an owner-only state dir (ST=~/.tmp/acc13; mkdir -p "$ST"; chmod 700 "$ST"). Write each vouch at least 5 minutes before the first apply when a contract was denied before (verdicts are cached, denials too).

A. run requester (foreground, OAuth QR)

1. Start: $SWC run requester --name mp --state-dir "$ST" -- --atproto-oauth-qr > "$ST/mp.log" 2>&1 &. Sign in from the pendingAction.url in the log with scripts/oauth-signin.ts --creds <file> --account <ACCT> --url "<url>" (read the url right before, never print or paste it).
2. Wait for $ST/requesters/mp/ready.json (written atomically once the root is Ready); eval "$($SWC run requester env mp --state-dir "$ST")" sets KUBECONFIG to the requester workspace; kubectl get marketrequesters.socialweb.computer app -o jsonpath='{.status.phase} {.status.did}' is Ready <did>; the ready file's identityMode is oauthQr.
3. Requester -> bidder vouch: sed -e "s/REPLACE-REQUESTER-REPO/repo-$(jq -r .did "$ST/requesters/mp/ready.json" | tr : -)/" -e "s/REPLACE-BIDDER-DID/<bidder did>/g" -e "s/2026-11-01T00:00:00Z/$(date -u +%Y-%m-%dT%H:%M:%SZ)/" examples/infra/vouch.yaml | kubectl apply -f -; AtprotoRecord vouch-bidder Applied with at://<requester did>/sh.tangled.graph.vouch/<bidder did>. When the session lacks the vouch scope, write the same record with goat record create from the requester account instead.
4. Bidder operator -> requester vouch: in the bidder kcp, vouch-bidder-operator.yaml with REPLACE-REQUESTER-DID (the operator sign-in of ../bidder.md), or goat from the operator account; check with goat get "at://<operator did>/sh.tangled.graph.vouch/<requester did>" | jq -r .kind -> vouch.
5. Apply: sed -e "s|ssh-ed25519 AAAAREPLACE-WITH-YOUR-PUBLIC-KEY you@laptop|$(cat "$(jq -r .sshPublicKey "$ST/requesters/mp/ready.json")")|" -e "s/REPLACE-BIDDER-DID/<bidder did>/" examples/infra/mixed-prod.yaml | kubectl apply -f -; kubectl wait ephemeralinfras.socialweb.computer/mixedprod --for=jsonpath='{.status.phase}'=Deployed --timeout=5m.
6. vm1: $SWC run requester ssh mp --state-dir "$ST" --contract einfra-mixedprod-vm1 --exec 'cat /root/mixedprod' prints mixedprod-vm-ok; the same with --transport webrtc --exec 'uname -m' prints x86_64.
7. sbx1: kubectl get computecontracts.socialweb.computer einfra-mixedprod-sbx1 -o jsonpath='{.status.contract.sandbox}' has stdout mixedprod-deno-ok sum=10 on linux/x86_64 and exitCode 0.
8. agent1: the requester log has requester_agent_bundle_placeholder (no bao step) and, after the receipt, requester_atirc_minted; then sed -e 's/quickstart-agent-agent1-start/mixedprod-agent1-start/' -e 's/einfra-quickstart-agent-agent1/einfra-mixedprod-agent1/' examples/infra/agent-start.yaml | kubectl apply -f -; in your atirc sub the agent nick <label>-<rfp rkey> JOINs; atirc pub '<nick>: uname -m' gets x86_64 back.
9. Restart mid-run: Ctrl-C (kill -INT the run requester): the ready file is removed, the kcp stops, the state stays; start the same command again (the same options before --, such as --network; nothing after -- is needed): same DID, ready file rewritten, every contract still Held, step 6 ssh works again, the agent still answers.
10. Delete: kubectl delete ephemeralinfras.socialweb.computer mixedprod: each contract releases first; agent1 status.atircRevokes is Done before the bidder's vm.terminated; the EphemeralInfra and its ComputeContracts are gone; the bidder host is empty (0008 watcher: no VM unit, no sandbox); the two vouch records are still in the repos (deletionPolicy: Retain; goat get both).
11. Stop: Ctrl-C the run requester.

B. install requester (systemd user unit)

1. sudo $SWC install requester --name mp --dry-run, then sudo $SWC install requester --name mp (account swc-requester, identity --atproto-oauth-qr by default); the sign-in URL: $SWC run requester signin mp --state-dir /var/lib/socialweb-computer/swc-requester (as root or the account); sign in with scripts/oauth-signin.ts.
2. R=/var/lib/socialweb-computer/swc-requester/requesters/mp/ready.json; as the account, kubectl --kubeconfig "$(jq -r .kubeconfig "$R")" reads MarketRequester app Ready.
3. Steps A3 to A8 as the account, $SWC run requester ssh mp --state-dir /var/lib/socialweb-computer/swc-requester.
4. Restart mid-run: systemctl --user restart socialweb-computer-requester@mp.service as the account (and upgrade to the same binary): same DID, kcp and OpenBao units adopted (same InvocationID), contracts still Held, ssh again.
5. Step A10, then sudo $SWC uninstall requester --account swc-requester (keeps the state dir; --purge removes it).

C. Gateway in the requester kcp (Cluster and k8s-cluster.yaml)

The gateway runs inside the requester's core (Gateway spec.requesterRef): its clusterprovider, k3s and kine are HostProcesses of that core, and every node is a contract of the same requester. Needs a subuid/subgid range for the user (check-host reports it).

1. $SWC run requester --name mk --state-dir "$ST" --with-gateway -- --atproto-oauth-qr > "$ST/mk.log" 2>&1 &; sign in as in A1; wait until the ready file names the gateway: jq -r .gateway "$ST/requesters/mk/ready.json" is not null.
2. $SWC run requester cluster mk --cluster c1 --state-dir "$ST"; then TK=$($SWC run requester kubeconfig mk --cluster c1 --state-dir "$ST") (retry until it prints a path: the Cluster is Ready).
3. kubectl --kubeconfig "$TK" apply -f examples/infra/k8s-cluster.yaml: a NodeClaim becomes an Infra in the requester kcp, its contract is bid and Held, the node VM fetches meshd by sha256 from the public bucket and joins k3s; poll until the quickstart-k8s pod is Ready (a kubectl wait before the pod exists returns no match); kubectl logs deploy/quickstart-k8s prints quickstart-k8s-ok.
4. Pin the node to a remote bidder by leaving only that bidder able to win (the run below suspended the local bidder: kubectl patch marketbidders.socialweb.computer app --type merge -p '{"spec":{"suspend":true}}' in its kcp, resumed after).
5. kubectl --kubeconfig "$TK" delete deployment quickstart-k8s: the node is reaped, its contract released, kubectl get nodes is empty (scale to zero); the bidder host is empty.
6. Ctrl-C the run requester.

Expected Observations

Event	Where	Typical
ready file written, identityMode oauthQr, DID of the account	requesters/mp/ready.json	11 s after the sign-in on an own network (quickstart-prod section 7)
vouch-bidder Applied, at://<requester did>/sh.tangled.graph.vouch/<bidder did>; operator record read back vouch	AtprotoRecord status, goat get	1-4 s
RFP of each contract carries requester-tangled-vouch; the bidder's run allows (operator -> requester present)	bidder PolicyWorkflowRun, requester log	first run after the records
vm1 Deployed, ssh prints mixedprod-vm-ok, x86_64	EphemeralInfra status.contracts[vm1], ssh	~20 s after the apply (0012: 19 s)
sbx1 Held with ComputeVM backend DenoSandbox, sandbox.stdout mixedprod-deno-ok sum=10 on linux/x86_64	contract status	~6 s (0012)
agent1: requester_agent_bundle_placeholder, requester_service_auth atircSigning oauth, requester_atirc_minted, nick JOIN after agent-start	requester log, atirc channel	Held ~20 s, JOIN within a minute of agent-start (0012 prod readings)
restart: same DID, ready file rewritten, contracts Held	ready file, ComputeContracts	ready again in seconds (quickstart-prod section 7: 1.4 s)
delete: requester_atirc_revoked, status.atircRevokes Done, then vm.delete, vm.terminated, contracts gone	requester log, bidder repo, kcp	under a minute
bidder host empty, vouch records present	0008 watcher, goat get	right after the delete

Prod Readings (2026-11-01, rc159 ec0f4849, account alice)

time (Z)	path, step	reading
10:37:20	A2	ready file, identityMode oauthQr, alice's DID
10:37:55	A5	mixed-prod Deployed in 35 s on the local bidder (install bidder --compute-kinds vm,deno): vm1, sbx1, agent1 Held
10:38	A6, A7	vm1 ssh iroh and webrtc exit 0; sbx1 stdout mixedprod-deno-ok sum=10 on linux/x86_64, exit 0
10:38:31	A8	placeholder bundle, mint, agent JOIN, answered x86_64
10:39	A9	restart: same DID, all Held, ssh again
10:40:16.805	A10	agent1 revoked, then the terminations at 10:40:17; 0 contracts in 5 s; host empty
11:19:31	B3 (rc159)	run requester ssh as the account: did not answer within 1m (the ProxyCommand ran under the account's nologin shell)
11:46:44	B1, B2 (rc160)	install requester (account swc-requester, socialweb.slice) printed the journal and run requester signin hints; the URL from run requester signin mp; Ready
11:47:14	B3	vm.yaml pinned to hat-7; ssh as the account: quickstart-vm-ok, x86_64, exit 0
11:47:15	B4	systemctl --user restart: same DID, contract Held, ssh quickstart-vm-ok again
11:47:35	B5	delete: 0 contracts; uninstall --purge removed the account; hat-7 empty
11:31:52	C1	--with-gateway, ready file names the gateway
11:32:03	C2	cluster c1 kubeconfig
11:33:04	C3, C4	node 2wjeqtgx-c1-node1 on hat-7 (remote, local bidder suspended), k3s v1.36.4+k3s1, pod Ready, quickstart-k8s-ok
11:35:41	C5	node reaped (hat-7 terminate 11:35:36, released 11:35:38), 0 nodes, hat-7 empty

No plaintext token, key or sign-in URL in any log (check 0 on every file).

Local Readings (2026-11-01, own market network mxp, main 6497fe61 dev build)

dev up --bidders 1 --compute-kinds vm,deno (default bidder-tangled-vouch, network-composed vouches), then run requester --name mp --network mxp (localRepo, handle root-mp.requester.local), ready in 11.5 s.

time (Z)	step	reading
08:38	kubectl apply --dry-run=server, the file as committed and filled in	ephemeralinfra.socialweb.computer/mixedprod created (server dry run), exit 0; no policy in any contract
08:38:38	apply, bidderDids = the network bidder	agent1 Failed at once, refused: ... root-mp.requester.local is a .local handle ... --atproto-handle ..., placeholder bundle written; vm1 and sbx1 Held on the pinned bidder (winner did) by 08:38:58
08:38:43	sbx1	sandbox {"exitCode":0,"stdout":"mixedprod-deno-ok sum=10 on linux/x86_64\n"}
08:39:05	run requester ssh mp --contract einfra-mixedprod-vm1	iroh: mixedprod-vm-ok, x86_64 (0.87 s); webrtc: x86_64 (0.68 s)
08:39:20	Ctrl-C	requester_contract_left_for_restart vm1, sbx1; kcp stopped, state kept
08:39:30	restart without --network	refused: "created with other options (network "mxp" ...)" (the network is part of the same command)
08:39:39	restart with --network mxp	ready in 1.6 s, same DID; 08:39:44 vm1 and sbx1 Failed the held contract is not resumed: the policy run for the recorded winner ... the policy engine is not running, then infra_ephemeral_complete, both released, the EphemeralInfra deleted

Pass Criteria

• kubectl apply --dry-run=server -f examples/infra/mixed-prod.yaml passes against the requester (A and B)
• no file and no command names a policy: requester-tangled-vouch and bidder-tangled-vouch run, and the contracts are bid only after both vouch records exist
• every contract lands on the pinned bidder (status.contract.winner.did is the DID put in for REPLACE-BIDDER-DID)
• vm1 ssh over iroh and webrtc, sbx1 stdout, agent1 JOIN and answer, all on the same requester
• a Ctrl-C (A) or unit restart (B) mid-run keeps the DID and every Held contract Held
• delete: every agent token revoked before vm.terminated (0004 pass criteria), the bidder host empty, both vouch records still present
• no token, password, key, cookie or sign-in URL on any terminal or in any log (plaintext check 0 on every log file)

Cleanup

• kubectl delete ephemeralinfras.socialweb.computer mixedprod (it ends itself at endWhen: Complete or the 1 h ttl)
• A: Ctrl-C the run requester; remove $ST only when the DID is not needed again
• B: sudo $SWC uninstall requester --account swc-requester (--purge for a throwaway account)
• the vouch records stay: remove a vouch only from the account itself (goat), never by a kcp delete
• post done on atirc; stop your atirc sub

Known Pitfalls

Symptom	Cause / fix
after a restart every Held contract is Failed the held contract is not resumed: ... the policy engine is not running	main 6497fe61 and rc157: fixed in rc158 (resume waits for the policy engine)
B: no sign-in URL in journalctl --user of the account	the user units of a system account log to the system journal, which the account cannot read: run requester signin mp --state-dir /var/lib/socialweb-computer/swc-requester (rc160), or as root journalctl _UID=<uid> _SYSTEMD_USER_UNIT=socialweb-computer-requester@mp.service
B: run requester ssh as the account does not answer within 1m	rc159: ssh ran its ProxyCommand with the account's nologin shell; rc160 runs it under /bin/sh
C: run requester cluster says the requester runs no gateway	rc159: the ready file was first written before the gateway was Ready; rc160 writes it once, with .gateway
sbx1 no bids received within 30s, stays without a bid	the pinned bidder does not serve deno: --compute-kinds vm,deno on run bidder, or install bidder --compute-kinds vm,deno again for an installed bidder
every contract no bids; bidder policy rejected ... not vouched	the operator -> requester record is missing (step A4); denials are cached 5 minutes
agent1 Failed before its RFP, refused: the atirc agent needs a requester handle that resolves to <did> ... --atproto-handle ...	the requester handle is .local (a localRepo identity, the default without an identity option) or does not resolve; use the OAuth account, or a local PDS on plc.directory with --atproto-handle <a domain you control> (../quickstart-prod.md section 5)
vouch-bidder (or the operator record) not Applied, the MarketIdentity has pendingAction signInAgain ScopeUpgrade	the QR page (qr.fedfork.com) granted a session without repo:sh.tangled.graph.vouch; the scope is optional, so the identity stays Ready: write the record with goat, or sign in again once the QR page asks for it
agent1 Failed secrets need the requester ingress (getSecrets) or secretsFallback: Sealed	--comms-mode Firehose has no inbound endpoint: add --secrets-fallback Sealed after --, or keep the default comms Both
a restart is refused	the options after -- differ from requester.json of the first start: give the same ones or none, or start a new name
contracts land on another bidder	REPLACE-BIDDER-DID left in one contract or a wrong DID: every contract of the file carries its own bidderDids