Acceptance 0013: Mixed prod (VM, Deno sandbox, agent VM on a user-configured bidder)
Proves ../../examples/infra/mixed-prod.yaml end to end against prod services: a
long-lived requester (run requester, then install requester) with a real atproto account, kubectl apply of one
EphemeralInfra pinned to a bidder the user configured, use each contract, restart the requester mid-run, delete. The
default policy runs on both sides; no file and no command names open (../quickstart-prod.md
sections 2, 5 and 7).
- Paths: A
run requesterin the foreground (OAuth QR); Binstall requester(systemd user unit of its own account); C the gateway in the requester kcp (run requester --with-gateway, a tenant cluster) - Plan: 0039 items F-MIXED-PROD (this runbook) and ACC39-MIXED-PROD (its live pass)
- Dev network, C (
0013:c, composition, own 2-bidder network on hat-8, main aefab467 dev build + the OIDC discovery fix), 2026-11-15 00:57-01:08Z, alice: 3 passes,pod_ready_s95.3 (the discovery race hit: the first fetch was rejected, the next one after the issuer was configured), 50.3, 45.1;nodes_zero_s155.5, 155.5, 155.4;seconds296.8, 235.4, 235.3. Before the fix, same net and binary:pod_ready_s275.7 (5 min discovery cache),seconds479.0 - Last pass A, B, C: 2026-11-13 03:11-04:44Z on prod rollout 20 (v0.0.0-rc165-pre1 038bd32e, binary 24a8b530 for the
CLI and
install requester), bob, no pin (the kit'smixed-prod-nopin.yaml; winner hat-8 every time); consent pages qr.fedfork.com for @bobvmbuilder.bsky.social (screenshots read). A: ready 42 s after start,oauthQr; Deployed 31 s (vm1, sbx1, agent1 Held); sbx1mixedprod-deno-ok sum=10; agent1 (bundle by A4b) association +32 s after agentStart,online and listening+141 s,uname -mansweredx86_64in 3.6 s; Ctrl-C and restart: ready 12 s, same DID, all Held,hostnameanswered 3 s; delete 10.8 s, R2 released first,requester_association_deleted, records 6/6, 10/10, 14/14, the 14 requester-repo URIs RecordNotFound x3 (the bidder's ownmarket.receiptstays), 0 market.event / vm.delete / rfp / accept left (fix 13), R2 and agent DIDsplc_tombstone, vouches kept, hat-8 empty. A6 re-checked on a vm1-only EphemeralInfra (the first apply carried an old key from the kit file:Permission denied (publickey), a harness fault): iroh 4.4 s, webrtc 1.4 s,mixedprod-vm-ok x86_64, release 10/10, RecordNotFound x3. The restart logged 10 xReconciler error the OpenBaoServer requester reports no address yet (phase Starting)at one instant (first start 0 ERROR). B (swc-requester, vm1 + sbx1): install exit 0 after the sign-in; ssh iroh 3.8 s, webrtc 1.3 s;systemctl --user restart: kcp, kine, front and bao units keep their InvocationID, same DID, ssh 0.98 s; delete 3 s, 8/8 RecordNotFound x3;uninstall --purge: no account, state, linger or install record (the installed requester'slogs/stayed empty and the journal had none of its lines, so B's ERROR count was not read). C: ready with the gateway 44 s after start, cluster c1 + kubeconfig 17 s, pod Ready 50 s after the apply (node*-c1-node1on hat-8),quickstart-k8s-ok; delete Deployment: nodes 0 in 156 s, node records 6/6,selfandnodes-<hex>mirrors gone, 0 ERROR - C also reads the index: after nodes 0
computer.socialweb.infra/selfis the account's, not the run's (it holds every deployment of the DID, live or inrecentEnded, and is deleted only when nothing is left in it), so C checks it apart from the run's records: it is gone, or neither itslivenor itsrecentEndednames this run'snodes-<hex>and it is not empty; the run's own records, thenodes-<hex>mirror among them, must be RecordNotFound (2026-11-18, alice:selfheld 32recentEndedentries of other deployments, so the old "self gone" read failed, as on 2026-11-14); not yet run with that check; the controller reconciles the index (infra_index_reconciled: entries whose record ended or is missing are dropped fromliveat start and on each index write) (a 2-bidder dev net 2026-11-14 never got a Ready quickstart-k8s pod in 15 min: the k8s node did not join there) - Previous pass C: 2026-11-12 21:12-21:19Z, kcp e190a3b1 (rollout 19, rc165 candidate): 0 ERROR, the infra
selfandnodes-<hex>mirror gone 20 s after nodes 0 (8e849b42). Before: 2026-11-12 20:02-20:10Z, kcp ac51b91e (rollout 18): node Ready,quickstart-k8s-ok, the requester logs 0 ERROR (no ComponentLink errors: fixed in 363dd814); nodes 0 in 156 s; but the emptiedcomputer.socialweb.infra/nodes-<hex>mirror stayed in the repo (infra_record_replacedwith 0 nodes; deleted by hand),selfwent. Last pass: 2026-11-12 18:46-19:00Z on prod as bob, kcp cdd93727 (release 6380eee4, rollout 16), A and C, no pin (winner hat-8): agent1 online,uname -manswered in 2.9 s, delete in 12 s (r2Released8.8 s), 45/45 RecordNotFound,core.log0 ERROR; C: node Ready,quickstart-k8s-ok, nodes 0 in 156 s,computer.socialweb.infra/{self,nodes-...}gone with the cluster (none left after the stop). Open: C's requester logsReconciler errorbus: no accepted ComponentLink allows root:EventStream.../watch-infra-nodes-<hex>(andwatch-infraindex-<repo>)events.subscribex18 while the node lives. Before: 2026-11-12 17:47-18:08Z, kcp 60d0d3b7 (release 0b68ceeb, rollout 15), path A, NObidderDidspin (requester discovery; winners hat-8, then hat-9): vm1 iroh + webrtc, sbx1, agent1 with its harness from the pre-written bundle (step A4b): association written 36 s after agent-start,online+136 s, messages both ways,uname -manswered in 2.9 s; delete: every contract record and agent1's grant + endorsement RecordNotFound x3, both DIDsplc_tombstone, hosts empty. 2026-11-11 (kcp f482083c, no pin): A, B (hat-7) and C (hat-8) pass except agent1 (no association on a declarative contract: fixed in 60d0d3b7). Earlier: 2026-11-10 as bob (bidder hat-9): A, B and C; 2026-11-01: A and C on rc159 (ec0f4849), B on the rc160 CLI (840f0437). Prod readings below
sequenceDiagram
participant U as user (kubectl, atirc)
participant K as requester kcp (run requester)
participant R as Requester app
participant B as user-configured bidder
participant I as irc.fedfork.com
U->>K: apply vouch.yaml, mixed-prod.yaml
K->>R: ComputeContract vm1, sbx1, agent1 (bidderDids pin)
R->>B: RFP (requester-tangled-vouch attached)
B->>B: bidder-tangled-vouch + attached policy: operator vouched requester?
B->>R: bid
R->>R: requester-tangled-vouch: requester vouched bidder?
R->>B: accept
B->>R: receipt, signed guest events at computed keys (onNetwork, hostKeys, vm.sandbox)
R->>I: chat.atirc.mint (agent1, leased on receipt + vm.terminated)
R->>K: ComputeContracts Held, EphemeralInfra Deployed
U->>B: ssh vm1 (iroh, webrtc); read sbx1 stdout
U->>K: agent-start ContractEvent -> guest.agentStart
U->>I: JOIN of the agent nick, prompt, answer
U->>K: Ctrl-C, restart: Held stays Held
U->>K: delete EphemeralInfra -> revoke -> vm.delete -> vm.terminated
Preconditions
| Item | Need |
|---|---|
| Bidder | a bidder the user configured: install bidder with the default policy (no --bidder-policy: bidder-tangled-vouch), ../bidder.md; for sbx1 it must serve deno: run bidder --compute-kinds vm,deno, or install bidder --compute-kinds vm,deno (written to its config.json; on an already installed bidder, run the install again with the flag, which restarts the bidder unit); optional --bidder-operator-signin <operator handle> for the operator vouch of step A4. A prod-qr install bidder lists https in its offering endpoints only with the bidder option market-https=true (config.json "bidder-options": "--market-https=true"); without it the offering lists iroh and webrtc |
| Bidder DID | MarketBidder app status.did in the bidder kcp (the operator reads it); it replaces REPLACE-BIDDER-DID in every contract of mixed-prod.yaml |
| Bidder association | the bidder DID is its operator's agent: agent-of/<operator DID> in its DID doc (written by the bidder) and, in the operator repo, market.association/<bidder DID> then link.ident.endorsement/<bidder DID> (written by the bidder with --bidder-operator-signin, needs the optional scope repo:link.ident.endorsement?action=create&action=delete, or by deno run -A --no-lock scripts/associate-bidder.ts --creds <file> --account <ACCOUNT> --agent-did <bidder DID>); check: socialweb-computer internal association <bidder DID> roots [operator DID] |
| Requester account | a real atproto account (handle resolves to its DID, PDS public) signed in by OAuth QR; the creds file owner-only (mode 0600), read only by scripts/oauth-signin.ts |
| Vouches | both directions: requester -> bidder by ../../examples/infra/vouch.yaml in the requester kcp (needs the optional scope repo:sh.tangled.graph.vouch) or goat; bidder operator -> requester by ../../examples/infra/vouch-bidder-operator.yaml in the bidder kcp (operator sign-in) or goat from the operator account, unless the requester account is that bidder's own user or one of the user's verified agents. Owner, 2026-11-18: "the vouch policy mode must also trust the own user and their associated DIDs/agents to N levels down". bidder-tangled-vouch (and requester-tangled-vouch) build the trust snapshot from the identlink walk of each party (trust.BuildSnapshot): each link verified both ways (the agent's DID doc agent-of/<principal>, the principal's link.ident.endorsement/<agent> and com.publicdomainrelay.temp.market.association/<agent>), N = 100 levels (identlink.DefaultMaxDepth), 256 lookups; a requester that shares a root with the bidder, or whose operator is on the bidder's side, is trusted (policytest cases operated-by-counterparty, same-verified-operator, two-levels-below-counterparty, counterparty-two-levels-below-self(-operator) and their forged and half-verified twins). The test runs, per bidder, the same walks of the account and of the bidder (com.fedfork.exec.identlink.walk) and reads the operator's vouch of the account; a bidder admits the account when that vouch exists or the walks say so |
| atirc | a token of your own (atirc token <nick>) to atirc sub the owner channel; never print it |
| Host | Linux, kubectl, jq, ssh, dumbpipe on PATH (the requester tries iroh first; a bidder without https, such as hat-9, is reached only over iroh or webrtc), the release or dev binary of the commit under test as $SWC; B: root on the requester host |
| Rules | README.md shared rules: announce on atirc first, one driver per account, release what you rent |
Steps
The composition starts its run requester with --bidder-dids <the target's bidders> (plan 0040 step 5): the list
filters discovery for every rental, a --with-gateway node's too; on a market network the network sets them.
ST is a NEW owner-only state dir for each run (ST=~/.tmp/acc13-$(date +%s); mkdir "$ST"; chmod 700 "$ST"): a
run requester --name mp on an existing dir resumes that dir's requester with its old account and session (it logs
run requester resumes an existing requester), never a new sign-in. Write each vouch at least 5
minutes before the first apply when a contract was denied before (verdicts are cached, denials too).
A. run requester (foreground, OAuth QR)
-
Start:
$SWC run requester --name mp --state-dir "$ST" --atproto-oauth-qr > "$ST/mp.log" 2>&1 &. Sign in from thependingAction.urlin the log withscripts/oauth-signin.ts --creds <file> --account <ACCT> --url "<url>"(read the url right before, never print or paste it). -
Wait for
$ST/requesters/mp/ready.json(written atomically once the root is Ready);eval "$($SWC run requester env mp --state-dir "$ST")"setsKUBECONFIGto the requester workspace;kubectl get marketrequesters.socialweb.computer app -o jsonpath='{.status.phase} {.status.did}'isReady <did>; the ready file'sidentityModeisoauthQr. -
Requester -> bidder vouch:
sed -e "s/REPLACE-REQUESTER-REPO/repo-$(jq -r .did "$ST/requesters/mp/ready.json" | tr : -)/" -e "s/REPLACE-BIDDER-DID/<bidder did>/g" -e "s/2026-11-01T00:00:00Z/$(date -u +%Y-%m-%dT%H:%M:%SZ)/" examples/infra/vouch.yaml | kubectl apply -f -; AtprotoRecordvouch-bidderApplied withat://<requester did>/sh.tangled.graph.vouch/<bidder did>. When the session lacks the vouch scope, write the same record withgoat record createfrom the requester account instead. -
Bidder operator -> requester vouch: in the bidder kcp,
vouch-bidder-operator.yamlwithREPLACE-REQUESTER-DID(the operator sign-in of ../bidder.md), orgoatfrom the operator account; check withgoat get "at://<operator did>/sh.tangled.graph.vouch/<requester did>" | jq -r .kind->vouch. Skip it for a bidder whose user is the requester account or one of the account's operators (Preconditions, Vouches). When the write needs the owner's account, it is owner-only: ask the owner to write it; an agent does not. 4b. agent1's harness (optional; without it the agent joins the mesh but nothing answers): write agent1's bundle into the requester's own OpenBao before the apply, with the placeholder/etc/atirc/atirc.envand an atirc agent config (as 0004--agent-config; mode 0600 file, never printed) at/etc/atirc/config.toml; a bundle that is there is kept, and the requester adds the association entries itself:D=$ST/requesters/mp/dev/mp/openbaoservers/requester/openbao BAO=$(find ~/.cache/socialweb-computer/runtime -maxdepth 2 -name bao -type f | head -1) export BAO_ADDR=$(kubectl get openbaoservers.socialweb.computer requester -o jsonpath='{.status.address}') export BAO_CACERT=$D/tls/ca.crt BAO_CLIENT_CERT=$D/clients/requester/client.crt BAO_CLIENT_KEY=$D/clients/requester/client.key (umask 077; jq -n --rawfile c ~/.config/atirc/config.toml '[{path: "/etc/atirc/atirc.env", value: "", mode: "0600"}, {path: "/etc/atirc/config.toml", value: $c, mode: "0600"}]' > "$ST/agent1-bundle.json") BAO_TOKEN=$($BAO write -field=token auth/cert/login name=requester) \ $BAO kv put -mount=requester contracts/einfra-mixedprod-agent1/secrets bundle=@"$ST/agent1-bundle.json" >/dev/null rm -f "$ST/agent1-bundle.json" -
Apply:
sed -e "s|ssh-ed25519 AAAAREPLACE-WITH-YOUR-PUBLIC-KEY you@laptop|$(cat "$(jq -r .sshPublicKey "$ST/requesters/mp/ready.json")")|" -e "s/REPLACE-BIDDER-DID/<bidder did>/" examples/infra/mixed-prod.yaml | kubectl apply -f -;kubectl wait ephemeralinfras.socialweb.computer/mixedprod --for=jsonpath='{.status.phase}'=Deployed --timeout=5m. -
vm1:
$SWC run requester ssh mp --state-dir "$ST" --contract einfra-mixedprod-vm1 --exec -- cat /root/mixedprodprintsmixedprod-vm-ok; the same with--transport webrtc --exec -- uname -mprintsx86_64. -
sbx1:
kubectl get computecontracts.socialweb.computer einfra-mixedprod-sbx1 -o jsonpath='{.status.contract.sandbox}'hasstdoutmixedprod-deno-ok sum=10 on linux/x86_64andexitCode0. -
agent1: the requester log has
requester_agent_bundle_placeholder(no bao step) and, after the receipt, the token-less mesh agent'srequester_agent_mesh_key_writtenandrequester_association_written(grant + endorsement at rkey = the in-VM R2 DID; the agent's peer record is R2's co-located account's, no record in the requester repo names it, 0004); thensed -e 's/quickstart-agent-agent1-start/mixedprod-agent1-start/' -e 's/einfra-quickstart-agent-agent1/einfra-mixedprod-agent1/' examples/infra/agent-start.yaml | kubectl apply -f -(ContractEventSubmitted). The owner mesh node (atirc mesh run --peer <requester_agent_identity peer>as the requester account) lists the agent and seesonline: <nick>(the agent posts it by itself; without it the entry FAILs); pub/sub both ways and a prompt as 0004 (agent1 hassshAuthorizedKeysfor the guest side). Without step A4b the agent has no harness and never answers. -
Restart mid-run: Ctrl-C (
kill -INTtherun requester): the ready file is removed, the kcp stops, the state stays; start the same command again (the same CLI options, such as--network; no application options are needed): same DID, ready file rewritten, every contract still Held, step 6 ssh works again, the agent still answers. -
Delete:
kubectl delete ephemeralinfras.socialweb.computer mixedprod: each contract releases first; agent1status.atircRevokesis Done before the bidder'svm.terminated; the EphemeralInfra and its ComputeContracts are gone; the bidder host is empty (0008 watcher: no VM unit, no sandbox); the requester log hasrequester_contract_records_deleted, and a getRecord of each contract record (accept, rfp, compute.vm, com.fedproxy.rbac; for agent1 alsomarket.associationandlink.ident.endorsementat rkey = its R2 DID) at its key answers RecordNotFound (read 3 times); the two vouch records are still in the repos (deletionPolicy: Retain;goat getboth). -
Stop: Ctrl-C the
run requester.
B. install requester (systemd user unit)
sudo $SWC install requester --name mp --dry-run, thensudo $SWC install requester --name mp(accountswc-requester, identity--atproto-oauth-qrby default); the sign-in URL:$SWC run requester signin mp --state-dir /var/lib/socialweb-computer/swc-requester(as root or the account); sign in withscripts/oauth-signin.ts.R=/var/lib/socialweb-computer/swc-requester/requesters/mp/ready.json; as the account,kubectl --kubeconfig "$(jq -r .kubeconfig "$R")"readsMarketRequester appReady.- Steps A3 to A8 as the account,
$SWC run requester ssh mp --state-dir /var/lib/socialweb-computer/swc-requester. - Restart mid-run:
systemctl --user restart socialweb-computer-requester@mp.serviceas the account (andupgradeto the same binary): same DID, kcp and OpenBao units adopted (same InvocationID), contracts still Held, ssh again. - Step A10, then
sudo $SWC uninstall requester --account swc-requester(keeps the state dir;--purgeremoves it).
C. Gateway in the requester kcp (Cluster and k8s-cluster.yaml)
The gateway runs inside the requester's core (Gateway spec.requesterRef): its clusterprovider, k3s and kine are
HostProcesses of that core, and every node is a contract of the same requester. Needs a subuid/subgid range for the
user (check-host reports it).
$SWC run requester --name mk --state-dir "$ST" --with-gateway --atproto-oauth-qr > "$ST/mk.log" 2>&1 &; sign in as in A1; wait until the ready file names the gateway:jq -r .gateway "$ST/requesters/mk/ready.json"is notnull.$SWC run requester cluster mk --cluster c1 --state-dir "$ST"; thenTK=$($SWC run requester kubeconfig mk --cluster c1 --state-dir "$ST")(retry until it prints a path: the Cluster is Ready).kubectl --kubeconfig "$TK" apply -f examples/infra/k8s-cluster.yaml: a NodeClaim becomes an Infra in the requester kcp, its contract is bid and Held, the node VM fetches meshd by sha256 from the public bucket and joins k3s; poll until thequickstart-k8spod is Ready (akubectl waitbefore the pod exists returns no match);kubectl logs deploy/quickstart-k8sprintsquickstart-k8s-ok.- Pin the node to a remote bidder by leaving only that bidder able to win (the run below suspended the local
bidder:
kubectl patch marketbidders.socialweb.computer app --type merge -p '{"spec":{"suspend":true}}'in its kcp, resumed after). kubectl --kubeconfig "$TK" delete deployment quickstart-k8s: the node is reaped, its contract released,kubectl get nodesis empty (scale to zero); the bidder host is empty;computer.socialweb.infra/nodes-<hex>answers RecordNotFound;computer.socialweb.infra/selfis gone, or names nonodes-<hex>of this run inliveorrecentEndedand is not empty.- Ctrl-C the
run requester.
Expected Observations
| Event | Where | Typical |
|---|---|---|
ready file written, identityMode oauthQr, DID of the account | requesters/mp/ready.json | 11 s after the sign-in on an own network (quickstart-prod section 7) |
vouch-bidder Applied, at://<requester did>/sh.tangled.graph.vouch/<bidder did>; operator record read back vouch | AtprotoRecord status, goat get | 1-4 s |
RFP of each contract carries requester-tangled-vouch; the bidder's run allows (operator -> requester present) | bidder PolicyWorkflowRun, requester log | first run after the records |
vm1 Deployed, ssh prints mixedprod-vm-ok, x86_64 | EphemeralInfra status.contracts[vm1], ssh | ~20 s after the apply (0012: 19 s) |
sbx1 Held with ComputeVM backend DenoSandbox, sandbox.stdout mixedprod-deno-ok sum=10 on linux/x86_64 | contract status | ~6 s (0012) |
agent1: requester_agent_bundle_placeholder, requester_service_auth atircSigning oauth, requester_atirc_minted, nick JOIN after agent-start | requester log, atirc channel | Held ~20 s, JOIN within a minute of agent-start (0012 prod readings) |
| restart: same DID, ready file rewritten, contracts Held | ready file, ComputeContracts | ready again in seconds (quickstart-prod section 7: 1.4 s) |
delete: requester_atirc_revoked, status.atircRevokes Done, then vm.delete, vm.terminated, contracts gone | requester log, bidder repo, kcp | under a minute |
contract records deleted: requester_contract_records_deleted, requester_association_deleted (agent1); accept, rfp, compute.vm, rbac, agent1's grant + endorsement RecordNotFound at their keys | requester log, getRecord in the requester repo | seconds after the release |
| bidder host empty, vouch records present | 0008 watcher, goat get | right after the delete |
Prod Readings (2026-11-01, rc159 ec0f4849, account alice)
| time (Z) | path, step | reading |
|---|---|---|
| 10:37:20 | A2 | ready file, identityMode oauthQr, alice's DID |
| 10:37:55 | A5 | mixed-prod Deployed in 35 s on the local bidder (install bidder --compute-kinds vm,deno): vm1, sbx1, agent1 Held |
| 10:38 | A6, A7 | vm1 ssh iroh and webrtc exit 0; sbx1 stdout mixedprod-deno-ok sum=10 on linux/x86_64, exit 0 |
| 10:38:31 | A8 | placeholder bundle, mint, agent JOIN, answered x86_64 |
| 10:39 | A9 | restart: same DID, all Held, ssh again |
| 10:40:16.805 | A10 | agent1 revoked, then the terminations at 10:40:17; 0 contracts in 5 s; host empty |
| 11:19:31 | B3 (rc159) | run requester ssh as the account: did not answer within 1m (the ProxyCommand ran under the account's nologin shell) |
| 11:46:44 | B1, B2 (rc160) | install requester (account swc-requester, socialweb.slice) printed the journal and run requester signin hints; the URL from run requester signin mp; Ready |
| 11:47:14 | B3 | vm.yaml pinned to hat-7; ssh as the account: quickstart-vm-ok, x86_64, exit 0 |
| 11:47:15 | B4 | systemctl --user restart: same DID, contract Held, ssh quickstart-vm-ok again |
| 11:47:35 | B5 | delete: 0 contracts; uninstall --purge removed the account; hat-7 empty |
| 11:31:52 | C1 | --with-gateway, ready file names the gateway |
| 11:32:03 | C2 | cluster c1 kubeconfig |
| 11:33:04 | C3, C4 | node 2wjeqtgx-c1-node1 on hat-7 (remote, local bidder suspended), k3s v1.36.4+k3s1, pod Ready, quickstart-k8s-ok |
| 11:35:41 | C5 | node reaped (hat-7 terminate 11:35:36, released 11:35:38), 0 nodes, hat-7 empty |
No plaintext token, key or sign-in URL in any log (check 0 on every file).
Prod Readings (2026-11-10, bob, bidder hat-9 on kcp 2d367fe8, hat-8 requester host)
| step | result |
|---|---|
| A1-A2 | run requester --name mp with --atproto-oauth-qr, signed in by scripts/oauth-signin.ts (creds 0600); ready file identityMode oauthQr |
| A3 | vouch-bidder Applied (bob -> hat-9) |
| A4 | not written: hat-9's operator is bob, the requester itself |
| A5-A7 | mixedprod Deployed; vm1 mixedprod-vm-ok over iroh, x86_64 over webrtc; sbx1 mixedprod-deno-ok sum=10 |
| A8 | mesh key, lease and peer record written; agentStart Submitted |
| A9 | Ctrl-C removes the ready file; the restart keeps the DID; 3 contracts Held; ssh again |
| A10 (first run) | FAIL: records kept, requester_record_delete_skipped (the QR session lacked every optional delete scope: qr.fedfork.com's client metadata was behind scope.go) |
| A10 (after the QR metadata fix, fresh sign-in, main ea6935f7 CLI) | requester_contract_records_deleted 6/6, 6/6, 10/10 and requester_agent_peer_deleted; no market record of the run left (the requester's badgeBlueKeys signer key stays by rule) |
| B | install requester --name mp as swc-requester, signed in; Ready, contracts Held, ssh and deno; systemctl --user restart keeps the kcp and kine InvocationIDs and the DID; delete releases (on the 2d367fe8 binary the policy and vm.delete records stayed: fixed in ea6935f7); uninstall requester --purge leaves no account, linger or state |
| C | run requester --name mk --with-gateway, Cluster c1, the node *-c1-node1 from hat-9 Ready in 50 s, quickstart-k8s-ok; delete Deployment: nodes 0 in 160 s; the node contract's records deleted 6/6; the Infra mirror computer.socialweb.infra/nodes-<hex> and self are kept by design and were deleted by hand |
Local Readings (2026-11-01, own market network mxp, main 6497fe61 dev build)
dev up --bidders 1 --compute-kinds vm,deno (default bidder-tangled-vouch, network-composed vouches), then
run requester --name mp --network mxp (localRepo, handle root-mp.requester.local), ready in 11.5 s.
| time (Z) | step | reading |
|---|---|---|
| 08:38 | kubectl apply --dry-run=server, the file as committed and filled in | ephemeralinfra.socialweb.computer/mixedprod created (server dry run), exit 0; no policy in any contract |
| 08:38:38 | apply, bidderDids = the network bidder | agent1 Failed at once, refused: ... root-mp.requester.local is a .local handle ... --atproto-handle ..., placeholder bundle written; vm1 and sbx1 Held on the pinned bidder (winner did) by 08:38:58 |
| 08:38:43 | sbx1 | sandbox {"exitCode":0,"stdout":"mixedprod-deno-ok sum=10 on linux/x86_64\n"} |
| 08:39:05 | run requester ssh mp --contract einfra-mixedprod-vm1 | iroh: mixedprod-vm-ok, x86_64 (0.87 s); webrtc: x86_64 (0.68 s) |
| 08:39:20 | Ctrl-C | requester_contract_left_for_restart vm1, sbx1; kcp stopped, state kept |
| 08:39:30 | restart without --network | refused: "created with other options (network "mxp" ...)" (the network is part of the same command) |
| 08:39:39 | restart with --network mxp | ready in 1.6 s, same DID; 08:39:44 vm1 and sbx1 Failed the held contract is not resumed: the policy run for the recorded winner ... the policy engine is not running, then infra_ephemeral_complete, both released, the EphemeralInfra deleted |
On today's main a run request release of an invocation that Failed before a VM was held, or is Released, only removes
it; no second release is needed.
Pass Criteria
kubectl apply --dry-run=server -f examples/infra/mixed-prod.yamlpasses against the requester (A and B)- no file and no command names a policy:
requester-tangled-vouchandbidder-tangled-vouchrun, and the contracts are bid only after both vouch records exist - every contract lands on the pinned bidder (
status.contract.winner.didis the DID put in forREPLACE-BIDDER-DID) - vm1 ssh over iroh and webrtc, sbx1 stdout, agent1 JOIN, its own
<nick> onlineline (seen by the owner node within 60 s of subscribing; none is a FAIL, 0004 pass criteria) and answer, all on the same requester - a Ctrl-C (A) or unit restart (B) mid-run keeps the DID and every Held contract Held
- delete: every agent token revoked before
vm.terminated(0004 pass criteria), the bidder host empty, every contract record of the requester RecordNotFound at its key, both vouch records still present - no tokens, passwords or session secrets in any log (plaintext check 0 on every log file); the short-lived QR sign-in URL is printed by design
Cleanup
kubectl delete ephemeralinfras.socialweb.computer mixedprod(it ends itself atendWhen: Completeor the 1 h ttl)- A: Ctrl-C the
run requester; remove$STonly when the DID is not needed again - B:
sudo $SWC uninstall requester --account swc-requester(--purgefor a throwaway account) - the vouch records stay: remove a vouch only from the account itself (
goat), never by a kcp delete - post done on atirc; stop your
atirc sub
Known Pitfalls
| Symptom | Cause / fix |
|---|---|
A: FAIL the agent never posted <nick> online | before atirc 58f46d0 the agent's first line was the model's choice; since, agent run posts it itself: 0004 Known pitfalls |
C: the tenant node joins 4 to 5 min after the apply (pod_ready_s 275.7; was 45 to 50): the node's socialweb-node-prep logs HTTP 400 could not load configuration from auth/jwt-<hash>/login, the cluster OpenBao error checking oidc discovery URL: issuer did not match ... got https://127.0.0.1:0/v1/identity/oidc | before the fix: the bidder's guest control plane fetched the OIDC discovery document 3 s before it configured the issuer and cached the OpenBao default for 5 min (oidcDocumentRefresh); a document that does not name the configured issuer is no longer cached (internal/guestidentity/service.go, bidder log refreshing the guest OIDC document failed; no document is cached yet ... does not name the configured issuer yet) |
C on a dev network: no node VM, ComputeContract Requesting / rfp_submitted, requester_submit_rfp_failed 401 from prod bidders | before 22b49fb2: the test host started run requester without --network, so it discovered the public relays and the prod bidders; requester_bidder_discovery must read relay:http://127.0.0.1:<port> and the network's bidders |
after a restart every Held contract is Failed the held contract is not resumed: ... the policy engine is not running | main 6497fe61 and rc157: fixed in rc158 (resume waits for the policy engine) |
B: no sign-in URL in journalctl --user of the account | the user units of a system account log to the system journal, which the account cannot read: run requester signin mp --state-dir /var/lib/socialweb-computer/swc-requester (rc160), or as root journalctl _UID=<uid> _SYSTEMD_USER_UNIT=socialweb-computer-requester@mp.service |
B: run requester ssh as the account does not answer within 1m | rc159: ssh ran its ProxyCommand with the account's nologin shell; rc160 runs it under /bin/sh |
C: run requester cluster says the requester runs no gateway | rc159: the ready file was first written before the gateway was Ready; rc160 writes it once, with .gateway |
sbx1 no bids received within 30s, stays without a bid | the pinned bidder does not serve deno: --compute-kinds vm,deno on run bidder, or install bidder --compute-kinds vm,deno again for an installed bidder |
every contract no bids; bidder policy rejected ... not vouched | the operator -> requester record is missing (step A4); denials are cached 5 minutes |
no bids though both vouches exist; internal association <bidder DID> has no roots | the bidder is not its operator's agent: no agent-of, or the operator's endorsement is missing (operator_endorsement_skipped: sign the operator in again with the optional scope) |
agent1 Failed before its RFP, refused: the atirc agent needs a requester handle that resolves to <did> ... --atproto-handle ... | the requester handle is .local (a localRepo identity, the default without an identity option) or does not resolve; use the OAuth account, or a local PDS on plc.directory with --atproto-handle <a domain you control> (../quickstart-prod.md section 5) |
vouch-bidder (or the operator record) not Applied, the MarketIdentity has pendingAction signInAgain ScopeUpgrade | the QR page (qr.fedfork.com) granted a session without repo:sh.tangled.graph.vouch; the scope is optional, so the identity stays Ready: write the record with goat, or sign in again once the QR page asks for it |
agent1 Failed secrets need the requester ingress (getSecrets) or secretsFallback: Sealed | --comms-mode Firehose has no inbound endpoint: add --secrets-fallback Sealed, or keep the default comms Both |
| a restart is refused | the application options differ from requester.json of the first start: give the same ones or none, or start a new name |
| contracts land on another bidder | REPLACE-BIDDER-DID left in one contract or a wrong DID: every contract of the file carries its own bidderDids |
| Held never reached after the accept | the winner's offering endpoints hold nothing the requester can use: no dumbpipe on PATH for iroh, and no https endpoint |
HTTP 400 on market.event at the bidder | the event is unsigned: the requester is an old build |