Acceptance 0001: account modes, per-component pause, and one VM per host
This is the runnable acceptance used for plans 0023 and 0024. It exercises:
- a localRepo bidder;
- an oauthQr bidder that receives the market over the prod PDS firehose;
- a localRepo gateway requester;
- per-component suspend without stopping the GuestControlPlane or the shared IngressProxy;
- typed Bidder settings and fail-closed overrides;
- a tenant k3s cluster that scales from zero to three market VMs and back to zero, placing one VM on each bidder host;
- ssh over WebRTC from the CLI requester to a guest on each bidder host (section 9, plan 0034 ACC34-CLI-SSH).
Run it from a host with goat, kubectl, ssh to socialweb.computer, hat-7
and hat-9, dumbpipe on PATH, and the credential files named below. The
requester tries iroh first when dumbpipe is on PATH; hat-9 offers no https, so
a requester host without dumbpipe cannot reach it. It changes the prod gateway
and the bidders. Do not run it without that authority.
Hosts for this pass (2026-11-10)
- hat-7 is john's prod bidder (
did:plc:7q5lnitxv4meizdon5fmhzrq). Its offering endpoints are iroh and webrtc (itsmarket-https=truetransition ended 2026-11-10). Do not re-identity hat-7 and do not end its session. - hat-9 is bob's bidder (
did:plc:4vcig7s43z3nyqrnepmcfzqt). Its offering endpoints are iroh and webrtc. - Both serve the compute kinds
vmanddeno. - local is hat-8's
swc-bidder(did:plc:2l2wdry77c7jtucbslstagkj, localRepo, associated to bob). It runs the same release as hat-7 and hat-9; a bidder on an older release with anendpointUrl-only offering must not take part (upgrade it, or keep its rootspec.suspend=true). - The oauthQr bidder parts (sections 2, 4, 7 and the section 9 sign-in) run on
hat-8 as a temporary prod-qr bidder for bob under a fresh account (2026-11-10:
swc-qr0001), so no existing install is touched. Read it and its kubeconfig where the steps sayhat-7; hat-7 is then paused with the others in sections 1 and 4 and is one more host in sections 5, 6 and 9. The oauthQr mode is a drop-in written before the first start, so no localRepo DID is ever created:
A=swc-qr0001; S=/var/lib/socialweb-computer/$A; H=bobvmbuilder.bsky.social
BIN=/var/tmp/swc-release/socialweb-computer-<sha12>
D=$S/.config/systemd/user/socialweb-computer-bidder@$H.service.d
sudo useradd --system --user-group --no-create-home --home-dir $S --shell /usr/sbin/nologin --comment socialweb-computer $A
sudo install -d -m 0700 -o $A -g $A $S
sudo install -d -m 0755 -o $A -g $A $S/.config $S/.config/systemd $S/.config/systemd/user "$D"
printf '[Service]\nExecStart=\nExecStart=%s/bin/current/socialweb-computer run bidder --handle %%i --fg --atproto-oauth-qr\n' "$S" |
sudo install -m 0644 -o $A -g $A /dev/stdin "$D/10-oauth-qr.conf"
sudo "$BIN" install bidder --account $A --adopt-account --handle $H --compute-kinds vm,deno --vm-slots 8 \
--guest-image-source file:///var/lib/socialweb-computer/guest-image
It comes up oauthQr AwaitingScan with no local Pds; section 2 signs bob in.
After the pass: pause it, sudo $S/bin/current/socialweb-computer uninstall bidder --account $A --purge, sudo userdel $A, then delete its records in
bob's repo with deleteRecord: market.offering/self,
market.heartbeat/self and its computer.socialweb.ingress.listen/market record. A
getRecord of each then answers RecordNotFound. Never delete badgeBlueKeys,
market.association, link.ident.endorsement or vouch records. A step that only reads may run on the existing
hat-9 bob bidder instead.
- The runner host is hat-8 (it has
kubectl,dumbpipe,denoand Chrome forscripts/oauth-signin.ts, and both credential files). Withoutgoatthere, the gateway XRPC calls take a service-auth token from alice's PDS (com.atproto.server.getServiceAuth?aud=$AUD&lxm=$N.<method>) and call$GW/xrpc/$N.<method>with curl; the session token stays in a mode-0600 file of the scratch dir, deleted afterwards. - Gateway steps are lead-approved. Take a backup first and verify after. Coordinate them with whoever redeploys the gateway.
- Market service auth uses a hostname-free audience:
<bidder DID>#pdr_temp_marketand<bidder DID>#pdr_temp_compute_event. The gateway cluster XRPC audience (AUDbelow) is unchanged. - Every market read is one getRecord at a computable key; no step lists records.
Section 9 last pass: 2026-11-13 01:16-01:25Z, every bidder and the gateway on rollout 20 (release/rc165
038bd32e = e190a3b1 + the startup-wait fix 5b75ac08 + the release-records fix, v0.0.0-rc165-pre1, binary
24a8b5307d19), CLI 24a8b5307d19 on hat-8, alice by QR (consent qr.fedfork.com for @aliceoa.bsky.social), no
bidders list, the other two MarketBidders suspended per host and resumed (Ready 6/6) after. Run on the three
prod bidders; the oauthQr host was not re-run (swc-qr0001 was removed after the rollout 19 pass). Fresh /
restarted ssh over WebRTC: local 2.7 / 5.7 s, hat-9 2.6 / 7.1 s, hat-7 2.5 / 2.0 s (rollout 19: ~64 s
restarted). Held, winner = $HOST, uid=0 both times, plaintext 0 / 0, released, hosts 0 ComputeVM / 0
firecracker, 10 logged alice URIs RecordNotFound x3. core.log waits / timeouts / "serves discovery" per run (3
core starts): 7/0/7, 6/0/6, 5/0/5. Each release left 0 market.event / compute.events.vm.delete (the 0006
release-records bug is fixed in the CLI). Sections 1-8 stand on the rollout 19 reading below.
Last pass: 2026-11-12 22:25-23:35Z, every bidder and the gateway on kcp e190a3b1 (release 66f3bd4d,
rollout 19), CLI 66f3bd4d on hat-8, alice requester, bob oauthQr bidder swc-qr0001 on hat-8 (installed for
the pass, removed after), no bidders list anywhere. Section 9 FAIL (restart time); every other section
passes. Preconditions: requester did:plc:jjcsg5fg3ujd22ywuxg4lr72 Associated=True, phase Ready, roots
[alice], approvedAssociations empty; three bidders bidder-tangled-vouch; both vouches; the three bidder
pairs agent-of only, roots [operator]. Section 1: hat-9, local and hat-7 Suspended
{"desired":6,"notReady":0,"ready":5,"suspended":1}, Bidder true Ready. Section 2: bob by QR (consent
qr.fedfork.com for @bobvmbuilder.bsky.social; the --done text timeout is harmless), root Ready 4/4,
EventStream gateway-pds Running, offering ["iroh","webrtc"], no endpointUrl. Section 3: Cluster Ready
40 s after createCluster, get --raw "$P/readyz" ok, 5 namespaces, k3s v1.36.4+k3s1. Section 3a: title and
noExpiry ok; page sign-in by its own #handle form (consent k8s.socialweb.computer); the runbook's curl
api() got 401 until it sent Origin: $GW (gateway WARN web_origin_refused, empty origin: a cookie POST
needs the page origin; the block now sends it). Then the page no-expiry kubeconfig (notAfter 9999-12-31)
lists 5 namespaces, $P/readyz ok, lists noExpiry, revoke -> refused (You must be logged in), $KC
still works; the same over xrpc. Section 4: first VM on the oauthQr bidder, pod Ready 90 s after the apply, 0
nodes 134 s after the scale, claims empty. Sections 5-6 (the oauthQr bidder paused after section 4 so the
three VMs land on three hosts): 1 -> local (98 s), 2 -> hat-7 (104 s), 3 -> hat-9 (83 s), each winner paused
before the next step: 3 pods on 3 distinct nodes, 3 distinct winners. A2 with all three winners paused:
3 -> 2 released hat-9 (155 s), 2 -> 1 hat-7 (155 s), 1 -> 0 local (144 s); each step one node gone, the
released host at 0 ComputeVM / FirecrackerVM / firecracker, vm.terminated deleted_on_request with a
vm.delete request ref; gateway 3 requester_released + 3 requester_contract_records_deleted (6 each);
cluster deleted. Section 7 (oauthQr): heartbeat frozen at 22:59:29Z for 180 s, root Ready
{"desired":4,"notReady":0,"ready":3,"suspended":1}, GuestControlPlane and ingress-iroh Ready, healthz
200; advanced 5 s after resume. Section 8 (hat-7): applied:true (generation 39), heartbeat every 5 s,
offering cid unchanged for 60 s, four offerings ["iroh","webrtc"] without endpointUrl, drift reverted
(ChildEditReverted, then NoDrift), typo Failed / InvalidKey, child generation unchanged. Section 9
(ACC34, fresh / restarted; consent qr.fedfork.com for @aliceoa.bsky.social): oauthQr 2.5 / 64.0 s, local
3.2 / 64.2 s, hat-9 2.6 / 67.2 s, hat-7 2.6 / 63.0 s; Held, Linux + uid=0 both times, winner = $HOST,
0 plaintext signaling lines, Released, requester_contract_records_deleted 12/12, every logged record URI
RecordNotFound three times, hosts empty. FAIL: the restarted invocation takes ~64 s, not ~3-7 s. Root cause
(cdd93727, internal/controllers/runtime/exportkinds.go awaitEndpoints): every core start waits for each
APIExportEndpointSlice's EndpointURLsReady condition, which kcp v0.33.0 never sets on these slices (they
carry only APIExportValid and PartitionValid, with the URL in status.endpoints). So all 10 exports log
"waits for its virtual workspace" at kcp ready + 0.2 s and "starts before its virtual workspace serves
discovery" (1m0s) at + 60.2 s, with 0 "serves discovery" successes (30 waits / 30 timeouts / 0 successes
over the 3 core starts of each invocation). The virtual workspace answers <url>/clusters/*/apis 200 the
whole time. Every core start pays it: CLI invocations, bidders (11 WARNs each at the 21:10Z restart), the
gateway (13). Also seen: each CLI release left its release market.event and compute.events.vm.delete
in alice's repo (the 0006 bug); the 8 records were deleted by hand. Bidder ERROR 0 on hat-7 and hat-9; hat-8 1
(firecrackervm write conflict on a HostProcess, retried, 0006). Gateway ERROR only the pre-existing
com.fedfork.irc.agentOperator ScopeMissing retries, plus 2 xrpc: unmapped error from a malformed
name== query (the runner's own mistake; the gateway does not map it to a 400). Cleanup: swc-qr0001
uninstalled --purge, userdel; bob's market.offering/self, market.heartbeat/self and
computer.socialweb.ingress.listen/market RecordNotFound; badgeBlueKeys, vouches, endorsements kept.
Previous pass: 2026-11-11 11:54-12:45Z, every bidder and the gateway on kcp e496cbff
(release d9438c15, ident.link associations), CLI d9438c15 on hat-8, alice requester,
bob oauthQr bidder swc-qr0001 on hat-8 (installed for the pass, removed after),
no bidders list anywhere (the gateway env names only a deny list). Association: requester
did:plc:jjcsg5fg3ujd22ywuxg4lr72 Associated=True, phase Ready, alsoKnownAs
[at://root-app.gateway.local, https://ident.link/agent-of/<alice>], endorsement + grant at rkey =
requester DID, walk roots [alice], approvedAssociations empty; the three bidder pairs (hat-7 -> john,
hat-9 and hat-8 -> bob): agent-of only, endorsement {agent}, grant without expiresAt, roots
[operator]. Section 1: hat-9, local and hat-7 Suspended {"desired":6,"notReady":0,"ready":5,"suspended":1},
Bidder true Ready. Section 2: bob signed in by QR (page "Session transferred"; the --done text
waits 60 s for "close this window", the page says "close this page": harmless), root Ready 4/4,
EventStream gateway-pds Running, offering ["iroh","webrtc"]. Section 3: Cluster Ready 34 s after
createCluster (a bare get --raw /readyz asks the host root, not the cluster: NotFound; namespaces list fine).
Section 4: first VM on the oauthQr bidder, pod Ready 76 s after the apply; zero nodes 139 s after the
scale. Section 5: oauthQr (73 s), local (52 s), hat-9 (80 s), hat-7 (77 s): four pods on four distinct
nodes. Section 6 (A2, all four winners paused): 4 -> 3 -> 2 -> 1 -> 0 released hat-9, oauthQr, hat-7,
local (155, 160, 155, 145 s), each step one requester_released, requester_contract_records_deleted
6/6, the released host at 0 ComputeVM / FirecrackerVM / firecracker, vm.terminated deleted_on_request
with a vm.delete request ref; cluster deleted. Section 7: heartbeat frozen at 12:27:04Z for 180 s,
root Ready {"desired":4,"notReady":0,"ready":3,"suspended":1}, GuestControlPlane and ingress-iroh
Ready, healthz 200; advanced 4 s after resume. Section 8 (hat-7): applied:true, heartbeat every 5 s,
offering cid unchanged for 60 s, four offerings ["iroh","webrtc"] without endpointUrl, drift
reverted (a direct edit of a composed child is reverted), typo Failed / InvalidKey, child
generation unchanged. Section 9 (ACC34, fresh / restarted): oauthQr 2.0 / 3.8 s, local 3.1 / 4.8 s,
hat-9 2.4 / 5.0 s, hat-7 2.3 / 3.6 s; every run Released, requester_contract_records_deleted 12/12,
every logged record URI RecordNotFound three times, 0 plaintext signaling lines, hosts empty. Bidder
ERROR 0 on all three; gateway ERROR only Reconciler error of the root:app AtprotoRecord
testaccount-alice-atirc-operator (com.fedfork.irc.agentOperator, ScopeMissing; Failed since
02:43Z, pre-existing). Cleanup: swc-qr0001 uninstalled with --purge, userdel; bob's
market.offering/self, market.heartbeat/self and computer.socialweb.ingress.listen/market
RecordNotFound.
Earlier pass: 2026-11-10 18:06-18:41Z, every bidder and the gateway on c770a887
(alice requester, bob oauthQr bidder swc-qr0001 on hat-8). Section 1: hat-9,
hat-7 and local Suspended, Bidder true Ready. Section 2: bob signed in by QR,
root Ready 4/4, EventStream gateway-pds Running, offering ["iroh","webrtc"].
Section 4: the first VM landed on the oauthQr bidder 49 s after the apply; back
to zero nodes in 2 min 34 s. Section 5: four pods on four distinct nodes, one
market VM on each bidder (oauthQr, local, hat-7, hat-9). Section 6 (A2): all
four winners paused, deployment deleted, every VM released (0 ComputeVMs, 0
firecracker on every host), cluster deleted. Section 7: heartbeat frozen at
18:21:54Z for 180 s, root Ready {"desired":4,"notReady":0,"ready":3,"suspended":1},
GuestControlPlane and ingress-iroh Ready, GCP healthz 200; advanced 15 s after
resume. Section 8: override applied:true (generation 8), heartbeat every 5 s,
offering cid unchanged for 60 s, all four offerings ["iroh","webrtc"] without
endpointUrl, drift reverted (a direct edit of a composed child is reverted),
typo Failed / InvalidKey with no child generation change. Section 9 (ACC34,
fresh / restarted invocation): hat-9 5.2 / 5.2 s, hat-7 21.6 / 4.8 s, local
3.7 / 6.9 s, oauthQr 9.1 / 3.8 s; every run Released,
requester_contract_records_deleted 12 of 12, every logged record URI
RecordNotFound three times, 0 plaintext signaling lines, hosts empty.
Gateway re-check (5e6c4e3d), 2026-11-11 01:32-01:58Z: the gateway on 5e6c4e3d
(redeployed 2026-11-10 19:23Z), every bidder on 253031e2 (rollout 10), alice
signed in at k8s.socialweb.computer by scripts/oauth-signin.ts --start-api;
sections 3, 1, 4, 5 and 6 only. No oauthQr bidder is installed (swc-qr0001 is
gone and hat-7 is localRepo, receiving over the relay and Jetstream), so section
2 does not apply and hat-7 is the receiving host of section 4. Section 3:
Cluster Ready 36 s after createCluster; the kubeconfig of getKubeconfig and of
the web POST /api/clusters/kubeconfig both name
https://kcp.socialweb.computer/clusters/root:2wjeqtgx:<cluster>:clusters, a
client certificate and no CA data; get --raw /readyz ok (the host root then still answered), k3s v1.36.4+k3s1,
namespaces default kube-node-lease kube-public kube-system users
(/api/config reports kcpPublicHost: null: the GatewayAPI does not pass it
to the cluster API; the kubeconfig takes it from SOCIALWEB_GATEWAY_KCP_PUBLIC_HOST).
Section 4: hat-9 and local Suspended, first VM on hat-7, pod Ready 54 s after
the scale, back to zero nodes in 149 s. Section 5: 1 -> hat-7 (53 s), pause, 2 ->
hat-8 (49 s), pause, 3 -> hat-9 (49 s): three pods on three distinct nodes, one
market VM on each bidder. Section 6, one step at a time after the resume: 3 -> 2
released the hat-9 VM (156 s), 2 -> 1 the hat-8 VM (155 s), 1 -> 0 the hat-7 VM
(133 s). Each step: one node gone, the other pods on their nodes, one
requester_released, one requester_contract_records_deleted (6 records), the
released host at 0 ComputeVM / 0 FirecrackerVM / 0 firecracker / 0 vm dirs, the
vm.terminated pinned deleted_on_request with a vm.delete request ref; 0
bidder ERROR. A pre-run VM (hat-8, before section 1) released while hat-8 was
paused (A2). Gateway 0 ERROR through the scale steps. After deleteCluster the
session owner of alice is pruned, and the declared-state testaccounts
AtprotoRecords of alice log Reconciler error LinkDenied every 5 s (also after
the 2026-11-10 pass, until the 19:20Z restart): the AtprotoRepo controller kept
the binding of the deleted AtprotoRepo (no Gone hook); fixed on main, a
record then waits for its repo binding (INFO).
This runbook was validated end to end on v0.0.0-rc81: three pods on three
distinct tenant nodes, one market VM on each bidder host, paused winners forcing
the next placement, and every VM released when the cluster returned to zero.
It passed again, all nine sections, on v0.0.0-rc142 (2026-10-31) with every
bidder on rc142, and on v0.0.0-rc154 (2026-11-01, gateway, ssh gateway and
every bidder on rc154): first VM on hat-7 in 43 s, one VM per host (hat-7,
local, hat-9), back to zero, heartbeat freeze and advance, offering unchanged,
drift reverted, typo fails closed InvalidKey, ACC34 exit 0 on every host
(fresh / restarted kcp: local 2.2 / 3.2 s, hat-9 3.1 / 2.2 s, hat-7 4.9 / 4.5 s),
no plaintext signaling in any log.
Plan 0026 (abstract kinds, the bus with default-deny ComponentLinks) changed
the object names and fields it reads; the commands below use the plan 0026
names. Every kind is read by its full resource name (for example
marketbidders.socialweb.computer), because a market kcp binds several
exports that share short names.
The authoritative OAuth role sets are in
internal/atproto/oauth/scope.go, ported from
typescript-helpers/lib/oauth-scope/mod.ts (the generator behind
did-key-associator/generated/oauth-scope.js). Do not substitute
transition:generic.
Variables
GW=https://k8s.socialweb.computer
AUD=did:web:k8s.socialweb.computer
N=com.publicdomainrelay.temp.cluster
CL=scale-auth-$(date +%s)
W=~/.tmp/auth-modes-0001
mkdir -p "$W"
chmod 700 "$W"
PROD_K=/var/lib/socialweb-computer/socialweb/kcp/admin.kubeconfig
PROD_ACCOUNT=2wjeqtgx
H7_K=$(ssh hat-7 'sudo -n -u swc-bidder sh -c "ls -t /var/lib/socialweb-computer/swc-bidder/handles/*/kcp/front.json | head -1 | sed s,front.json,admin.kubeconfig,"')
H9_K=$(ssh hat-9 'sudo -n -u swc-bidder sh -c "ls -t /var/lib/socialweb-computer/swc-bidder/handles/*/kcp/front.json | head -1 | sed s,front.json,admin.kubeconfig,"')
LOCAL_K=$(sudo -n -u swc-bidder sh -c "ls -t /var/lib/socialweb-computer/swc-bidder/handles/*/kcp/front.json | head -1 | sed s,front.json,admin.kubeconfig,")
The gateway account workspace name (PROD_ACCOUNT) appears in the cluster
provider log line home":"root:<account>:<cluster>". It is stable for a state
dir.
Requester association
The gateway requester must be associated before it accepts any bid. Check it before creating a cluster.
The requester MarketIdentity lives in root:app; the gateway's components
(Pds/pds, Requester/requester, GatewayAPI/gateway-api) live in root.
The association is the ident.link pair + grant
(../design/ident-link-associations.md):
the gateway adds https://ident.link/agent-of/<alice DID> to the requester
DID's alsoKnownAs (its local PDS plc-rotation.key; a failure logs
gateway_requester_agent_of), and alice's repo holds
com.publicdomainrelay.temp.market.association/<requester DID> {agent,
createdAt} and link.ident.endorsement/<requester DID> {agent}: declared in
deploy/testaccounts/alice/identity.yaml, or written by
scripts/associate-bidder.ts from the Alice account. The identity controller
walks the requester DID and reports the roots in status.associations and the
Associated condition. SOCIALWEB_GATEWAY_APPROVED_ASSOCIATIONS
(comma-separated DIDs in the gateway env) pre-approves operators in
spec.approvedAssociations; it is not needed when the walk reaches alice. Do
not patch the MarketIdentity spec: the Gateway root owns it and server-side
apply reverts the edit.
ID_ARGS="--kubeconfig $PROD_K --server https://127.0.0.1:6443/clusters/root:app"
MI=marketidentities.socialweb.computer
ALICE=did:plc:lpfuqerea3deuoyrn7ojser4; APDS=https://jellybaby.us-east.host.bsky.network
ASSOC=$(ssh socialweb.computer "sudo -n -u socialweb env HOME=/var/lib/socialweb-computer/socialweb kubectl $ID_ARGS get $MI requester -o jsonpath='{.status.conditions[?(@.type==\"Associated\")].status}'")
RI_DID=$(ssh socialweb.computer "sudo -n -u socialweb env HOME=/var/lib/socialweb-computer/socialweb kubectl $ID_ARGS get $MI requester -o jsonpath='{.status.did}'")
printf 'requester=%s associated=%s\n' "$RI_DID" "$ASSOC"
curl -s "https://plc.directory/$RI_DID" | jq -c .alsoKnownAs # holds https://ident.link/agent-of/$ALICE
for c in link.ident.endorsement com.publicdomainrelay.temp.market.association; do
curl -s "$APDS/xrpc/com.atproto.repo.getRecord?repo=$ALICE&collection=$c&rkey=$RI_DID" | jq -c '.value // .error'
done # {agent: RI_DID}, {agent: RI_DID, createdAt}
if [ "$ASSOC" != True ]; then
deno run -A --no-lock scripts/associate-bidder.ts --creds ~/.aliceoa_bsky_social_source \
--account ALICE --agent-did "$RI_DID" | jq -c .
fi
socialweb-computer internal association "$RI_DID" | jq -c '{roots, errors}' # roots [ALICE]
ssh socialweb.computer "sudo -n -u socialweb env HOME=/var/lib/socialweb-computer/socialweb kubectl $ID_ARGS get $MI requester -o jsonpath='{.spec.approvedAssociations}{\"\n\"}{.status.associations}{\"\n\"}{.status.phase}{\"\n\"}'"
A missing agent-of entry is a PLC op on the gateway host (lead go first):
socialweb-computer identity agent-of add <alice DID> --did <requester DID> --pds-state-dir "$(kubectl $ID_ARGS get pds pds -o jsonpath='{.spec.stateDir}')" --drop-borrowed-handle
(the Pds stateDir is <instance state>/components/Pds-pds unless the root overrides it).
Wait until the identity phase is Ready, status.associations names alice's
DID and the Requester child reports spec.operatorDID with it.
Without that, every bid is denied by requester-tangled-vouch even though
bidder policy runs succeed.
Bidder policy: bidder-tangled-vouch on every prod bidder
Every prod bidder runs bidder-tangled-vouch. It is the MarketBidder default
when spec.policy is unset, and install bidder writes it to config.json as
bidder-policy. open runs only when an operator sets it explicitly
(--bidder-policy open or spec.policy: open); it is never acceptable on a
prod bidder. The bidder bids only for a requester in its operator's vouch
graph, so the acceptance requester alice (did:plc:lpfuqerea3deuoyrn7ojser4)
must be vouched by the bidder operator: bob (did:plc:ocnuqjlzgohypj3zoznermms)
and john (did:plc:5svqtrhheairglgiiyvutzik) each hold an
sh.tangled.graph.vouch record with rkey alice's DID. The CLI requester's
default requester-tangled-vouch needs the other direction: alice vouches bob
and john.
ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl get bidders.socialweb.computer bidder -o jsonpath='{.spec.policy}{\"\n\"}'"
ssh hat-9 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H9_K kubectl get bidders.socialweb.computer bidder -o jsonpath='{.spec.policy}{\"\n\"}'"
sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$LOCAL_K \
kubectl get bidders.socialweb.computer bidder -o jsonpath='{.spec.policy}{"\n"}'
for OP in did:plc:ocnuqjlzgohypj3zoznermms did:plc:5svqtrhheairglgiiyvutzik; do
goat get "at://$OP/sh.tangled.graph.vouch/did:plc:lpfuqerea3deuoyrn7ojser4" | jq -r .kind
done
Each Bidder prints bidder-tangled-vouch, and each vouch prints vouch.
Each localRepo bidder is its operator's agent by the ident.link pair + grant
(the bidder writes agent-of with its local PDS plc-rotation.key; the
operator's records are declared in deploy/testaccounts/{john,bob}/identity.yaml
or written by scripts/associate-bidder.ts --agent-did <bidder DID>):
pair() { # <bidder DID> <operator DID>
curl -s "https://plc.directory/$1" | jq -c .alsoKnownAs
P=$(curl -s "https://plc.directory/$2" | jq -r '.service[] | select(.id=="#atproto_pds") | .serviceEndpoint')
for c in link.ident.endorsement com.publicdomainrelay.temp.market.association; do
curl -s "$P/xrpc/com.atproto.repo.getRecord?repo=$2&collection=$c&rkey=$1" | jq -c '.value // .error'
done
socialweb-computer internal association "$1" | jq -c '{roots, errors}'
}
pair did:plc:7q5lnitxv4meizdon5fmhzrq did:plc:5svqtrhheairglgiiyvutzik # hat-7 -> john
pair did:plc:4vcig7s43z3nyqrnepmcfzqt did:plc:ocnuqjlzgohypj3zoznermms # hat-9 -> bob
pair did:plc:2l2wdry77c7jtucbslstagkj did:plc:ocnuqjlzgohypj3zoznermms # hat-8 swc-bidder -> bob
Each alsoKnownAs holds https://ident.link/agent-of/<operator DID> and no
at://<operator handle>, each endorsement is {agent: <bidder DID>}, each
grant {agent, createdAt} with no expiresAt, and each walk's roots are
[operator DID].
Service auth for the gateway
The gateway XRPC calls are authenticated as the Alice operator account.
if ! goat account check-auth >/dev/null 2>&1; then
set -a
. ~/.aliceoa_bsky_social_source
set +a
GOAT_USERNAME=$ALICE_ATPROTO_HANDLE GOAT_PASSWORD=$ALICE_ATPROTO_PASSWORD goat account login
fi
tok() { goat account service-auth --aud "$AUD" --lxm "$N.$1"; }
xrpc() {
verb=$1
method=$2
shift 2
goat xrpc "$verb" "$GW" "$N.$method" "$@" "Authorization:Bearer $(tok "$method")"
}
1. Pause hat-9 and local, leave hat-7 receiving
Pause a bidder with the root's spec.suspend. The Bidder process keeps
running with spec.suspendParticipation=true: it withdraws its offering,
declines new RFPs, still fulfils an accept of a bid it published before the
pause, and still reads the market inbox, so it still releases the VMs it runs
(acceptance row A2). The root reports Suspended; the
GuestControlPlane, Pds, IngressProxy, Services and Exposures stay Ready.
ssh hat-9 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H9_K kubectl patch marketbidders.socialweb.computer app --type merge -p '{\"spec\":{\"suspend\":true}}'"
sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$LOCAL_K \
kubectl patch marketbidders.socialweb.computer app --type merge -p '{"spec":{"suspend":true}}'
Check both roots and the Bidder spec:
ssh hat-9 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H9_K kubectl get marketbidders.socialweb.computer app -o jsonpath='{.status.phase}{\"\n\"}{.status.componentSummary}{\"\n\"}'"
sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$LOCAL_K \
kubectl get marketbidders.socialweb.computer app -o jsonpath='{.status.phase}{"\n"}{.status.componentSummary}{"\n"}'
sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$LOCAL_K \
kubectl get bidders.socialweb.computer bidder -o jsonpath='{.spec.suspendParticipation} {.status.phase}{"\n"}'
Both roots must be Suspended; the Bidder reports true Ready.
A component override ({"spec":{"components":{"Bidder":{"suspend":true}}}})
pauses only the Bidder's participation: it sets the Bidder's
spec.suspendParticipation=true, not spec.suspend (suspend stops
participation, never obligations). The Bidder process keeps running: it
withdraws its offering, stops its heartbeat, declines new RFPs and accepts, and
still releases the VMs it runs on vm.delete, over XRPC or the firehose. The
root stays Ready with componentSummary
{"desired":5,"notReady":0,"ready":4,"suspended":1} on a localRepo bidder
(Bidder, GuestControlPlane, Pds, PolicyEngine and one more since rc81) and the Bidder reports
true Ready. A Requester override works the same way. Other component kinds
keep spec.suspend, which stops the component. Section 7 uses it.
2. Resume hat-7 and point it at the gateway PDS firehose
Read the gateway PDS ingress URL from its Exposure. Then apply it on hat-7 as a
RelayEventSource and name it in the root's spec.comms.eventSourceRefs. It
receives market.rfp and market.accept from this stream. The offering lists
the bidder's endpoints as endpoints: [{kind, address}] with kind iroh,
webrtc or https. A prod-qr bidder lists https only with the bidder option
market-https=true (config.json "bidder-options": "--market-https=true").
hat-7 and hat-9 list iroh and webrtc (no prod bidder lists https since 2026-11-10). The requester
reads the winner's offering to reach it.
PDS_URL=$(ssh socialweb.computer "sudo -n -u socialweb env HOME=/var/lib/socialweb-computer/socialweb kubectl --kubeconfig $PROD_K get exposures.exposure.socialweb.computer svc-pds-pds-ingress -o jsonpath='{.status.address}'")
ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl apply -f -" <<YAML
apiVersion: events.socialweb.computer/v1alpha1
kind: RelayEventSource
metadata: {name: gateway-pds}
spec: {url: "$PDS_URL"}
---
apiVersion: events.socialweb.computer/v1alpha1
kind: EventSource
metadata: {name: gateway-pds}
spec: {backendRef: {group: events.socialweb.computer, kind: RelayEventSource, name: gateway-pds}}
YAML
ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl patch marketbidders.socialweb.computer app --type merge -p '{\"spec\":{\"suspend\":false,\"comms\":{\"eventSourceRefs\":[{\"kind\":\"EventSource\",\"name\":\"gateway-pds\"}]}}}'"
Both objects and spec.comms.eventSourceRefs survive a bidder restart: the run
command applies the root without that field, so its server-side apply keeps the
operator's value (D03).
The root must become Ready and one EventStream must be running:
ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl get marketbidders.socialweb.computer app -o jsonpath='{.status.phase}{\"\n\"}{.status.message}{\"\n\"}'"
ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl get eventstreams.events.socialweb.computer,eventsources.events.socialweb.computer"
If the MarketIdentity is AwaitingScan, run the headless sign-in with Bob's
credential file. Read the current QR url immediately before the command; it
expires after 5 minutes.
U=$(ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl get marketidentities.socialweb.computer bidder -o jsonpath='{.status.pendingAction.url}'")
deno run -A --no-lock scripts/oauth-signin.ts --creds ~/.bobvmbuilder_bsky_social_source \
--account BOB --url "$U" --done "Session transferred"
The page ends on "Session transferred" (a refused transfer on "Transfer failed": the sign-in then fails). On staging the
bidder is one of the staging network (install bidder --market-network-file), so the link is at the network's own QR
origin (status.network.qrOrigin, tests/compositions/staging/qr.sh), never qr.fedfork.com. The stored scope must be the authoritative bidder set:
ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl get marketidentities.socialweb.computer bidder -o jsonpath='{.status.phase}{\"\n\"}'"
3. Create the tenant cluster
xrpc procedure createCluster name="$CL"
for i in $(seq 1 60); do
phase=$(xrpc query listClusters | jq -r --arg n "$CL" '.clusters[] | select(.name==$n) | .phase')
[ "$phase" = Ready ] && break
sleep 5
done
[ "$phase" = Ready ] || { echo "cluster $CL is $phase"; exit 1; }
xrpc procedure getKubeconfig name="$CL" ttlSeconds:=28800 | jq -r .kubeconfig > "$W/kubeconfig"
chmod 600 "$W/kubeconfig"
KC=$W/kubeconfig
# `get --raw` takes an absolute path on the host: prefix the kubeconfig's /clusters/... server path, or it asks
# kcp.socialweb.computer/readyz itself (404 since the client certificate front, kcp e496cbff)
P=$(kubectl --kubeconfig "$KC" config view --minify -o jsonpath='{.clusters[0].cluster.server}' | sed 's#https://[^/]*##')
kubectl --kubeconfig "$KC" get --raw "$P/readyz" # ok
3a. The page, a no-expiry kubeconfig, revoke
Last pass: 2026-11-12 20:13-20:17Z, kcp ac51b91e (rollout 18), alice, headless Chrome (scripts/oauth-signin.ts --start-api --cookie-file, then the page's own /api/* calls in the page context, puppeteer-core): consent page
k8s.socialweb.computer for @aliceoa.bsky.social; cluster r18c1 Ready 36 s after create; a 1 h and a no-expiry
kubeconfig (notAfter 9999-12-31, noExpiry: true) both list namespaces (5) and get --raw "$P/readyz" (the
server path prefixed, section 3) answers ok; a bare /readyz asks the host root: NotFound); kubeconfigs lists both; revoking the no-expiry one: kubectl
refused at once (You must be logged in to the server), listed revoked: true, the 1 h one still works; deleting the
cluster: the 1 h one refused within 1 s, the cluster list empty 16 s later. Do not press Sign out on the
page as alice: it revokes every kubeconfig ($KC too) and ends the gateway's alice session (DELETE /session did
that on this pass; alice signed in again at once with oauth-signin.ts --start-api).

The page signs in (its own #handle form), mints with ttlSeconds 0, lists and revokes:
curl -fsS "$GW/" | grep -q '<title>socialweb.computer k8s</title>'
curl -fsS "$GW/api/config" | jq -e .noExpiry
deno run -A --no-lock scripts/oauth-signin.ts --creds ~/.aliceoa_bsky_social_source --account ALICE \
--url "$GW/" --done "Signed in as" --cookie-file "$W/jar.json"
api() { # a cookie POST without Origin: $GW is refused (401, gateway WARN web_origin_refused)
curl -fsS -H 'Content-Type: application/json' -H "Origin: $GW" -d "$2" "$GW/api/clusters/$1" \
-H "Cookie: swc_session=$(jq -r '.[] | select(.name=="swc_session") | .value' "$W/jar.json")"
}
api kubeconfig "{\"name\":\"$CL\",\"ttlSeconds\":0}" | jq -r .kubeconfig > "$W/noexpiry.kubeconfig"
chmod 600 "$W/noexpiry.kubeconfig"
kubectl --kubeconfig "$W/noexpiry.kubeconfig" get ns
S=$(api kubeconfigs "{\"name\":\"$CL\"}" | jq -r '[.kubeconfigs[] | select(.noExpiry and (.revoked|not))] | sort_by(.createdAt) | last | .serial')
api kubeconfigs/revoke "{\"name\":\"$CL\",\"serial\":\"$S\"}" | jq -e .revoked
sleep 2
! kubectl --kubeconfig "$W/noexpiry.kubeconfig" get ns
kubectl --kubeconfig "$KC" get ns
The same over xrpc:
xrpc procedure getKubeconfig name="$CL" ttlSeconds:=0 | jq -r .kubeconfig > "$W/noexpiry-xrpc.kubeconfig"
chmod 600 "$W/noexpiry-xrpc.kubeconfig"
kubectl --kubeconfig "$W/noexpiry-xrpc.kubeconfig" get ns
S=$(xrpc query listKubeconfigs name=="$CL" | jq -r '[.kubeconfigs[] | select(.noExpiry and (.revoked|not))] | sort_by(.createdAt) | last | .serial')
xrpc procedure revokeKubeconfig name="$CL" serial="$S" | jq -e .revoked
sleep 2
! kubectl --kubeconfig "$W/noexpiry-xrpc.kubeconfig" get ns
Pass: the page answers, noExpiry is true, each no-expiry kubeconfig lists namespaces, its serial lists
with noExpiry, and after its revoke kubectl is refused (Unauthorized) while $KC still works.
4. First scale: one oauthQr VM on hat-7, then zero
With hat-9 and local paused, only hat-7 can bid. The deployment uses required
pod anti-affinity on kubernetes.io/hostname, so each replica needs a different
node, and each node is one market VM.
kubectl --kubeconfig "$KC" apply -f - <<'YAML'
apiVersion: apps/v1
kind: Deployment
metadata:
name: scale-check
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: scale-check
template:
metadata:
labels:
app: scale-check
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: scale-check
topologyKey: kubernetes.io/hostname
containers:
- name: run
image: docker.io/library/busybox:1.36.1
command: ["sh", "-c", "sleep 3600"]
YAML
kubectl --kubeconfig "$KC" wait --for=condition=Ready pod -l app=scale-check --timeout=15m
kubectl --kubeconfig "$KC" get pods -o wide
Read the winner and the VM:
NC="https://127.0.0.1:6443/clusters/root:$PROD_ACCOUNT:$CL"
ssh socialweb.computer "sudo -n -u socialweb env HOME=/var/lib/socialweb-computer/socialweb kubectl --kubeconfig $PROD_K --context root --server $NC get nodeclaims.socialweb.computer -o wide"
get nodeclaims -o wide prints only the name and age; the winner is
status.contract.winner.did, the phase status.phase (Bidding,
Provisioning, Joining, Running, Released) and the node's workload
grant status.grant. Read the three bidder DIDs once:
D7=$(ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl get marketbidders.socialweb.computer app -o jsonpath='{.status.did}'")
D9=$(ssh hat-9 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H9_K kubectl get marketbidders.socialweb.computer app -o jsonpath='{.status.did}'")
DL=$(sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$LOCAL_K kubectl get marketbidders.socialweb.computer app -o jsonpath='{.status.did}')
printf 'hat-7=%s\nhat-9=%s\nlocal=%s\n' "$D7" "$D9" "$DL"
Only hat-7 is unpaused, so the first winner must be $D7:
WINNER=$(ssh socialweb.computer "sudo -n -u socialweb env HOME=/var/lib/socialweb-computer/socialweb kubectl --kubeconfig $PROD_K --context root --server $NC get nodeclaims.socialweb.computer -o jsonpath='{.items[0].status.contract.winner.did}'")
[ "$WINNER" = "$D7" ] || { echo "first VM landed on $WINNER, not hat-7 ($D7)"; exit 1; }
kubectl --kubeconfig "$KC" scale deployment scale-check --replicas=0
for i in $(seq 1 120); do
nodes=$(kubectl --kubeconfig "$KC" get nodes --no-headers 2>/dev/null | wc -l)
[ "$nodes" = 0 ] && break
sleep 5
done
[ "$nodes" = 0 ] || { echo "cluster still has $nodes nodes"; exit 1; }
The provider holds an idle node for two minutes before it requests release, so this loop can take several minutes after the last pod goes away. That hold is normal, not a hung scale-down.
That proves the oauthQr bidder on hat-7 alone takes the cluster from zero to one
node and back to zero. The NodeClaim list is empty or every claim is Released.
The release is not complete until the bidder has destroyed the VM. Check that
hat-7 holds no Running ComputeVM for this cluster:
ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl get computevms.socialweb.computer -o wide"
5. One VM per host: resume hat-9 and local
Resume the two paused bidders, then run the same deployment one VM at a time. After each VM is Ready, pause the bidder that won it; the next scale-up must use the next host.
ssh hat-9 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H9_K kubectl patch marketbidders.socialweb.computer app --type merge -p '{\"spec\":{\"suspend\":false}}'"
sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$LOCAL_K \
kubectl patch marketbidders.socialweb.computer app --type merge -p '{"spec":{"suspend":false}}'
ssh hat-9 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H9_K kubectl wait --for=condition=Ready bidders.socialweb.computer/bidder --timeout=120s"
sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$LOCAL_K \
kubectl wait --for=condition=Ready bidders.socialweb.computer/bidder --timeout=120s
kubectl --kubeconfig "$KC" scale deployment scale-check --replicas=1
kubectl --kubeconfig "$KC" wait --for=condition=Ready pod -l app=scale-check --timeout=15m
WINNER=$(ssh socialweb.computer "sudo -n -u socialweb env HOME=/var/lib/socialweb-computer/socialweb kubectl --kubeconfig $PROD_K --context root --server $NC get nodeclaims.socialweb.computer -o jsonpath='{.items[0].status.contract.winner.did}'")
case "$WINNER" in
"$D7") HOST=hat-7 ;;
"$D9") HOST=hat-9 ;;
"$DL") HOST=local ;;
*) echo "winner $WINNER is not one of the three market bidders"; exit 1 ;;
esac
printf 'first winner=%s host=%s\n' "$WINNER" "$HOST"
Pause that winner:
case "$HOST" in
hat-7)
ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl patch marketbidders.socialweb.computer app --type merge -p '{\"spec\":{\"suspend\":true}}'"
;;
hat-9)
ssh hat-9 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H9_K kubectl patch marketbidders.socialweb.computer app --type merge -p '{\"spec\":{\"suspend\":true}}'"
;;
local)
sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$LOCAL_K \
kubectl patch marketbidders.socialweb.computer app --type merge -p '{"spec":{"suspend":true}}'
;;
esac
Wait until that root reports Suspended before the next scale-up. The pause is
asynchronous: the root reports Suspended only once its Bidder runs the new
generation with suspendParticipation; scaling up earlier lets the paused
bidder still bid on the next RFP, and an accept of a bid it already published
is honoured.
case "$HOST" in
hat-7) ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl wait --for=jsonpath='{.status.phase}'=Suspended marketbidders.socialweb.computer/app --timeout=300s" ;;
hat-9) ssh hat-9 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H9_K kubectl wait --for=jsonpath='{.status.phase}'=Suspended marketbidders.socialweb.computer/app --timeout=300s" ;;
local) sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$LOCAL_K \
kubectl wait --for=jsonpath='{.status.phase}'=Suspended marketbidders.socialweb.computer/app --timeout=300s ;;
esac
Scale to two; wait, read the second winner, and pause it too:
kubectl --kubeconfig "$KC" scale deployment scale-check --replicas=2
kubectl --kubeconfig "$KC" wait --for=condition=Ready pod -l app=scale-check --timeout=15m
ssh socialweb.computer "sudo -n -u socialweb env HOME=/var/lib/socialweb-computer/socialweb kubectl --kubeconfig $PROD_K --context root --server $NC get nodeclaims.socialweb.computer -o wide"
Repeat the host map/pause block for the new winner. Scale to three; the final winner must be the one host still unpaused:
kubectl --kubeconfig "$KC" scale deployment scale-check --replicas=3
kubectl --kubeconfig "$KC" wait --for=condition=Ready pod -l app=scale-check --timeout=15m
kubectl --kubeconfig "$KC" get pods -o wide
ssh socialweb.computer "sudo -n -u socialweb env HOME=/var/lib/socialweb-computer/socialweb kubectl --kubeconfig $PROD_K --context root --server $NC get nodeclaims.socialweb.computer -o wide"
Pass condition: three pods are Ready, the NodeClaims name three distinct
winners, and the three NODE names in get pods -o wide are distinct.
6. Resume paused winners, scale to zero, delete the cluster
A paused winner (root spec.suspend) still reads the market inbox and
releases its VM on the requester's vm.delete (A2). To prove A2, run the
scale to zero below before the resume block and check computevms on every
paused host; otherwise resume first. A winner paused with the component
override releases its VM the same way.
ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl patch marketbidders.socialweb.computer app --type merge -p '{\"spec\":{\"suspend\":false}}'"
ssh hat-9 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H9_K kubectl patch marketbidders.socialweb.computer app --type merge -p '{\"spec\":{\"suspend\":false}}'"
sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$LOCAL_K \
kubectl patch marketbidders.socialweb.computer app --type merge -p '{"spec":{"suspend":false}}'
ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl wait --for=condition=Ready bidders.socialweb.computer/bidder --timeout=120s"
ssh hat-9 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H9_K kubectl wait --for=condition=Ready bidders.socialweb.computer/bidder --timeout=120s"
sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$LOCAL_K \
kubectl wait --for=condition=Ready bidders.socialweb.computer/bidder --timeout=120s
Scale down one replica at a time, 3 -> 2 -> 1 -> 0 (by hand with three bidders; composition `0001:run` scales 1 -> 2 -> 1 -> 0: a staging network has two bidders or more). Each step must remove
exactly one node and release exactly one VM before the next step starts:
```bash
for n in 2 1 0; do
kubectl --kubeconfig "$KC" scale deployment scale-check --replicas=$n
for i in $(seq 1 120); do
nodes=$(kubectl --kubeconfig "$KC" get nodes --no-headers 2>/dev/null | wc -l)
[ "$nodes" = $n ] && break
sleep 5
done
[ "$nodes" = $n ] || { echo "cluster has $nodes nodes, want $n"; exit 1; }
kubectl --kubeconfig "$KC" get nodes -o wide
kubectl --kubeconfig "$KC" get pods -o wide
ssh socialweb.computer "sudo -n -u socialweb env HOME=/var/lib/socialweb-computer/socialweb kubectl --kubeconfig $PROD_K --context root --server $NC get nodeclaims.socialweb.computer -o jsonpath='{range .items[*]}{.metadata.name} {.status.phase} {.status.contract.winner.did}{\"\\n\"}{end}'"
# the orphan check of every bidder host below, then pin the released VM's vm.terminated
done
At each step: the pods still run on distinct nodes, the claim of the node that
left is gone, its bidder host has one firecracker fewer, and the gateway logs
one requester_released and one requester_contract_records_deleted. Each
step takes about 2.5 min: the provider holds an idle node for two minutes.
Check every bidder host (the orphan check of
0008): the ComputeVM and FirecrackerVM of the released
VM are gone, and pgrep -c -x firecracker counts only the VMs still Running.
ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl get computevms.socialweb.computer,firecrackervms.socialweb.computer; pgrep -c -x firecracker"
ssh hat-9 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H9_K kubectl get computevms.socialweb.computer,firecrackervms.socialweb.computer; pgrep -c -x firecracker"
sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$LOCAL_K \
kubectl get computevms.socialweb.computer,firecrackervms.socialweb.computer; pgrep -c -x firecracker
Pin the released VM's vm.terminated with one getRecord (0008 "Pinning the
vm.terminated record", step 2). The gateway journal names the receipt URI
(at://<bidder DID>/com.publicdomainrelay.temp.market.receipt/receipt-<hex>);
the record is deleted_on_request with a vm.delete request ref:
RK=termination-payload-$(printf 'terminated:%s' "$RECEIPT_URI" | sha256sum | cut -c1-32)
goat get "at://<bidder DID>/com.publicdomainrelay.temp.compute.events.vm.terminated/$RK" | jq -c '{reason, request: .request.uri}'
After the last step the claim list is empty and no host runs a firecracker. Delete the deployment:
kubectl --kubeconfig "$KC" delete deployment scale-check
Delete the cluster and the scratch directory:
xrpc procedure deleteCluster name="$CL"
rm -rf "$W"
7. Per-component pause checks (A1-A4)
On the oauthQr bidder, define the heartbeat reader:
BOB=$(ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl get marketidentities.socialweb.computer bidder -o jsonpath='{.status.did}'")
PDS=$(ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl get marketidentities.socialweb.computer bidder -o jsonpath='{.status.pdsUrl}'")
hb() { curl -sS "$PDS/xrpc/com.atproto.repo.getRecord?repo=$BOB&collection=com.publicdomainrelay.temp.market.heartbeat&rkey=self" | jq -r .value.createdAt; }
Pause only Bidder and wait until the Bidder runs the paused generation
(spec.suspendParticipation is true and status.observedGeneration equals
metadata.generation); a heartbeat sent between the patch and that point is
the old generation's last one:
ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl patch marketbidders.socialweb.computer app --type merge -p '{\"spec\":{\"components\":{\"Bidder\":{\"suspend\":true}}}}'"
ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl get bidders.socialweb.computer bidder -o jsonpath='{.spec.suspendParticipation} {.metadata.generation} {.status.observedGeneration}{\"\n\"}'"
Then record the heartbeat, wait 180 s, and compare:
hb
sleep 180
hb
createdAt must be identical. On the oauthQr bidder the root must be Ready with
{"desired":4,"notReady":0,"ready":3,"suspended":1} (it runs no Pds), the Bidder reports
spec.suspendParticipation=true and phase Ready; GuestControlPlane stays
Ready; the IngressProxy stays Ready; the GuestControlPlane ingress URL answers:
ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl get marketbidders.socialweb.computer app -o jsonpath='{.status.phase}{\"\n\"}{.status.componentSummary}{\"\n\"}'"
ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl get guestcontrolplanes.socialweb.computer,ingressproxies.socialweb.computer"
curl -sS "https://$(ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl get exposures.exposure.socialweb.computer svc-guestcontrolplane-guest-kcp-ingress -o jsonpath='{.status.address}'" | sed 's#https://##')/healthz"
Resume, wait for Ready, and prove the heartbeat advances:
ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl patch marketbidders.socialweb.computer app --type merge -p '{\"spec\":{\"components\":{\"Bidder\":null}}}'"
ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl wait --for=condition=Ready bidders.socialweb.computer/bidder --timeout=120s"
hb
8. Typed settings, drift and fail-closed overrides
On any Ready bidder, set a 5 s heartbeat and a 15 s offering refresh:
K=$H7_K
HOST=hat-7
ssh "$HOST" "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$K kubectl patch marketbidders.socialweb.computer app --type merge -p '{\"spec\":{\"components\":{\"Bidder\":{\"suspend\":false,\"settings\":{\"bidder\":{\"heartbeatIntervalSeconds\":5,\"offeringRefreshSeconds\":15}}}}}}'"
ssh "$HOST" "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$K kubectl wait --for=condition=Ready bidders.socialweb.computer/bidder --timeout=120s"
ssh "$HOST" "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$K kubectl get marketbidders.socialweb.computer app -o jsonpath='{.status.componentOverrides}'"
The override entry must show "applied":true and the child generation. Sample
the heartbeat twice, 10 s apart; createdAt advances about every 5 s. Read the
offering record cid twice, 60 s apart: it must not change. A refresh re-reads
the offering, corrects it only when it drifted and checks relay visibility; four
refreshes of a correct offering write nothing to the PDS.
Read the offering endpoints with one getRecord (D7 and D9 from section 4):
for D in "$D7" "$D9"; do
goat get "at://$D/com.publicdomainrelay.temp.market.offering/self" | jq -c '[.endpoints[].kind], has("endpointUrl")'
done
hat-7 and hat-9 each print ["iroh","webrtc"] (any
order); each prints false: no offering carries endpointUrl.
A direct child edit is reverted and reported:
ssh "$HOST" "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$K kubectl patch bidders.socialweb.computer bidder --type merge -p '{\"spec\":{\"suspend\":true}}'"
sleep 5
ssh "$HOST" "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$K kubectl get bidders.socialweb.computer bidder -o jsonpath='{.spec.suspend}{\"\n\"}'"
ssh "$HOST" "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$K kubectl get marketbidders.socialweb.computer app -o jsonpath='{.status.conditions}' | jq -c '.[]|select(.type==\"ComponentDrift\")'"
The child suspend is empty/false within seconds. ComponentDrift=True names
Bidder/bidder only until the next pass sees the child match again; then it is
False NoDrift, so read it right after the edit or in the core log. A typo key fails closed:
ssh "$HOST" "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$K kubectl patch marketbidders.socialweb.computer app --type merge -p '{\"spec\":{\"components\":{\"Bldier\":{\"suspend\":true}}}}'"
ssh "$HOST" "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$K kubectl get marketbidders.socialweb.computer app -o jsonpath='{.status.phase}{\"\n\"}{.status.conditions}' | jq -c '.[]|select(.type==\"ComponentOverridesValid\")'"
The phase is Failed and the condition is False/InvalidKey; no child
generation changed. Remove the typo and the test settings when done:
ssh "$HOST" "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$K kubectl patch marketbidders.socialweb.computer app --type merge -p '{\"spec\":{\"components\":null}}'"
9. ACC34: ssh over WebRTC from the CLI on each bidder
Plan 0034 ACC34-CLI-SSH. Each bidder runs a release with plan 0034: its
socialweb-guest-downloads asset serves socialweb-guest-agent to its guests
at http://192.0.2.1:<port>/v1/downloads, and its guest event queue and
compute.guest.subscribeEvents are up. The requester is the CLI of the same
release, on this host, against the prod market. The guest reaches
stun:stun.cloudflare.com:3478 and nothing else is used for ICE.
hat-7 signs in again after a scope change
An oauthQr session keeps the scope it was granted at sign-in. Plan 0034 added
com.publicdomainrelay.temp.compute.events.guest.publicKey and
com.publicdomainrelay.temp.compute.events.guest.signalWebRTC to the bidder
and requester role scopes (internal/atproto/oauth/scope.go). hat-7's identity
stays Ready on its old session after the upgrade, and nothing asks for a new
QR on its own; its PDS then refuses the bidder's writes of those records, so
the guest's publicKey and answer never reach the requester and ACC34 times out
waiting for the guest's publicKey on hat-7. After any release that changes a
role scope, before this section:
- Check that the qr.fedfork.com client metadata scope covers every bidder and requester scope token of the release.
- End hat-7's old session so the identity reports
AwaitingScanorSessionExpiredwith apendingAction.url. - Sign bob in again with the commands of section 2 (read the current QR url
right before
scripts/oauth-signin.ts; it expires after 5 minutes), and wait forReady.
On the 2026-11-10 pass these three steps run on the hat-8 prod-qr bidder, not on hat-7: hat-7's session is not ended (Hosts for this pass).
hat-9 and local are localRepo bidders; their records go to their own PDS and need no sign-in.
Per bidder
Run the block once per host, with the other two bidders paused as in section 1
so that only $HOST bids. Wait until both paused roots report Suspended
before the request.
SWC=socialweb-computer
R=~/.tmp/acc34
mkdir -p "$R"
chmod 700 "$R"
HOST=hat-7
NAME=acc34-$HOST-$(date +%s)
$SWC run request --state-dir "$R" --name "$NAME" --hold \
--tunnel ssh:22:webrtc --transport webrtc --atproto-oauth-qr \
--exec -- sh -c 'uname -a; id' 2>&1 | tee "$R/$NAME-request.log"
$SWC run request ssh "$NAME" --state-dir "$R" --transport webrtc \
--exec -- sh -c 'uname -a; id' 2>&1 | tee "$R/$NAME-ssh.log"
The requester is alice, signed in by QR: the CLI requester's identity on the prod market is
--atproto-oauth-qr (README, Rent a VM from the CLI). The default localRepo makes a new DID
per invocation that waits in AwaitingApproval for alice's endorsement + grant, and with
--skip-qr it has no operator, so no prod bidder is vouched. While run request waits, it
prints identity requester pendingAction scanQr url <url> expiresAt <RFC 3339> on stderr, and
again for each new URL. From a second shell, sign alice in with the newest one:
URL=$(command grep -oE 'pendingAction (scanQr|signInAgain) url [^ ]*' "$R/$NAME-request.log" | tail -1 | cut -d' ' -f4)
deno run -A --no-lock scripts/oauth-signin.ts --creds ~/.aliceoa_bsky_social_source --account ALICE --url "$URL"
The wait has no deadline while the identity waits on the scan; run request goes on once it
prints signed in as.
Pass conditions for each host:
run requestreachesHeldand the command's output showsLinuxanduid=; ssh ran withStrictHostKeyChecking=yesagainst the host keys the guest reported (a host key error fails the run).- The second
run request ssh(the invocation's kcp started again from its state dir) also printsuid=. - Record the Held to shell time: the
secondsof thessh returnedline with"transport":"webrtc". On the local bidder in a market network it was 2.16 s on a fresh invocation and about 7 s after the invocation's kcp restarted (2026-10-24). - The winner is
$HOST: while the contract is held,$HOSTlists a Running ComputeVM for the requester DID (run request list --jsoncarries no winner). - No log line holds signaling plaintext or a private key:
command grep -c -E 'a=candidate|AGE-SECRET-KEY' "$R/$NAME-request.log" "$R/$NAME-ssh.log"
Both counts are 0.
Release, then check the host holds no ComputeVM for it:
$SWC run request release "$NAME" --state-dir "$R"
run request release ends Released. The requester log has
requester_contract_records_deleted: its accept, rfp, compute.vm and
com.fedproxy.rbac records are gone (with an R2, also the grant and endorsement
at rkey = R2 DID). A
getRecord of each at its key in alice's repo answers RecordNotFound; read it 3
times. A second run request release of a Released invocation only removes it.
Read computevms.socialweb.computer -o wide on $HOST as in section 6; no ComputeVM of this contract is left. Resume
the two paused bidders as in section 6 and repeat for the next host. ACC34
passes when every bidder host passes (2026-11-10: local, hat-9, hat-7 and the oauthQr bidder).
When the requester waits for the guest's publicKey or answer and times out, read
the winning bidder's core journal for "the requester refused the market event"
and "submitting the market event to the requester failed" before retrying, and
on hat-7 recheck the scope note above. A bidder HTTP 400 on market.event
means an unsigned event: the requester is an old build. Guest events are signed
and stored at computed keys.