~ knotless Search/
↑↓ move↵ open⇧↵ all results

johnandersen777.bsky.social/socialweb-computer-kcp
Switch repository
Loading…

0
0
main
Switch branches or tags
651 lines · 33.9 KB Raw History

Acceptance 0001: account modes, per-component pause, and one VM per host

This is the runnable acceptance used for plans 0023 and 0024. It exercises:

  • a localRepo bidder;
  • an oauthQr bidder that receives the market over the prod PDS firehose;
  • a localRepo gateway requester;
  • per-component suspend without stopping the GuestControlPlane or the shared IngressProxy;
  • typed Bidder settings and fail-closed overrides;
  • a tenant k3s cluster that scales from zero to three market VMs and back to zero, placing one VM on each bidder host;
  • ssh over WebRTC from the CLI requester to a guest on each bidder host (section 9, plan 0034 ACC34-CLI-SSH).

Run it from a host with goat, kubectl, ssh to socialweb.computer, hat-7 and hat-9, and the credential files named below. It changes the prod gateway and the bidders. Do not run it without that authority.

This runbook was validated end to end on v0.0.0-rc81: three pods on three distinct tenant nodes, one market VM on each bidder host, paused winners forcing the next placement, and every VM released when the cluster returned to zero. It passed again, all nine sections, on v0.0.0-rc142 (2026-10-31) with every bidder on rc142, and on v0.0.0-rc154 (2026-11-01, gateway, ssh gateway and every bidder on rc154): first VM on hat-7 in 43 s, one VM per host (hat-7, local, hat-9), back to zero, heartbeat freeze and advance, offering unchanged, drift reverted, typo fails closed InvalidKey, ACC34 exit 0 on every host (fresh / restarted kcp: local 2.2 / 3.2 s, hat-9 3.1 / 2.2 s, hat-7 4.9 / 4.5 s), no plaintext signaling in any log. Plan 0026 (abstract kinds, the bus with default-deny ComponentLinks) changed the object names and fields it reads; the commands below use the plan 0026 names. Every kind is read by its full resource name (for example marketbidders.socialweb.computer), because a market kcp binds several exports that share short names.

The authoritative OAuth role sets are in internal/atproto/oauth/scope.go, ported from typescript-helpers/lib/oauth-scope/mod.ts (the generator behind did-key-associator/generated/oauth-scope.js). Do not substitute transition:generic.

Variables

GW=https://k8s.socialweb.computer
AUD=did:web:k8s.socialweb.computer
N=com.publicdomainrelay.temp.cluster
CL=scale-auth-$(date +%s)
W=~/.tmp/auth-modes-0001
mkdir -p "$W"
chmod 700 "$W"

PROD_K=/var/lib/socialweb-computer/socialweb/kcp/admin.kubeconfig
PROD_ACCOUNT=2wjeqtgx
H7_K=$(ssh hat-7 'sudo -n -u swc-bidder sh -c "ls /var/lib/socialweb-computer/swc-bidder/handles/*/kcp/admin.kubeconfig | head -1"')
H9_K=$(ssh hat-9 'sudo -n -u swc-bidder sh -c "ls /var/lib/socialweb-computer/swc-bidder/handles/*/kcp/admin.kubeconfig | head -1"')
LOCAL_K=$(sudo -n -u swc-bidder sh -c "ls /var/lib/socialweb-computer/swc-bidder/handles/*/kcp/admin.kubeconfig | head -1")

The gateway account workspace name (PROD_ACCOUNT) appears in the cluster provider log line home":"root:<account>:<cluster>". It is stable for a state dir.

Requester association

The gateway requester must be associated before it accepts any bid. Check it before creating a cluster; a gateway restart can leave the operator claim missing.

The requester MarketIdentity lives in root:app; the gateway's components (Pds/pds, Requester/requester, GatewayAPI/gateway-api) live in root. The identity controller associates the requester from the operator's verified claim record: scripts/associate-bidder.ts writes it from the Alice account and the requester's association watch reads it from the bsky JetStream. The identity controller reports the result in status.associations and the Associated condition. SOCIALWEB_GATEWAY_APPROVED_ASSOCIATIONS (comma-separated DIDs in the gateway env) pre-approves operators in spec.approvedAssociations; it is not needed when the claim is verified. Do not patch the MarketIdentity spec: the Gateway root owns it and server-side apply reverts the edit.

ID_ARGS="--kubeconfig $PROD_K --server https://127.0.0.1:6443/clusters/root:app"
MI=marketidentities.socialweb.computer
ASSOC=$(ssh socialweb.computer "sudo -n -u socialweb env HOME=/var/lib/socialweb-computer/socialweb kubectl $ID_ARGS get $MI requester -o jsonpath='{.status.conditions[?(@.type==\"Associated\")].status}'")
RI_DID=$(ssh socialweb.computer "sudo -n -u socialweb env HOME=/var/lib/socialweb-computer/socialweb kubectl $ID_ARGS get $MI requester -o jsonpath='{.status.did}'")
printf 'requester=%s associated=%s\n' "$RI_DID" "$ASSOC"
if [ "$ASSOC" != True ]; then
  OP_DID=$(deno run -A --no-lock scripts/associate-bidder.ts --creds ~/.aliceoa_bsky_social_source \
    --account ALICE --requester-did "$RI_DID" | jq -r .operatorDid)
  printf 'operator=%s\n' "$OP_DID"
fi
ssh socialweb.computer "sudo -n -u socialweb env HOME=/var/lib/socialweb-computer/socialweb kubectl $ID_ARGS get $MI requester -o jsonpath='{.spec.approvedAssociations}{\"\n\"}{.status.associations}{\"\n\"}{.status.phase}{\"\n\"}'"

Wait until the identity phase is Ready, status.associations names the operator DID and the Requester child reports spec.operatorDID with it. Without that, every bid is denied by requester-tangled-vouch even though bidder policy runs succeed.

Bidder policy: bidder-tangled-vouch on every prod bidder

Every prod bidder runs bidder-tangled-vouch. It is the MarketBidder default when spec.policy is unset, and install bidder writes it to config.json as bidder-policy. open runs only when an operator sets it explicitly (--bidder-policy open or spec.policy: open); it is never acceptable on a prod bidder. The bidder bids only for a requester in its operator's vouch graph, so the acceptance requester alice (did:plc:lpfuqerea3deuoyrn7ojser4) must be vouched by the bidder operator: bob (did:plc:ocnuqjlzgohypj3zoznermms) and john (did:plc:5svqtrhheairglgiiyvutzik) each hold an sh.tangled.graph.vouch record with rkey alice's DID. The CLI requester's default requester-tangled-vouch needs the other direction: alice vouches bob and john.

ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl get bidders.socialweb.computer bidder -o jsonpath='{.spec.policy}{\"\n\"}'"
ssh hat-9 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H9_K kubectl get bidders.socialweb.computer bidder -o jsonpath='{.spec.policy}{\"\n\"}'"
sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$LOCAL_K \
  kubectl get bidders.socialweb.computer bidder -o jsonpath='{.spec.policy}{"\n"}'
for OP in did:plc:ocnuqjlzgohypj3zoznermms did:plc:5svqtrhheairglgiiyvutzik; do
  goat get "at://$OP/sh.tangled.graph.vouch/did:plc:lpfuqerea3deuoyrn7ojser4" | jq -r .kind
done

Each Bidder prints bidder-tangled-vouch, and each vouch prints vouch.

Service auth for the gateway

The gateway XRPC calls are authenticated as the Alice operator account.

if ! goat account check-auth >/dev/null 2>&1; then
  set -a
  . ~/.aliceoa_bsky_social_source
  set +a
  GOAT_USERNAME=$ALICE_ATPROTO_HANDLE GOAT_PASSWORD=$ALICE_ATPROTO_PASSWORD goat account login
fi
tok() { goat account service-auth --aud "$AUD" --lxm "$N.$1"; }
xrpc() {
  verb=$1
  method=$2
  shift 2
  goat xrpc "$verb" "$GW" "$N.$method" "$@" "Authorization:Bearer $(tok "$method")"
}

1. Pause hat-9 and local, leave hat-7 receiving

Pause a bidder with the root's spec.suspend. The Bidder process keeps running with spec.suspendParticipation=true: it withdraws its offering, declines new RFPs, still fulfils an accept of a bid it published before the pause, and still reads the market inbox, so it still releases the VMs it runs (acceptance row A2). The root reports Suspended; the GuestControlPlane, Pds, IngressProxy, Services and Exposures stay Ready.

ssh hat-9 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H9_K kubectl patch marketbidders.socialweb.computer app --type merge -p '{\"spec\":{\"suspend\":true}}'"
sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$LOCAL_K \
  kubectl patch marketbidders.socialweb.computer app --type merge -p '{"spec":{"suspend":true}}'

Check both roots and the Bidder spec:

ssh hat-9 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H9_K kubectl get marketbidders.socialweb.computer app -o jsonpath='{.status.phase}{\"\n\"}{.status.componentSummary}{\"\n\"}'"
sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$LOCAL_K \
  kubectl get marketbidders.socialweb.computer app -o jsonpath='{.status.phase}{"\n"}{.status.componentSummary}{"\n"}'
sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$LOCAL_K \
  kubectl get bidders.socialweb.computer bidder -o jsonpath='{.spec.suspendParticipation} {.status.phase}{"\n"}'

Both roots must be Suspended; the Bidder reports true Ready.

A component override ({"spec":{"components":{"Bidder":{"suspend":true}}}}) pauses only the Bidder's participation: it sets the Bidder's spec.suspendParticipation=true, not spec.suspend (suspend stops participation, never obligations). The Bidder process keeps running: it withdraws its offering, stops its heartbeat, declines new RFPs and accepts, and still releases the VMs it runs on vm.delete, over XRPC or the firehose. The root stays Ready with componentSummary {"desired":5,"notReady":0,"ready":4,"suspended":1} on a localRepo bidder (Bidder, GuestControlPlane, Pds, PolicyEngine and one more since rc81) and the Bidder reports true Ready. A Requester override works the same way. Other component kinds keep spec.suspend, which stops the component. Section 7 uses it.

2. Resume hat-7 and point it at the gateway PDS firehose

Read the gateway PDS ingress URL from its Exposure. Then apply it on hat-7 as a RelayEventSource and name it in the root's spec.comms.eventSourceRefs. The oauthQr Bidder advertises no inbound market endpoint; it receives market.rfp and market.accept from this stream.

PDS_URL=$(ssh socialweb.computer "sudo -n -u socialweb env HOME=/var/lib/socialweb-computer/socialweb kubectl --kubeconfig $PROD_K get exposures.exposure.socialweb.computer svc-pds-pds-ingress -o jsonpath='{.status.address}'")
ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl apply -f -" <<YAML
apiVersion: events.socialweb.computer/v1alpha1
kind: RelayEventSource
metadata: {name: gateway-pds}
spec: {url: "$PDS_URL"}
---
apiVersion: events.socialweb.computer/v1alpha1
kind: EventSource
metadata: {name: gateway-pds}
spec: {backendRef: {group: events.socialweb.computer, kind: RelayEventSource, name: gateway-pds}}
YAML
ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl patch marketbidders.socialweb.computer app --type merge -p '{\"spec\":{\"suspend\":false,\"comms\":{\"eventSourceRefs\":[{\"kind\":\"EventSource\",\"name\":\"gateway-pds\"}]}}}'"

Both objects and spec.comms.eventSourceRefs survive a bidder restart: the run command applies the root without that field, so its server-side apply keeps the operator's value (D03).

The root must become Ready and one EventStream must be running:

ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl get marketbidders.socialweb.computer app -o jsonpath='{.status.phase}{\"\n\"}{.status.message}{\"\n\"}'"
ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl get eventstreams.events.socialweb.computer,eventsources.events.socialweb.computer"

If the MarketIdentity is AwaitingScan, run the headless sign-in with Bob's credential file. Read the current QR url immediately before the command; it expires after 5 minutes.

U=$(ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl get marketidentities.socialweb.computer bidder -o jsonpath='{.status.pendingAction.url}'")
deno run -A --no-lock scripts/oauth-signin.ts --creds ~/.bobvmbuilder_bsky_social_source \
  --account BOB --url "$U" --done "You can close this window"

A timeout waiting for the final page text is harmless when the identity is Ready afterwards. The stored scope must be the authoritative bidder set:

ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl get marketidentities.socialweb.computer bidder -o jsonpath='{.status.phase}{\"\n\"}'"

3. Create the tenant cluster

xrpc procedure createCluster name="$CL"
for i in $(seq 1 60); do
  phase=$(xrpc query listClusters | jq -r --arg n "$CL" '.clusters[] | select(.name==$n) | .phase')
  [ "$phase" = Ready ] && break
  sleep 5
done
[ "$phase" = Ready ] || { echo "cluster $CL is $phase"; exit 1; }
xrpc procedure getKubeconfig name="$CL" ttlSeconds:=28800 | jq -r .kubeconfig > "$W/kubeconfig"
chmod 600 "$W/kubeconfig"
KC=$W/kubeconfig

4. First scale: one oauthQr VM on hat-7, then zero

With hat-9 and local paused, only hat-7 can bid. The deployment uses required pod anti-affinity on kubernetes.io/hostname, so each replica needs a different node, and each node is one market VM.

kubectl --kubeconfig "$KC" apply -f - <<'YAML'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: scale-check
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: scale-check
  template:
    metadata:
      labels:
        app: scale-check
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app: scale-check
              topologyKey: kubernetes.io/hostname
      containers:
        - name: run
          image: docker.io/library/busybox:1.36.1
          command: ["sh", "-c", "sleep 3600"]
YAML
kubectl --kubeconfig "$KC" wait --for=condition=Ready pod -l app=scale-check --timeout=15m
kubectl --kubeconfig "$KC" get pods -o wide

Read the winner and the VM:

NC="https://127.0.0.1:6443/clusters/root:$PROD_ACCOUNT:$CL"
ssh socialweb.computer "sudo -n -u socialweb env HOME=/var/lib/socialweb-computer/socialweb kubectl --kubeconfig $PROD_K --context root --server $NC get nodeclaims.socialweb.computer -o wide"

get nodeclaims -o wide prints only the name and age; the winner is status.contract.winner.did, the phase status.phase (Bidding, Provisioning, Joining, Running, Released) and the node's workload grant status.grant. Read the three bidder DIDs once:

D7=$(ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl get marketbidders.socialweb.computer app -o jsonpath='{.status.did}'")
D9=$(ssh hat-9 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H9_K kubectl get marketbidders.socialweb.computer app -o jsonpath='{.status.did}'")
DL=$(sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$LOCAL_K kubectl get marketbidders.socialweb.computer app -o jsonpath='{.status.did}')
printf 'hat-7=%s\nhat-9=%s\nlocal=%s\n' "$D7" "$D9" "$DL"

Only hat-7 is unpaused, so the first winner must be $D7:

WINNER=$(ssh socialweb.computer "sudo -n -u socialweb env HOME=/var/lib/socialweb-computer/socialweb kubectl --kubeconfig $PROD_K --context root --server $NC get nodeclaims.socialweb.computer -o jsonpath='{.items[0].status.contract.winner.did}'")
[ "$WINNER" = "$D7" ] || { echo "first VM landed on $WINNER, not hat-7 ($D7)"; exit 1; }
kubectl --kubeconfig "$KC" scale deployment scale-check --replicas=0
for i in $(seq 1 120); do
  nodes=$(kubectl --kubeconfig "$KC" get nodes --no-headers 2>/dev/null | wc -l)
  [ "$nodes" = 0 ] && break
  sleep 5
done
[ "$nodes" = 0 ] || { echo "cluster still has $nodes nodes"; exit 1; }

The provider holds an idle node for two minutes before it requests release, so this loop can take several minutes after the last pod goes away. That hold is normal, not a hung scale-down.

That proves the oauthQr bidder on hat-7 alone takes the cluster from zero to one node and back to zero. The NodeClaim list is empty or every claim is Released. The release is not complete until the bidder has destroyed the VM. Check that hat-7 holds no Running ComputeVM for this cluster:

ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl get computevms.socialweb.computer -o wide"

5. One VM per host: resume hat-9 and local

Resume the two paused bidders, then run the same deployment one VM at a time. After each VM is Ready, pause the bidder that won it; the next scale-up must use the next host.

ssh hat-9 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H9_K kubectl patch marketbidders.socialweb.computer app --type merge -p '{\"spec\":{\"suspend\":false}}'"
sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$LOCAL_K \
  kubectl patch marketbidders.socialweb.computer app --type merge -p '{"spec":{"suspend":false}}'
ssh hat-9 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H9_K kubectl wait --for=condition=Ready bidders.socialweb.computer/bidder --timeout=120s"
sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$LOCAL_K \
  kubectl wait --for=condition=Ready bidders.socialweb.computer/bidder --timeout=120s

kubectl --kubeconfig "$KC" scale deployment scale-check --replicas=1
kubectl --kubeconfig "$KC" wait --for=condition=Ready pod -l app=scale-check --timeout=15m
WINNER=$(ssh socialweb.computer "sudo -n -u socialweb env HOME=/var/lib/socialweb-computer/socialweb kubectl --kubeconfig $PROD_K --context root --server $NC get nodeclaims.socialweb.computer -o jsonpath='{.items[0].status.contract.winner.did}'")
case "$WINNER" in
  "$D7") HOST=hat-7 ;;
  "$D9") HOST=hat-9 ;;
  "$DL") HOST=local ;;
  *) echo "winner $WINNER is not one of the three market bidders"; exit 1 ;;
esac
printf 'first winner=%s host=%s\n' "$WINNER" "$HOST"

Pause that winner:

case "$HOST" in
  hat-7)
    ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl patch marketbidders.socialweb.computer app --type merge -p '{\"spec\":{\"suspend\":true}}'"
    ;;
  hat-9)
    ssh hat-9 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H9_K kubectl patch marketbidders.socialweb.computer app --type merge -p '{\"spec\":{\"suspend\":true}}'"
    ;;
  local)
    sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$LOCAL_K \
      kubectl patch marketbidders.socialweb.computer app --type merge -p '{"spec":{"suspend":true}}'
    ;;
esac

Wait until that root reports Suspended before the next scale-up. The pause is asynchronous: the root reports Suspended only once its Bidder runs the new generation with suspendParticipation; scaling up earlier lets the paused bidder still bid on the next RFP, and an accept of a bid it already published is honoured.

case "$HOST" in
  hat-7) ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl wait --for=jsonpath='{.status.phase}'=Suspended marketbidders.socialweb.computer/app --timeout=300s" ;;
  hat-9) ssh hat-9 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H9_K kubectl wait --for=jsonpath='{.status.phase}'=Suspended marketbidders.socialweb.computer/app --timeout=300s" ;;
  local) sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$LOCAL_K \
           kubectl wait --for=jsonpath='{.status.phase}'=Suspended marketbidders.socialweb.computer/app --timeout=300s ;;
esac

Scale to two; wait, read the second winner, and pause it too:

kubectl --kubeconfig "$KC" scale deployment scale-check --replicas=2
kubectl --kubeconfig "$KC" wait --for=condition=Ready pod -l app=scale-check --timeout=15m
ssh socialweb.computer "sudo -n -u socialweb env HOME=/var/lib/socialweb-computer/socialweb kubectl --kubeconfig $PROD_K --context root --server $NC get nodeclaims.socialweb.computer -o wide"

Repeat the host map/pause block for the new winner. Scale to three; the final winner must be the one host still unpaused:

kubectl --kubeconfig "$KC" scale deployment scale-check --replicas=3
kubectl --kubeconfig "$KC" wait --for=condition=Ready pod -l app=scale-check --timeout=15m
kubectl --kubeconfig "$KC" get pods -o wide
ssh socialweb.computer "sudo -n -u socialweb env HOME=/var/lib/socialweb-computer/socialweb kubectl --kubeconfig $PROD_K --context root --server $NC get nodeclaims.socialweb.computer -o wide"

Pass condition: three pods are Ready, the NodeClaims name three distinct winners, and the three NODE names in get pods -o wide are distinct.

6. Resume paused winners, scale to zero, delete the cluster

A paused winner (root spec.suspend) still reads the market inbox and releases its VM on the requester's vm.delete (A2). To prove A2, run the scale to zero below before the resume block and check computevms on every paused host; otherwise resume first. A winner paused with the component override releases its VM the same way.

ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl patch marketbidders.socialweb.computer app --type merge -p '{\"spec\":{\"suspend\":false}}'"
ssh hat-9 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H9_K kubectl patch marketbidders.socialweb.computer app --type merge -p '{\"spec\":{\"suspend\":false}}'"
sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$LOCAL_K \
  kubectl patch marketbidders.socialweb.computer app --type merge -p '{"spec":{"suspend":false}}'
ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl wait --for=condition=Ready bidders.socialweb.computer/bidder --timeout=120s"
ssh hat-9 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H9_K kubectl wait --for=condition=Ready bidders.socialweb.computer/bidder --timeout=120s"
sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$LOCAL_K \
  kubectl wait --for=condition=Ready bidders.socialweb.computer/bidder --timeout=120s

kubectl --kubeconfig "$KC" delete deployment scale-check
for i in $(seq 1 120); do
  nodes=$(kubectl --kubeconfig "$KC" get nodes --no-headers 2>/dev/null | wc -l)
  [ "$nodes" = 0 ] && break
  sleep 5
done
[ "$nodes" = 0 ] || { echo "cluster still has $nodes nodes"; exit 1; }
ssh socialweb.computer "sudo -n -u socialweb env HOME=/var/lib/socialweb-computer/socialweb kubectl --kubeconfig $PROD_K --context root --server $NC get nodeclaims.socialweb.computer -o wide"

The claim list is empty (or every claim is Released). Check every bidder host: no Running ComputeVM may remain, and no socialweb-vm-* unit may remain for this cluster.

ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl get computevms.socialweb.computer -o wide"
ssh hat-9 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H9_K kubectl get computevms.socialweb.computer -o wide"
sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$LOCAL_K \
  kubectl get computevms.socialweb.computer -o wide

Delete the cluster and the scratch directory:

xrpc procedure deleteCluster name="$CL"
rm -rf "$W"

7. Per-component pause checks (A1-A4)

On the oauthQr bidder, define the heartbeat reader:

BOB=$(ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl get marketidentities.socialweb.computer bidder -o jsonpath='{.status.did}'")
PDS=$(ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl get marketidentities.socialweb.computer bidder -o jsonpath='{.status.pdsUrl}'")
hb() { curl -sS "$PDS/xrpc/com.atproto.repo.getRecord?repo=$BOB&collection=com.publicdomainrelay.temp.market.heartbeat&rkey=self" | jq -r .value.createdAt; }

Pause only Bidder and wait until the Bidder runs the paused generation (spec.suspendParticipation is true and status.observedGeneration equals metadata.generation); a heartbeat sent between the patch and that point is the old generation's last one:

ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl patch marketbidders.socialweb.computer app --type merge -p '{\"spec\":{\"components\":{\"Bidder\":{\"suspend\":true}}}}'"
ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl get bidders.socialweb.computer bidder -o jsonpath='{.spec.suspendParticipation} {.metadata.generation} {.status.observedGeneration}{\"\n\"}'"

Then record the heartbeat, wait 180 s, and compare:

hb
sleep 180
hb

createdAt must be identical. On hat-7 the root must be Ready with {"desired":3,"notReady":0,"ready":2,"suspended":1}, the Bidder reports spec.suspendParticipation=true and phase Ready; GuestControlPlane stays Ready; the IngressProxy stays Ready; the GuestControlPlane ingress URL answers:

ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl get marketbidders.socialweb.computer app -o jsonpath='{.status.phase}{\"\n\"}{.status.componentSummary}{\"\n\"}'"
ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl get guestcontrolplanes.socialweb.computer,ingressproxies.socialweb.computer"
curl -sS "https://$(ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl get exposures.exposure.socialweb.computer svc-guestcontrolplane-guest-kcp-ingress -o jsonpath='{.status.address}'" | sed 's#https://##')/healthz"

Resume, wait for Ready, and prove the heartbeat advances:

ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl patch marketbidders.socialweb.computer app --type merge -p '{\"spec\":{\"components\":{\"Bidder\":null}}}'"
ssh hat-7 "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$H7_K kubectl wait --for=condition=Ready bidders.socialweb.computer/bidder --timeout=120s"
hb

8. Typed settings, drift and fail-closed overrides

On any Ready bidder, set a 5 s heartbeat and a 15 s offering refresh:

K=$H7_K
HOST=hat-7
ssh "$HOST" "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$K kubectl patch marketbidders.socialweb.computer app --type merge -p '{\"spec\":{\"components\":{\"Bidder\":{\"suspend\":false,\"settings\":{\"bidder\":{\"heartbeatIntervalSeconds\":5,\"offeringRefreshSeconds\":15}}}}}}'"
ssh "$HOST" "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$K kubectl wait --for=condition=Ready bidders.socialweb.computer/bidder --timeout=120s"
ssh "$HOST" "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$K kubectl get marketbidders.socialweb.computer app -o jsonpath='{.status.componentOverrides}'"

The override entry must show "applied":true and the child generation. Sample the heartbeat twice, 10 s apart; createdAt advances about every 5 s. Read the offering record cid twice, 60 s apart: it must not change. A refresh re-reads the offering, corrects it only when it drifted and checks relay visibility; four refreshes of a correct offering write nothing to the PDS.

A direct child edit is reverted and reported:

ssh "$HOST" "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$K kubectl patch bidders.socialweb.computer bidder --type merge -p '{\"spec\":{\"suspend\":true}}'"
sleep 5
ssh "$HOST" "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$K kubectl get bidders.socialweb.computer bidder -o jsonpath='{.spec.suspend}{\"\n\"}'"
ssh "$HOST" "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$K kubectl get marketbidders.socialweb.computer app -o jsonpath='{.status.conditions}' | jq -c '.[]|select(.type==\"ComponentDrift\")'"

The child suspend is empty/false within seconds. ComponentDrift=True names Bidder/bidder only until the next pass sees the child match again; then it is False NoDrift, so read it right after the edit or in the core log. A typo key fails closed:

ssh "$HOST" "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$K kubectl patch marketbidders.socialweb.computer app --type merge -p '{\"spec\":{\"components\":{\"Bldier\":{\"suspend\":true}}}}'"
ssh "$HOST" "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$K kubectl get marketbidders.socialweb.computer app -o jsonpath='{.status.phase}{\"\n\"}{.status.conditions}' | jq -c '.[]|select(.type==\"ComponentOverridesValid\")'"

The phase is Failed and the condition is False/InvalidKey; no child generation changed. Remove the typo and the test settings when done:

ssh "$HOST" "sudo -n -u swc-bidder env HOME=/var/lib/socialweb-computer/swc-bidder KUBECONFIG=$K kubectl patch marketbidders.socialweb.computer app --type merge -p '{\"spec\":{\"components\":null}}'"

9. ACC34: ssh over WebRTC from the CLI on each bidder

Plan 0034 ACC34-CLI-SSH. Each bidder runs a release with plan 0034: its socialweb-guest-downloads asset serves socialweb-guest-agent to its guests at http://192.0.2.1:<port>/v1/downloads, and its guest event queue and compute.guest.subscribeEvents are up. The requester is the CLI of the same release, on this host, against the prod market. The guest reaches stun:stun.cloudflare.com:3478 and nothing else is used for ICE.

hat-7 signs in again after a scope change

An oauthQr session keeps the scope it was granted at sign-in. Plan 0034 added com.publicdomainrelay.temp.compute.events.guest.publicKey and com.publicdomainrelay.temp.compute.events.guest.signalWebRTC to the bidder and requester role scopes (internal/atproto/oauth/scope.go). hat-7's identity stays Ready on its old session after the upgrade, and nothing asks for a new QR on its own; its PDS then refuses the bidder's writes of those records, so the guest's publicKey and answer never reach the requester and ACC34 times out waiting for the guest's publicKey on hat-7. After any release that changes a role scope, before this section:

  1. Check that the qr.fedfork.com client metadata scope covers every bidder and requester scope token of the release.
  2. End hat-7's old session so the identity reports AwaitingScan or SessionExpired with a pendingAction.url.
  3. Sign bob in again with the commands of section 2 (read the current QR url right before scripts/oauth-signin.ts; it expires after 5 minutes), and wait for Ready.

hat-9 and local are localRepo bidders; their records go to their own PDS and need no sign-in.

Per bidder

Run the block once per host, with the other two bidders paused as in section 1 so that only $HOST bids. Wait until both paused roots report Suspended before the request.

SWC=socialweb-computer
R=~/.tmp/acc34
mkdir -p "$R"
chmod 700 "$R"
HOST=hat-7
NAME=acc34-$HOST-$(date +%s)
$SWC run request --state-dir "$R" --name "$NAME" --hold \
  --tunnel ssh:22:webrtc --transport webrtc --exec "uname -a; id" \
  -- --atproto-oauth-qr 2>&1 | tee "$R/$NAME-request.log"
$SWC run request ssh "$NAME" --state-dir "$R" --transport webrtc \
  --exec "uname -a; id" 2>&1 | tee "$R/$NAME-ssh.log"

The requester is alice, signed in by QR: the CLI requester's identity on the prod market is -- --atproto-oauth-qr (README, Rent a VM from the CLI). The default localRepo makes a new DID per invocation that waits in AwaitingApproval for alice's association claim, and with -- --skip-qr it has no operator, so no prod bidder is vouched. While run request waits, it prints identity requester pendingAction scanQr url <url> expiresAt <RFC 3339> on stderr, and again for each new URL. From a second shell, sign alice in with the newest one:

URL=$(command grep -oE 'pendingAction (scanQr|signInAgain) url [^ ]*' "$R/$NAME-request.log" | tail -1 | cut -d' ' -f4)
deno run -A --no-lock scripts/oauth-signin.ts --creds ~/.aliceoa_bsky_social_source --account ALICE --url "$URL"

The wait has no deadline while the identity waits on the scan; run request goes on once it prints signed in as.

Pass conditions for each host:

  • run request reaches Held and the command's output shows Linux and uid=; ssh ran with StrictHostKeyChecking=yes against the host keys the guest reported (a host key error fails the run).
  • The second run request ssh (the invocation's kcp started again from its state dir) also prints uid=.
  • Record the Held to shell time: the seconds of the ssh returned line with "transport":"webrtc". On the local bidder in a market network it was 2.16 s on a fresh invocation and about 7 s after the invocation's kcp restarted (2026-10-24).
  • The winner is $HOST: while the contract is held, $HOST lists a Running ComputeVM for the requester DID (run request list --json carries no winner).
  • No log line holds signaling plaintext or a private key:
command grep -c -E 'a=candidate|AGE-SECRET-KEY' "$R/$NAME-request.log" "$R/$NAME-ssh.log"

Both counts are 0.

Release, then check the host holds no ComputeVM for it:

$SWC run request release "$NAME" --state-dir "$R"

run request release ends Released. Read computevms.socialweb.computer -o wide on $HOST as in section 6; no ComputeVM of this contract is left. Resume the two paused bidders as in section 6 and repeat for the next host. ACC34 passes when all three hosts (local, hat-9, hat-7) pass.

When the requester waits for the guest's publicKey or answer and times out, read the winning bidder's core journal for "the requester refused the market event" and "submitting the market event to the requester failed" before retrying, and on hat-7 recheck the scope note above.