~ knotless Search/
↑↓ move↵ open⇧↵ all results

socialweb.computer/socialweb-computer-kcp

socialweb.computer: a compute market on atproto (kcp)

1 collaborator
0
0
139 lines · 9.6 KB Raw History

Acceptance 0003: CLI requester, ssh over WebRTC, per bidder host (ACC34)

Proves the CLI requester (socialweb-computer run request) rents a VM on each prod bidder and opens ssh to the guest over a WebRTC datachannel, both from a fresh invocation kcp and after that kcp restarts from its state dir.

  • Plan 0034 ACC34-CLI-SSH; the long form with the scope note is 0001 section 9
  • One run per host: only $HOST bids: --allow-bidder-dids <$HOST bidder DID> (or the other two bidders suspended)
  • Last pass: 2026-11-13 01:25-01:35Z, bidders + CLI on rollout 20 (release/rc165 038bd32e, v0.0.0-rc165-pre1, binary 24a8b5307d19) on hat-8, alice by QR (consent page: client qr.fedfork.com for @aliceoa.bsky.social), no bidders list (the other two MarketBidders suspended, resumed and Ready 6/6 after each host). local / hat-9 / hat-7: Held, winner = the unpaused host (Running ComputeVM), ssh over WebRTC fresh 4.0 / 2.6 / 9.5 s, restarted invocation 6.7 / 5.2 / 5.5 s (was ~65 s on rollout 19), uid=0 from both invocations, exit 0, plaintext 0 / 0, run request release -> released, hosts 0 ComputeVM / 0 firecracker, the 10 alice URIs each run logged RecordNotFound x3. core.log per run (3 core starts) waits / timeouts / "serves discovery": 5/0/5, 8/0/8, 6/0/6. Each release left 0 market.event / compute.events.vm.delete / rfp / accept in alice's repo (createdAt since the run start)
  • Previous run (FAIL): 2026-11-12 23:57Z-2026-11-13 00:17Z, bidders + CLI on kcp e190a3b1 (release 66f3bd4d, rollout 19) on hat-8, alice by QR (consent page: client qr.fedfork.com for @aliceoa.bsky.social), no bidders list (the other two MarketBidders suspended, resumed after each host): FAIL on the restarted-kcp ssh only. local / hat-9 / hat-7: identity Ready -> guest on network 49 / 37 / 32 s, Held, ssh over WebRTC fresh 3.9 / 3.3 / 3.1 s, restarted invocation 64.7 / 67.6 / 66.2 s (expected 2-10 s): every core start logs 10 the APIExport provider waits for its virtual workspace to serve discovery then 10 WARN ... starts before its virtual workspace serves discovery after the 1 m timeout (30/30 per run, 3 core starts; 0001 section 9, cdd93727 awaitEndpoints waits for an EndpointURLsReady condition kcp v0.33 never sets). Everything else passes: uid=0 from both invocations, exit 0, StrictHostKeyChecking=yes, winner = the unpaused host (Running ComputeVM), plaintext 0 / 0 on all six logs, run request release Released, hosts 0 ComputeVM / 0 firecracker, the 10 alice URIs each run logged (rfp, accept, 4 market.event, 2 guest.publicKey, 2 guest.signalWebRTC) RecordNotFound x3. Each release left its release market.event and compute.events.vm.delete (the 0006 release-records bug; 6 deleted by hand); the run's badgeBlueKeys signer binding (market_requester, stable rkey per state dir) stays by design
  • Last pass: bidders rollout 6 (2d367fe8; hat-7 iroh + webrtc only, market-https off), CLI 41319a3a on hat-8, 2026-11-10 16:39Z hat-9 and 16:44Z hat-7, alice pdsAccount: Held in 63 / 66 s, ssh over WebRTC exit 0 fresh (5.5 / 2.8 s) and from a restarted invocation (4.8 / 2.3 s), plaintext 0 / 0, Released; every record of alice's the run logged (rfp, accept, 4 market.event incl. the restarted invocation's signal envelope, 2 guest.publicKey, 2 guest.signalWebRTC) answers getRecord 400 after release (22479e17); the bidder's own event records stay in the bidder's repo. Before: kcp 438ca0fa bidders (p2p rollout 5), CLI 69aa8b0f, 15:13-15:17Z (also rollout 2, 13:49-13:56Z)
sequenceDiagram
  participant C as CLI requester (alice, oauthQr)
  participant Q as qr.fedfork.com
  participant B as bidder $HOST
  participant G as guest VM
  C->>C: pendingAction scanQr url (stderr)
  Q-->>C: session transfer (signed in as alice)
  C->>B: RFP -> bid -> accept, over the bidder's offering endpoints (iroh; https only if advertised)
  B->>G: boot, hostKeys, publicKey, onNetwork
  C->>G: WebRTC offer/answer (signalWebRTC events), STUN only
  C->>G: ssh (StrictHostKeyChecking=yes) uname -a; id
  C->>B: release -> vm.terminated

Preconditions

ItemNeed
Hostsssh to hat-7, hat-9; local bidder on this host; kubeconfigs as in 0001 Variables
Biddersall three bidder-tangled-vouch, Ready, same rc as the CLI; each advertises its ingress in market.offering/self endpoints (iroh by default; webrtc once its computer.socialweb.ingress.listen/market record is written, at once for a localRepo bidder, for an OAuth one after a sign-in grants the optional ingress listen/answer scopes; https only with market-https=true, the hat-7 transition): the requester calls the first it reaches, service auth aud <bidder DID>#pdr_temp_market (no endpointUrl, no fedproxy hostname)
Requesteralice signed in by QR (--atproto-oauth-qr), or with an app password (--atproto-identity-mode pdsAccount --atproto-handle aliceoa.bsky.social, the password in ATPROTO_PASSWORD of that one command); default localRepo makes an unassociated DID and gets no bids
Vouchesbob + john vouch alice; alice vouches bob + john
Signera second shell (or the drives-oauth agent) opens the newest pendingAction url and signs alice in; creds file never printed
hat-7oauthQr bidder: after a role-scope change re-sign bob first (0001 section 9)
ICEguest reaches stun:stun.cloudflare.com:3478, nothing else

Steps

Announce first: linux-kcp -> all: ACC34 on <host> now; hold VM tests.

SWC=socialweb-computer
R=~/.tmp/acc34; mkdir -p "$R"; chmod 700 "$R"
HOST=<local|hat-9|hat-7>
# 1. suspend the other two MarketBidders (spec.suspend=true), wait both Suspended
<kubectl for other host> patch marketbidders.socialweb.computer app --type merge -p '{"spec":{"suspend":true}}'
# 2. fresh invocation: request + hold + first ssh
NAME=acc34-$HOST-$(date +%s)
$SWC run request --state-dir "$R" --name "$NAME" --hold \
  --tunnel ssh:22:webrtc --transport webrtc --atproto-oauth-qr \
  --exec -- sh -c 'uname -a; id' > "$R/$NAME-request.log" 2>&1 &
# 3. sign alice in with the newest url from the log (second shell / drives-oauth)
command grep -oE 'pendingAction (scanQr|signInAgain) url [^ ]*' "$R/$NAME-request.log" | tail -1
# 4. wait for uid= in the request log and the --hold process to exit
# 5. restarted invocation kcp: second ssh from the same state dir
$SWC run request ssh "$NAME" --state-dir "$R" --transport webrtc --exec -- sh -c 'uname -a; id' > "$R/$NAME-ssh.log" 2>&1
# 6. winner check (host lists a Running ComputeVM for alice), plaintext check
command grep -c -E 'a=candidate|AGE-SECRET-KEY' "$R/$NAME-request.log" "$R/$NAME-ssh.log"
# 7. release, then resume the two paused bidders
$SWC run request release "$NAME" --state-dir "$R"

The scripted form (pause, sign-in watch, both ssh runs, winner, release, resume) lives on the lead host as acc34.sh <host>; it is a wrapper of the commands above.

Expected observations

Request log phases ("phase": lines), rc142 runs:

Steplocalhat-9hat-7
start -> AwaitingScan20 s21 s20 s
scan -> identity Ready17 s19 s18 s
Requesting -> Held31 s40 s29 s
ssh returned, fresh kcp (seconds)2.5 s5.2 s8.9 s
ssh returned, restarted kcp6.1 s4.9 s3.6 s
  • signed in as aliceoa.bsky.social, then Held
  • {"msg":"ssh returned","transport":"webrtc","exit":0,"seconds":N} in both logs
  • output contains Linux and uid=
  • whole host run (pause -> resume) ~3 min; all three ~10 min
  • 2026-11-10 (p2p, offering endpoints, alice pdsAccount, --ready-timeout 10m): hat-9 start -> Held + first ssh 63 s, ssh 3.8 s fresh / 10.5 s restarted; hat-7 62 s, 5.2 s / 3.5 s; plaintext 0/0, both Released
  • 2026-11-10 rollout 5: hat-9 64 s, ssh 3.4 s / 2.1 s; hat-7 60 s, 3.1 s / 2.1 s; plaintext 0/0; each run left one market.event (the restarted invocation's signal) in the repo: delete it
  • earlier baselines: 2026-10-24 local 2.16 s fresh / ~7 s restarted; rc118 CLI run RFP -> Held 29 s, ssh 4.4 s

Pass criteria

  • Each host: Held, uid= from both invocations, exit 0
  • ssh ran with StrictHostKeyChecking=yes against guest-reported host keys
  • Winner = $HOST (Running ComputeVM for alice DID there while held)
  • Plaintext check counts 0 / 0 (no ICE candidates, no age keys in logs)
  • run request release ends Released; $HOST has no ComputeVM / firecracker left
  • All 3 hosts pass, paused bidders resumed (Ready)

Cleanup

  • Resume both paused bidders (spec.suspend=false), check all three Ready
  • Leftover VM: 0007
  • chmod 700 state dir stays; delete old $R/acc34-* state dirs when done

Known pitfalls

SymptomCause / fix
second ssh exit 255 after 120 s (seen once on local, 18:15Z)first invocation still held the shared state-dir kcp; wait for the --hold process to exit, rerun
invocation kcp did not come up on releasesame: release before the hold process exited
timeout waiting for guest publicKey on hat-7bob's oauthQr session predates a scope change; re-sign bob (0001 section 9)
no bidsalice not signed in by QR (localRepo DID waits in AwaitingApproval), or a vouch missing
every canary lands on the old rcother bidders not paused; suspend them as above
this host reaches none of <did>'s endpoints / no submitRfpthe bidder's offering has no endpoint the CLI dials (iroh needs dumbpipe on the CLI host; https only when advertised)
guest publicKey / answer timeoutread the winning bidder journal for the requester refused the market event / submitting the market event to the requester failed