johnandersen777.bsky.social/socialweb-computer-kcp

Switch repository

Loading…

plan: "0033"
title: "Observe and manage a fleet of bidders: what the lead does by hand today, and the objects that should do it"
status: todo
source: >
  2026-10-24, written by the lead while running prod (the gateway socialweb.computer and the bidders
  local, hat-9 and hat-7) through rc83..rc98 and while a browser-hosted kcp rents VMs from those
  bidders. The user asked for every lesson to be written down for the observability story and for a
  plan to manage a fleet of bidders, grown as the work goes. This file is that record: the
  learnings section is appended as they come, and the items turn the manual steps into objects.
principle: >
  A fleet is observed through objects and records that already exist (each bidder's MarketBidder
  root status, its market heartbeat and offering records, its MarketIdentity, its ComputeVMs and
  Bids), read from one place, not by ssh into each host. Nothing new runs on a bidder host to be
  observed; nothing reaches a host from CI (prod installs stay on the lead's machine, user decision
  2026-10-24). Secrets never enter a fleet view.
state_rules:
  - Read AGENTS.md first. status is one of todo, doing, done, blocked. The lead flips statuses.
  - No Go tests (AGENTS.md, plan 0027). Proof is the build gates and the live runs below.
learnings:
  - id: L1-HOW-THE-LEAD-SEES-THE-FLEET
    text: >
      Today the lead sees each host through ssh: ~/.tmp/0026/w7/b.sh runs kubectl against the
      bidder's own kcp as root on local, hat-9 and hat-7 (and the gateway's root and account
      workspaces), and journalctl _UID=<swc-bidder> reads the core's log. There is no single view;
      every check in the acceptance script (docs/acceptance/0001-auth-modes.md) is an ssh per host.
  - id: L2-HEALTH-SIGNALS-THAT-WORKED
    text: >
      The signals that answered "is this bidder healthy" during the rc runs: the MarketBidder root
      phase and message (Ready, or the exact wait); status.componentSummary
      {desired, notReady, ready, suspended}; the market heartbeat record
      com.publicdomainrelay.temp.market.heartbeat/self createdAt (every 60 s; stale means not
      bidding); the offering record com.publicdomainrelay.temp.market.offering (its cid must not
      churn); the MarketIdentity phase (Ready, AwaitingScan for a lost oauthQr session); the event
      streams' and sources' Ready conditions; the deny count in the journal (A7: 0 on every host).
  - id: L3-FAILURES-THE-FLEET-HAD
    text: >
      Failures seen on the real fleet and what showed them: hat-7 lost its oauthQr session twice
      (refresh-token rotation race between several in-memory session copies; journal
      "invalid_grant: Invalid refresh token", MarketIdentity AwaitingScan; fixed e671cd9e, plan
      0030); a paused winner still won the next VM (root Suspended written before the Bidder ran the
      paused generation; A2 distinct winners 2; fixed 3f47b6bd/a9f18ce0); a heartbeat sent 1 s after
      a pause patch looked like a pause failure (the check sampled before the Bidder ran the paused
      generation; runbook fixed 351852b1); a stuck root after its MarketIdentity was deleted (no
      requeue; a5936dfe). None of these raised anything by itself; each was found by a scripted check.
  - id: L4-DEPLOY-FACTS
    text: >
      A release is CI-built (tag run about 16-30 min; the guest image is fetched by its pin instead
      of rebuilt when the pins are unchanged) and installed from this machine: download from the
      public bucket, sha256 against SHA256SUMS, README install commands over ssh on each hat,
      scripts/deploy.sh for the gateway. The local release build of the same commit is byte-identical
      to CI's (rc93). A CI flake (one failed apt download) was fixed by retrying all download errors
      (a86026a6). Install order local, hat-9, hat-7, gateway; each bidder is Ready again within
      about 1 min and keeps its DID, association and (since plan 0030) its OAuth session.
  - id: L5-TIMINGS
    text: >
      From the acceptance runs (UTC 2026-10-24): RFP to a Running VM 41-53 s on local, hat-9 and
      hat-7; tenant cluster stop to restart Ready 32-59 s; a Gateway root delete with a busy tenant
      cluster 16-18 s; scale to 0 releases 3 VMs in about 2.5 min (the k3s graceful stop of the
      guest is about 90 s and is not visible to the requester).
  - id: L6-TOOLING-TRAPS
    text: >
      In the lead's Bash tool, grep is a ugrep shell function that holds each line in a pipe until
      the next one: watches through it looked like 38-100 s event lag. Use command grep in streaming
      pipelines. Editing a running acceptance script garbles bash's read of it: run a frozen copy.
  - id: L7-RESTART-READINESS
    text: >
      A bidder core restart used to leave the bidder Provisioning for about 5 minutes and 7 retries:
      its ingress Exposure was checked before the Bidder registered its handler, nothing re-triggered
      the check, and the Bidder waits for that Exposure's address (fixed c8147d96: a bus handler
      registration re-triggers the checks). After the fix and status.appliedSet (25ee9d70) a bidder
      core is Ready 8-9 s after it starts and a gateway core about 18 s (local network as25,
      2026-10-24). A fleet view should report time-to-Ready per host after each restart; a slow one is
      the first sign of an ordering race like this one.
  - id: L8-START-ORDER-CYCLES
    text: >
      A component that waits for its own Exposure before registering its handler waits forever,
      because the Exposure needs the handler to become Ready (condition HandlerMissing). The Bidder
      had it (fixed 46eada60); on 2026-10-24 a browser-hosted kcp renting a VM through our Requester
      showed the same cycle in the Requester (phase Starting, "the requester market URL is not
      resolved yet" every ~11 s, no RFP ever sent; contract vmshell-apv725). On our own networks the
      10 s timeout let it limp on, so only a stricter host exposed it. Signals worth a fleet alert: a
      component in phase Starting for more than a minute, an Exposure Ready=False with reason
      HandlerMissing, and a contract Pending with no RFP record.
  - id: L9-BROWSER-REQUESTER-CHAIN
    text: >
      The browser-hosted requester test (2026-10-24, peer page on kcpwasm, alice, local bidder) has
      exposed one link of the requester's dependency chain per run: run 2 the ingress Exposure (wait
      cycle, fixed c69d7325); run 3 the AtprotoRepo repo.write provider missing from the browser core
      ("bus: no provider serves this kind and capability"), with the Requester restarting under
      backoff (7 restarts in 70 s) while the contract stayed Pending and no bidder saw an RFP. A fleet
      or requester view should show, per contract, which dependency the requester is waiting on
      (handler, Exposure address, repo provider, identity) instead of a bare Pending; the component's
      restarts count and last error are already in its status and are the fastest signal.
  - id: L10-SILENT-WATCHDOG-RESTARTS
    text: >
      From 2026-10-23 17:06Z to the rc101 rollout every bidder core (local, hat-9, hat-7) was aborted by
      systemd's watchdog every ~7 minutes (WatchdogSec=7min; "Failed with result 'watchdog'"): the core
      fed the watchdog only from devcore's OnTick, which stops once the stack settles (fixed 81c172f9,
      notify.KeepAlive). It ran unnoticed for about 16 hours through several acceptance runs, because
      each core came back Ready within seconds and every acceptance row reads state, not uptime. Only a
      Go stack dump in a journal watch for something else showed it. A fleet view must report per host
      the unit's NRestarts, its last result (watchdog, signal, exit) and the core's uptime; any restart
      the lead did not cause is an alert. After rc101 (installed 09:27-09:28Z) no host logged a watchdog kill
      through 09:37:39Z, past the 7-minute window; the last kills were 09:26:55 (local), 09:27:09 (hat-9)
      and 09:27:32 (hat-7), each just before its install.
  - id: L11-BROWSER-REQUESTER-BID-PATH
    text: >
      Runs 6-9 of the browser-hosted requester (2026-10-24) each stopped one link further along the bid
      path, and none of the stops raised anything on the bidder side: run 6 no bidder candidates (the
      contract's spec.bidderDids was never read; fixed 267ae116, requester_no_bidders now names each
      dropped candidate and why); run 8 the bid published but never delivered, because an OAuth
      requester published no submitBid and took bids only from the firehose, which the browser build
      lacks (fixed d6937f50: OAuth requesters take bids over their ingress); run 9 the bid POSTed
      into the tab and answered 503 every 2 s for minutes, because the browser's Exposure kept the
      stopped first Requester's handler (fixed on the browser side). The bidder retried that 503 at a
      fixed 2 s until the bid's 24 h expiry, and a bid not yet delivered was held only in memory, lost
      on a core restart. A fleet view should show per bid: published, delivered (or the last delivery
      error and attempt count), accepted; a bid published but undelivered for more than the bid window
      is the signal that a requester cannot be reached.
  - id: L12-BROWSER-REQUESTER-END-TO-END
    text: >
      Run 12 (2026-10-24, alice in a browser-hosted kcp, Bidders=local, spec.tunnels wootty) passed
      end to end on rc101 bidders: click 09:51:35Z, discovery from the contract's bidderDids 16 s,
      bid in the tab 17 s, accept 20 s, receipt verified 30 s, secrets fetched 36 s, iroh onNetwork
      38 s, Held 40 s; the tagged wootty onNetwork came 72 s after the iroh one (opentunnel bring-up
      in the guest, the longest step), the shell answered uname/id/hostname as root, and release
      took 3 s with nothing left on the host. Runs 10 and 11 had failed on OAuth scopes: bsky.social
      asks for the update action on every putRecord, so a requester scope must grant create and
      update on each collection it writes at a fixed rkey (fixed ca681018, 5f31bd25). A fleet view
      should keep per contract these step times; a step that is suddenly slower on one host (the
      tunnel step above all) is the first sign of trouble there.
  - id: L13-IDENTITY-MODE-SHAPES-REACHABILITY
    text: >
      The three bidders were not alike: local and hat-9 run identity mode localRepo, hat-7 oauthQr.
      Until rc103 an OAuth-mode bidder put no endpoint in its offering and heard RFPs only through its
      firehose sources (gateway-pds on hat-7), so the gateway's RFPs reached it but a requester posting
      to bsky.social never did (browser run 14, 'requester_no_bidders: the offering names no endpoint',
      seen at once thanks to 267ae116). On rc103 hat-7 served the browser end to end (run 15: Held 30 s,
      shell as root). After each install hat-7's first offering write failed ('the repo has no bound
      backend') and the 60 s refresh corrected it, so for a minute a restarted OAuth bidder advertises
      a stale offering. A fleet view should list per bidder its identity mode, comms mode, the offering's
      endpoint and the offering's age, and flag a bidder whose offering has no endpoint.
  - id: L14-ABANDONED-GUESTS
    text: >
      A browser requester tab closed one second after it began a release (the lead's driver, run 15,
      2026-10-24 10:51Z) left its guest Running on hat-7 for 20 minutes, until the lead deleted the
      ComputeVM by hand; the prod acceptance that followed waited 11.5 minutes in its scale-to-0
      section on that unrelated VM (A2 'vms=1'). Nothing in the market protocol bounds a guest's life:
      the compute.vm and RFP records carry no lease, the release deadline in the requester's status is
      8 days, and a bidder keeps a guest whose requester has gone for good. A fleet view must list
      every Running guest per bidder with its requester, age and last requester contact; a guest with
      no requester contact past a lease is an alert, and see F5.
  - id: L15-ERROR-LINES-THAT-ARE-NOT-ERRORS
    text: >
      Until rc108 every core start and stop wrote 'Reconciler error' lines that were not failures:
      reconciles cut by the shutdown context (about 22 per bidder stop), and controllers that reached a
      repo, Pds, EventSource or sink before its provider registered (one error per race, then a backoff
      retry). They buried the real ones; a lost release on an unbound repo (fixed 29ed308e) looked like
      the same noise. Since rc108 such waits are status reasons with a bind or registration requeue,
      shutdown cancels are Debug, and an rc108 core logged 0 'Reconciler error' lines from start
      through a full acceptance run on every host (2026-10-24 15:18-15:32Z). A fleet view can now alert
      on any 'Reconciler error' at all.
items:
  - id: F1-INVENTORY
    title: "A fleet inventory object: which bidders exist, where, on which version"
    depends_on: []
    decision: >
      To be designed. Candidate: a cluster-scoped BidderHost (or Fleet with spec.hosts) in the lead's
      own kcp naming each host (ssh target, bidder DID, expected version, role) so the rollout and
      the acceptance scripts read one list instead of a hard-coded host set.
    files: []
    done_when: "To be written when designed."
    status: todo
  - id: F2-HEALTH-FROM-RECORDS
    title: "Fleet health read from public records, with no access to the host"
    depends_on: []
    decision: >
      To be designed. Every bidder already publishes its heartbeat and offering records in its repo
      and on the relay; a fleet view can subscribe (EventStream, plan 0026 W10) to those collections
      for the fleet's DIDs and report per bidder: last heartbeat, offering cid, and staleness.
    files: []
    done_when: "To be written when designed."
    status: todo
  - id: F3-SESSION-AND-IDENTITY-ALERTS
    title: "Raise a lost account session instead of finding it in a journal"
    depends_on: []
    decision: >
      To be designed. A MarketIdentity leaving Ready (AwaitingScan, Failed SubjectMismatch,
      oauth_session_expired) should surface in the fleet view with the reason; today it is visible
      only on the host.
    files: []
    done_when: "To be written when designed."
    status: todo
  - id: F4-ROLLOUT-AS-OBJECT
    title: "A rollout the lead declares, with per-host steps and readings"
    depends_on: [F1-INVENTORY]
    decision: >
      To be designed. rollout-ci.sh and the acceptance script become a declared rollout of a version
      to the inventory, executed from the lead's machine (no CI access to hosts), recording per host
      the installed sha256, the Ready time and the acceptance rows.
    files: []
    done_when: "To be written when designed."
    status: todo
  - id: F5-GUEST-LEASE
    title: "A guest's life is bounded by a lease the requester renews"
    depends_on: []
    decision: >
      To be designed, and a market protocol change, so the user decides it. Candidate: the compute.vm
      (or RFP) record carries a lease (seconds); the requester renews it while it holds the guest (a
      renewal record or an XRPC to the bidder's ingress), and the bidder releases a guest whose lease
      ran out, writing vm.terminated as for a release. Today an abandoned guest lives until someone
      deletes its ComputeVM (L14).
    files: []
    done_when: "To be written when designed."
    status: todo