plan: "0033" title: "Observe and manage a fleet of bidders: what the lead does by hand today, and the objects that should do it" status: todo source: > 2026-10-24, written by the lead while running prod (the gateway socialweb.computer and the bidders local, hat-9 and hat-7) through rc83..rc98 and while a browser-hosted kcp rents VMs from those bidders. The user asked for every lesson to be written down for the observability story and for a plan to manage a fleet of bidders, grown as the work goes. This file is that record: the learnings section is appended as they come, and the items turn the manual steps into objects. principle: > A fleet is observed through objects and records that already exist (each bidder's MarketBidder root status, its market heartbeat and offering records, its MarketIdentity, its ComputeVMs and Bids), read from one place, not by ssh into each host. Nothing new runs on a bidder host to be observed; nothing reaches a host from CI (prod installs stay on the lead's machine, user decision 2026-10-24). Secrets never enter a fleet view. state_rules: - Read AGENTS.md first. status is one of todo, doing, done, blocked. The lead flips statuses. - No Go tests (AGENTS.md, plan 0027). Proof is the build gates and the live runs below. learnings: - id: L1-HOW-THE-LEAD-SEES-THE-FLEET text: > Today the lead sees each host through ssh: ~/.tmp/0026/w7/b.sh runs kubectl against the bidder's own kcp as root on local, hat-9 and hat-7 (and the gateway's root and account workspaces), and journalctl _UID= reads the core's log. There is no single view; every check in the acceptance script (docs/acceptance/0001-auth-modes.md) is an ssh per host. - id: L2-HEALTH-SIGNALS-THAT-WORKED text: > The signals that answered "is this bidder healthy" during the rc runs: the MarketBidder root phase and message (Ready, or the exact wait); status.componentSummary {desired, notReady, ready, suspended}; the market heartbeat record com.publicdomainrelay.temp.market.heartbeat/self createdAt (every 60 s; stale means not bidding); the offering record com.publicdomainrelay.temp.market.offering (its cid must not churn); the MarketIdentity phase (Ready, AwaitingScan for a lost oauthQr session); the event streams' and sources' Ready conditions; the deny count in the journal (A7: 0 on every host). - id: L3-FAILURES-THE-FLEET-HAD text: > Failures seen on the real fleet and what showed them: hat-7 lost its oauthQr session twice (refresh-token rotation race between several in-memory session copies; journal "invalid_grant: Invalid refresh token", MarketIdentity AwaitingScan; fixed e671cd9e, plan 0030); a paused winner still won the next VM (root Suspended written before the Bidder ran the paused generation; A2 distinct winners 2; fixed 3f47b6bd/a9f18ce0); a heartbeat sent 1 s after a pause patch looked like a pause failure (the check sampled before the Bidder ran the paused generation; runbook fixed 351852b1); a stuck root after its MarketIdentity was deleted (no requeue; a5936dfe). None of these raised anything by itself; each was found by a scripted check. - id: L4-DEPLOY-FACTS text: > A release is CI-built (tag run about 16-30 min; the guest image is fetched by its pin instead of rebuilt when the pins are unchanged) and installed from this machine: download from the public bucket, sha256 against SHA256SUMS, README install commands over ssh on each hat, scripts/deploy.sh for the gateway. The local release build of the same commit is byte-identical to CI's (rc93). A CI flake (one failed apt download) was fixed by retrying all download errors (a86026a6). Install order local, hat-9, hat-7, gateway; each bidder is Ready again within about 1 min and keeps its DID, association and (since plan 0030) its OAuth session. - id: L5-TIMINGS text: > From the acceptance runs (UTC 2026-10-24): RFP to a Running VM 41-53 s on local, hat-9 and hat-7; tenant cluster stop to restart Ready 32-59 s; a Gateway root delete with a busy tenant cluster 16-18 s; scale to 0 releases 3 VMs in about 2.5 min (the k3s graceful stop of the guest is about 90 s and is not visible to the requester). - id: L6-TOOLING-TRAPS text: > In the lead's Bash tool, grep is a ugrep shell function that holds each line in a pipe until the next one: watches through it looked like 38-100 s event lag. Use command grep in streaming pipelines. Editing a running acceptance script garbles bash's read of it: run a frozen copy. - id: L7-RESTART-READINESS text: > A bidder core restart used to leave the bidder Provisioning for about 5 minutes and 7 retries: its ingress Exposure was checked before the Bidder registered its handler, nothing re-triggered the check, and the Bidder waits for that Exposure's address (fixed c8147d96: a bus handler registration re-triggers the checks). After the fix and status.appliedSet (25ee9d70) a bidder core is Ready 8-9 s after it starts and a gateway core about 18 s (local network as25, 2026-10-24). A fleet view should report time-to-Ready per host after each restart; a slow one is the first sign of an ordering race like this one. - id: L8-START-ORDER-CYCLES text: > A component that waits for its own Exposure before registering its handler waits forever, because the Exposure needs the handler to become Ready (condition HandlerMissing). The Bidder had it (fixed 46eada60); on 2026-10-24 a browser-hosted kcp renting a VM through our Requester showed the same cycle in the Requester (phase Starting, "the requester market URL is not resolved yet" every ~11 s, no RFP ever sent; contract vmshell-apv725). On our own networks the 10 s timeout let it limp on, so only a stricter host exposed it. Signals worth a fleet alert: a component in phase Starting for more than a minute, an Exposure Ready=False with reason HandlerMissing, and a contract Pending with no RFP record. - id: L9-BROWSER-REQUESTER-CHAIN text: > The browser-hosted requester test (2026-10-24, peer page on kcpwasm, alice, local bidder) has exposed one link of the requester's dependency chain per run: run 2 the ingress Exposure (wait cycle, fixed c69d7325); run 3 the AtprotoRepo repo.write provider missing from the browser core ("bus: no provider serves this kind and capability"), with the Requester restarting under backoff (7 restarts in 70 s) while the contract stayed Pending and no bidder saw an RFP. A fleet or requester view should show, per contract, which dependency the requester is waiting on (handler, Exposure address, repo provider, identity) instead of a bare Pending; the component's restarts count and last error are already in its status and are the fastest signal. - id: L10-SILENT-WATCHDOG-RESTARTS text: > From 2026-10-23 17:06Z to the rc101 rollout every bidder core (local, hat-9, hat-7) was aborted by systemd's watchdog every ~7 minutes (WatchdogSec=7min; "Failed with result 'watchdog'"): the core fed the watchdog only from devcore's OnTick, which stops once the stack settles (fixed 81c172f9, notify.KeepAlive). It ran unnoticed for about 16 hours through several acceptance runs, because each core came back Ready within seconds and every acceptance row reads state, not uptime. Only a Go stack dump in a journal watch for something else showed it. A fleet view must report per host the unit's NRestarts, its last result (watchdog, signal, exit) and the core's uptime; any restart the lead did not cause is an alert. After rc101 (installed 09:27-09:28Z) no host logged a watchdog kill through 09:37:39Z, past the 7-minute window; the last kills were 09:26:55 (local), 09:27:09 (hat-9) and 09:27:32 (hat-7), each just before its install. - id: L11-BROWSER-REQUESTER-BID-PATH text: > Runs 6-9 of the browser-hosted requester (2026-10-24) each stopped one link further along the bid path, and none of the stops raised anything on the bidder side: run 6 no bidder candidates (the contract's spec.bidderDids was never read; fixed 267ae116, requester_no_bidders now names each dropped candidate and why); run 8 the bid published but never delivered, because an OAuth requester published no submitBid and took bids only from the firehose, which the browser build lacks (fixed d6937f50: OAuth requesters take bids over their ingress); run 9 the bid POSTed into the tab and answered 503 every 2 s for minutes, because the browser's Exposure kept the stopped first Requester's handler (fixed on the browser side). The bidder retried that 503 at a fixed 2 s until the bid's 24 h expiry, and a bid not yet delivered was held only in memory, lost on a core restart. A fleet view should show per bid: published, delivered (or the last delivery error and attempt count), accepted; a bid published but undelivered for more than the bid window is the signal that a requester cannot be reached. - id: L12-BROWSER-REQUESTER-END-TO-END text: > Run 12 (2026-10-24, alice in a browser-hosted kcp, Bidders=local, spec.tunnels wootty) passed end to end on rc101 bidders: click 09:51:35Z, discovery from the contract's bidderDids 16 s, bid in the tab 17 s, accept 20 s, receipt verified 30 s, secrets fetched 36 s, iroh onNetwork 38 s, Held 40 s; the tagged wootty onNetwork came 72 s after the iroh one (opentunnel bring-up in the guest, the longest step), the shell answered uname/id/hostname as root, and release took 3 s with nothing left on the host. Runs 10 and 11 had failed on OAuth scopes: bsky.social asks for the update action on every putRecord, so a requester scope must grant create and update on each collection it writes at a fixed rkey (fixed ca681018, 5f31bd25). A fleet view should keep per contract these step times; a step that is suddenly slower on one host (the tunnel step above all) is the first sign of trouble there. - id: L13-IDENTITY-MODE-SHAPES-REACHABILITY text: > The three bidders were not alike: local and hat-9 run identity mode localRepo, hat-7 oauthQr. Until rc103 an OAuth-mode bidder put no endpoint in its offering and heard RFPs only through its firehose sources (gateway-pds on hat-7), so the gateway's RFPs reached it but a requester posting to bsky.social never did (browser run 14, 'requester_no_bidders: the offering names no endpoint', seen at once thanks to 267ae116). On rc103 hat-7 served the browser end to end (run 15: Held 30 s, shell as root). After each install hat-7's first offering write failed ('the repo has no bound backend') and the 60 s refresh corrected it, so for a minute a restarted OAuth bidder advertises a stale offering. A fleet view should list per bidder its identity mode, comms mode, the offering's endpoint and the offering's age, and flag a bidder whose offering has no endpoint. - id: L14-ABANDONED-GUESTS text: > A browser requester tab closed one second after it began a release (the lead's driver, run 15, 2026-10-24 10:51Z) left its guest Running on hat-7 for 20 minutes, until the lead deleted the ComputeVM by hand; the prod acceptance that followed waited 11.5 minutes in its scale-to-0 section on that unrelated VM (A2 'vms=1'). Nothing in the market protocol bounds a guest's life: the compute.vm and RFP records carry no lease, the release deadline in the requester's status is 8 days, and a bidder keeps a guest whose requester has gone for good. A fleet view must list every Running guest per bidder with its requester, age and last requester contact; a guest with no requester contact past a lease is an alert, and see F5. - id: L15-ERROR-LINES-THAT-ARE-NOT-ERRORS text: > Until rc108 every core start and stop wrote 'Reconciler error' lines that were not failures: reconciles cut by the shutdown context (about 22 per bidder stop), and controllers that reached a repo, Pds, EventSource or sink before its provider registered (one error per race, then a backoff retry). They buried the real ones; a lost release on an unbound repo (fixed 29ed308e) looked like the same noise. Since rc108 such waits are status reasons with a bind or registration requeue, shutdown cancels are Debug, and an rc108 core logged 0 'Reconciler error' lines from start through a full acceptance run on every host (2026-10-24 15:18-15:32Z). A fleet view can now alert on any 'Reconciler error' at all. items: - id: F1-INVENTORY title: "A fleet inventory object: which bidders exist, where, on which version" depends_on: [] decision: > To be designed. Candidate: a cluster-scoped BidderHost (or Fleet with spec.hosts) in the lead's own kcp naming each host (ssh target, bidder DID, expected version, role) so the rollout and the acceptance scripts read one list instead of a hard-coded host set. files: [] done_when: "To be written when designed." status: todo - id: F2-HEALTH-FROM-RECORDS title: "Fleet health read from public records, with no access to the host" depends_on: [] decision: > To be designed. Every bidder already publishes its heartbeat and offering records in its repo and on the relay; a fleet view can subscribe (EventStream, plan 0026 W10) to those collections for the fleet's DIDs and report per bidder: last heartbeat, offering cid, and staleness. files: [] done_when: "To be written when designed." status: todo - id: F3-SESSION-AND-IDENTITY-ALERTS title: "Raise a lost account session instead of finding it in a journal" depends_on: [] decision: > To be designed. A MarketIdentity leaving Ready (AwaitingScan, Failed SubjectMismatch, oauth_session_expired) should surface in the fleet view with the reason; today it is visible only on the host. files: [] done_when: "To be written when designed." status: todo - id: F4-ROLLOUT-AS-OBJECT title: "A rollout the lead declares, with per-host steps and readings" depends_on: [F1-INVENTORY] decision: > To be designed. rollout-ci.sh and the acceptance script become a declared rollout of a version to the inventory, executed from the lead's machine (no CI access to hosts), recording per host the installed sha256, the Ready time and the acceptance rows. files: [] done_when: "To be written when designed." status: todo - id: F5-GUEST-LEASE title: "A guest's life is bounded by a lease the requester renews" depends_on: [] decision: > To be designed, and a market protocol change, so the user decides it. Candidate: the compute.vm (or RFP) record carries a lease (seconds); the requester renews it while it holds the guest (a renewal record or an XRPC to the bidder's ingress), and the bidder releases a guest whose lease ran out, writing vm.terminated as for a release. Today an abandoned guest lives until someone deletes its ComputeVM (L14). files: [] done_when: "To be written when designed." status: todo