~ knotless Search/
↑↓ move↵ open⇧↵ all results

socialweb.computer/socialweb-computer-kcp

socialweb.computer: a compute market on atproto (kcp)

1 collaborator
0
0

fixes for the causes of gate failure found since roll two, the dumbpipe v0.40.1 bump; entry 0007:ingress (staging 48)

Merged
Merged johnandersen777.bsky.social asked to merge into main from socialweb.computer/socialweb-computer-kcp:phase-2

Plan 0040 decision 78: no roll while a known cause of gate failure is unfixed. Five fixes and the dumbpipe bump (decision 79), each reviewed and run out of band first, and one new staging entry. Entries: staging 48 (new: 0007:ingress), prod 31. Tree: start-warn 00e47278 = stack6 1c028e5d + the bump (bump6 cf34bc81) + the two start-up log levels (bind-label 287b641c, start-warn).

Pieces:

  • health-pid (tests only): bidder.health reads the current core process only (roll two's 0008:run counted the old process's shutdown errors); logs.fetchRuns takes its absolute dir; 0014:exhausted counts a capacity decline.

  • bid-capacity: the room is decided once per RFP, when its bid is created, on every intake path (submitRfp and the market stream); a later submitRfp answers the recorded decline; an intake that cannot read the RFP is retried; the requester's accept replacement budget is per waiting round and a 409 past it waits for capacity; 0014:giveup and 0014:wait fail if a bidder held full both declines and bids in one round. The ssh gateway's sign-in ends once the account's link exists.

  • ingress-rereg: a replaced ingress listener's publications are re-registered; an Exposure is Ready only once dumbpipe listens; no stored build-specific dumbpipe path; no xrpc fallback for a named IngressProxy not registered yet. New staging entry 0007:ingress (exclusive).

  • signin-retry: a transient createSession failure keeps the MarketIdentity Pending and is retried; Failed stays for a refused credential, a DNS not-found and a certificate error; the intended 10 s per call applies (a 5 s client cut it).

  • stage-budget (tests only): a host script states its budget; the test host waits that long plus 120 s (it cut every script at 120 s).

  • the bump: dumbpipe 0.39.0 -> 0.40.1 (pins, guest image 7e5dc836 with 0.40.1 baked, the cloud-init default). Why: in the stack6 cycle iroh.service restarted once on bidder-2 (07:12:33Z) and once on the gateway VM (07:29:38Z), each "Endpoint dropped without calling Endpoint::close" (both survived): the one known cause still open. Decision 79 drops the two-release order: with ingress-rereg in the same release the new core resolves dumbpipe from its own record and an old core rewrites its own path. The guest's setup-iroh fetch is limited to 3 tries of 10 s when the image has a dumbpipe (worst case 34 s, then the image's own runs); it is main's loop when the image has none.

  • bind-label (owner, 2026-11-22: a healed retry is not an ERROR): hostprocess.Supervise logs WARN while it restarts and ERROR once a failing streak lasts the set apply's deadline (1 min; the streak ends only after a run that stayed up that long). Cause, on main since 2026-10-21: sets provider and applications both apply APIBinding policy at core start and kcp refuses one when they overlap; it failed 0006:run in the stack7 hand cycle (one start-up ERROR, healed in 2 s). The duplicated writer is recorded for a later plan.

  • start-warn (same ruling): the reflector's "Failed to watch" with a NotFound error for a type logs WARN while its streak is under the cache-sync deadline (2 min), ERROR after (internal/controllers/runtime/logbridge.go). Cause: at core start the exposures virtual workspace serves discovery about 0.6 s before it serves the resource; the first list gets 404 and is retried. Seen once, on a bidder's first start in the stack8 hand cycle.

Run out of band (plan 0040 Findings has each run and cause):

  • health-pid 8e49c3a0: full hand cycle 47 of 47 on stg-031952 (~/.tmp/cycle-health/saved/); shown not vacuous.
  • bid-capacity, on stg-033371 with its own build 08ffeb14: 0014:giveup 5 of 5, 0014:wait 7 of 8 (the FAIL: the rental's createSession timed out before any RFP), 0014:impossible PASS; bidder journals: every RFP arrived over the stream first, a full bidder declined at intake, 0 accept refusals for capacity (~/.tmp/bid-capacity/rig4/). ssh sign-in: 30 fresh sign-ins on the fix under three loads, 0 refusals, 0.7 to 4.1 s added at sign-in.
  • ingress-rereg, on stg-033371 with its own build 60d0cdf8: switch from a pre-fix build on both bidders, 0007:ingress PASS twice, 0007:upgrade and 0007:fence PASS, 0007:ingress on main 7b0afc69 FAIL as it must (1 started, 0 listening after a replacement); 0 ERROR lines in 11 bidder core processes (~/.tmp/ingress-rereg/slot/saved/). Local: 5 upgrades with a guest held, 0 ERROR in each new core's first 600 s. Rollback to stable 324f69a1: the old core listens again 8 to 18 s after its start on staging; local, an iroh-only RFP answered 3 of 3 after the downgrade and 3 of 3 after the upgrade back.
  • signin-retry ae1db64f, local market, a proxy holding createSession: main fails 3 of 3 with the staging error; the fix signs in (8 s hold: first attempt; 40 s hold: after retries); a wrong password, a typo'd host and an untrusted certificate fail in under 10 s; a PDS that never answers ends at the CLI's 5 min deadline (~/.tmp/signin-retry/logs/).
  • stage-budget 1c028e5d: local harness with the real script code (old: cut at 120.0 s; new: completes at 153.7 s; a hang is still cut) (/.tmp/stage-budget/proof3.log); on stg-277608 with the fixed test host, by hand: 0007:upgrade, 0007:fence, 0007:restart, 0006:run, 0008:run PASS (/.tmp/cycle-stack5/saved/hand/).
  • the bump, on stg-113817 (~/.tmp/ingress-rereg/slot6/): both bidders switched 0.39.0 -> 0.40.1 with 0 ERROR; 0007:upgrade 324f69a1 <-> bump PASS (different pins, both directions), the old core listening 8.9 and 8.1 s after its start; 0007:run and 0007:restart PASS; a guest of the bump requester on a bump bidder fetched nothing and runs dumbpipe 0.40.1. setup-iroh, the rendered script run locally: fetched in under 1 s; black-holed and a silent listener fall back in 19 s; a 200 KB/s source falls back at 34 s with an image dumbpipe and is fetched (17 s) with none.
  • Hand cycles of the combined tree, full list, fresh network, CI's default input:
    • stack4 7b0f1bc3 (stg-033371): 44 of 48; the four FAILs came from earlier heads of two branches, each corrected.
    • stack5 df3b1c2c (stg-277608): gate: staging: 47 of 48 PASS, wall 7311 s (~/.tmp/cycle-stack5/saved/final/); the FAIL, 0007:upgrade: the 120 s cut of the stage fetch (stage-budget).
    • stack6 1c028e5d (stg-113817): gate: staging: 48 of 48 PASS, wall 7347 s (~/.tmp/cycle-stack6/saved/final/). 0 bus link denied, 0 bidder_payload_refused, 0007:upgrade with real fetches (17 s and 40 s).
    • the hand cycle ran on stack7 43297eda = this tree + the colored help change; this tree itself: stack6 48 of 48 + the bump's switch proof. stack7 43297eda (bump6 + cli-help): stopped as a survey at 10 of 48 done, 9 PASS, 1 FAIL (09:01Z; ~/.tmp/cycle-stack7/saved/final/); iroh.service restarts 0 and "Endpoint dropped" 0 on all four VMs in 54 min; FAIL 0006:run (the line above). stack8 70321d69 (bind-label + cli-help): stopped as a survey at 3 of 48 done, 2 PASS, 1 FAIL (0006:run on bob3: that line; 09:33Z; ~/.tmp/cycle-stack8/results.txt) (the start-warn line above). stack9 1d794a7b (this tree + cli-help): gate: staging: 48 of 48 PASS, wall 7632 s (stg-090117, 09:45-11:52Z; ~/.tmp/cycle-stack9/saved/final/): iroh.service restarts 0 and "Endpoint dropped" 0 on all four VMs; dumbpipe 0.40.1 never exited under a running core (0.39.0 did 3 times under the old binaries inside 0007); the APIBinding race fired once on bidder-1 at WARN and 0006:run passed (startupErrors 0). start-warn: a harness with real 404/403 list errors through the real handler (transient: WARN; permanent: ERROR from 2 min; 403: ERROR), /.tmp/bind-label/proof-watch/. bind-label: a harness on the real Supervise (never up: ERROR at the bound; crash loop: ERROR at bound + one attempt; one transient: 1 WARN; healthy then one failure: 1 WARN) and 10 fresh dev up rounds, 0 ERROR (/.tmp/bind-label/)
  • Checks on the head: go build, go vet, gofmt, deploy/check.sh --fast, cargo check of the test host.

Not proven / not exercised:

  • This exact tree has no full hand cycle of its own (decision 79): the cycled tree is this one plus the CLI help change.
  • An installed ssh gateway re-rendering its iroh-dial HostProcess to the new dumbpipe tree across the upgrade: proven in prod's layout on hat-8 (~/.tmp/gw-dial/): systemctl restart of the --fg gateway re-renders the dialer onto the 0.40.1 tree (extracted before the dialer starts), dialer down about 21 s, 0 ERROR, new connections to 0.39.0 and 0.40.1 listeners PASS, rollback the same; an open session breaks at the restart as on every release; one host, no real signed-in ssh session
  • The ssh sign-in wait closing a window: main 7b0afc69 gave 0 refusals in 40 sign-ins too (failure not reproduced; its worst session wait was 5.5 s against the 5 s settle). Shown not to regress.
  • signin-retry on staging: 0 retry lines in the stack5 and stack6 cycles (no sign-in was slow). Why one createSession took over 5 s on stg-033371 is not established.
  • stage-budget's slow path on staging: the fetches seen since were fast. Why one took 119 s is not established.
  • setup-iroh's fallback on staging (no knob to black-hole one guest's GitHub); a bash older than 4.4 was not run.
  • A session through an adopted 0.39.0 dialer to a 0.40.1 guest by hand; hand rollbacks with forced iroh on staging with different pins (0007:upgrade's downgrade is the staging instance; the local 3 of 3 stands).
  • ingress-rereg: the crash-restart path. 0007:upgrade does not assert per-bidder iroh reachability of the from binary.
  • Prod: the bidder-side changes (capacity intake, listener re-registration, the bump) run on prod only once promote moves the bidders; 0007:ingress is staging only.

Known, not in this pull (plan 0040 Findings):

  • one ERROR "Failed to watch ... context canceled" (controller marketbidder, an apibindings watch) 4 s after the start of an OLD-binary core (324f69a1) inside 0007:upgrade on stg-090117: no health reading counted it; first seen; the same controller code is in this tree, cause not established: being root-caused while the roll runs (lead's call, owner asleep: his order is one green roll by noon).
  • A guest fetches dumbpipe from GitHub at boot when its requester and its image have different versions: CI's 4 network VMs at stand-up, an old stable CLI on updated bidders, new requesters after a bidder rollback (no sha256 check: docs/security/proposed-controls.md).
  • Three outer bounds of the test host shorter than their step's stated worst case (dev up 200 s, requester stop 120 s, page-round 900 s): latent, never observed; raising them alone was reviewed not safe.
  • Provision StateSyncs are never deleted.
  • Other silent bid failures (requester unresolvable, invalid signature, invalid RFP, policy and scope denials).
  • The rc186 binary's shutdown ERROR lines; the prod ops reader's whole-second journal window (hostread).
  • A staging run with a listed suite passes with zero entries run (CI's own path is not affected).

🤖 Generated with Claude Code

Some checks were cancelled

2 cancelled, 1 passed

Merged as 0019f5f