0
0
fixes for the causes of gate failure found since roll two, the dumbpipe v0.40.1 bump; entry 0007:ingress (staging 48)
MergedSome checks were cancelled
2 cancelled, 1 passed
1 passed check
releaseSuccess in 38m 19sMerged as 0019f5f
2 cancelled, 1 passed
Plan 0040 decision 78: no roll while a known cause of gate failure is unfixed. Five fixes and the dumbpipe bump (decision 79), each reviewed and run out of band first, and one new staging entry. Entries: staging 48 (new: 0007:ingress), prod 31. Tree: start-warn 00e47278 = stack6 1c028e5d + the bump (bump6 cf34bc81) + the two start-up log levels (bind-label 287b641c, start-warn).
Pieces:
health-pid (tests only): bidder.health reads the current core process only (roll two's 0008:run counted the old process's shutdown errors); logs.fetchRuns takes its absolute dir; 0014:exhausted counts a capacity decline.
bid-capacity: the room is decided once per RFP, when its bid is created, on every intake path (submitRfp and the market stream); a later submitRfp answers the recorded decline; an intake that cannot read the RFP is retried; the requester's accept replacement budget is per waiting round and a 409 past it waits for capacity; 0014:giveup and 0014:wait fail if a bidder held full both declines and bids in one round. The ssh gateway's sign-in ends once the account's link exists.
ingress-rereg: a replaced ingress listener's publications are re-registered; an Exposure is Ready only once dumbpipe listens; no stored build-specific dumbpipe path; no xrpc fallback for a named IngressProxy not registered yet. New staging entry 0007:ingress (exclusive).
signin-retry: a transient createSession failure keeps the MarketIdentity Pending and is retried; Failed stays for a refused credential, a DNS not-found and a certificate error; the intended 10 s per call applies (a 5 s client cut it).
stage-budget (tests only): a host script states its budget; the test host waits that long plus 120 s (it cut every script at 120 s).
the bump: dumbpipe 0.39.0 -> 0.40.1 (pins, guest image 7e5dc836 with 0.40.1 baked, the cloud-init default). Why: in the stack6 cycle iroh.service restarted once on bidder-2 (07:12:33Z) and once on the gateway VM (07:29:38Z), each "Endpoint dropped without calling Endpoint::close" (both survived): the one known cause still open. Decision 79 drops the two-release order: with ingress-rereg in the same release the new core resolves dumbpipe from its own record and an old core rewrites its own path. The guest's setup-iroh fetch is limited to 3 tries of 10 s when the image has a dumbpipe (worst case 34 s, then the image's own runs); it is main's loop when the image has none.
bind-label (owner, 2026-11-22: a healed retry is not an ERROR): hostprocess.Supervise logs WARN while it restarts and ERROR once a failing streak lasts the set apply's deadline (1 min; the streak ends only after a run that stayed up that long). Cause, on main since 2026-10-21: sets
providerandapplicationsboth apply APIBindingpolicyat core start and kcp refuses one when they overlap; it failed 0006:run in the stack7 hand cycle (one start-up ERROR, healed in 2 s). The duplicated writer is recorded for a later plan.start-warn (same ruling): the reflector's "Failed to watch" with a NotFound error for a type logs WARN while its streak is under the cache-sync deadline (2 min), ERROR after (internal/controllers/runtime/logbridge.go). Cause: at core start the exposures virtual workspace serves discovery about 0.6 s before it serves the resource; the first list gets 404 and is retried. Seen once, on a bidder's first start in the stack8 hand cycle.
Run out of band (plan 0040 Findings has each run and cause):
/.tmp/stage-budget/proof3.log); on stg-277608 with the fixed test host, by hand: 0007:upgrade, 0007:fence, 0007:restart, 0006:run, 0008:run PASS (/.tmp/cycle-stack5/saved/hand/).~/.tmp/cycle-stack7/saved/final/); iroh.service restarts 0 and "Endpoint dropped" 0 on all four VMs in 54 min; FAIL 0006:run (the line above). stack8 70321d69 (bind-label + cli-help): stopped as a survey at 3 of 48 done, 2 PASS, 1 FAIL (0006:run on bob3: that line; 09:33Z;~/.tmp/cycle-stack8/results.txt) (the start-warn line above). stack9 1d794a7b (this tree + cli-help): gate: staging: 48 of 48 PASS, wall 7632 s (stg-090117, 09:45-11:52Z;~/.tmp/cycle-stack9/saved/final/): iroh.service restarts 0 and "Endpoint dropped" 0 on all four VMs; dumbpipe 0.40.1 never exited under a running core (0.39.0 did 3 times under the old binaries inside 0007); the APIBinding race fired once on bidder-1 at WARN and 0006:run passed (startupErrors 0). start-warn: a harness with real 404/403 list errors through the real handler (transient: WARN; permanent: ERROR from 2 min; 403: ERROR),/.tmp/bind-label/proof-watch/. bind-label: a harness on the real Supervise (never up: ERROR at the bound; crash loop: ERROR at bound + one attempt; one transient: 1 WARN; healthy then one failure: 1 WARN) and 10 fresh/.tmp/bind-label/)dev uprounds, 0 ERROR (Not proven / not exercised:
~/.tmp/gw-dial/):systemctl restartof the--fggateway re-renders the dialer onto the 0.40.1 tree (extracted before the dialer starts), dialer down about 21 s, 0 ERROR, new connections to 0.39.0 and 0.40.1 listeners PASS, rollback the same; an open session breaks at the restart as on every release; one host, no real signed-in ssh sessionKnown, not in this pull (plan 0040 Findings):
🤖 Generated with Claude Code