0
0
plan 0040: roll three's test red three times, causes read; no roll until the staging overlay is understood
docs/plans/0040-findings.md+47−0
| @@ -4118,3 +4118,50 @@ Each: what was seen (with the log), the cause, the fix, its proof. | ||
already logs "the reconcile failed; it retries with backoff" at WARN should hand controller-runtime a requeue with | ||
that delay instead of the error, and escalate to ERROR itself after a bound, so controller-runtime's own "Reconciler | ||
error" never doubles a retried failure at ERROR. | ||
| ||
- 2026-11-22 12:36Z, ROLL THREE (main 0019f5f5): `test` RED IN 80 S, OUTSIDE THE TREE: THE RUNNER'S OWN PROCESS LIMIT. | ||
SEEN: release green (39 min); `test` rented its VM, then "the guest requester was not associated: | ||
computer.socialweb.vm.associate: the bundle entries: spawn ssh Resource temporarily unavailable (os error 11)", | ||
"stopped", a misleading "killed after 1800s"; promote filed failure without running (nothing reached prod). CAUSE, | ||
read on the runner host (the gateway host, container `knotless-runner`, up 4 days, `--pids-limit 256`): 173 zombie | ||
`ssh` processes, all children of `knotless-run` (never reaped; the oldest as old as the container, the newest 8 h), | ||
pids.current 199 of 256; the association's bundle step spawns several ssh at once and hit the limit (EAGAIN). The | ||
kernel log has "fork rejected by pids controller" for this container's scope. Not this tree: no step of the | ||
composition had started. FIX NOW (ops, no code): `systemctl restart knotless-fork-runner.service` at 12:37Z (state in | ||
`/var/lib/knotless-fork/runner`, sign-in kept): pids.current 11, zombies 0; then `test` and `promote` asked again at | ||
main's head (docs/ci-debug.md "After a red test", case 3). FOLLOW-UP (knotless fork, a plan after the rolls): | ||
knotless-run must wait for the ssh children it spawns (about 40 zombies a day at this week's run count: the limit is | ||
reached after about 5 days up); until then: before a roll, read the container's pids.current | ||
(`docker inspect`/the scope's `pids.current`) and restart the runner when it is over 100. LESSON: the pre-roll checks | ||
covered the bidder hosts' disk and load but not the runner. | ||
| ||
- 2026-11-22 12:37-18:15Z, ROLL THREE (main 0019f5f5): `test` RED TWICE MORE: THE STAGING OVERLAY OVER THE PUBLIC IROH RELAYS. | ||
ATTEMPT 2 (12:37-15:04Z): `gate: staging: 42 of 48 PASS`, wall 6305 s. FAIL 0007:restart "adopt: the host script | ||
did not end in 120 s", 0014:abandon "un: the host script did not end in 120 s", then within 15 s 0014:backtoback, | ||
0014:overcommit, 0014:memfit, 0015:pressure "getRecord failed with NoSubscriber" for bidder-1's own PDS (log | ||
`~/.tmp/roll3/test2.log`; notes `~/.tmp/roll3/rootcause.md`). Both host calls are `bidder.logLines` (BIDDER_LOG): | ||
the bidder account's whole journal since the first core start crosses ssh over the overlay (184 MB in a 7 day | ||
sample) and the test host filters it; 13 to 19 s in the stack9 hand cycle, over 120 s here. The lead started | ||
attempt 3 before pulling attempt 2's artifacts: the rerun pruned them (a mistake: docs/ci-debug.md). | ||
ATTEMPT 3 (15:06Z, cancelled 18:15Z after 3 h): read from inside over `knotless run attach` at 18:07Z: 20 entries | ||
done, 6 PASS, 14 FAIL, the errors all over the list ("the ssh forward through root@10.44.0.2 did not open in 30 s", | ||
"i/o timeout", "the guest did not come on network within 2m0s", "adopt: the host script did not end in 120 s"). | ||
MEASURED on that network (`~/.tmp/ci-attach/*-0019.txt`): from the atproto VM ping loss 95 to 100% to every peer; | ||
TCP connect over swmesh0: bidder-1 <-> bidder-2 7 to 10 ms both ways, bidder-2 -> atproto 8 to 10 ms, atproto -> | ||
bidder-2 2 to 15 s, atproto <-> bidder-1 (both on hat-7) 1 to 11 s, every pair with the gateway 0.5 to 5 s with | ||
failures. meshd on all four VMs: `relay Default, discovery N0` (the public n0 relays), and each logged WARN "The | ||
relay is rate-limiting this endpoint; outbound relay traffic is being throttled" (atproto and gateway 8 s after | ||
their start, the bidders 3 min later). bidder-1 itself: core up, load low, memory and disk fine, 0 interface | ||
errors. So: a pair with a direct path is fast; a pair that falls back to the public relay is throttled, and a bulk | ||
transfer on it starves that endpoint. NOT ESTABLISHED: which pairs are direct and why the others are not (meshd | ||
logs no path kind at INFO; guests sit behind pasta NAT on hosts behind one router), whether healthy networks log | ||
the same WARN (no meshd log of a healthy network was saved), and whether the throttle follows our volume (nine | ||
hand cycles in 24 h) or the relay's own state. The placement the lead first suspected (bidder-1 beside atproto on | ||
hat-7) also held in the green rolls of 93fd8c11 and 324f69a1: withdrawn as the cause. Not this tree: BIDDER_LOG and | ||
meshd's relay default predate it. NEXT: a hand network that measures the overlay (path kind per pair, idle and | ||
under load, relay disabled); the test-host fix `log-filter` 2f399ec8 (the filter runs on the bidder VM: 184 MB -> | ||
2 KB per call, reviewed safe); the relay the overlay uses is an owner decision. Also ready, reviewed: | ||
`watch-cancel` 0e08da46 (kcpapply's short-lived watch logs its own cancel at ERROR; seen once on an old-binary | ||
core). LESSONS: seven hand cycles never ran in CI's placement or with CI's runner; a CI run gives no per-entry | ||
result until it ends: read it from inside (`knotless run attach`) at the first doubt, and pull a failed run's | ||
artifacts before asking for a rerun. | ||
.plans.yaml+3−3
| @@ -16,9 +16,9 @@ plans: | ||
progress: | ||
staging: | ||
target: "CI test 48 of 48 PASS, 0 FAIL (no SKIP state), on step 21's second roll's commit" | ||
now: "45 of 47 in CI on main 7b0afc69 (roll two, test 16:53-19:38Z 11-21: RED: 0008:run (health counted the old process's shutdown errors), 0014:exhausted (cause open)). Last green: 47 of 47 on 324f69a1 (11-21)." | ||
now: "42 of 48 in CI on main 0019f5f5 (roll three, test attempt 2, 12:37-15:04Z 11-22: RED; attempt 3 cancelled at 6 PASS of 20: the staging overlay, Findings). By hand on this tree + cli-help: 48 of 48 (stack9, stg-090117). Last green in CI: 47 of 47 on 324f69a1 (11-21)." | ||
log: "knotless run list knotless://socialweb.computer/socialweb-computer-kcp --commit 7b0afc69f4a2 --workflow test --log" | ||
waits_on: "roll one (this pull): CI test on the landed commit" | ||
waits_on: "the staging overlay (public relay throttling): measurement, then the owner's decision on the relay" | ||
prod_ci: | ||
target: "CI promote 31 of 31 both-entries PASS as prod bob and alice, on step 21's second roll" | ||
now: "31 of 31 PASS in CI at main 324f69a1 (promote v0.0.1-rc3, 07:31-08:39Z 11-21: green, wall 3718 s; stable v0.0.1). Before: 28 of 31 at 93fd8c11 (red, WebRTC from the release VM: fixed by decision 75)." | ||
| @@ -35,4 +35,4 @@ plans: | ||
log: "bash scripts/check-docs.sh" | ||
waits_on: "the two rolls" | ||
status: "steps 0-5 done (promote asked-only; promote.sh --dry-run rehearsed on the release image on main de0b91ee; hand builds to local/, rehearsed: publish+verify local/de0b91ee, ci-staging-binary local:); prod on rc174; no stable" | ||
next: "12:00Z 2026-11-22: owner (asleep): at least one green roll by noon Pacific (20:00Z). This pull = roll one: main + health-pid, bid-capacity, ingress-rereg, signin-retry, stage-budget, the dumbpipe v0.40.1 bump with its guest image (decision 79), the two start-up log levels (bind-label, start-warn; owner: a healed retry is not an ERROR). Out of band: stack6 1c028e5d 48 of 48 (the six fixes); the bump's switch proof on stg-113817 and the ssh gateway in prod's layout; stack9 1d794a7b (this tree + cli-help) gate: staging: 48 of 48 PASS, wall 7632 s (~/.tmp/cycle-stack9/saved/final/). Open, lead's call: one Failed to watch context canceled ERROR on an old-binary core inside 0007:upgrade, first seen, being root-caused during the roll. Then: watch release, test, promote; red promote: roll back at once; green: cli-help (roll two, help text only)." | ||
next: "18:20Z 2026-11-22: roll three (main 0019f5f5): release green; test red three times, none traced to the tree: 80 s on the runner container's process limit (173 zombie ssh; runner restarted); 42 of 48 (two bidder.logLines calls over 120 s, then NoSubscriber); cancelled after 3 h at 6 PASS of 20 (the staging overlay over the public iroh relays, throttled: Findings). Promote never ran; prod on stable v0.0.1. No roll until the overlay is understood: a hand network measures it now. Ready, reviewed, not landed: log-filter 2f399ec8, watch-cancel 0e08da46; roll two tree staged (roll2 ee2c30ce). The relay the staging overlay uses is an owner decision. 0 green rolls in a row." | ||