0
0
ci-debug: mistakes 14 to 18 from roll three
docs/ci-debug.md+6−0
| @@ -497,6 +497,11 @@ The lead's own, on plan 0040 (2026-11-20). Each has the rule it left; plan 0040 | ||
| 11 | Accepted a test entry whose precondition was "the previous release lacks the feature" (`0007:fence` took its from binary from the previous rc) | The entry failed for ever the moment its own roll went green; caught before the next roll, one more fix | Ask of every new entry: does it still pass after it lands, on the roll after, and after a rollback | | ||
| 12 | Added a pinned-binary bump (dumbpipe) to a fix stack late, proven by pins and a file diff, without switching a running bidder between the two builds | The full hand cycle found both bidders deaf over iroh after the switch, in both directions: on prod it would have been an outage at the roll and at the rollback | A change of a pinned binary or of anything a core stores across restarts is proven by `0007:upgrade` alone first, both directions, before it joins any stack | | ||
| 13 | Kept adding fixes to the stack as I found intermittents | Five hand cycles and no second roll in a day | Freeze the stack once it holds what failed; anything new that is not a regression of the stack goes to its own roll | | ||
| 14 | Rolled after checking the four bidder hosts but not the runner | `test` red in 80 s: the runner container held 173 zombie ssh, 199 of its 256 processes | Before a roll read the runner container's `pids.current` on the gateway host; over 100: `systemctl restart knotless-fork-runner.service` (no run active) | | ||
| 15 | Asked for a rerun of a red `test` before pulling its artifacts | The rerun pruned them; the bidder's side of the failure was never read | Pull a failed run's artifacts (`knotless run artifacts`, a fork-built CLI) before any rerun | | ||
| 16 | Waited on a CI `test` by its state word only | Three hours on a run that was 6 PASS of 20 | At the first doubt read the run from inside: `knotless run attach`, then `swc-test-host attach --ci` (state, the gate dir's RESULT lines); a pass ends near 2 h | | ||
| 17 | Took seven hand cycles as a rehearsal of CI | CI's placement (its own VM on the hosts) and its runner were never exercised by hand | Before the roll of a batch, one hand network in CI's layout with the overlay measured (pair matrix); save a meshd log of a healthy network | | ||
| 18 | Named a cause (VM placement) from one run's layout | Two green rolls had the same layout | Compare with the runs that passed before naming a cause | | ||
| ||
Before a pull is queued, each of these is a yes: | ||
| ||
| @@ -506,6 +511,7 @@ Before a pull is queued, each of these is a yes: | ||
- Anything that differs on prod (hosts, routers, repo sizes, a requester inside a guest, guests booted by the old | ||
bidder) has a rehearsal that has that difference, or is written in the pull as unproven with the reason. | ||
- The tree in the pull is the tree that passed. | ||
- The runner container's process count was read (mistake 14); a healthy network's overlay (meshd) log is saved. | ||
| ||
## Do not waste time | ||