"8a1b58dd67c32c937c4c3257ece9dcbcbd53d87c92e1749906da49a6489bc49b": "cloud-init --version: the base carries cloud-init, checked in its dpkg status",
"fe76a87a62dc5484bc6e777dc04b55c15c3f2eebd2bf7345f9788fbec438b140": "dumbpipe v0.39.0: baked from the versions.json dumbpipe pin of the image's arch as the asset /usr/local/bin/dumbpipe, with dumbpipe.version from the overlay"
"9f6d4e35fa37405a63e3a37936db9d5751dba5b74e4163362590a53fe74bd7d1": "dumbpipe v0.40.1: baked from the versions.json dumbpipe pin of the image's arch as the asset /usr/local/bin/dumbpipe, with dumbpipe.version from the overlay"
@@ -188,6 +188,41 @@ The guest requester's own atirc token mint answers 403 "association not verified
(0005's note; the requester retries it, the rental does not wait on it): the entry counts those lines
(`readings.atirc_mint_403`) instead of failing fast on them.
## The market endpoint over iroh survives a new ingress instance (`0007:ingress`, staging)
A bidder's market endpoint over iroh is the Exposure `svc-bidder-bidder-iroh`, published through the IngressProxy
`ingress-iroh` (a dumbpipe listener per registration). Any new generation of that IngressProxy, and a rebuild after its
service exited, makes the core's runtime stop the service instance and build a new one with no registration; the
exposure publisher re-registers on the new instance (its fingerprint holds the instance, `component.Entry.Attempt`) and
the Exposure is Ready only once dumbpipe printed its ticket. Before the fix the publication stayed on the dead instance
and the bidder was deaf over iroh until its core restarted (docs/plans/0040-findings.md, "A BIDDER GOES DEAF OVER IROH
WHEN ITS STORED DUMBPIPE PATH CHANGES").
Test: `tests/compositions/0007` entry `ingress`; staging only (it replaces each bidder's IngressProxy), exclusive. The
trigger is the bidder root's own component override, the one kube write the test host may make on a bidder (its guard:
a patch of the MarketBidder, restored after): `spec.components["IngressProxy/ingress-iroh"].settings.ingressProxy
.keepaliveMilliseconds` (an xrpc-subscription setting the iroh provider never reads). The compose pass rewrites the
IngressProxy spec, the generation changes and the core's runtime (`internal/component/runtime.go` Reconcile: generation
changed, stop, start) replaces the instance while the core stays up: the path a compose rewrite of
`spec.iroh.dumbpipePath` takes after a switch between binaries whose pinned dumbpipe differs, without changing the pin.
A core restart (`0007:restart`) is no such trigger: a new process starts a new publisher with nothing published; an
upgrade (`0007:upgrade`) is one only when the stored spec differs (the first start of a release on the previous one's
stored path). There is no operator command for one component of an installed bidder (`dev restart <component>` reads a
dev instance's record) and no kubectl on the staging bidders. Host call `bidder.dumbpipe {name}` (the dumbpipe children
of the bidder's run bidder process: pid, executable, `--host`, read from /proc over ssh, no sudo); `request.start` takes
`bidderEndpoints` (SOCIALWEB_REQUESTER_BIDDER_ENDPOINTS of that run request). Per bidder, three rounds (keepalive 6000 ms,
6001 ms, then the override as it was), each:
- the MarketBidder patch; then, read every 3 s for at most 180 s: an `ingressproxy_iroh_started` after the patch, an
`ingressproxy_iroh_dumbpipe_listening` of `svc-bidder-bidder-iroh` after the LAST started, and the IngressProxy Ready
at its generation (`readings.rounds`: seconds to heal, lines, generation, address)
then a live dumbpipe child of the core on the address of the last listening line (`readings.dumbpipe`), and a run
limited to that bidder (`--allow-bidder-dids`, bidder endpoints `iroh` only, `--exec -- id`) exits 0 (its submitRfp
over iroh reached the bidder and was answered) and leaves no record. Any failure while an override is applied (a round
that does not heal, the deadline, a failed call) puts it back first; a run killed from outside (SIGTERM, on-fail cleanup)
is put back by the test host's end: the kube transport's ledger restores every spec the run patched (`kube.cleanup`).
## Preconditions
- Authority: the owner asked for it, or the VM is matched to a contract you are ending (stray test / orphan). Never an owner-held VM unless the owner says so
@@ -258,4 +293,4 @@ Bidder journal, in order (hat-7 example, 2026-10-31 22:51Z):
| token still answers | token was plain (pre-lease or test mint); revoke it by subhandle |
| ComputeVM `Lost` after `uninstall --force` | expected: forced uninstall does not run the finalizer path; do not use it to end one VM |
| requester resumes later and re-releases | harmless: release is idempotent; the contract is already Released |
| `--expires-in` round fails `no bids received within 30s`; bidder `bidder_payload_invalid` "the RFP payload's expiresAt ... has passed" (2026-11-12) | the CLI fixes `expiresAt` when the invocation starts, before the QR sign-in; a sign-in later than the duration expires the RFP. Sign in right when the QR url appears (it can take 2+ min) |
| `--expires-in` round fails `no bids received within 30s`; bidder `bidder_payload_invalid` (since branch bid-capacity: "a bid failed" reason `DeclinedUnservable` at submitRfp, `bidder_payload_refused` kind unservable for an RFP seen over the firehose only) "the RFP payload's expiresAt ... has passed" (2026-11-12) | the CLI fixes `expiresAt` when the invocation starts, before the QR sign-in; a sign-in later than the duration expires the RFP. Sign in right when the QR url appears (it can take 2+ min) |
@@ -47,7 +47,7 @@ from the bidders' offerings (`maxCpus`) less the guests already on their kcp, so
| `impossible` | the target's bidders | 0 | none | requests nobody can take fail fast, never wait: more vCPUs than the largest offering's maxCpus, and (when no offering sets nestedVirt) a nestedVirt request, and a request whose `--expires-in 1s` is over before the requester sends an RFP (the contract's deadline: no RFP goes out): each Failed within 120 s naming why ("maxCpus", "nestedVirt", "deadline"), nothing Held, no `requester_waiting_for_capacity`. The requester judges this over the candidates with a fresh heartbeat: a relay still lists a removed bidder's repo (its heartbeat and offering gone), and that candidate does not make the request wait |
| `giveup` | first 2 of the target | 1 | both bidders full (one pinned 1 h filler each) | one rental with `--expires-in 150s` has no room: it waits in rounds (at least one `requester_waiting_for_capacity`) and fails at its deadline with "gave up waiting for capacity after N rounds" |
| `wait` | first 2 of the target | 1 | both bidders full | one rental with `--expires-in 15m` waits; after two rounds a filler is released: the rental is Held on the freed bidder within 300 s of the release (about one backoff and a boot); readings `waitRounds`, `heldAfterFreeS` |
| `exhausted` | first 2 of the target | 3 | `min(free)/2 + 1` (one per bidder) | required: one rental is refused by both, its bids run out, its new RFP (the refusers excluded, as specified) finds no bid: the rental WAITS for capacity (`requester_waiting_for_capacity`, rounds with a backoff of 30 s, 1 min, 2 min, a fresh RFP asking every bidder each time) and is released by the test |
| `exhausted` | first 2 of the target | 3 | `min(free)/2 + 1` (one per bidder) | required: one rental is refused for capacity by both (each once: a 409 to its accept, or `declined: capacity` to its RFP when the bidder was full as the RFP came; readings `rounds[].rentals[].capacityRefusals`), its bids run out, its new RFP (the refusers excluded, as specified) finds no bid: the rental WAITS for capacity (`requester_waiting_for_capacity`, rounds with a backoff of 30 s, 1 min, 2 min, a fresh RFP asking every bidder each time) and is released by the test |
| `overcommit` | the target's first | `overcommit` + 1 (input `overcommit`, default 2), at once | 2, with room left for exactly `overcommit` of them | `overcommit` Held, the last refused 409 `CapacityExceeded` by the bidder, which finds no bid and waits for capacity (plan 0041) |
| `memfit` | the target's first | fit + 1 (fit at most 3), at once, 1 vCPU each | sized from the room the bidder computes (`bidder.meminfo`: the host's MemAvailable less, per live VM, its committed memory above its cgroup's anon) so it is fit and a half guests | fit Held, the last refused at admission (409 `CapacityExceeded` naming the memory, or no bid) and waiting; one Held guest released, the waiting one Held within 5 min (owner decision 73: a guest is admitted only when MemAvailable, less the memory every admitted guest has not taken yet, holds it) |
@@ -129,8 +129,14 @@ $B dev down $N
## Pass criteria
- Every rental Held (`exhausted`: all but one), each on one of the entry's bidders and never on a bidder that refused it
- `exhausted`: the one rental left without a VM was refused (409) by every bidder of the entry, logged
`requester_bids_exhausted_new_rfp`, and then waits for capacity (at least one `requester_waiting_for_capacity` line) instead of failing; the test releases it
- `exhausted`: the one rental left without a VM was refused for capacity by every bidder of the entry (a 409
`CapacityExceeded` to its accept, or `requester_bidder_declined` kind `capacity` to its RFP: the same check, `vmsize.go`
`capacityRefused`, at either time; which one is the order the RFP and another rental's admission reach the bidder), each
before its first wait (a waiting round's RFP asks every bidder again and does not explain the wait); after a 409 it logged
`requester_bids_exhausted_new_rfp` (with no bid at all there is no accept and nothing to ask again); and then it waits for
capacity (at least one `requester_waiting_for_capacity` line) instead of failing; the test releases it. A bidder that did not
answer (unreachable, `requester_submit_rfp_failed`, or a bid dropped after submitRfp with no answer, which the bidder no
longer does: plan 0040 Findings "THE BID DECISION MAKES NO CAPACITY DECISION") fails it
- No bidder over its vCPUs: this run's VMs on a bidder fit its free vCPUs; its kcp's live ComputeVMs' `spec.vm.cpus`
sum to at most its `maxCpus`
- Every refusal is a 409 `CapacityExceeded`; every refused accept record is kept (x3), the rule above
- stack7 43297eda (bump6 + cli-help), stg-513391: stopped as a survey at 10 of 48 done, 9 PASS, 1 FAIL
(09:01Z; `~/.tmp/cycle-stack7/saved/final/`); iroh.service restarts 0 and "Endpoint dropped" 0 on all four
VMs in 54 min. FAIL 0006:run:
one start-up ERROR on bidder-1's first start, "the HostProcess supervisor stopped; starting it again" (sets
`provider` and `applications` both apply APIBinding `policy`; kcp refuses one when they overlap; healed 2 s
later). On main since 2026-10-21; first hit. Fix (owner: a healed retry is not an ERROR): `bind-label`
287b641c: WARN while retrying, ERROR once a failing streak lasts the set apply's deadline (1 min). The four
network VMs fetched dumbpipe v0.40.1 in 0 to 1 s each at stand-up.
- stack8 70321d69 (bind-label + cli-help), stg-858914: stopped as a survey at 3 of 48 done, 2 PASS, 1 FAIL (0006:run on bob3: that line; 09:33Z; `~/.tmp/cycle-stack8/results.txt`). Seen at
its start: one start-up ERROR on bidder-2's first start, "Failed to watch ... the server could not find the
requested resource ... *v1alpha1.Service" (the exposures virtual workspace serves discovery about 0.6 s before
it serves the resource; the reflector retried and healed in 0.4 s). Fix: `start-warn` 00e47278: that reflector
line logs WARN while its streak is under the cache-sync deadline (2 min), ERROR after. No timing shift from the
bump was found (10 local starts each on the stack6 and stack8 release builds); the races follow host load.
Sweep of every saved journal: no other ERROR within 600 s of a bidder core start (`~/.tmp/bind-label/sweep.txt`).
- stack9 1d794a7b (this tree 00e47278 + cli-help): gate: staging: 48 of 48 PASS, wall 7632 s (stg-090117,
09:45-11:52Z; `~/.tmp/cycle-stack9/saved/final/`): iroh.service restarts 0 and "Endpoint dropped" 0 on all
four VMs; dumbpipe 0.40.1 never exited under a running core (0.39.0 did 3 times under the old binaries
inside 0007); the APIBinding race fired once on bidder-1 at WARN and 0006:run passed (startupErrors 0)
- one ERROR "Failed to watch ... context canceled" (controller marketbidder, an apibindings watch) 4 s after
the start of an OLD-binary core (324f69a1) inside 0007:upgrade on stg-090117: no health reading counted it;
first seen; the same controller code is in this tree, cause not established: being root-caused while the
roll runs (lead's call, owner asleep: his order is one green roll by noon);
This one cycle is the out of band run for both pulls (decision 79).
- Open through the next roll:
- a guest still fetches dumbpipe from GitHub at boot when its requester and its image have different versions:
CI's 4 network VMs at stand-up, an old stable CLI on updated bidders, new requesters after a bidder rollback;
setup-iroh's fallback to the image's own dumbpipe is proven locally only;
- an installed ssh gateway re-rendering its iroh-dial HostProcess to the new dumbpipe tree across the upgrade:
proven in prod's layout on hat-8 (`~/.tmp/gw-dial/`): `systemctl restart` of the `--fg` gateway re-renders
the dialer onto the 0.40.1 tree (extracted before the dialer starts), dialer down about 21 s, 0 ERROR, new
connections to 0.39.0 and 0.40.1 listeners PASS, rollback the same; an open session breaks at the restart as
on every release; one host, no real signed-in ssh session
- 0007:upgrade does not assert per-bidder iroh reachability of the from binary (follow-up);
- three outer bounds of the test host shorter than their step's stated worst case (`dev up` 200 s, requester stop
120 s, page-round 900 s): latent, never observed; raising them alone was reviewed NOT SAFE (Findings);
- provision StateSyncs are never deleted; the local market delivers no RFP over the market stream, so a change
to RFP intake is proven on staging only.
- Next: the roll of this pull (release, test, promote); then `cli-help` (decision 74) for the second roll.
- Landed without CI (docs, site): the site from `site/` with `/install.sh`, white pages, build-time highlighting,
the lexicon and XRPC reference, lexicon tables and glossary generated from `lexicons/`; the flow map; plan 0053
(rename to `computer.socialweb.*` and `goat lex publish`, after Done when; decision 59).
@@ -137,7 +114,7 @@ green only when both are):
release gate checks, the one build, publish bucket main/<sha>/binaries/{..., SHA256SUMS}
| green
test one ephemeral staging network; every entry
below (46), as staging's own accounts
below (48), as staging's own accounts
(drivers alice1..alice4)
| green
promote on the release VM, as socialweb.computer (release-workload):
@@ -220,7 +197,7 @@ DNS (`/etc/systemd/resolved.conf.d/public.conf`): the region's resolver answers
## Entries
`tests/entries.txt` (47; `tests/gate.sh` reads it): *staging* entries run on staging only (16); the rest run on both (31). 0010 (iOS) was
`tests/entries.txt` (48; `tests/gate.sh` reads it): *staging* entries run on staging only (17); the rest run on both (31). 0010 (iOS) was
removed (owner, 2026-11-17).
| Entry | Proves |
@@ -240,6 +217,7 @@ removed (owner, 2026-11-17).
| 0007:webrtc | getSecrets over the WebRTC channel (iroh off by the contract's user_data): log and contract say webrtc, files written, nothing left |
| 0007:upgrade | *staging* (every bidder changes binary): a guest placed before its bidder's self-update still gets its binaries from the new core and is Held |
| 0007:fence | *staging* (a bidder changes binary under a running guest): the new core reloads the guest's nftables rules (decision 75); from inside the guest, WebRTC to a guest of the same bidder fails before and works after; its open ssh survives |
| 0007:ingress | *staging* (replaces each bidder's IngressProxy `ingress-iroh` by its MarketBidder's component override, restored): after the last of three replacements the market Exposure listens again (dumbpipe_listening after the last started, a live dumbpipe child) and a run limited to that bidder over iroh exits 0 |
| 0008:run | a VM's timeline on its host, release, vm.terminated, records gone, orphan check on every host |
| 0009:run | the browser page rents per bidder over webrtc, a shell over the datachannel, nothing left |
@@ -335,3 +335,9 @@ The 256 byte cap of an image marker: `RequestedMarkerMax` (requested.go), `MaxGu
- Threat: anyone holding a valid service-auth token for the relay (any DID; aud the relay, lxm the method) asks a nonce for another DID, signs it with its own key and subscribes as that DID; the relay then routes `<that DID's label>.xrpc.fedproxy.com` (a bidder's or requester's market endpoint: submitRfp, submitAccept, submitEvent, getSecrets with the guest's workload token) to it, until the DID's own subscriber connects again and replaces it. The relay's DID allowlist (`didAllowed`), when set, limits which DIDs, not who may claim one. Read from the code by the lexicon sweep (2026-11-19, `lexicons/com/fedproxy/temp/xrpc/registration.json`), not tried.
- Status: proposed; not built.
## Guest boot: the dumbpipe fetch checks no sha256
- What: the guest's cloud-init install script checks the dumbpipe archive against the versions.json sha256 of its arch (as `deploy/bidder/preinstall-market.json` already does for the image build) and refuses a mismatch.
- Where: `internal/atproto/cloudinit/modules.go` (the dumbpipe install script: `curl`, `tar -xzf`, `install`; no `sha256sum -c`). It runs only when the image's `/usr/local/bin/dumbpipe.version` differs from `cloudinit.DumbpipeVersionDefault` (a guest booted from an older image than its bidder's cloud-init).
- Threat: a changed or substituted release asset on GitHub (or a path to it) becomes the guest's ssh listener (`iroh.service`, root) with no check; the image build and the host embed are pinned, this path is not.