Acceptance 0014: accept fallback and committed capacity
Proves a bidder never takes more VMs than it has room for, and that a requester whose accept is refused for capacity
gets its VM elsewhere on the same RFP: the bidder answers the accept's submitAccept with HTTP 409
CapacityExceeded (no receipt), the requester keeps that accept (a bidder may still act on it; old records are not deleted), denies the bidder for the contract and takes the
next bid; with no bid left it deletes the RFP and issues a new one without the refusers.
- Code:
internal/market/bidder/vmsize.go(committed: the one count, live ComputeVMs plus the capacity holds of the open provision StateSyncs, one per accept;admitAccept: the one admission, at a submitAccept or a firehose accept, which refuses 409 or writes the hold into the accept's provision StateSync; no capacity check after it),internal/market/requester/runner.go(replaceDeclined,reRFPAfterRefusals, up to 3 replacements) - Test:
tests/compositions/0014(moduletests/modules/acc-0014), entriesrun,concurrent,exhausted,giveup,wait,impossible,backtoback,overcommitandmemfit, through the kcp CLI requester (run request); staging only. The same fallback in the atlibs requester module (the cicd composition's) istests/compositions/knotlessentryfallback(concurrent workflows, one accept refused 409, the checks still succeed, the refused accept kept) - Last pass: see Evidence
sequenceDiagram
participant R1 as rental 1
participant R2 as rental 2
participant X as bidder X (room for 1)
participant Y as bidder Y
R1->>X: RFP 1; X bids (room)
R2->>X: RFP 2; X bids (room)
R2->>Y: RFP 2; Y bids
R1->>X: accept 1 -> reserved, receipt
R2->>X: accept 2 -> 409 CapacityExceeded, no receipt
R2->>R2: requester_submit_accept_declined, requester_accept_declined_replace, keep accept 2, deny X
R2->>Y: accept (Y's bid on RFP 2) -> receipt, Held
Why the rentals are concurrent
A bidder checks capacity twice: when it bids (its live guests plus reservations of other RFPs) and when it gets the
accept (the same, then it reserves). A bidder that is already full when an RFP arrives does not bid at all, so filler
VMs rented first make the requester skip that bidder, never a 409. A 409 needs room taken between a bidder's bid and
the accept: rentals whose RFPs land together, each first picking the bid that arrived first. The test sizes every VM
from the bidders' offerings (maxCpus) less the guests already on their kcp, so each bidder has room for exactly
fit of them.
| entry | bidders | rentals | VM vCPUs | refusal |
|---|---|---|---|---|
run | first 2 of the target | 2 | min(free)/2 + 1 (one per bidder) | a reading: both rentals may pick the same bidder first, and the refused one falls back to the other; up to 2 rounds; the rentals end on different bidders (scaling across bidders) |
concurrent | first 2 of the target | 4, two rounds | min(free)/2 (two per bidder) | a reading; once all four are Held each accept is submitted 5 times more at once to its winner (each answered 202, one ComputeVM per receipt, no bidder over its room); after the release every released accept is submitted 5 times again and the same round runs a second time: all four Held with no wait for capacity (the released room is back, a late re-send holds nothing) |
backtoback | the target's first | 6 (input count), one after another | 1 (input cpus), mem 1G | none: every VM Held |
impossible | the target's bidders | 0 | none | requests nobody can take fail fast, never wait: more vCPUs than the largest offering's maxCpus, and (when no offering sets nestedVirt) a nestedVirt request, and a request whose --expires-in 1s is over before the requester sends an RFP (the contract's deadline: no RFP goes out): each Failed within 120 s naming why ("maxCpus", "nestedVirt", "deadline"), nothing Held, no requester_waiting_for_capacity. The requester judges this over the candidates with a fresh heartbeat: a relay still lists a removed bidder's repo (its heartbeat and offering gone), and that candidate does not make the request wait |
giveup | first 2 of the target | 1 | both bidders full (one pinned 1 h filler each) | one rental with --expires-in 150s has no room: it waits in rounds (at least one requester_waiting_for_capacity) and fails at its deadline with "gave up waiting for capacity after N rounds" |
wait | first 2 of the target | 1 | both bidders full | one rental with --expires-in 15m waits; after two rounds a filler is released: the rental is Held on the freed bidder within 300 s of the release (about one backoff and a boot); readings waitRounds, heldAfterFreeS |
exhausted | first 2 of the target | 3 | min(free)/2 + 1 (one per bidder) | required: one rental is refused for capacity by both (each once: a 409 to its accept, or declined: capacity to its RFP when the bidder was full as the RFP came; readings rounds[].rentals[].capacityRefusals), its bids run out, its new RFP (the refusers excluded, as specified) finds no bid: the rental WAITS for capacity (requester_waiting_for_capacity, rounds with a backoff of 30 s, 1 min, 2 min, a fresh RFP asking every bidder each time) and is released by the test |
| overcommit | the target's first | overcommit + 1 (input overcommit, default 2), at once | 2, with room left for exactly overcommit of them | overcommit Held, the last refused 409 CapacityExceeded by the bidder, which finds no bid and waits for capacity (plan 0041) |
| memfit | the target's first | fit + 1 (fit at most 3), at once, 1 vCPU each | sized from the room the bidder computes (bidder.meminfo: the host's MemAvailable less, per live VM, its committed memory above its cgroup's anon) so it is fit and a half guests | fit Held, the last refused at admission (409 CapacityExceeded naming the memory, or no bid) and waiting; one Held guest released, the waiting one Held within 5 min (owner decision 73: a guest is admitted only when MemAvailable, less the memory every admitted guest has not taken yet, holds it) |
A bidder commits cores x cpu-overcommit vCPUs (default 2), its maxMem in guest memory plus each VM's overhead
(contract/host.VMCommitMiB: the guest's memory plus the larger of 512M and a tenth of it) and its guest quota
(maxConcurrentGuests, 0: none): every entry reads the commitment from the bidder's own Bidder spec.cpuOvercommit (an
input overcommit must equal it) and sizes a bidder's room as the least of maxCpus x that, maxMem and the quota, less
its live guests and the fillers; it fills the vCPUs with pinned fillers of at most maxCpus each (one VM is at most the
cores), and the check after each round fails a bidder over its vCPUs, its memory or its quota. Run overcommit against a
network brought up with --cpu-overcommit N with --input '{"overcommit":N}'. overcommit at 2: two
2-vCPU guests accepted at once, the third declined capacity; at 1 (a network up with --cpu-overcommit 1): the second declined.
The leg "a guest above the cores is declined unservable" is impossible (maxCpus + 1 is under cores x 2, so only the static
check refuses it).
Every rental is pinned to the entry's bidders (--allow-bidder-dids); mem 1G (input mem).
Big bidders (more than 8 free vCPUs, e.g. a dev network's bidders, which offer the whole host) first get one filler
VM each, pinned to that bidder (--allow-bidder-dids <it>), sized to leave room for exactly fit guests of 2 vCPUs
(input cpus); the rentals are then 2-vCPU guests. Without fillers a 16-CPU host gets four 8-vCPU guests booting at
once, and on a loaded host they miss the requester's 2 min on-network window. The fillers are released with the
rentals and their records checked too.
backtoback
VMs booted one after another on one bidder reuse its VM slots: each unit was just reset when the next is placed. Before
f15dda15 the placer took a unit that was merely loaded (inactive) for placed, awaitBoot read "went inactive (result
success, status 0) before its VMM answered" and killed the VM some 100 ms after the hand-off; the requester saw "the
bidder reported vm.terminated before the guest came up". The entry rents, waits for Held, releases, count times,
each pinned to the bidder; a rental that ends before Held is recorded and the run goes on. It needs the bidder's VMs
placed by a service manager (--vm-host-backend systemd-user, as prod): with the proc backend the race does not show.
Inputs: count (6), cpus (1), mem ("1G"), overlap (false: with true the next VM is requested the moment the
previous one is Held, while its release runs, so the next unit is placed as the last is reset). The race is rare (about one
boot in a hundred on prod), so a pass proves little: run with --input '{"count":12,"overlap":true}'.
Pass: every VM Held (none Released or Failed before it, onNetwork in the readings for the ones that ended), the
bidder's log gains no "before its VMM answered" line over the run (bidder.logCount: the network bidder's core.log, an
installed bidder's journal), every record gone.
Preconditions
- A market network the run owns: staging (
tests/compositions/staging,bidders>= 2) or a dev market network (dev up --profile market-network --bidders 3), its target file fromtests/compositions/staging/target.sh - Nothing else renting on those bidders during the run (the sizing reads their live guests once, at the start)
- The filling entries are
stagingintests/entries.txt(the gate does not run them on prod): they fill bidders to capacity, and prod bidders are shared
Steps
cd tests/modules && cargo wasm && cd ..
cd compositions/0014 && deno run -A --no-lock compose.ts > composition.json && cd ../..
SWC_TARGET=<staging target.json> cargo run -p swc-test-host -- compositions/0014 --account alice1 --entry run
SWC_TARGET=<staging target.json> cargo run -p swc-test-host -- compositions/0014 --account alice1 --entry concurrent
SWC_TARGET=<staging target.json> cargo run -p swc-test-host -- compositions/0014 --account alice1 --entry exhausted
A dev market network on a host (hat-8: --asset-cache under /var/tmp, the default cache is not traversable by the
VM user):
B=<socialweb-computer>; N="--instance capt --state-dir /var/tmp/capt/net"
$B dev up $N --profile market-network --bidders 3 --network-timeout 15m --asset-cache /var/tmp/capt/assets
SWC=$B SWC_STATE=/var/tmp/capt/net SWC_INSTANCE=capt SWC_CREDS=/var/tmp/capt/creds \
tests/compositions/staging/target.sh capt0001 > /var/tmp/capt/target.json
$B dev down $N
Expected observations
- Each bidder's offering record
com.publicdomainrelay.temp.market.offering/selfcarriesmaxCpus(the host's CPUs) - The refused rental's request log, in order:
requester_submit_accept_declined(status409,body.errorCapacityExceeded,body.message"... running and reserved guests hold N of its M"),requester_accept_declined_replace(bidder), thencontractHeld with a receipt from another bidder; with no bid left,requester_bids_exhausted_new_rfp(declined) and a new RFP - The refusing bidder's log:
bidder_accept_refused_capacity(the async provision path, from the accept record on the firehose, and thesubmitAcceptpath)
Pass criteria
- Every rental Held (
exhausted: all but one), each on one of the entry's bidders and never on a bidder that refused it exhausted: the one rental left without a VM was refused for capacity by every bidder of the entry (a 409CapacityExceededto its accept, orrequester_bidder_declinedkindcapacityto its RFP: the same check,vmsize.gocapacityRefused, at either time; which one is the order the RFP and another rental's admission reach the bidder), each before its first wait (a waiting round's RFP asks every bidder again and does not explain the wait); after a 409 it loggedrequester_bids_exhausted_new_rfp(with no bid at all there is no accept and nothing to ask again); and then it waits for capacity (at least onerequester_waiting_for_capacityline) instead of failing; the test releases it. A bidder that did not answer (unreachable,requester_submit_rfp_failed, or a bid dropped after submitRfp with no answer, which the bidder no longer does: plan 0040 Findings "THE BID DECISION MAKES NO CAPACITY DECISION") fails it- No bidder over its vCPUs: this run's VMs on a bidder fit its free vCPUs; its kcp's live ComputeVMs'
spec.vm.cpussum to at most itsmaxCpus - Every refusal is a 409
CapacityExceeded; every refused accept record is kept (x3), the rule above run: at least one refusal (readings:rounds[].refusals,maxRefusalsOneRental: 2 is a rental refused by its first two picks and Held on the third)- After the release: every record of every rental RecordNotFound, no ComputeVM of the run's receipts on any bidder kcp within 3 min
Cleanup
The run releases every rental (run request release) and checks; --on-fail freeze keeps them for attach, then
cleanup releases them.
Known pitfalls
| Symptom | Cause / fix |
|---|---|
run fails "no accept refused in 3 rounds" | every round's first picks were distinct (a race); run again. A requester picks the first free bid that arrives |
| "the bidders have room for N VMs" | guests of another run on the bidders; wait for them to go or use another network |
a rental fails requester_no_bidders after requester_bids_exhausted_new_rfp in run or concurrent | every bidder refused it: a bidder had less room than its live guests show. Before 91103758 a reservation whose ComputeVM was released before the bidder's next capacity check counted for its whole 10 min hold (phantom "hold 16 of its 16" with one VM); run the entries back to back only on bidders with that fix |
the guest did not come on network within 2m0s | a loaded host booting several large guests at once (hat-8, four 8-vCPU guests, 2m10s); rerun when the host is quieter |
Evidence
Dev market network capt on hat-8 (3 bidders, --vmm qemu, 16 CPUs each in their offering), binary bcf42c0d
(capacity 8b04fd2c + the reservation fix 91103758), test host and modules of this branch, account alice (generated):
-
concurrentPASS 2026-11-13 21:38:47-21:40:58Z: fillers 12 vCPUs on each bidder (Held 35 s), four 2-vCPU rentals Held in 56 s; one refused 409 by bidder zdt45... (requester_submit_accept_declined,requester_accept_declined_replace) and Held on tngqo...; 2 per bidder, liveCpus 16/16 on both; 19 records gone, no ComputeVM left -
runPASS 2026-11-13 21:41:08-21:44:43Z: fillers 14 vCPUs on each of 3 bidders, three 2-vCPU rentals Held in 73 s; rental 1 refused by lqkfi... then tngqo... (two 409s) and Held on zdt45... (the third); rental 2 refused once; 3 refusals, liveCpus 16/16 on all three; 21 records gone -
exhaustedPASS 2026-11-13 21:51:55-21:53:35Z: fillers 14 vCPUs on each of 2 bidders, three 2-vCPU rentals; two Held (one per bidder, liveCpus 16/16), rental 0 refused 409 by both,requester_bids_exhausted_new_rfp, its new RFP Failed at once withrequester_no_bidders(both "denied by the contract"); 12 records gone (the old RFP too) -
Env failures on the way (not the code under test): four 8-vCPU guests at once on a loaded hat-8 missed the 2 min on-network window (hence the fillers); one
exhaustedtry lost a 2-vCPU guest the same way (21:48Z) -
Found by the first runs (22c89875): a reservation whose ComputeVM was released before the bidder's next capacity check counted for its whole 10 min hold: bidder-2 refused 409 "hold 16 of its 16" with one live VM, a rental was refused by both bidders and failed (fixed in 91103758)
-
port-wait-for-capacity (requester that waits for a slot), capt dev network (hat-8, 2 bidders, 2026-11-14), CLI and net core = branch 77029b12 vs main 3c725c6f:
exhaustedPASS 2/2 (the unplaced rental waits, released by the test) and FAILS on main ("failed instead of waiting for capacity: requester_no_bidders");giveup(both bidders filled by pinned fillers, rental--expires-in 150s) PASS 2/2 with "gave up waiting for capacity after 2 rounds" after onerequester_waiting_for_capacityline, FAILS on main (Failed after one 30 s round, no "gave up");wait(--expires-in 15m, a filler released after two waiting rounds) PASS 2/2: Held 77.1 s after the release (readingsheldAfterFreeS,waitRounds2,waitSlot2-vCPU), FAILS on main (the rental fails after one 30 s round);concurrentandbacktobackPASS on the fix,runneeds 3 bidders (skipped on this 2-bidder net) -
fail-fast-nobody-serves (port-wait requester that tells "no capacity" from "nobody can serve"), capt dev network (hat-8, 2 bidders, 2026-11-14), fix 9113a163 vs main aeb3ada0:
0016:badshafix PASS 41.2 s and 42.1 s (the error names why:no bidder can serve this request: <bidder>: guestimage: the requested image is refused: ... hashes to ..., not the requested ...), main did not finish in 420 s (it waits for capacity);impossible(17 vCPUs against maxCpus 16) fix PASS: Failed 10 s after the request,requester_no_bidders: ... the offering's maxCpus is below the request: 16 (the request asks 17), main FAIL (requester_waiting_for_capacityround 1);wait,exhausted,giveupstill wait on the fix (PASS 238.7 s, 97.4 s, 213.4 s). Not provable on this net: the nestedVirt case (both dev bidders offer nestedVirt), an image request against an offering withoutimages(both set it), a bidder too old to answerdeclined
abandon (an accept that cannot be read)
0014:abandon submits an accept to the first bidder whose record does not exist (what a requester that deleted its accept before the
bidder read it leaves): the bidder retries the provision (bidder_accept_unresolved: kind notFound or unreachable, the tries and the
age) and, after 5 failed tries and an intent 3 min old (30 min for an unreachable PDS), abandons it: one bidder_accept_abandoned
(accept, kind, tries, age, reservationReleased), its reservation released, the intent Obsolete (the outbox never retries it) and
the accept remembered; nothing more is logged for it. About 8 min on a dev net (the outbox stalls after its first burst of retries and tries again 5 min later; 328 tries by then).
Pass: exactly one abandon within 9 min, no the provision of an accept failed line, no retry line in the 2 min after.