chadtmiller.com/knotless
git that lives in your Atmosphere account
Gateway pushes log each part's time; landings read a space side by side
MergedThis adds a gateway push timing line, reads the queue once in landing_refusal, and reads a private space's collections eight at a time. I found nothing that has to change.
space_entries:bufferedkeeps the read order. Entries therefore still come out member by member, collection by collection, as before, and reactions are still filtered by their subject afterwards. Concurrency is limited toSPACE_READS.member()is now called for all DIDs at once, which is fine.landing_refusal: it now reads the queue beforelands's early returns and before its own refusals for non-landing shapes (several moves, force). A push that is refused anyway now pays one queue read it didn't before. That is rare and cheap next to what it saves, so I'd leave it.Timedandpush_by:Timedlogs fromDrop, so every exit is covered._turnis declared aftertimed, so the lock is released before the line is written. Bothpush_bycallers pass aSpent.pack_mscan't go negative, because the upload time is always part of the elapsed time.- Docs and changeset: the gateway and observability docs and the changeset are included.
I did not build or run this, so I'm relying on CI for compile, clippy and the coverage gate.
The only change since my approval is in space_entries. The per-collection read moved into a listed helper, and the futures are now built before they are awaited so the request future stays Send. buffered keeps the order, so the zip of collections with results still lines up. The behaviour is unchanged. I did not build or run it, so I'm relying on CI for the compile check.
The only change since my approval is in Target::entries. The owner's blocks are now read before the space entries instead of beside them. Behaviour is the same, and the only cost is a little latency. It also removes a possible Send or borrow problem with the joined futures. The side-by-side reads inside space_entries stay. The PR text still says entries reads the owner's blocks beside them, so update that line. I did not build or run this.
The only change since my approval is in the changeset. It no longer says the owner's blocks are read beside the space entries, which matches the code. That was the stale line I flagged last round. Nothing else changed, and I did not build or run it.
8 passed, 3 skipped
Pushes through the gateway took 5 to 16 s on knotless.dev with nothing in their traces saying where. Each now logs one
gateway pushline with what each part took, and the part that looked slowest, a runner landing a private repository's queue reading every member's records in its space one round trip after another, twice, reads them once, eight at a time.Claude Opus 5.5 · high
What changes
A
gateway pushline for each push.dev.knotless.pushRefslogs, however it ends (answered, refused, or dropped by a client that left), the repository, the caller (did), the refs (count), the pack's size (bytes),duration_ms, and each part in turn:auth_ms(the caller's token),turn_ms(waiting for the repository's turn),records_ms(reading it as its owner),mirror_ms(syncing the gateway's mirror),index_ms(indexing the pack sent),gate_ms(branch rules, the merge gate, a runner's landing check), and frompush::push_by,pack_ms(checking the updates, packing what they bring),upload_ms(the pack's blobs) andwrite_ms(applyWrites). A part never reached is left out, so the part a push stopped in is whatduration_mshas beyond the rest. At info, at warn when slow, asmirror syncis.push_bytakes a&mut push::Spentto say what its parts took.A runner's landing reads the queue once.
queue::landing_refusalread every forge record of the repository forauto_merge::lands, then again for the batch check. It now reads them once and hands them tolands.A private repository's forge records are read side by side.
Target::space_entriesread each member's eleven collections (the ten forge ones and reactions) one after another, eachlist_alla round trip at least; it now reads them eight at a time, as a fetch downloads packs. The CLI, the runner and the gateway all read private pulls and issues through this.Why
Prod evidence (
admin_logs,admin_stats, 2026-11-10):pushRefshad a 3.4 s mean over 607 requests, 378 over 2 s;gateway.push4.0 s mean, from 3 accounts. The slow ones in the last hours, 15.2 s (trace 072f43585a0885da4323eae33976f04d), 14.2 s (025f44e9d440399f47d3f387c2de5733) and 11.3 s (c3595f3ed9e660fa5395dfa7deeff146), were alone in their minutes, so not waiting on another push's turn, and their traces held only the request line, but for oneuploadBlobof 2.1 s in the 14.2 s one. The trace is not lost: thatuploadBlobline carries it. The calls are simply underKNOTLESS_SLOW_MS(1 s) each, so at debug, and many of them add up. The one public push via the gateway in that window (the runner landing knotless's release, 2.7 s, a7291ad5af9a84d9371a3481803d9f13) shows the shape: tengetForgepages read one after another for the landing check.The slow pushes are to spec-pds, private, whose only writer besides its owner is the runner
did:plc:4bocpkn5zkwe4wuu4plrj5kt(itsmainis aqueuebranch): they are the runner landing the merge queue. For a private repository that check reads the whole space as the owner: each member's eleven collections in turn, a call each to a Workers PDS whose calls take hundreds of ms, and before this did it twice for a one-ref landing.knotless pull listof spec-pds, the same read, takes 7.6 s from here. That is most of a 10 to 15 s push. Reading them side by side brings the read to a few rounds; the line then shows what is left (gate_msfor this,write_msfor the PDS's own writes).Not changed: the turn lock is held across the push's calls on purpose (the merge gate and landing read the repository as
writerread it, under the turn); the gateway's mirror settling to the snapshot's refs is inmirror::settle, being changed elsewhere, and runs in a task of its own, so its git lines are in no request's trace.Changeset in
.changeset/private-forge-reads-side-by-side.md.How it was checked
Not built or tested on this machine, as the repository asks; CI ran check, test (with the coverage gate), web, lexicons, shots, docs and e2e, all passing. The gateway's push tests run the new line on every push; the private-repository flows run
space_entries. A first version also read the owner's blocks beside the space's records; with it,assignees_and_reviews_maintainers_ask_for(a public repository's, which that path does not touch) failed in both harness runs, reading the appview before it had heard of a write; without it the test passed, so it was left out.Shortcomings
The phase timings name where a slow push spends its time; whether the side-by-side reads bring spec-pds pushes down as far as expected shows only in the next
gateway pushlines on prod (look atgate_ms). Each pull waiting in the queue still opens its head repository in turn during the landing check. A space whose PDS limits concurrent reads below eight would be read more slowly than this expects.Review
Start at
pushin crates/knotless-serve/src/gateway.rs andTarget::space_entriesin crates/knotless/src/forge.rs.