chadtmiller.com/knotless

Switch repository

Loading…

What changes

At most four diffs are read at once. files in crates/knotless-serve/src/web.rs, which getCommit, getThread (a pull's files), getPullDiff and getCompare all read a diff through, takes a turn of a semaphore of four (DIFFING) before it runs git, as blames (BLAMING) and archives (ARCHIVING) already do. The rest wait their turn.

A diff's rows are highlighted off the async threads. Drawing the files with their rows, which parses the whole of each side with tree-sitter, runs in spawn_blocking, as getLastCommits' walk and the cards' drawing do.

docs/design/serve.md says both, under Highlighting and diffs, and the Crawlers section points there.

Why

From admin_logs, 2026-11-08 03:11:20 to 03:12:25 UTC (the bulk of the burst; four stragglers to 03:46):

• Shape. Request lines over 2 s overlapped 75 deep at 03:12:11; git lines over 1 s (only slow ones are logged) overlapped 63 deep at 03:11:57, 294 of 313 on the one mirror of chadtmiller.com/knotless. The slow page lines came from five browser agents, the reduced Android "Linux; Android 10; K" Chrome 131 for 33 of them and Windows Chrome 144 to 149 and macOS Chrome for the rest, on blob, check, commits, blame, issue and pull pages. No request line carries the client's address, so how many hosts it was cannot be told; the slow commit pages' own lines are mostly missing, as a request whose client hangs up logs none, while the pages' getCommit behind them went on (trace 57252dba638aa0b4ff7133f8699280e1: getRepo, then getCommit 29.8 s, no page line).
• Which git. By git-seconds: cat-file 807 s (186 lines, 136 of them under getCommit), git -c … show/diff 201 s (57, about half each getCommit and getThread), then diff, log, rev-list, rev-parse from a pull's range, 16 to 32 s each. getCommit?firstRows=true: 71 slow, median 9.8 s, max 29.8 s; getThread?filesOnly=true: 29, median 5.5 s, max 23.2 s.
• Everything else waited too. getTree and getLastCommits 2.3 to 2.9 s, getProfile and getActorRepos 2.4 to 5.8 s, an outgoing call answered with a 404 in 2.3 s (trace 03d75dc8e4eda2e38631de293988b04a). None of those run git per request.

The cat-file lines are the tell: texts reads a handful of blobs with cat-file --batch, which takes milliseconds, yet took 1 to 26 s. Its output is drained through a pipe by the runtime, and serve's runtime has as many threads as the container has cores, two. files highlighted every drawn file inline on those threads, "a good part of a second for a long file" by our own notes, for dozens of requests at once; meanwhile 60-odd git processes shared the same two cores. So git's output went unread, every request's future went unpolled, and the whole site stalled behind the crawler's diffs.

Holding diffs to four at a time keeps the burst's git processes and highlighting to what the cores can do, and moving the highlighting off the runtime keeps the two async threads free for every other answer. A crawler of a hundred commit pages then queues on itself: each diff takes its uncontended time, a few hundred milliseconds, four at a time, and its own pages wait in turn, while trees, profiles, pushes and pulls' conversations answer as before.

Others considered:

• Bounding every git process in knotless::git. Mirror syncs, the gateway's pushes and runner checkouts use the same Git; a crawl would queue a push. Bounding what a crawler can ask for, as blames and archives are, keeps it to the burst.
• Limiting pages per client. No log line names the client's address, the pages' own queries reach serve as knotless-pages whoever asked, and the burst spread over five agents at a page or two a second. A limit tight enough to stop it would catch someone opening a pull's tabs, and there is nothing yet to measure it against.
• Caching getCommit's answer by sha. It never changes, but the crawler asked for 49 different commits in its 71 slow getCommit lines, so a cache would have spared at most a third, and none of a crawler walking new pages. Highlighting, the dearest part, is already kept by text for the last 128 texts, so a repeat of the same commit parses nothing again.

Under a repeat of the same burst: its diffs run four at a time and its pages wait for their turn; nothing else waits on them but the CPU the four take, and a person's commit page waits behind at most the crawler diffs already queued, each taking its uncontended time.

How it was checked

Nothing was run here, the build, clippy and tests are left to CI. Behaviour is unchanged but for when a diff runs; the existing diff tests cover files with and without rows.

Shortcomings

A pull's other git (log, rev-list, rev-parse, diff for its range) is not bounded, nor is a blob page's highlighting; neither was slow but for the stall. A person's commit page waits behind a crawler's queued diffs, as no request says who asked. A crawler's page abandoned by its client still has its diff read when its turn comes.

Changeset: none