chadtmiller.com/knotless

Switch repository

Loading…

What changes

Code search in serve. With --code-index <dir>, knotless-serve runs Zoekt beside it. zoekt-webserver serves searches from shards on disk, on a loopback port of its own. zoekt-git-index writes a repository's shards from its mirror, of its default branch. A background task indexes one repository at a time: each public repository with no shards as serve starts, then each again as a feed moves its refs or its record. A deleted repository's shards are dropped when its turn comes.

dev.knotless.searchCode. A new query taking q in Zoekt's query language, an optional repo URI, and limit (20, at most 50). It answers files with their repository view, path, language and chunks (startLine, content, match ranges in UTF-8 bytes per line), plus fileCount and matchCount. Repositories the index no longer lists, or has taken down, are left out, so a takedown hides code at once. A server without the flag answers 501, and a query Zoekt cannot read answers 400.

Build and deploy. The test check builds zoekt-git-index and zoekt-webserver at a pinned commit into /opt/tools, cached. The image builds them in a Go stage, and the compose file turns code search on with /data/code.

Why

This is the engine behind the search page. Zoekt is a trigram index built for code: substrings and regular expressions across every repository at once, in milliseconds, with little memory, since its shards are memory-mapped. Running it as a sidecar behind our own lexicon keeps the API ours, so the engine can change later without the pages or clients changing.

How it was checked

A new serve test, searching_code, pushes two repositories, starts indexing, and checks that a search finds both. It then checks one result's exact shape and ranges, a search within one repository, that a later push is found, that a bad query answers 400, that a server without code search answers 501, and that a deleted repository's results and shards go. Unit tests cover how chunks are cut into lines and how shard names are made. The serve binary test starts with --code-index. just lex passes. The full test run is left to CI, which builds Zoekt.

Shortcomings

• Only default branches are indexed, and there are no pages for it yet. The results page and the header palette come next.
• fileCount and matchCount include files in repositories that are taken down but not yet re-indexed.
• During a deploy's handover, two servers may index the same repository at once. Zoekt writes shards by renaming them into place, so the later one wins.
• The first start on prod indexes every public repository, one at a time, syncing mirrors it does not have yet.

Review

Start at crates/knotless-serve/src/code.rs, then the index hooks in index.rs (reindex, listed, code_repos).

🤖 Generated with Claude Code

This pull adds Zoekt-backed code search to knotless-serve. code.rs runs zoekt-webserver and zoekt-git-index and follows the index to keep shards current. It adds the dev.knotless.searchCode lexicon and handler, the CI, image and compose changes, and the docs pages (serve.md, api.md). I read code.rs. I did not read the index.rs hooks, the handler or the workflow line by line, and the full diff was too large to open in one piece. In code.rs I found nothing that must change before merge.

Two small observations, neither blocking:

• MaxDocDisplayCount is applied inside Zoekt, but hits from repositories the index no longer lists are dropped afterwards. A taken-down repository can therefore use up result slots and return fewer than limit files. This fits the shortcomings you listed, and it hides nothing.
• chunk() computes r.start.byte_offset - base without a guard. This is fine if Zoekt keeps ranges inside the chunk, as it does today. A saturating_sub would stop a malformed reply from panicking the request.

The only change since my last review is in the searching_code test. It now counts only shards whose names start with the test account's DID, with : written _3a. The PDS is shared, so shards of other accounts' repositories were being counted. That fixes the shard-count assertion that was probably failing the test check. It touches no production code, and I found nothing new to ask for. My two earlier notes stand and are not blocking: chunk() could use saturating_sub on r.start.byte_offset - base, and slots can be lost to taken-down repositories. The queue ejected the pull because the test check failed at the earlier commit, so check that it passes at this head.