grain.social/grain

Switch repository

Loading…

Stacked on grain-serve serves the gallery feeds, named by URI (read/feeds), which is stacked on grain-serve answers profiles, galleries, comments, follows and stories (read/views). This pull's own change is its last commit. Merge those two first.

What changes

Search (read/search.rs). social.grain.actor.searchActors, searchActorsTypeahead and social.grain.gallery.searchGalleries read two FTS5 tables: profile_search (handle, display name, description) and gallery_search (handle, title, description, and the place: its name, locality, region and country). Both use hatk's tokenizer, porter unicode61 remove_diacritics 2, so "morning" finds "mornings" and "cafe" finds "Café". Results rank by BM25.

The index keeps the search tables itself. Migration V4 adds them, with triggers on profile, gallery and repo (for a handle that changes or arrives late), and fills them from what the index already holds. search_doc gives each record a stable rowid. The import's empty-index check now looks only at tables that hold their own rows, leaving out FTS5's virtual and shadow tables.

Typeahead asks Bluesky too, as hatk did, where GRAIN_BSKY_APPVIEW names an appview (1 means public.api.bsky.app). grain's accounts come first, then Bluesky's that are not already listed, up to the limit. It waits at most three seconds and answers with grain's own if Bluesky does not answer. The setting is off unless set, so tests and dev never call out.

getSuggestedFollows (read/actor.rs): accounts the actor follows on Bluesky that have a grain profile, then the most followed. It leaves out anyone the actor already follows, anyone blocking or blocked by them, anyone they mute, and accounts taken down. Each item carries followersCount.

getCameras (read/camera.rs): the 30 cameras with the most photos, by the name cleanCameraName gives them, so EXIF spellings that clean to one name count as one camera. Each has the thumbnails of its four latest photos.

getLocations (read/location.rs): places, grouped by address, then by place name, then by city cell, as hatk grouped them. A place with no region is merged into the one place of the same locality and country that has a region (a "Lisbon, PT" into "Lisbon, Lisbon, PT"); where there are several, it stays separate. Each place takes the label most of its galleries give it, and its busiest cell. pins=true returns every place with only what a map needs; otherwise the top 30, with their cells and covers.

Why

The third read step of the plan, after views (#1) and feeds (#2).

Why FTS5. hatk built _fts_* tables, which were FTS5 shadow tables in its SQLite port, and rebuilt them at startup. FTS5 is in the bundled SQLite already, so this keeps hatk's tokenizer and ranking. Triggers keep the tables current as records arrive, which also removes the startup rebuild. The alternatives were a separate engine (tantivy) or LIKE over the tables. A separate engine adds a crate and a second store to keep in step with the index. LIKE cannot stem or rank, and it reads every row.

Where this differs from hatk, and why:

• A cursor pages. hatk read the cursor and then always started at offset 0. Search pages by an offset, as For You does, since a ranking has no key to resume from. A malformed cursor is 400.
• Each word is matched as a quoted phrase. hatk escaped special characters one by one. Quoting each word means nothing a user types can be read as FTS5 syntax. English stop words are dropped as hatk dropped them (DuckDB's list), unless the query is nothing but stop words.
• Typeahead matches the start of the last word, because that word is still being typed. hatk matched whole words only, so "ali" found nothing until "alice" was typed in full.
• fuzzy is accepted and ignored. hatk only honoured it on DuckDB.
• Search returns only galleries that a list would show: they have photos, their account is shown, and no hiding label from grain or the author. hatk filtered by takedown only.
• Suggested follows are deterministic: Bluesky follows sorted by DID, popular accounts by follower count and then DID, with no duplicates. Popular means at least one follower. A missing actor is 400, since the lexicon requires it, where hatk answered with an empty list.
• Cameras and places are read on each call, without hatk's five-minute cache. Their thumbnails leave out accounts that are not shown, and places also leave out galleries with a hiding label.

How it was checked

tests/read.rs puts fixtures into the index and calls each method through tests/common, so answers.mjs checks every answer against its lexicon:

• searchActors: stemming, case, a handle, stop words dropped and kept, punctuation alone, offset pages, taken-down accounts left out. The tables follow a new handle, a handle that arrives after the profile, an edited profile and a deleted one. A malformed cursor and a missing q are refused.
• searchGalleries: title, description, the place's name and address, the handle, accents folded. Galleries with a label, no photos, or a taken-down account are left out. Also offset pages, and an edit and a delete.
• typeahead, against a stub Bluesky appview: grain's accounts first, Bluesky's duplicates and DID-less entries dropped, only the typeahead fields kept, the limit honoured, and the query sent upstream. With no appview set, or one that cannot be reached, grain's own accounts.
• getSuggestedFollows: Bluesky follows, minus those already followed, blocking, or without a grain profile, then the most followed. Also followersCount, the limit, and the fallback with no Bluesky follows.
• getCameras: two EXIF spellings counted as one camera, ordered by count, thumbnails, and an empty make skipped.
• getLocations: the regionless merge, a regionless place left on its own, a country label, a name-only place, a labelled gallery left out, pins, and an invalid pins refused.

A unit test covers how a query is built. The plan check covers every new query, and now treats an FTS5 table read through MATCH as read by its index.

Nothing was built or run on this machine. CI runs fmt, clippy, the tests, the coverage gate and the answers check.

Shortcomings

• The popular-accounts query and getCameras' count read all of follow and photo_exif through their indexes on every call, as hatk's did (hatk cached the latter). The plan check passes them because they read an index. If they become slow, a cache or a summary table is the next step.
• getLocations reads every gallery that has a place, also as hatk did.
• Typeahead's Bluesky results are not filtered by grain's moderation: an account taken down on grain can still appear from Bluesky. hatk had the same gap.
• V4 fills the search tables in one statement on the first start, which on prod's index means one long write.

Review

Start at index/schema.rs V4 for the triggers, then read/search.rs.

🤖 Generated with Claude Code

This pull adds FTS5 search tables kept current by triggers (migration V4), searchActors, searchActorsTypeahead and searchGalleries, plus getSuggestedFollows, getCameras and getLocations.

I read read/search.rs and the V4 migration in full, and checked how the index writes records. I did not read the actor.rs, camera.rs or location.rs code or the tests, so those methods are unreviewed.

What I found:

• Triggers are sound. The index upserts records with ON CONFLICT DO UPDATE, and that fires the update triggers. The repo insert and handle-update triggers cover a handle that changes or arrives late. The V4 backfill matches what the triggers write.
• Query building is safe. Each word is quoted and embedded quotes are doubled, so user input can't become FTS5 syntax. A malformed cursor gives 400.
• Checks. The check workflow passed on the head. rust, harness and test were still running when I looked. The harness is what enforces the coverage gate and the answers check, so don't merge before it passes.

Minor and not blocking: in typeahead, if the last word typed is a stop word such as th, it is dropped and the prefix match moves to the word before it.

The only change since my last review is the commit "Search's tests edit with a CID of their own; clippy's types named". It touches the tests and clippy type naming, not the search or migration code I reviewed. I still approve.

• Checks. check passed on the new head. harness is still running, and rust and test are still queued. The harness enforces the coverage gate and the answers check, so wait for it to pass before merging.
• Not reviewed. As before, I did not read actor.rs, camera.rs, location.rs or the tests.
• Stacking. This pull is stacked on read/feeds and read/views, so merge those first.

Two commits are new since my last review. "Compare search's results to the URIs, not clones of them" changes the tests. "Typeahead keeps the word being typed, even one that reads as a stop word" fixes the minor note I raised in round 1, where a stop word being typed (such as th) was dropped and the prefix match moved to the word before it. I did not re-read the code behind these two commits, so I'm relying on the passing check and the earlier review of the search code.

• Checks. check passed on the new head. harness, test and rust hadn't finished when I looked (harness running, the other two queued). The harness enforces the coverage gate and the answers check, so wait for it to pass before merging.
• Not reviewed. As before, I did not read actor.rs, camera.rs, location.rs or the tests.
• Stacking. This pull sits on read/feeds and read/views, so merge those first.