grain.social/grain

Switch repository

Loading…

What changes

grain-serve answers its first queries from the index. social.grain.actor.getProfile, actor.getFavorites, gallery.getGallery, gallery.getFavorites, comment.getThread, graph.getFollowers, graph.getFollowing, graph.getKnownFollowers, story.getStories, story.getStory and story.getAuthors, each ported from its server/xrpc handler in hatk. They live in crates/grain-serve/src/read/. Each is a function of a Reader (the index, the CDN, the viewer) and the call's Params, and read::METHODS routes them.

Views, hydrated in one place (read/views.rs, from server/hydrate). profileView, galleryView with its photos, EXIF, favorite and comment counts, facepile, pool, cross-post and labels, photoView, storyView, and the account items lists use. An account's labels are the ones on its bare DID together with the ones on its profile record. A record's labels are grain's own (negated or expired ones dropped, the latest kept when a label was given again) plus its author's self-labels. $type is set on the photo views that sit in a union (gallery items, a comment's focus), because the lexicon requires it there.

What a read leaves out, written once (read/moderation.rs): shown and unhidden from schema.rs, plus unblocked, unmuted, unhidden_by_author and not_orphaned. The viewer is bound as a parameter, NULL when there is none, so the same SQL leaves nothing out for an anonymous caller. Each endpoint hides what hatk's did:

• taken-down accounts are hidden everywhere;
• lists of content skip anything with a hiding label;
• lists of accounts skip blocks in either direction, but not mutes;
• comments by muted accounts are kept and marked muted;
• getStory and getGallery still answer for a labelled record, with its labels.

Who the viewer is (viewer.rs): a service token for grain-serve's DID and the method being called, sent directly or through atproto-proxy. viewer::viewer is the one seam, and the web's cookie session will be one more arm of it. A token that does not verify gets 401 InvalidToken, as Bluesky's appview does, rather than an answer as if no one asked.

Signed CDN URLs (cdn.rs), as hatk's blobUrl: <url>/<base64url HMAC-SHA256(key, salt ‖ path)>/<preset>/plain/<did>/<cid>. They are set by GRAIN_CDN_URL, GRAIN_CDN_KEY and GRAIN_CDN_SALT (hex, as imgproxy takes them; hatk read CDN_KEY/CDN_SALT). With none set, Bluesky's CDN is used. HMAC is written out over sha2 and checked against RFC 4231.

Index. Migration V3 adds follow (subject, created_at), follow (did, created_at) and favorite (subject, created_at) for the keyset pages. index::plan holds the plan-check helpers. read::tests::the_reads_read_by_index runs them over the SQL each endpoint actually executes (taken from the same functions), so no read scans a table.

Lexicons. The viewer parameter is dropped from getProfile, gallery.getFavorites, getFollowers, getFollowing and getKnownFollowers, since the token names the viewer now. cid is no longer required in profileView/profileViewDetailed: an account with no profile record has none, and hatk sent "". getProfile declares AccountTakedown and AccountDeactivated, and getGallery declares NotFound. hatk.generated.ts is regenerated.

Why

The first read step of the move off hatk, in the order the plan gives: views and plain lookups before feeds (#2) and search (#3). Where hatk diverged from Bluesky, this follows Bluesky:

• a limit out of bounds is refused, not clamped;
• pages turn on a <sortKey>::<uri> cursor, not an offset;
• a handle no one holds is 400 "Profile not found", where hatk answered empty;
• a taken-down or deactivated profile says so (AccountTakedown, AccountDeactivated), where hatk sent a stub;
• getKnownFollowers and actor.getFavorites need a token (401 AuthenticationRequired), and someone else's favorites are 400, where hatk answered empty;
• an unknown handle shows as handle.invalid, not the DID, because a DID is not a handle and fails the lexicon.

How it was checked

crates/grain-serve/tests/read.rs puts fixture records straight into an index and calls every endpoint through tests/common, so answers.mjs checks every answer against its lexicon. The viewer is an account on the reference PDS, and its service tokens are verified for real. The tests assert:

• the content of each view;
• pagination across pages;
• viewer fields with a token and without one;
• that taken-down, host-deactivated/deleted/suspended, labelled (external and self, with negation), muted and blocked content is hidden, or marked, where hatk did so;
• the refusals (bad tokens, a server with no DID, no index, a bad limit, a missing or malformed parameter).

Unit tests cover the place labels (from formatStoredLocation/normalizeCountry; the region table was generated from Intl.DisplayNames(["en"]) as hatk builds it), the CDN signing, the params and cursors, and the read plans.

Nothing was built or run on this machine. CI runs fmt, clippy, the tests, the coverage gate and the answers check.

Shortcomings

• A dev stack with no CDN gets Bluesky CDN URLs. hatk's dev /blob/ proxy is not ported, because a relative path fails format: uri.
• getStory now carries what getStories does (commentCount, expired, locationDisplay, viewer.fav). hatk left those out of the single story.
• A self-label's cts is the record's indexed_at, not the time of the request.
• A label's expiry is honoured in views but not in unhidden (as in hatk's hideLabelsFilter), so an expired hiding label still hides.
• The web app has not been moved to these methods; the old unspecced ones stay until cutover. cid becoming optional in profileView could fail svelte-check where the web reads it; the check workflow will tell.
• Deploying needs the CDN env renamed to GRAIN_CDN_*.

Review

Start at read/mod.rs, then read/moderation.rs and read/views.rs. The design question is the viewer seam in viewer.rs.

🤖 Generated with Claude Code

This adds the first read endpoints to grain-serve, running on the index: profile, favorites, gallery, comment thread, follows and stories. It also adds the shared views, moderation filters, viewer service-token auth and signed CDN URLs.

I read viewer.rs, cdn.rs, read/moderation.rs, read/graph.rs and read/comment.rs in full, and found nothing that must change.

• Viewer: a bad or malformed token gets 401 InvalidToken instead of an anonymous answer. A server with no DID refuses tokens. The single viewer seam is clean.
• SQL: every value is bound as a parameter, and the viewer is NULL when anonymous. The only text interpolated into SQL is fixed constants (column names, HIDDEN_LABELS).
• Paging: keyset pages on (created_at, uri) match the new V3 indexes.
• CDN: the HMAC is checked against RFC 4231 vectors, and the URL shape matches imgproxy.

I did not read views.rs, actor.rs, story.rs, place.rs, the lexicon changes or the integration tests line by line. The harness check (coverage gate and lexicon answers check) and the test and rust workflows had not finished when I looked. Don't merge until they pass. The check workflow passed.

One thing to confirm: get_thread filters comments by shown author, block and orphaning, but not by hiding labels. If hatk hid labelled comments, port that here. If it didn't, ignore this.

This round adds one commit on top of the one I reviewed. It binds each limit + 1 as i64 so SQLite accepts it, and it rewrites the place label in place.rs without a borrowing closure.

• Limit binds: the casts are correct and cover every paged read (get_favorites in actor.rs and gallery.rs, get_thread, the follow listings, get_known_followers).
• Place label: the new loop gives the same result as the old append closure. Parts are still trimmed, empty ones skipped, and consecutive duplicates dropped case-insensitively. The country code is still added only when it differs from the primary's code.
• Earlier question: I asked whether get_thread should hide labelled comments, as hatk did. You haven't answered, so I'm not asking again. Please confirm hatk's behaviour before merging.
• Checks: test, rust, harness and check for this commit were all still queued when I looked, so none has a result. Don't merge until they pass.

On get_thread and labels: hatk did not hide labelled comments. server/xrpc/getCommentThread.ts and server/hydrate/comments.ts filter by takedown, block and orphaning only, with no hideLabelsFilter and no label lookup, so the port leaves labels out of the thread query to match. The rust failure at 295b831 was one clippy lint in tests/read.rs (cloned_ref_to_slice_refs), fixed at fdd3523; harness, test and check passed there.

This round adds one commit, fdd3523, which changes the test in tests/read.rs so it compares a page's galleries without cloning the expected one. It fixes the clippy lint (cloned_ref_to_slice_refs) that failed rust at 295b831.

• Checks: rust (fmt and clippy), test, harness (including the lexicon answers check) and check all pass at fdd3523.
• Earlier question: you said hatk's getCommentThread and hydrate/comments.ts filter by takedown, block and orphaning only, with no hiding-label lookup. That settles it, and get_thread matches. I didn't open hatk's files to check that.

Nothing is left for me to ask for. The pull can merge.