grain.social/grain

Switch repository

Loading…

What changes

An index for grain-serve. crates/grain-serve/src/index is one SQLite file in WAL mode, written through one connection, with its tables made by ordered migrations (PRAGMA user_version; a step is only ever appended, and an index from a newer grain is refused). schema.rs documents the tables:

• a table per collection grain indexes, rebuildable from the network: gallery, gallery_item, photo, photo_exif, profile, follow, block, favorite, comment, story, group_item, group_submission, group_decline, group_declaration (fyi.opensocial.declaration), bsky_post, bsky_follow, germ_declaration, plus self_label for galleries' and stories' self-labels;
• repo (tracked accounts: handle, backfill state, and what the host says of the account) and stream_cursor;
• what the network cannot rebuild: takedown (admin takedowns, apart from repo), label (an account's label is on its bare DID), report, mute, preference, push_token, story_view, space_invite, space_support.

Each record row is its envelope (uri, cid, did, indexed_at) and then its fields as typed columns (records.rs); structured values (blobs, facets, locations) are JSON columns, as hatk kept them, so grain-serve import is a column-for-column copy that keeps indexed_at. A gallery's sort_at (the earlier of created_at and indexed_at, as _galleryTable.ts computes it) and h3 are generated columns.

Records are put by URI and CID. Index::put checks the record against its lexicon and refuses one that does not fit; putting the same CID again changes nothing (Applied::Unchanged), another CID replaces the row and keeps its first indexed_at; Index::delete removes it. purge removes everything an account wrote.

grain::lexicon: hatk's validator (@bigmoves/lexicon's validateRecord) ported, with the lexicons of every indexed collection built in. Same rules, three spec-following differences: maxLength counts UTF-8 bytes (not UTF-16 units), a datetime with an offset is valid whatever its UTC date (hatk compared the literal date with the UTC one), and a DID may hold colons past its method.

Indexes for the feeds. Each index serves a query of the feeds (following, for you, recent, actor, camera, hashtag, location, group, groups), their hydration, or the moderation filters (schema::shown, schema::unhidden). the_feeds_read_by_index plans those queries against prod-sized statistics, as knotless's plans.rs does, and fails on any full scan of a table.

Why

Step one of the data layer on feature/rust: the schema the Jetstream ingest (stacked on this) writes into, and the import fills at cutover.

Ordered migrations rather than knotless's drop-the-cache-on-version: grain's records come from a costly import and backfill, so the schema moves forward in place; any index or column a step needs is made in that same step, so nothing runs against a column that does not exist yet (hatk's DDL-before-migrate crash loop).

Kept from hatk's index: every record lexicon grain's queries read (app.bsky.graph.follow for suggested follows and com.germnetwork.declaration for the profile, besides social.grain.*, fyi.opensocial.declaration and app.bsky.feed.post). Left out: app.bsky.actor.profile, postgate and threadgate, which hatk indexed only because their lexicons are vendored and no grain query reads. _classifications is not here yet (scores can be recomputed); OAuth state lives with grain::session.

Changed from hatk: a redelivered or edited record keeps its first indexed_at (hatk's INSERT OR REPLACE reset it each time), and an admin takedown is its own table rather than a _repos.status value a backfill could overwrite.

How it was checked

Unit tests in index/tests.rs and lexicon.rs: upsert idempotence and indexed_at kept, delete, every refusal (URI, collection, CID, $type, lexicon, row), a record of every collection round-tripped through its typed row, the tables' columns against the rows', accounts, takedowns, cursor and labels persisting across a reopen, migrations (a later step, a newer index, a failing step), the query plans, and every validator rule. Nothing was built or run locally; CI runs fmt, clippy and the harness.

Shortcomings

• The plan check here is a unit test over hand-written versions of the feeds' queries; the queries themselves arrive with the feeds, and the harness's plans.mjs guard over every statement run is a follow-up.
• No PDS is involved: the index takes records as given. A test that writes social.grain.* records to the reference PDS and indexes the read-back belongs with social.grain.sync.indexRecords.
• Records are applied in arrival order with no rev check, as hatk's are.

Review

The design: index/schema.rs first (its module doc and V1), then index/mod.rs::put_at.

🤖 Generated with Claude Code

This adds the SQLite index for grain-serve: ordered migrations, one table per collection, a put that checks the lexicon and keys on URI and CID, and the ported grain::lexicon validator. I read index/mod.rs and index/schema.rs closely. I did not read the 3,000-line diff beyond them, so records.rs, lexicon.rs and the tests got no review from me. I found nothing that must change.

• put_at: the checks run in a sound order. An unchanged CID returns before any write. indexed_at is left out of the conflict SET, so the first value is kept. The self_label rows are rewritten in the same transaction as the record row.
• Migrations: each step runs in its own transaction, and a newer index is refused.
• set_host_status: the CASE reads the old host_status, because SQLite evaluates SET expressions against the pre-update row. A returning account is therefore set back to pending as the doc comment says.
• purge: the at://<did>/ to at://<did>0 range works, since 0 is the byte after /.
• Minor: delete removes the self_label rows and the record row as two separate statements without a transaction. A failure between them would leave a record with no labels, so you could wrap them in a transaction.

The check, harness, test and rust checks were still queued or running when I looked. The PR says nothing was built locally, so merge only once they pass.

The new commit changes only tests, and I found nothing that must change.

• Plan check: scans now reads the EXPLAIN QUERY PLAN rows with raw_query, so statements with unbound parameters can be planned. This is a sound fix.
• Broken JSON column: the test now corrupts facets instead of location. I did not check that gallery reads facets as a JSON column, so I'm relying on the commit message.
• Grapheme test: the test string is now e\u{301} repeated 65 times. It is 65 graphemes, each two code points and three bytes, so it stays under any byte limit and triggers only the grapheme limit. This is better than the old emoji string.

My earlier minor suggestion, wrapping delete in a transaction, was optional and is not done. I don't hold the approval on it. The check, harness, test and rust checks were still queued or running when I looked, so merge only once they pass.

The new commit wraps delete in a transaction, so the self_label rows and the record row are removed together. That was my one optional suggestion, and it is done. The change is small and correct: db is now mut for transaction(), and commit runs after both statements. I found nothing that must change. The check, harness, test and rust checks were still queued or running when I looked, so merge only once they pass.