grain.social/grain

Switch repository

Loading…

What changes

social.grain.sync.indexRecords is served. The apps and the command line write to their own PDS with com.atproto.repo.*, then call it with a service token and the 1 to 25 URIs they wrote. grain-serve verifies the token with the core's auth::verify (aud its own DID, lxm this method, the signature against the issuer's DID document), refuses a URI outside the caller's repo (RepoMismatch) or of a collection the index does not keep (UnknownCollection) before reading anything, reads each record back with com.atproto.repo.getRecord from the PDS the caller's DID document names, and puts it by URI and CID, or deletes it when the PDS says RecordNotFound (answered in notFound). It answers {indexed, notFound} only after the index has them. The caller is tracked from then on, so Jetstream keeps what they write next.

sync::read_back is the read-back path, one function the web's own writes can call with the session's DID and PDS: it does the repo and collection checks itself, reads the records at once, applies each one read whatever became of the others, and reads a URI named twice once.

grain-serve has a DID. GRAIN_PUBLIC_URL names where it is reached, and its DID is the did:web of that host, as knotless-serve's is of its --public-url: https://grain.social is did:web:grain.social, the DID the web's /.well-known/did.json already names. GRAIN_INDEX names the index's file. Without both, indexRecords answers MethodNotImplemented. Settings come from grain_serve::Settings, the router takes an App (context, index, DID), and XRPC failures are xrpc::Failure.

Why

A write should be readable as soon as it returns, without waiting for Jetstream, and the server should take nothing from the client but the URIs: content only ever comes from the PDS. Putting by URI and CID makes Jetstream's later delivery of the same record a no-op.

How it was checked

crates/grain-serve/tests/index_records.rs, against the reference PDS through the harness: an account writes social.grain.gallery and social.grain.graph.follow records, asks its PDS for a token with getServiceAuth, and calls indexRecords through tests/common, so answers.mjs checks every answer. It covers records indexed, edited, named twice and named again (unchanged by CID), deleted on the PDS and never written (notFound, removed from the index), the refusals (another repo, an unknown collection, not a record URI, 0 and 26 URIs, input that is not {uris}, no token, a token for another server, for another method, signed by another key, expired, not a JWT), a record that does not fit its lexicon, a PDS that does not answer, and a server with no index or DID. Unit tests cover the settings, the DID and how each failure answers. Built, linted and run on the runner only.

social.grain.* writes on the test PDS. @atproto/pds 0.5.43 resolves no lexicon over the network for a record write (repo/prepare.js: "@TODO add support for lexicon resolution"; the LexResolver its context builds is not used by writes). It validates against the schemas it bundles, which include copies of social.grain.gallery, social.grain.photo and social.grain.favorite; a record of any other $type, written with validate unset as the apps write them, is kept with validationStatus: "unknown". So the tests write as the apps do, with nothing reaching for grain.social, and the lexicon-mismatch test writes a follow with no subject, which the PDS keeps unchecked and grain-serve refuses.

Coverage: 99.31% of lines and 98.73% of functions, over the 99.1 / 98.4 gate; answers.mjs passes on every answer.

Shortcomings

• A record on the PDS that does not fit its lexicon answers InvalidRequest naming it, and whatever the index held for it stays; the lexicon has no output for it.
• A failure part way (a PDS that stops answering for one record) answers the error after applying the records that were read; calling again is harmless.
• The PDS's bundled copies of the gallery, photo and favorite schemas may lag grain's own; a gallery the current lexicon allows could be refused by that PDS. That is the PDS's write, not this method's.
• grain-serve does not serve its own /.well-known/did.json yet; the web does, at grain.social.
• This touches the router, main.rs and Settings, as the Jetstream pull does; whichever lands second merges the other's settings in.

Review

Start at crates/grain-serve/src/sync.rs (read_back, then index_records), then the tests.

🤖 Generated with Claude Code

This pull serves social.grain.sync.indexRecords. It verifies the service token with auth::verify (aud the server's did:web, lxm this method). It refuses URIs outside the caller's repo or outside the indexed collections before reading anything. It reads each record back from the PDS in the caller's DID document, puts it by URI and CID, and deletes it on RecordNotFound. I found nothing that must change.

• Lexicon: the method, its 1–25 bound and the RepoMismatch and UnknownCollection errors match lexicons/social/grain/sync/indexRecords.json. Other failures use XRPC's own error names.
• read_back: it checks all URIs first, dedupes, and reads concurrently. It then applies every record that was read and returns the first error, so a retry after a partial failure is safe. The caller is tracked only after the reads, so a refused call tracks nobody.
• Tests: they run through the harness, so answers.mjs checks every answer. They cover the token failures, the refusals, the lexicon mismatch, a PDS that is down, and a server with no index or DID.

I did not read the check results. The change was built and tested on the runner only.

The DID document's PDS URL is fetched without any host filtering. That is existing behaviour in grain::identity, not part of this pull, but it is worth a follow-up now that an authenticated caller can trigger the fetch.

The new commit only changes tests, and my earlier approval still stands. The reference PDS bundles gallery, photo and favorite schemas, so it rejected the untitled gallery the lexicon-mismatch test used. The test now writes a follow with no subject, which the PDS keeps unchecked, and it checks that the index holds no Follow for it. The blanket validationStatus == "unknown" assertion is gone and the module comment says why. The pds-down test now uses std::slice::from_ref, which doesn't change what it checks. I didn't read the check results.