grain.social/grain

Switch repository

Loading…

What changes

grain::repo reads a whole repo export and checks it while reading. It gives out each record in key order, rebuilds the tree's root from those records, checks that root against the one the commit names, and checks the commit's signature against the account's key. repo::open looks at the first bytes to tell zstd, STAR-lite and CAR apart. Archive::record gives out records, and Archive::finish gives back the checked Commit once the rebuilt root matches. Nothing it gives out counts as proven before finish.

• repo::star: STAR-lite v0 (*l\0), the format hubble serves. It reads in one streaming pass and holds nothing but one tree node per layer. It also has a writer, which the tests use and which a server could use to serve the format.
• repo::car: CARv1, as a PDS's getRepo answers. The header must say version 1. Each block is checked against its CID as it arrives and written to a spill file (any Read + Write + Seek), at most MAX_CAR (16 GiB) spilled in all. Only a CID and an offset per block stay in memory. Once every block is in, the tree is walked from the commit, and its records are rebuilt into a root the same way STAR-lite's are. So a tree whose blocks all hash correctly but which is shaped other than atproto would shape it (keys in the wrong order, the wrong layering) is refused.
• repo::mst: layer, RootBuilder, and tree nodes encoded and decoded.
• repo::verify_commit: checks a commit's DID, its revision (a TID), its tree and its signature, which is over the commit's DRISL less sig.
• drisl gains cid, cid_text, link_to, and encode, which sorts map keys canonically.

New dependency: zstd (default features off). The branch has feature/rust (indexRecords) merged in.

Why

This is the first step of grain-serve's backfill, on the design the owner approved. Repos come from hubble as STAR-lite, streamed and verified, with a fallback to the account's PDS as a CAR. This pull is the generic part. It sits in crates/grain, depends only on drisl and crypto, and is written so knotless can copy it back for its own backfill issue.

• RootBuilder follows opensocial's (packages/group-sites/src/repoArchive.ts) and the reference algorithm in the STAR-lite spec.
• The varint framing is knotless's STAR v1 reader's (crates/knotless/src/space_sync.rs), changed to read from a stream rather than a slice: shortest form, at most 6 bytes, a clean end allowed only between frames.
• The size limits are the spec's: keys up to 830 bytes, records up to 1 MiB, commits up to 4096 bytes.

Unlike opensocial, a CAR is not held whole in memory. Its blocks go to a spill file.

How it was checked

CI only, as AGENTS.md asks. Nothing was built or run here.

• Unit tests (src/repo/tests.rs, mst.rs, drisl.rs): repos signed in the test, read back as STAR-lite (plain and zstd) and as CARs. Hand-built trees of one and two layers are checked against RootBuilder.
  • Refused: a changed record, a dropped record, swapped keys, a commit signed by another key or for another DID, and a stream cut short at every byte offset (plain and zstd).
  • Every framing error: a bad magic, a non-dag-cbor root, an empty, oversized or non-map commit, a commit carrying its own data, overlong or non-shortest varints, an oversized key or record, a key that is not text, and a body that fails partway.
  • CAR failures: a header of another version, a missing block, a block that does not match its CID, a CID atproto does not use, a header with no root, a commit that is not CBOR, a flat tree whose blocks all hash correctly, a node whose keys go backwards.
• Against the reference PDS (tests/repo.rs): an account with 151 records (several layers deep). Its getRepo CAR is read and checked against the signing key from PLC, and its revision matches getLatestCommit. The same repo is written out as STAR-lite and read back, plain and zstd. Tampered and truncated copies of both are refused. After a delete, the next export has a later revision and lacks that record.
• The MST layer vectors, the empty-tree CID and a one-key tree were worked independently in Python.

Shortcomings

• Not run against the real hubble. The STAR-lite fixtures are made by this crate's own writer from the reference PDS's repos, so an STAR-lite archive that hubble writes differently from the spec (say, field order in its partial commit) would only show up in the worker pull or in production.
• The spec gives the empty repo's CID as bafyreihmh6…. atproto's empty node, {e: [], l: null}, is bafyreie5737…, and that is what this code checks for. The spec looks wrong here. An empty repo from hubble would show which.
• The 16 GiB cap is not exercised by a test; its check is one comparison before each block is written.

Review

The first of two stacked pulls. The worker, backfill/worker, is open on top of this one. Start at src/repo/mod.rs, then mst.rs; the tests show what is refused.

🤖 Generated with Claude Code

This pull adds grain::repo. It reads a repo export as STAR-lite or CAR, with or without zstd. It rebuilds the MST root from the records and compares it to the commit's root, and it checks the commit signature before any record is given out. I read mod.rs, star.rs, car.rs and mst.rs in full. I did not read the test files in the diff.

I found nothing that must change:

• Framing and limits: the varint reader enforces shortest form and a 6-byte cap. Key, record and commit sizes are bounded before allocation. A stream cut short fails either inside a frame or in finish, because the rebuilt root no longer matches.
• RootBuilder: the fold logic is correct, including the layer gaps. A node with only a left link is kept, as the atproto reference does. The ordering check, key > last, also rejects duplicate keys.
• CAR walk: the stack order gives in-order traversal. Every block is checked against its CID, and the rebuilt root rejects trees shaped other than atproto's.
• Not proven before finish: the API and docs say so, and callers must honour that.

Two small things, neither blocking:

• Car::read doesn't check the header's version is 1.
• The CAR spill file and its in-memory index have no total size cap. Callers should bound the body when it comes from an untrusted source. The worker pull is a good place for that.

The check, test, rust and harness runs are all still queued. Please confirm they pass, including the coverage gate in harness/cov.json, before this lands.

The only change since my last review is a rename. Archive::next, Car::next and StarLite::next are now record(), to avoid clippy's should_implement_trait lint. The tests are updated to match, and a Vec in tests.rs became an array. I searched for call sites still using the old name and found none; the one remaining .next()? match is an unrelated string split in grain-serve. The behaviour I reviewed before is unchanged.

The PR body still says Archive::next, so update it to record. The check, test, rust and harness runs are still queued for the new commit. Confirm they pass, including the coverage gate, before it lands.

Since my last review, the two points I raised are done. Car::read now refuses a header whose version isn't 1 (car.rs:76-77). Total spilled bytes are capped at MAX_CAR = 16 GiB, checked before each block is written (car.rs:95-98). The cap counts duplicate blocks too, so repeated blocks can't get around it. The newest commit is otherwise a merge of feature/rust.

I found nothing else that must change. The PR body still says Archive::next; it should say record.

The check, test, rust and harness runs are still queued for this commit. Confirm they pass, including the coverage gate, before it lands.

Landed in feature/rust with the worker pull (grain-serve backfills the index from hubble, or each account's PDS), fast-forwarded to 1b26083, which contains this branch's head 3b70890. All four checks passed there, and that pull's approval at 1b26083 covered these commits too. Closing this one, since its content is merged.