grain.social/grain

Switch repository

Loading…

What changes

Stacked on backfill/star (Core reads whole repos as STAR-lite or CAR, checked as they stream). Its commits show here until it lands. What this pull adds is crates/grain-serve. feature/rust (indexRecords) is merged in.

grain_serve::backfill reads every tracked account's repo whole and makes it the index's.

• Which accounts. It asks the collection directories who writes the signal collections (social.grain.actor.profile, fyi.opensocial.declaration). The directories are lightrail, then Bluesky's relay, using listReposByCollection and paging by cursor; it moves to the next directory when one does not answer. It also reads whatever the index queues itself: a newly tracked account (Index::track, as Jetstream calls it), an account whose host says it is back (set_host_status), and whatever admin entry points ask for. For those it exposes Backfill::add_repos and Backfill::resync, ready for social.grain.admin.addRepos and resyncRepos.
• Each repo. It asks hubble for STAR-lite (accept: application/x.microcosm.star-lite, accept-encoding: zstd). The body streams through a bounded channel into grain::repo on a blocking thread, so a repo is never held whole and the checks never run on the async runtime. If hubble has no copy, answers an error, goes quiet, cuts off, asks to be left alone, or serves a copy that fails its checks, it reads getRepo from the account's PDS as a CAR, spilled to a temp file, and checks that the same way. Only records of collections the index keeps are kept, and nothing is written until the whole archive has checked out.
• Which hosts it will ask. A DID document names whatever PDS it likes, so a PDS is asked only on https and not at an IP address outside the public internet (loopback, private, link-local, shared, reserved, multicast, documentation, IPv4-mapped forms of those, and anything outside IPv6 global unicast). Hostnames go through a resolver that keeps only public addresses, so a name pointing inward is refused too. No redirect is followed. A PDS refused this way is given up on at once. GRAIN_BACKFILL_PRIVATE=1 (Config::private) lifts all of this for dev and tests.
• Newer revision wins. The index gains record_rev: the revision each record was last written or deleted at (migration V2). Backfill writes with Index::put_rev at the revision it read the repo at, and the index answers Applied::Stale where it already has the record, or its deletion, from that revision or later. Writes that name no revision (put and delete, as Jetstream and indexRecords call them today) are stamped with the time they were applied, as a TID, so a repo exported before them never overwrites them. After applying, Index::backfilled deletes whatever the index holds for that account that is older than the repo and absent from it. It then drops the deletions the repo already covers, and records the account's rev (which never goes backwards), backfilled_at and state active.
• The queue is the repo table, with a new last_error column.
  • Concurrency is bounded at 5 repos and 2 requests per host.
  • A 429 pauses that host for its Retry-After (at most an hour).
  • Failures retry after 60s, doubling up to 6h, and give up after 8 tries.
  • A PDS's 4xx gives up at once, as for a deactivated account.
  • Accounts on atproto.brid.gy are marked skipped, as hatk's skipPds did.
  • Deletions older than a week are forgotten, so record_rev does not grow without bound.
• Settings. GRAIN_BACKFILL=1 turns the worker on. It is off by default, as in tests and dev, and needs GRAIN_INDEX. GRAIN_BACKFILL_DIRECTORIES, _MIRROR, _SKIP, _CONCURRENCY and _PRIVATE override the defaults. They are read into grain_serve::Settings as its backfill field. Settings::backfill(&app) gives the Backfill handle (none when unset, an error when there is no index), and main spawns its run(). The admin endpoints will take the same handle.

Why

This is the backfill step of the move off hatk, on the design the owner approved, which is opensocial's (packages/group-sites/src/networkIndex.ts): directories for discovery, hubble's STAR-lite first, the PDS's CAR as fallback, a SQLite queue, newer revision wins.

Opensocial keeps a revision on every record in its own table. grain's record tables are shared with every other writer, so the revisions sit beside them in record_rev, which covers tombstones too, rather than as a column on each table. The stamp for writes with no revision makes the Jetstream pull safe as it stands: it does not carry a commit's rev yet.

How it was checked

CI only. Nothing was built or run here.

• tests/backfill.rs, against the reference PDS, with an in-process stand-in serving both lightrail's listReposByCollection (one DID a page, behind a dead first directory) and hubble's getRepo:
  • Discovery and both paths. Three accounts are discovered and read. One comes from the mirror's STAR-lite, which was made from the PDS's own CAR and zstd'd. One the mirror lacks, and one the mirror has forged; both are read from the PDS, and the forged title never reaches the index. A second read writes nothing.
  • Newer revision wins. A gallery edited and one deleted "by the stream" after the repo was written both stand. A record the index had from long before, and the repo no longer has, is deleted.
  • Failures. A DID PLC does not know retries after 60s, then 120s, then is given up on, and resync queues it afresh. A deactivated account is given up on at once. A mirror that answers 500, is too slow to answer, stalls mid-body or cuts the connection falls back to the PDS. A 429 pauses the mirror before it is asked again. A skipped host is never read. Without private, the reference PDS (http, loopback) is given up on before it is asked.
  • The worker. It reads what discovery finds, what add_repos adds and what resync queues, and still reads queued accounts when no directory answers.
  • Settings and Settings::backfill.
• Unit tests in backfill.rs: public and non-public addresses of each kind, the PDS URL check, and the resolver refusing localhost and 127.0.0.1 and keeping a public address.
• Index unit tests: put_rev and delete_rev against older and newer revisions, stream stamps, a revision ahead of the clock, and purge. Also the queue (pending, queued, resync, backfill_failed, backfill_skipped), backfilled's deletions, and forget_deletions. The queue and revision queries are added to the plan check, so they are shown to read by index.
• The migrations test now holds for any number of steps.

Shortcomings

• Not run against the real hubble or lightrail. The stand-ins answer as opensocial's code expects them to: {repos: [{did}], cursor}, and STAR-lite served raw-zstd. Real paging limits, rate limits, and whether hubble sends zstd as content-encoding or as the body are untested. Either kind of zstd is read, since zstd is recognised by its magic.
• No redirects applies to hubble and the directories too. If hubble redirects getRepo to a CDN, that read fails and falls back to the PDS; following a redirect would need each hop checked, which this does not do.
• The configured mirror and directories are trusted: an IP literal there is asked whatever it is. Only a hostname goes through the public-only resolver.
• Jetstream should pass each commit's rev (put_rev and delete_rev) once its Op carries it. Until then, a PDS whose clock runs ahead of grain's could see an older export overwrite a newer stream write. That is a follow-up for index/jetstream.
• SIGNALS is defined here as well as in the Jetstream pull. Whichever merges second should take the other's.
• A worker task that panics leaves its DID out of the busy set's cleanup, so that DID is not retried until restart.
• Records are applied one transaction each. A very large repo holds the index's write lock in many short turns rather than one long one.

Review

The design is worth a look, especially record_rev and the stamp for writes that carry no revision, in index/mod.rs and index/backfill.rs. Then backfill.rs, from read_and_apply, and public, may_ask and PublicOnly for the address policy.

🤖 Generated with Claude Code

This pull adds grain_serve::backfill. It discovers accounts from the collection directories, reads each repo from hubble as STAR-lite or from the account's PDS as a CAR, and checks it. It adds record_rev, so the newer revision wins. I read backfill.rs, index/backfill.rs and the revision logic in index/mod.rs. I did not read the full 168 KB diff (it includes the stacked backfill/star commits), tests/backfill.rs, the schema migration or lib.rs. All four checks were still queued, so none has a result.

The revision design holds together. I checked newer/stamp, the stale-write handling, backfilled's deletions and the tombstone cleanup, and found no fault.

Must change: the PDS URL comes from the DID document and is fetched unchecked. read_and_apply takes the endpoint from identity::service and fetch GETs {pds}/xrpc/com.atproto.sync.getRepo with no checks. Anyone can publish a did:web document, or a PLC one, and write a signal-collection record. They can then point the endpoint at loopback, a cloud metadata address or another internal host, and grain-serve will request it.

• The client follows redirects by default, so a public-looking PDS can redirect inward.
• The failure text, including the status, is stored in last_error.

Before this runs with GRAIN_BACKFILL=1:

• Accept only https endpoints, apart from an explicit test or dev override.
• Refuse hosts that are loopback, private, link-local or otherwise non-public. Check the resolved address, for example with a custom resolver, so a hostname that resolves to a private address is refused too.
• Disable redirects, or re-validate each hop.

The tests use a local stand-in server, so they will need that override.

The shortcomings you list (a panicked task's DID staying busy, and the Jetstream rev follow-up) are acceptable for now.

The one blocker from round 1 is still open. The PDS URL comes from the account's DID document, and getRepo is still fetched with no checks. I found none in this head (a31b6fa).

• Backfill::new in crates/grain-serve/src/backfill.rs builds a plain reqwest::Client with only a user agent and a connect timeout. Redirects are on by default, and no scheme or host check exists. A search for "redirect" in grain-serve finds nothing.
• Anyone can publish a did:web or PLC document that points at loopback, a cloud metadata address or another internal host. They then write a signal-collection record, and grain-serve will request that address once GRAIN_BACKFILL=1 is set. A public PDS can also redirect inward.
• The failure text, including the status, is stored in last_error.

What to change:

• Accept only https PDS endpoints, apart from an explicit test or dev override. The tests' local stand-in will need that override.
• Refuse hosts that are loopback, private, link-local or otherwise non-public. Check the resolved address, for example with a custom DNS resolver, so a hostname that resolves to a private address is refused too.
• Set redirect(Policy::none()) on the client, or re-validate each hop.

The same applies to the hubble mirror URL if it can come from anywhere other than config.

All four checks are still queued, so none has a result.

The blocker from rounds 1 and 2 is fixed, so I approve.

• https only: may_ask in backfill.rs refuses a PDS that is not on https. read_and_apply calls it before anything is fetched, and a refusal gives up at once.
• IP literals: a PDS given as an IP address must be public. Url parsing normalises odd forms such as decimal IPs, so they reach the same check.
• Hostnames: Backfill::new installs the PublicOnly resolver unless private is set, so a name that resolves inward is refused.
• Redirects: the client sets redirect(Policy::none()).
• Test override: GRAIN_BACKFILL_PRIVATE / Config::private lifts all of this, and the tests use it.

Unit tests cover the URL check and the resolver.

The four checks on this head are still queued, so none has a result. Make sure they pass before you merge. The shortcomings you list (the configured mirror and directories are trusted, a panicked task's DID stays busy, and Jetstream passing rev later) are acceptable for now.