~ knotless Search/
↑↓ move↵ open⇧↵ all results

grain.social/grain

photography-focused photo sharing platform

grain.social
1 collaborator
0
2

server: new photos, stories and avatars warm the CDN, and what goes is purged

Merged
Merged chadtmiller.com asked to merge into main from grain.social/grain:serve/image-warm

What changes

New images are warmed on the CDN. When the stream, refreshRecords or a web write puts a photo, story or profile in the index, new or changed (Index::kept), grain-serve fetches its image through cdn.grain.social at once, at the presets the views draw it at: feed_thumbnail and feed_fullsize for photos and stories, avatar for avatars (src/edge.rs). The index tells the keeper on a bounded channel and goes on: nothing that indexes waits for the CDN. At most --cdn-warm requests (4 by default, 0 turns it off) go at a time, and one that fails is logged and dropped. Each URL is asked once, with Accept: image/jpeg. ACCEPTS is where image/avif goes once the CDN varies on it. Nothing hidden is warmed: if an account is taken down or gone, or a hiding label stands on the record, its account or a gallery it is in, a later write does not put back what a takedown purged (Index::shown_images). Backfill warms nothing, and Bluesky's CDN is never warmed or purged.

What goes is purged from Bunny. These purge every URL grain signs for the image (all six presets) through Bunny's POST /purge?url=, with BUNNY_API_KEY: a takedown of an account (all its photos, stories and avatar) or of a record (a gallery's photos, or a photo's, story's or profile's own image), a photo, story or profile deleted from the index, and an account purged, whether its host deleted it or deleteAccount ran. Purges go one at a time. A 429 or a 5xx is retried after Retry-After (or 1s), up to 10 tries, then logged. With no key, nothing is purged. --bunny-api overrides https://api.bunny.net for tests.

OpenGraph cards and the classifiers read the *_og presets, and cards are JPEG. avatar_og, feed_thumbnail_og and feed_fullsize_og are pinned to ext:jpeg in imgproxy's IMGPROXY_PRESETS, so they stay JPEG when the browser presets move to AVIF. On Bluesky's CDN, which has no such presets, their namesakes are used. Cards are encoded as JPEG q85 (image/jpeg, still public, max-age=300) rather than PNG, and are kept as Bytes, so a cached card is no longer copied on every answer. png moves to dev-dependencies.

Why

Step 2 of the image tuning plan. Stories are slow mostly because the cache is cold: a story lives a day, few people see it, and each viewer is usually the first at their PoP, so they pay the PDS fetch (p50 1 s, eurosky 4 s+) plus imgproxy's encode. Warming at index time means the first viewer gets a hit instead. The infra change to cache for a year (hetzner-infra) leaves a taken-down image reachable at its URL for that year, hence the purge. A Bunny URL purge also deletes from Perma-Cache. grain-serve's image crate cannot decode AVIF, so what it reads itself is pinned to JPEG presets before AVIF is turned on.

No pixels change. The live box's IMGPROXY_PRESETS defines each _og preset with exactly its twin's options (avatar_og=rs:fill:1000:1000:1:1/g:ce/q:85/ext:jpeg = avatar, feed_thumbnail_og=rs:fit:1000:0/q:70/ext:jpeg = feed_thumbnail, feed_fullsize_og=q:90/ext:jpeg = feed_fullsize), checked against the running container's environment.

How it was checked

On the runner. New tests, all against a stand-in CDN and Bunny (tests/cdn):

  • tests/edge.rs covers:
    • each collection warmed at its presets;
    • updates warmed again, while unchanged records and records without images are not;
    • nothing warmed for an account taken down, a photo taken down, or a photo in a gallery taken down;
    • a CDN that answers errors or never answers holding up no index write;
    • the channel dropping work when it is full or closed;
    • purges on delete, takedown, account purge, a Bunny 429, repeated 5xx, a refused key and an unreachable API;
    • Settings::edge only starting for grain's own CDN with something to do;
    • the web's createRecord warming a photo and a story uploaded to the reference PDS.
  • jetstream.rs: what the stream brings is warmed, and a deleted photo and a deleted account are purged.
  • refresh_records.rs: a photo written to the reference PDS and refreshed is warmed; deleted and refreshed, it is purged.
  • moderation.rs: updateSubjectStatus takedowns of a record and of an account purge them.
  • deletes.rs: deleteAccount purges the account's photo.
  • og.rs: cards are read back as JPEG, and their images come from the _og presets.

Shortcomings

  • Before deploy, BUNNY_API_KEY (the Bunny account API key) has to be added to /etc/grain-serve/serve.env on the host. Without it, warming still works and purging does nothing.
  • A Bunny URL purge clears one exact URL. If presets are added later (step 4's w640/w1440), they belong in cdn::PRESETS, or they will not be purged.
  • Deletions found by backfill's reconciliation are not purged. Neither is an avatar replaced by a newer one: the old image stays cached until it expires.
  • Warming asks with image/jpeg only. Step 5 adds image/avif to ACCEPTS once Bunny varies on it.
  • Warm requests are spawned as tasks and wait for a turn, so during a long CDN stall they pile up at the arrival rate (each one times out after 60s once it is sent).
  • Purges run at roughly Bunny's exact-purge rate (about 5/s). Taking down an account with thousands of photos takes minutes, six URLs each.

Review

The design: src/index/edge.rs (what the index says, and where) and src/edge.rs (the keeper).

Approved

This pull warms the CDN for new or changed photos, stories and avatars, and purges Bunny for what is deleted or taken down. It also moves the OpenGraph cards and the classifiers onto the JPEG-pinned *_og presets, and the cards are now encoded as JPEG. I read src/edge.rs, src/index/edge.rs, cdn.rs, lib.rs, og/, classify, the call sites (jetstream, sync, write, moderation) and the deploy files. I did not read the test files (tests/cdn, edge.rs and the others) or the new unit tests in src. I didn't see the run results either.

The design holds up:

  • The index tells the keeper over a bounded try_send channel, so no index write waits on the CDN.
  • kept warms only Created and Updated records.
  • Purge image sets are read inside the delete or purge transaction, before the rows go.
  • Purges are serialized, and a 429 or 5xx is retried within a bound.
  • The Purge Debug impl hides the key.
  • Bluesky's CDN is never touched, because Cdn::own gates the keeper.

Two small points, neither blocking:

  • A takedown purges Bunny, but a later stream update of a taken-down record still warms its image again. If takedowns are meant to keep an image off the CDN, kept would need to skip taken-down subjects.
  • Warm tasks are spawned without limit before they take a semaphore turn. A long CDN stall with the full 10k queue would hold many parked tasks. That is fine at today's volume.

The pull's own Shortcomings list already names the other gaps: backfill deletions and replaced avatars are not purged, and cdn::PRESETS must grow with new presets.

Approved

I read the change since my approval at 7f9232f. It is a refactor of read and record_images into a shared image row reader, rustfmt fixes in the tests, and one new test assertion. The assertion checks that with no key, a purge reaches nothing. Behavior is unchanged, and what I approved still stands.

One thing to check: Edge::run now does recv().await.expect("the index holds the sender") where it used to end quietly on None. If the index is dropped at shutdown, or a test drops it, the keeper task panics instead of finishing. A panic in a detached tokio task is only logged, so it breaks nothing. It does add a noisy panic line on shutdown. Going back to while let or let Some(..) else { break } would avoid that, if you don't need the loop to be infallible.

On the review: (1) Fixed in 919d4b0. The keeper now warms only what is shown (Index::shown_images): nothing of an account taken down or gone, and nothing with a hiding label on the record, its account or a gallery it is in. A later write cannot put back what a takedown purged. Tested in what_is_taken_down_is_not_warmed_again. (2) Left as is. A parked warm task is small, the parking grows only at the arrival rate (about 260 images a day), and each request times out after ctx.http's 60s once it gets a turn. Taking the turn before spawning would make the receive loop wait on the CDN, and purges queued behind a stall would be dropped with the warms. Listed under Shortcomings.

Approved

I read the change since my approval at e2e00a0. It answers the first point in my earlier review, that a later write could warm an image a takedown had purged.

Index::shown_images replaces images for warming. It returns an image only when the account is shown (shown), and no standing hiding label is on the record, its account or a gallery it is in (unhidden). Edge::warm now calls it. The shown and unhidden helpers exist in schema.rs and fit how they are used here. The new test what_is_taken_down_is_not_warmed_again covers a taken-down account, photo and gallery.

One thing to check: the delete tests in tests/edge.rs and tests/deletes.rs now call shown_images where they used to call images. So the old unfiltered images call has no test left. If the purge path still reads images through record_images, which sits outside this diff, it stays unfiltered, as it should. A takedown has to purge the images it hides.

I did not see the run results.

Approved

Since my approval at 919d4b0, the only change is in Edge::run. It now uses while let Some(change) = from.recv().await, so the keeper ends quietly when the index drops its sender. That answers my earlier note about the panic on shutdown. Nothing else changed, so my approval stands. I did not see the run results.

On the review: (1) Edge::run is back to while let Some(change) = from.recv().await (f4bfb71), so a dropped sender ends the keeper quietly instead of panicking. Through Edge::start it never ends in practice, since the index holds the sender and the keeper holds the index, so 15857b7 adds a unit test, edge::tests::the_keeper_ends_when_its_sender_is_gone, that drops the sender and awaits run, which keeps the loop's end covered (gate holds). (2) Nothing calls an unfiltered images reader for warming any more: warming reads Index::shown_images, while purging reads edge::record_images (index/mod.rs:415 and index/edge.rs:102), unfiltered as it should be, so a takedown purges what it hides. moderation.rs and edge.rs cover that path (takedown purges, purges on delete). Checks pass at 15857b7.

Approved

Since my approval at f4bfb71, the only change is a unit test in src/edge.rs. It checks that the keeper's run ends once its sender is dropped, which covers the earlier shutdown note. No behavior changes, so my approval stands. I did not see the run results.

Merged as 371a2e7