{"item":{"uri":"at://did:plc:bcgltzqazw5tb6k2g3ttenbj/dev.knotless.pull/3mza7hjmguydu","kind":"pull","title":"The sitemap names the profiles and galleries a search engine may show","state":"closed","author":{"did":"did:plc:bcgltzqazw5tb6k2g3ttenbj","handle":"chadtmiller.com","avatar":"https://cdn.bsky.app/img/avatar/plain/did:plc:bcgltzqazw5tb6k2g3ttenbj/bafkreig5bcb7dl7aphw6eg4e7kjkvpdshfexdvk7jldg53ypre6ecdhesu@jpeg"},"createdAt":"2026-11-01T06:44:04.581Z","comments":5},"body":"## What changes\n\n`social.grain.server.getSitemap` answers a page of the paths a search engine may index, for the web's `/sitemap.xml` and `/sitemap/-.xml` (`read/sitemap.rs`): `kind` is `galleries` (the latest first, by when each was made or first indexed, whichever is earlier, as the feeds order them) or `profiles` (by DID), `page` from 0, 10,000 entries a page, and `pageCount`. Each entry is a path named by the account's handle where it has a valid one, and its DID where not, with `lastmod`, in UTC: a gallery's `updatedAt`, or its `createdAt`; a profile's latest listed gallery.\n\nA search engine can neither blur nor warn, so a gallery is listed only where its author is shown, no label of grain's stands on it or on its author (`unlabeled`, whatever the label says), its author gave it no self-label, and it has a photo. A profile is listed where it has such a gallery and no label stands on its profile record. What the index holds is what the network serves, so nothing in a space is listed.\n\nOne of the pulls closing what the clients call and grain-serve did not serve; the table of them all is in `gaps/graph`'s body (at://did:plc:bcgltzqazw5tb6k2g3ttenbj/dev.knotless.pull/3mza75hfdbrq6).\n\n## Why\n\nThe web's sitemap routes call it on the server, and grain-serve answered 501. knotless has no sitemap (its pages are found by their links, and `robots.txt` keeps crawlers to them), so this follows grain's reads and moderation filters.\n\n## Differences from the old server\n\n- Galleries run by `sort_at` (the earlier of made and first indexed), which an index serves, rather than `createdAt` alone, so a clock set ahead does not move one up.\n- A page is not kept an hour: each call asks the index. The pages' queries read by index; the counts read every gallery, as a sitemap names them all.\n- `kind` other than the two, and a `page` that is not a whole number from 0, are refused (400), where the old server read them as `galleries` and 0.\n- `lastmod` is written by SQLite in UTC to the millisecond; the old server wrote JavaScript's `toISOString`, the same text.\n\n## How it was checked\n\n`crates/grain-serve/tests/sitemap.rs`: galleries with a photo listed in order with their `lastmod` (an `updatedAt` with an offset written in UTC), one without a photo, one labelled, one self-labelled, an author labelled and one taken down left out, an account with no handle named by DID; profiles listed and one whose record is labelled left out; past the last page empty; a bad kind or page refused. The plan check covers the pages' queries. Run here once (it needs no PDS); the rest is the runner's.\n\n## Shortcomings\n\n- No answer is cached. A crawler reading every page counts every gallery each time; at grain's size that is milliseconds, and a cache can come when it is not.\n\n🤖 Generated with [Claude Code](https://claude.com/claude-code)\n","html":"

What changes

social.grain.server.getSitemap answers a page of the paths a search engine may index, for the web's /sitemap.xml and /sitemap/<kind>-<page>.xml (read/sitemap.rs): kind is galleries (the latest first, by when each was made or first indexed, whichever is earlier, as the feeds order them) or profiles (by DID), page from 0, 10,000 entries a page, and pageCount. Each entry is a path named by the account's handle where it has a valid one, and its DID where not, with lastmod, in UTC: a gallery's updatedAt, or its createdAt; a profile's latest listed gallery.

A search engine can neither blur nor warn, so a gallery is listed only where its author is shown, no label of grain's stands on it or on its author (unlabeled, whatever the label says), its author gave it no self-label, and it has a photo. A profile is listed where it has such a gallery and no label stands on its profile record. What the index holds is what the network serves, so nothing in a space is listed.

One of the pulls closing what the clients call and grain-serve did not serve; the table of them all is in gaps/graph's body (Accounts mute and unmute, and list whom they block and mute).

Why

The web's sitemap routes call it on the server, and grain-serve answered 501. knotless has no sitemap (its pages are found by their links, and robots.txt keeps crawlers to them), so this follows grain's reads and moderation filters.

Differences from the old server

Galleries run by sort_at (the earlier of made and first indexed), which an index serves, rather than createdAt alone, so a clock set ahead does not move one up.
A page is not kept an hour: each call asks the index. The pages' queries read by index; the counts read every gallery, as a sitemap names them all.
kind other than the two, and a page that is not a whole number from 0, are refused (400), where the old server read them as galleries and 0.
lastmod is written by SQLite in UTC to the millisecond; the old server wrote JavaScript's toISOString, the same text.

How it was checked

crates/grain-serve/tests/sitemap.rs: galleries with a photo listed in order with their lastmod (an updatedAt with an offset written in UTC), one without a photo, one labelled, one self-labelled, an author labelled and one taken down left out, an account with no handle named by DID; profiles listed and one whose record is labelled left out; past the last page empty; a bad kind or page refused. The plan check covers the pages' queries. Run here once (it needs no PDS); the rest is the runner's.

Shortcomings

No answer is cached. A crawler reading every page counts every gallery each time; at grain's size that is milliseconds, and a cache can come when it is not.

🤖 Generated with Claude Code

\n","base":"feature/rust","head":{"repo":{"did":"did:plc:e7rftrdyz5e2rw4y6ocszew2","handle":"grain.social","name":"grain"},"branch":"gaps/sitemap"},"comments":[{"uri":"at://did:plc:kqi2glvzerg4dt66k64co2sl/dev.knotless.comment/3mza7ixwnff2b","author":{"did":"did:plc:kqi2glvzerg4dt66k64co2sl","handle":"review.knotless.dev","bot":true},"createdAt":"2026-11-01T06:44:53.150Z","body":"This adds `social.grain.server.getSitemap` with pages of galleries and profiles. It filters on author shown, no grain label on the gallery or its author, no self-label, and at least one photo. I found nothing that must change. The filters reuse the existing `shown` and `unlabeled` helpers. Bad `kind` and `page` values are refused with 400. The offset arithmetic saturates, and the page count is correct. The test covers the cases the description lists, and the plan check covers the new queries.\n\nOne small point, not blocking. A profile's `lastmod` is `max(g.created_at)` over the raw column, while a gallery's `lastmod` is `coalesce(updated_at, created_at)`. `max` compares strings, so a `createdAt` written with a UTC offset can pick the wrong \"latest\" gallery. Using `sort_at`, or `max` over the normalized value, would make the two consistent.","html":"

This adds social.grain.server.getSitemap with pages of galleries and profiles. It filters on author shown, no grain label on the gallery or its author, no self-label, and at least one photo. I found nothing that must change. The filters reuse the existing shown and unlabeled helpers. Bad kind and page values are refused with 400. The offset arithmetic saturates, and the page count is correct. The test covers the cases the description lists, and the plan check covers the new queries.

One small point, not blocking. A profile's lastmod is max(g.created_at) over the raw column, while a gallery's lastmod is coalesce(updated_at, created_at). max compares strings, so a createdAt written with a UTC offset can pick the wrong \"latest\" gallery. Using sort_at, or max over the normalized value, would make the two consistent.

\n","verdict":"approve","commit":"ab3fa56b50cb7d8511b9ffdf72f5899e8968c244"},{"uri":"at://did:plc:kqi2glvzerg4dt66k64co2sl/dev.knotless.comment/3mzaadmesaz2b","author":{"did":"did:plc:kqi2glvzerg4dt66k64co2sl","handle":"review.knotless.dev","bot":true},"createdAt":"2026-11-01T06:59:47.000Z","body":"Carried forward: the change this pull makes at `852e9b181d0de1cdc137510a7f2ac93f20b98602` is the one reviewed at `ab3fa56b50cb7d8511b9ffdf72f5899e8968c244`, so that review stands. What is new comes from its base.","html":"

Carried forward: the change this pull makes at 852e9b181d0de1cdc137510a7f2ac93f20b98602 is the one reviewed at ab3fa56b50cb7d8511b9ffdf72f5899e8968c244, so that review stands. What is new comes from its base.

\n","verdict":"approve","commit":"852e9b181d0de1cdc137510a7f2ac93f20b98602"},{"uri":"at://did:plc:kqi2glvzerg4dt66k64co2sl/dev.knotless.comment/3mzaagirxhm2b","author":{"did":"did:plc:kqi2glvzerg4dt66k64co2sl","handle":"review.knotless.dev","bot":true},"createdAt":"2026-11-01T07:01:23.900Z","body":"This adds `social.grain.server.getSitemap`, with pages of galleries and profiles. A gallery is listed only if its author is shown, no grain label stands on it or its author, it has no self-label, and it has a photo. A profile also needs an unlabelled profile record.\n\nMy earlier point is fixed. A profile's `lastmod` is now `max` over `strftime(..., coalesce(updated_at, created_at))`, the same UTC-normalized value a gallery uses, so an offset in `createdAt` or `updatedAt` can't pick the wrong latest gallery. The test checks this: alice's `+02:00` `updatedAt` comes out as `07:00:00.000Z` in both the gallery entry and the profile entry.\n\nI found nothing else that must change. Bad `kind` and `page` values are refused with 400, the offset saturates, and `pageCount` rounds up correctly. The test covers each case the description lists, and the plan check includes both page queries.","html":"

This adds social.grain.server.getSitemap, with pages of galleries and profiles. A gallery is listed only if its author is shown, no grain label stands on it or its author, it has no self-label, and it has a photo. A profile also needs an unlabelled profile record.

My earlier point is fixed. A profile's lastmod is now max over strftime(..., coalesce(updated_at, created_at)), the same UTC-normalized value a gallery uses, so an offset in createdAt or updatedAt can't pick the wrong latest gallery. The test checks this: alice's +02:00 updatedAt comes out as 07:00:00.000Z in both the gallery entry and the profile entry.

I found nothing else that must change. Bad kind and page values are refused with 400, the offset saturates, and pageCount rounds up correctly. The test covers each case the description lists, and the plan check includes both page queries.

\n","verdict":"approve","commit":"66ad04c999bb673c727528ca8ac612ec50e8ee20"},{"uri":"at://did:plc:bcgltzqazw5tb6k2g3ttenbj/dev.knotless.comment/3mzaagsatosth","author":{"did":"did:plc:bcgltzqazw5tb6k2g3ttenbj","handle":"chadtmiller.com","avatar":"https://cdn.bsky.app/img/avatar/plain/did:plc:bcgltzqazw5tb6k2g3ttenbj/bafkreig5bcb7dl7aphw6eg4e7kjkvpdshfexdvk7jldg53ypre6ecdhesu@jpeg"},"createdAt":"2026-11-01T07:01:33.825Z","body":"Profile lastmod: fixed at 66ad04c. It is now `max` over each gallery's own lastmod normalized to UTC (`strftime` of `coalesce(updated_at, created_at)`), so it agrees with the galleries' sitemap and an offset cannot pick the wrong one.","html":"

Profile lastmod: fixed at 66ad04c. It is now max over each gallery's own lastmod normalized to UTC (strftime of coalesce(updated_at, created_at)), so it agrees with the galleries' sitemap and an offset cannot pick the wrong one.

\n"},{"uri":"at://did:plc:bcgltzqazw5tb6k2g3ttenbj/dev.knotless.comment/3mzafjx5slg2b","author":{"did":"did:plc:bcgltzqazw5tb6k2g3ttenbj","handle":"chadtmiller.com","avatar":"https://cdn.bsky.app/img/avatar/plain/did:plc:bcgltzqazw5tb6k2g3ttenbj/bafkreig5bcb7dl7aphw6eg4e7kjkvpdshfexdvk7jldg53ypre6ecdhesu@jpeg"},"createdAt":"2026-11-01T08:32:48.341Z","body":"Landed on feature/rust at fc5f686 through the batch pull at://did:plc:bcgltzqazw5tb6k2g3ttenbj/dev.knotless.pull/3mzadedttgi2b, checked there as a whole. This pull's head is in feature/rust's history.","html":"

Landed on feature/rust at fc5f686 through the batch pull Seven reviewed pulls land on feature/rust together, checked once, checked there as a whole. This pull's head is in feature/rust's history.

\n"}],"reviews":[{"reviewer":{"did":"did:plc:kqi2glvzerg4dt66k64co2sl","handle":"review.knotless.dev","bot":true},"verdict":"approve","uri":"at://did:plc:kqi2glvzerg4dt66k64co2sl/dev.knotless.comment/3mzaagirxhm2b","commit":"66ad04c999bb673c727528ca8ac612ec50e8ee20","outdated":false,"maintainer":false}],"closes":[],"checks":[],"minutes":[{"runner":{"did":"did:plc:4bocpkn5zkwe4wuu4plrj5kt","handle":"runner.knotless.dev","bot":true},"minutes":2505,"freeing":[{"at":"2026-11-25T00:00:00Z","minutes":2},{"at":"2026-11-27T00:00:00Z","minutes":24},{"at":"2026-11-28T00:00:00Z","minutes":10},{"at":"2026-11-29T00:00:00Z","minutes":7},{"at":"2026-11-30T00:00:00Z","minutes":310},{"at":"2026-12-01T00:00:00Z","minutes":1355},{"at":"2026-12-02T00:00:00Z","minutes":509},{"at":"2026-12-06T00:00:00Z","minutes":43},{"at":"2026-12-07T00:00:00Z","minutes":185},{"at":"2026-12-08T00:00:00Z","minutes":16},{"at":"2026-12-09T00:00:00Z","minutes":38}],"updatedAt":"2026-11-10T01:31:15.298Z"},{"runner":{"did":"did:plc:ejh2v3z5ieqhqgz3ybk6r35x","handle":"ci.knotless.dev","bot":true},"minutes":28,"freeing":[{"at":"2026-11-29T00:00:00Z","minutes":2},{"at":"2026-11-30T00:00:00Z","minutes":7},{"at":"2026-12-01T00:00:00Z","minutes":1},{"at":"2026-12-06T00:00:00Z","minutes":6},{"at":"2026-12-07T00:00:00Z","minutes":8}],"updatedAt":"2026-11-08T00:05:20.251Z"}],"changes":{"repo":{"did":"did:plc:e7rftrdyz5e2rw4y6ocszew2","handle":"grain.social","name":"grain"},"commits":[],"files":[],"head":"66ad04c999bb673c727528ca8ac612ec50e8ee20"},"card":"/grain.social/grain/pull/did:plc:bcgltzqazw5tb6k2g3ttenbj/3mza7hjmguydu/card.png?v=a65f440af9256789"}

grain.social/grain Switch repository Loading…

The sitemap names the profiles and galleries a search engine may show

grain.social/grain

Switch repository

Loading…