chadtmiller.com/knotless

Switch repository

Loading…

What changes

serve holds crawlers to robots.txt. A page request whose User-Agent says it is a bot, a crawler or a spider, and whose path and query web/static/robots.txt disallows, gets a 403 before it is handed to the pages. serve reads the same file, built in with include_str!, so the two cannot drift. It matches the rules as crawlers do: * for any run, $ for the end, escapes decoded on both sides (so q=state:closed and q=state%3Aclosed match the same rule), and the longest matching rule deciding, with Allow winning a tie.

A link preview (Slack, Discord, Twitter, Telegram, WhatsApp, LinkedIn, embed services) is not held to it: someone shared that link, and a commit or check link pasted in a chat should still unfurl. Browsers, the CLI, and the harness's own clients are never matched.

Why

The robots rules for list filters landed yesterday (Crawlers skip list filters, and a page read retries a lost connection), and left a rate limit out until they were measured. Measured now, from the request logs:

• In 3 minutes of the info log, 43 of 500 requests were Amazonbot, every one on /chadtmiller.com/knotless/issues?q=… with up to nine label and author terms, all disallowed.
• In an hour of requests over 500 ms, the slowest page was one of those: 1.85 s, with its queries taking about 170 ms. The rest was the page server, which draws one page at a time, so a person's page waits behind it.
• GPTBot no longer showed up on filters in that sample, so the rules are working for the crawlers that re-read them. Amazonbot has not re-read them, or ignores them.

Turning these away in serve costs a string match, with no page drawn and no listItems asked. A 403 rather than a 429: a 429 tells Google to slow its whole crawl, and these URLs should not be crawled at all. Doing it in serve rather than Caddy keeps it tested, and it ships with each deploy, with nothing to change on the host.

How it was checked

pages_drawn_elsewhere now sends twelve requests through the pages proxy:

• 403: GPTBot on a label filter, a list's __data.json, a non-main tree, /search, and a closed filter with another term.
• 200: GPTBot on the closed list (escaped and not), its next page, the plain list, tree/main, and a profile.
• 200: Slack's link expander on a commit, and a browser on a filter.

The build, clippy and coverage are left to CI.

Shortcomings

A crawler that calls itself a browser is not caught. One that says neither bot, crawl nor spider is not either. Only page requests are checked: an XRPC call a crawler makes directly is answered as before. A new link previewer that names itself a bot gets a 403 on disallowed pages until it is added to the list.

🤖 Generated with Claude Code

This makes the pages proxy return 403 to a bot, crawler or spider User-Agent that asks for a path web/static/robots.txt disallows. It reads the same file through include_str! and matches it with *, $, decoded escapes, longest rule wins, and Allow winning a tie. Link previewers are exempt.

I traced the matcher against the shipped rules and found no bug. * runs, a final $ after a * part, the closed-list Allow rules beating Disallow: …issues?q=, and tree/main beating tree/ all come out as the new test expects. The twelve test cases cover both outcomes and the main edges. The design page and the robots.txt comment are updated, and no change file is needed because only serve changes.

Two minor notes, neither blocking:

• Because both sides are decoded, a crafted q=state%3Aclosed%26label:x decodes to q=state:closed&label:x. That still hits the Disallow, but a %26cursor= smuggled into the value would hit the Allow for &cursor=. Real crawlers decode the same way, so this only matters for an adversarial one.
• The bot substring check will also catch any in-house monitor whose User-Agent contains "bot". That only matters if one fetches a disallowed page.

Carried forward: the change this pull makes at 4748f8c320de2b242ac8cf2a11f48fda5dc3b36b is the one reviewed at 0e0704b36ed2db866e1d6c1c2a38eb597b011578, so that review stands. What is new comes from its base.