chadtmiller.com/knotless

Switch repository

Loading…

What changes

List filters are closed to crawlers. robots.txt now disallows ?q= on a repository's issues and pulls, and those lists' __data.json, while leaving the plain open and closed lists and their pagination open. The links in a filter menu and on a row's labels are rel="nofollow".

A page read survives a lost connection. serve's client for the pages app drops an idle connection after 4 seconds, and a GET or HEAD whose connection fails is asked once more. A write is never asked again, as it may have been done.

Why

Each label and author link adds a term to the list's query, so every mix of them is a new URL, and crawlers follow them without end. Over 6 hours on prod, about 72k of 130k requests from outside had 3 or more label terms, from GPTBot (42.6k requests in all), GoogleOther (about 30k) and Amazonbot (5.4k), nearly all on two repositories. They set off about 63k listItems calls too. It costs little today, with serve at 15% CPU, but it grows with repositories and labels and fills the logs.

The closed lists stay open so that crawlers can still find closed issues and pulls. robots.txt does the real work here: nofollow is only a hint.

The 19 pages 502s in the logs were resets within milliseconds, and all reached crawlers. reqwest keeps an idle connection 90 seconds by default and node closes one after 5, which would produce this race, though it is not proven to be the cause. One of the 502s also lines up with the deploy swapping the pages container, which the retry covers.

A rate limit is left out until these changes are measured.

How it was checked

pages_drawn_elsewhere now has an app that drops its first connection: a GET through it gets 200 from the second try, and a POST gets 502, then 200 from the next POST. The rules in robots.txt were run against 14 sample URLs with a small longest-match script. The rest is left to CI.

After it deploys, crawler requests with label terms get counted again from user_agent.original, over the same 6 hours.

Shortcomings

A crawler that ignores robots.txt is not stopped. Robots rules can take a day or so to be read again. The filter pages' HTML changes, but nothing drawn moves.

🤖 Generated with Claude Code

This pull does two things. robots.txt now disallows ?q= on issue and pull lists and their __data.json, and the filter links get rel="nofollow". serve's pages client now drops an idle connection after 4 seconds and retries a failed GET or HEAD once.

The robots.txt rules look right. Longest-match makes the state%3Aclosed Allow lines win over the shorter ?q= Disallow. The retry is limited to safe methods, and the body and headers are cloned per attempt. The new test covers both the retried read and the write that gets a 502 and is not retried. I couldn't check one thing from here. The closed-list Allow lines only work if the app emits exactly q=state%3Aclosed, with that encoding and that parameter order. If it writes state:closed or puts cursor first, the closed lists are blocked too. Check that against a real closed-list link.

I didn't run the checks. The body leaves them to CI.

The only change since my approval at b2357fd is 3a62255, which reformats the Label.svelte link the way oxfmt asks. The behaviour is the same, so my earlier review stands. The robots.txt rules, the retry limited to GET and HEAD, and the test for the retried read and the unretried write are all unchanged.

My earlier question is still open: do the closed-list Allow lines match what the app emits? They only work if the app writes exactly q=state%3Aclosed, with cursor after it. I searched web/src for a literal state:closed and found none, but I did not find where the link is built. Check one real closed-list link. If it differs, the closed lists are blocked too.

I did not run the checks.