~ knotless Search/
↑↓ move↵ open⇧↵ all results

chadtmiller.com/knotless
Switch repository
Loading…

git that lives in your Atmosphere account

knotless.dev
1 collaborator

knotless 0.13.0 Latest 2 packages, 1 image

1
10

The pages' own queries are counted as pages, not as a bot

Merged
Merged chadtmiller.com asked to merge into main from chadtmiller.com/knotless:serve/pages-client
1 reviewer 1 approved
Reviewers
review.knotless.devRequired reviewer for main Approved
chadtmiller.comopened

The stats counted 2.37M requests in 14 days from a crawler called node: those were the pages' own XRPC queries as they draw each page, since Node's fetch calls itself node, which the stats take for a script. The pages now name themselves knotless-pages, and the stats count them as pages, apart from every client. A quick look will do.

Claude Opus 5.5 · high

What changes

The pages name their queries. handleFetch in web/src/hooks.server.ts, which hands the pages' XRPC queries to knotless-serve, sets User-Agent: knotless-pages.

The stats count them apart. usage::client takes exactly knotless-pages as the kind pages, before the bot words, so they leave the bot count and the crawlers' agents list. A script calling itself node is still a bot. deploy/README.md (Usage) names the new kind.

Why

admin_stats over 14 days: clients bot 4,710,738, of which the agent node was 2,373,102, more than every real crawler together. The traces show node requests sharing the trace of the page request they drew (9ae9278d4b58ed7c02da91671e1f34b7: /chadtmiller.com/group-host from a browser, its getRepo, getTree, listBranches and others as node). So the bot count roughly doubled, and the crawler list was led by our own server. The request log's user_agent.original for those lines now says knotless-pages too.

How it was checked

usage::tests::clients takes knotless-pages as pages and node as bot. Left to CI.

Shortcomings

The iOS app calls itself Knotless/5 CFNetwork/…, which client takes for the CLI, since it starts knotless/ ignoring case. Not changed here.

review.knotless.devreviewed Approved

The pages' XRPC queries now send User-Agent: knotless-pages. usage::client counts that exact agent as pages, ahead of the bot check. I found no problems.

  • client only adds crawler agents to the list when the kind is bot (usage.rs near line 289), so pages leaves both the bot count and that list.
  • The usage-hit endpoint only takes browser clients, so pages doesn't reach it.
  • A script calling itself node is still a bot, and the test covers that.
  • I didn't run the tests; the pull leaves them to CI.
All checks have passed

8 passed, 2 skipped

8 passed checks checkSuccess in 3sdeploySuccess in 8m 13sdocsSuccess in 2se2eSuccess in 5slexiconsSuccess in 2sshotsSuccess in 3stestSuccess in 4swebSuccess in 3s
2 skipped checks publish-lexiconsSkippedreleaseSkipped
Merged as 91a3761