~ knotless Search/
↑↓ move↵ open⇧↵ all results

chadtmiller.com/3n2r55xzxxjol

lex-space: reading a whole space in the browser, and two things to fix

0
0
main
33 lines · 2.0 KB Raw History

lex-space: reading a whole space in the browser

Notes from measuring how an app on a group host scales when the browser reads every member's repo in a space itself (lex-space's allRecords), with no index service in between.

Measurements

Against a local group host world, debug builds, on a loaded machine:

RoomCold openDownloadedClient holdsViews in the client8 readers at once
30 records0.25–0.6 s15 KB0.3 MBunder 0.1 msunder 1 s each
2,065 records0.5–0.8 s0.7 MB8 MB2 ms or less1.6–2.2 s each
19,175 records3.3–3.6 s, 3.2 s of it client CPU checking records5.3 MB72 MB3–20 ms15–17 s each

Working out views in the client (counts, threads, topics by latest reply) is never the cost. Getting and checking the records is.

1. getRepoSnapshot hashes every record twice

Reading a repo, it hashes each record once to check the records and again to compute heads. At 19k records nearly all of a cold open is that client CPU.

Checking without computing heads would roughly halve it (an estimate, not measured).

2. allRecords keeps nothing between calls

Every reader pays the full read on every open.

Reading on is cheap at any size: about 6 requests, 6 KB and 150 ms after new writes, and one request when nothing changed. lex-space already has what it needs (writers({cursor}), and pullRepo from kept heads), but allRecords doesn't use it.

The fix is to keep each repo's head and records, in memory while the page is open and in IndexedDB between visits, and read on from there. Then a member pays the full read once per device. This matters more than #1 at scale.

Where that leaves an app

The browser alone serves groups up to a few thousand records in a space. Past that, an app should read from an index service: one that registers as a syncer of the space (authorizeSpaceSync with notify), keeping counts, links, orders and hiding.