chadtmiller.com/pds.js

Switch repository

Loading…

What changes

A commit deletes what it leaves unreachable. As the reference PDS drops a commit's removedCids, commitUnderLock now deletes three kinds of block. The first is the old tree's nodes it read that the new tree no longer reaches (unreachedNodes in mst.js). The second is the records it replaced or deleted that no other record still has as its CID (the reference's getDuplicateRecordCids, here recordCidsAmong). The third is the previous commit block. Until now nothing deleted a block, so every old tree node and every deleted record stayed in storage.

getRepo reads the block table and streams it. Once the table holds only what the head reaches, getRepo reads it 500 blocks at a time and sends each page as it goes. It no longer walks the tree block by block and builds the whole CAR first. The reference PDS reads its own table the same way.

One sweep for what was stored before. A store keeps a flag, blocksCollected. Until it is set, the first getRepo sweeps under the repo lock. It walks the head, keeps that plus every record's CID, deletes the rest a page at a time, then sets the flag. A rebuilt tree or an imported CAR clears the flag again.

sync.getRecord sends the proof path, the nodes a lookup of the key reads, rather than every node of the tree.

A backup snapshot retries from the new head when a commit lands during its walk and drops a node it had not reached, up to three heads.

The new port methods are optional, so a store without them (read-only, the test doubles) keeps every block and walks as before. Both SQL adapters have them, and one conformance suite runs over each, the Cloudflare one through the D1 shim.

Why

A Durable Object answers every request in turn. On a whole-repo walk the account's own requests waited behind it: measured at 3.5 s, and 8.8 s once 1295d196 made the walk take turns. microcosm's lightrail calls getRepo after every collection deletion it cannot prove. Reading pages straight from the table is far fewer, cheaper reads, and streaming lets requests run between pages. The table could not be read as the repo until commits stopped leaving garbage in it.

How it was checked

npm run ci: 3259 tests, typecheck, lint, coverage.

• packages/core/test/collect-blocks.test.js drives a server through 60 creates, 20 updates and 30 deletes. It checks that the stored blocks are exactly what the head reaches, that a deleted record's block goes and a shared one stays, that getRepo sweeps and then serves the table, that a rebuild is swept, and that getRecord carries only the path. Each half of the fix was disabled in turn, and its tests failed.
• test/backup-api.test.js covers the snapshot retry.

The workerd e2e suite was not run: its harness binds port 8787, which another app holds on this machine.

Shortcomings

• The first getRepo after upgrading holds writes while it sweeps: a walk of the whole repo, then one pass over the table.
• A commit landing while getRepo streams can delete blocks the stream has not sent yet. The consumer then gets a CAR missing part of its root's tree and asks again. The reference's stream has the same race.

🤖 Generated with Claude Code