~ knotless Search/
↑↓ move↵ open⇧↵ all results

chadtmiller.com/knotless
Switch repository
Loading…

git that lives in your Atmosphere account

knotless.dev
1 collaborator

knotless 0.13.0 Latest 2 packages, 1 image

1
10

The harness PDS keeps its databases in memory where it can

Closed
Closed chadtmiller.com asked to merge into main from chadtmiller.com/knotless:perf/pds-tmpfs
1 reviewer 1 approved
Reviewers
review.knotless.devRequired reviewer for main Approved
chadtmiller.comopened

What changes

The harness PDS keeps its databases in /dev/shm. harness/pds.mjs points the PDS's temp directory (where the dev-env puts each account's SQLite database, the blobstore and its certificate) at a directory in /dev/shm when that has 2 GiB free, and removes it when the harness stops. KNOTLESS_PDS_DISK=1 keeps the data on disk. Without a roomy /dev/shm, as on a Mac, nothing changes. The PDS code is the unchanged npm one.

Why

Every PDS write is a SQLite commit across several databases, each waiting on an fsync. Measured on the runner, an fsync of 4 KiB in /tmp takes 21.8 ms; in /dev/shm 0.01 ms. A pull create or git push does several of them, which is most of the 0.5 to 1.5 s a call costs there against tens of milliseconds on an idle local disk. The test binaries wait on this one disk through the two shared PDSes.

How it was checked

CI's test check. Before and after times are in a comment.

Shortcomings

If a runner's /dev/shm is under 2 GiB the change does nothing there. Memory used by the databases counts against the job's cgroup.

Changeset: none

chadtmiller.com added 2 commits
review.knotless.devreviewed Approved

This points the harness PDS's temp directory at a fresh /dev/shm/knotless-pds-* when that has 2 GiB free, and removes it on exit. KNOTLESS_PDS_DISK=1 opts out. I found nothing that must change.

  • TMPDIR is set before the certs mkdtempSync(join(tmpdir(), ...)) call and before the dev-env import. os.tmpdir() reads the environment on each call, so the certificate and the PDS data both land in the new directory.
  • shutdown ends in process.exit(0), and SIGTERM, SIGINT and stdin close all reach it. The exit handler therefore runs on every normal stop.
  • A SIGKILL or a crash leaves the directory behind, and it counts as RAM until the host is cleaned. This is only worth knowing about on a long-lived dev machine. A stale-directory sweep at startup would fix it, but I wouldn't hold the pull for that.
  • The docs paragraph in docs/development.md matches the behaviour. The Changeset: none line is right, since nothing under crates/ changes.