chadtmiller.com/knotless
git that lives in your Atmosphere account
7
# Performance
On a real network, an agent spends most of its time waiting on round trips
to the PDS, so every operation is planned around them.
## Round trips
| Operation | Round trips, warm |
| --- | --- |
| `git push` | 2: one to read the refs and newest packs (three reads in parallel), one `applyWrites` |
| `git fetch` | 1 when the new packs are under 192 KiB, as they arrive inside the listing; larger ones add parallel blob reads |
| `knotless commit`, no checkout | 2, plus 3 per lost race |
| A ref's compare-and-swap | 0 extra: its claim record goes in the same `applyWrites` |
"Warm" means after the first call: the identity, session and repository key
are cached on disk, and every request in a process shares one connection.
[Storage](../design/storage.md) explains each count.
## Benchmarks
Measured with `cargo run --release --example bench -- --rtt 50` (`just
bench 50`): the npm reference PDS (`@atproto/pds` 0.5.43, SQLite) on the
same machine, behind a proxy adding 50 ms of round-trip time, on an
Apple-silicon laptop.
| | p50 | p95 |
| --- | --- | --- |
| `git push`, one small commit | 223 ms | 276 ms |
| `git fetch`, one new commit | 177 ms | 211 ms |
| `knotless commit`, no checkout | 366 ms | 596 ms |
| `git clone`, 21 packs | 240 ms | 253 ms |
| `git clone`, compacted | 158 ms | 172 ms |
| `git clone --depth 1` via `knotless-serve` | 208 ms | 299 ms |
| One file via the `knotless-serve` API, warm | 45 ms | 57 ms |
| 16 agents pushing 16 branches at once | 443 ms total | |
| 16 agents committing to one branch at once | 10.9 s total, 97 pushes | |
## Many writers on one branch
A branch is one compare-and-swap, so of the writers pushing straight to one
branch, one lands per round. The reference PDS also serializes an account's
writes. That is why the last row above is slow. Give each agent its own
branch and merge, or send commits through a
[gateway](../design/gateway.md#commit), which queues them per branch and
lands each queue as one push.
## Large repositories
Measured by pushing three real repositories into the npm reference PDS
(`@atproto/pds` 0.5.43, SQLite) on the same Apple-silicon laptop, with no
round-trip time added, so the times are mostly CPU and disk. Over a real
network, add the time it takes to move the bytes. Each repository's default
branch went first, with its whole history, then its tags.
| | golang/go | python/cpython | nodejs/node |
| --- | --- | --- | --- |
| History | 68,207 commits, 500 tags | 133,994 commits, 685 tags | 49,240 commits, 969 tags |
| On the PDS, compacted | 450 MB | 723 MB | 1.53 GB |
| First push, the branch | 6.5 s | 10.7 s | 15.4 s |
| Push of the tags | 6.3 s | 18.7 s | 35.9 s |
| `git clone`, cold | 20 s | 38 s | 51 s |
| `git push` of one small commit | 0.2 s | 0.2 s | 0.2 s |
| `git fetch` of it | 0.1 s | 0.2 s | 0.2 s |
| `knotless compact` | 25 s | 44 s | 79 s |
| Peak memory: push, clone, compact | 0.9, 1.3, 1.9 GB | 1.4, 2.1, 2.1 GB | 2.8, 4.3, 3.7 GB |
A repository's size does not slow everyday work. A push or fetch moves
only its own pack, so one small commit takes the same time in a 1.5 GB
repository as in a new one. Size shows up in four places:
- **Memory.** The helper holds a whole pack in memory, and more than once:
`pack-objects` is read into one buffer, which a push then copies into its
blob parts, and a fetch joins every part before `index-pack` reads it.
Cloning or compacting takes two to three times the repository's packed
size. Streaming the pack to and from the blobs would keep this flat.
- **Packs after the first.** Every pack is self-contained, so a later push
cannot store objects as deltas against what the PDS already holds.
Pushing node's tags added 780 MB to its 1.44 GB branch. Compaction folds
them back into one pack, here of 1.53 GB.
- **Write points.** A PDS with rate limits on, as bsky.social has, gives an
account 5000 write points an hour and 35000 a day, counting every write
inside an `applyWrites`, so batching saves round trips but not points. A
new ref is one create, 3 points, so about 1600 new refs fit in an hour,
and node's 969 tags push in one go. Running out stops more than the
push: until the hour is up, every write from the account fails, a
one-commit push, compaction, and any other app writing as that account.
A push the limit stops keeps what landed, and names the refs it did not
push and when to push again. A repository with several thousand tags
takes more than an hour; a record holding many tags at once, as git's
`packed-refs` does, would make them a few writes, at the cost of a
compare-and-swap for each tag.
- **Blob uploads.** The same PDS allows 1000 `uploadBlob` a day, at 4 MB a
part. Node's branch took about 360 parts, its tags about 200, and a
compaction would take about 380 more, so importing a repository this
size and compacting it uses most of a day's uploads.
## Web pages
`knotless-serve` reads what a page shows from a mirror's objects in
process, in about a millisecond warm; a highlighted 40 KB source file takes
a few more. Drawing the page takes a few milliseconds more. A social card
takes about 5 ms to draw the first time, is about 30 KB, and is served from
memory after that (`just bench-cards` measures it).
## Caches in knotless-serve
| What | Kept |
| --- | --- |
| Social cards | by version, the oldest dropped past 512; at most one drawn per core |
| Pictures for cards | 128, each fetched within 2 s and at most 1 MiB |
| Avatars and email claims | 600 s; a PDS that did not answer is asked again |
| Languages bars | by tree, 1024 trees, cleared all at once past that |
| Objects read in process | 4 MiB of object cache per thread's handle on a mirror |