How repositories are stored

A knotless repository lives in its owner's atproto account and nowhere else. Nothing is installed on the owner's PDS, the server that holds the account's records and blobs. Every git operation is a handful of ordinary calls to that server. This page explains which records hold a repository, and how push, fetch and compaction work within what a stock PDS allows. For the protocol itself, see https://atproto.com.

What a stock PDS offers

Everything has to work against the reference PDS (@atproto/pds) as shipped, under its default limits. That server offers records (JSON documents in a collection, each addressed by a record key, or rkey), blobs (binary data stored beside records), and applyWrites, one call that creates, updates and deletes several records, all or none.

It offers no locks. It has no per-record compare-and-swap either: the only guard applyWrites takes is swapCommit, which names the account's latest commit and so conflicts with every other write to the account. Only an account can write to its own records, so a collaborator cannot push straight to the owner's repository. And every call is a round trip that a person, or an agent running a loop, waits on.

Limit	Reference PDS	What it forces
Blob upload	5 MB (PDS_BLOB_UPLOAD_LIMIT), advertised by describeServer	Large packs are split into parts
JSON request body	about 1 MB	Packs ride inside a record only when small
Writes per applyWrites	200	Large pushes span several calls
listRecords	filters by collection, not by key prefix	One repository's keys form one range
Write points, when rate limits are on (as on bsky.social)	5000 an hour per account: create 3, update 2, delete 1	A push is few writes
uploadBlob, when rate limits are on	1000 a day	Small packs use no blobs

The records

A repository is five kinds of record, all in the owner's account, with schemas in lexicons/dev/knotless/.

Collection	Keyed by	Holds
dev.knotless.repo	the repository's name	its identity and settings
dev.knotless.ref	<key>.<h(ref)>	where one ref points, and its generation
dev.knotless.claim	<key>.<h(ref)>.<generation>	one move of one ref
dev.knotless.pack	<key>.<tid>	a git pack, inline or over blobs
dev.knotless.redirect	a former name	where a renamed repository went

A private repository stores the same records inside a space, a permissioned area of the account that only its members can read (see access.md). A public repository is read with no authentication.

The repo record is keyed by the repository's name, so a clone URL resolves with one getRecord. Because the name doubles as a record key, it must be 1 to 100 bytes of [A-Za-z0-9._~-], and not . or ... Its key field is a TID, a time-ordered record key, minted when the repository is created and never reused. Every other record is keyed by key, not by the name, so a rename rewrites only the repo record and its redirects (see identity.md).

A repo record is never written over by a new one of the same name, since its refs and packs would be left under a key nothing names. Creating, forking, accepting a transfer and renaming first look for a repository under the name, public or in the owner's space of that name, and refuse with <url> already exists before writing anything. The write itself does not count on the PDS refusing a create over a record it holds: some PDSes, pds.js among them, treat that as an update. A public record is written with com.atproto.repo.putRecord and a null swapRecord, which the PDS refuses when any record is there. A space's putRecord has no swap, so a private record is an applyWrites create, behind the look. The record also holds what the repository says it is (description), the links its pages show (its website first, up to five, each an http or https address), the default branch, collaborators, runners, gateway, fork and transfer links, branch rules, the labels its pulls and issues may carry, the triagers whose labels count, and the agents whose pulls run as they move, as code no one vouches for (see runner.md), and archivedAt while it is archived and read only (see collaboration.md). A client that rewrites it keeps fields a newer client wrote that it does not know.

A ref record holds the full ref name, its target commit, and a generation: a counter that starts at 1 and goes up every time the ref moves. A deleted ref keeps its record with no target, so its generation never repeats.

A claim records one move of one ref. Its key ends in the generation the ref moved from, and its body holds from, to and the pack written with the move. A delete's has no to. A new ref has no claim: creating its record fails if one exists, which is all a claim would guard, and its record at generation 1 is its reflog entry. A claim is never updated. Its key is the ref's compare-and-swap, and its body is a reflog entry.

A claim, and a ref record, written by the gateway for anyone but the owner names them as pusher: a collaborator, or a runner landing a merge queue. An owner's own push leaves it out. Readers credit the move to the pusher while the repository record names them among its collaborators or runners, and to the owner otherwise, since the owner's account wrote it.

A pack record holds one git pack: every object reachable from its tips and not from its bases, the objects its history needs and does not hold. A pack is self-contained, with no thin deltas, so it can be indexed alone. It is also immutable. Exactly one of data (the pack inline) and parts (the blobs it was split over, in order) is present. A pack written by compaction also lists, in supersedes, the keys of the packs it replaced.

A rename leaves a redirect under the old name, pointing at the repo record by AT-URI and key, and repoints every older redirect, so an old name is one hop from the current one. A redirect is followed only within the account, and only to a repo record with the key it names. Creating a repository under a redirected name replaces the redirect.

Details:

• Every write is com.atproto.repo.applyWrites (com.atproto.space.applyWrites in a space) with #create, #update and #delete writes, never swapCommit. The one putRecord is the creation of a public repo record outside a push, with a null swapRecord (above).
• A push's pack has its tips sorted and de-duplicated; a compaction pack lists one per live ref, in ref-record order. bases are omitted when empty. objects comes from the pack header; size is the byte length, which a fetch checks against what it downloads.
• A redirect of a private repository renamed since it was made carries space, the space its records live in.
• A lookup follows up to 4 redirect hops, so a read can race a rename.

Deleting a repository

knotless delete (the delete_repo tool, the pages' Delete) deletes every record keyed by the repository's key, its refs, claims, packs, releases and reruns, and what its owner wrote about it in their own account: their pulls, issues, comments, states, labels, hides, queue entries and blocks naming it, their star of it, and their reactions to those, unless another account took it up, whose copy shows what was said on this one. Blobs go with the records that named them, as the PDS reaps a blob no record references. What other accounts wrote about it stays in their accounts. The repo record goes last, in a write of its own, with the redirect a repository taken up by another account leaves under its name: a delete cut short finds the record still there, and run again carries on where it stopped. A private repository's space is deleted instead, and everything in it goes with it.

Deleting your knotless data

knotless delete-data (the delete_account_data tool, the pages' account settings) deletes every record an account holds in a dev.knotless.* collection, whichever its PDS's describeRepo lists, and leaves the atproto account and other apps' records alone. It says first what there is: each collection's count, and the repositories, those another account pushes to or reads named, so their owner can transfer them first. Then it deletes, as the account, in this order:

1. the spaces the account owns, each private repository's and those a rename left, as deleting a private repository does;
2. each public repository, as knotless delete does;
3. what is left, a collection at a time, 100 records an applyWrites.

It works a part at a time, so a run cut short carries on when run again, and the pages show how far it has got. Once nothing is left, the appview forgets the account (serve.md). Records the account wrote in other accounts' spaces, as a member of their private repositories, are theirs to keep and stay.

Keys, and one repository as one range

listRecords lists a collection, and a collection holds the records of every repository in the account. Reading one repository's refs must not mean reading all of them. The keys solve this.

Every record of a repository except the repo record is keyed <key>.<suffix>, so a repository's records sit together in each collection. A scan starts from the cursor <key>/. The character / (0x2F) sorts just after . (0x2E), so <key>/ is above every <key>.… key. The reference PDS lists highest keys first and pages below the cursor, so the scan walks the repository's records and stops at the first key outside the prefix. Every TID is 13 symbols long, so no repository's key is a prefix of another's.

A ref name can hold characters a record key cannot. h(ref) turns any name git allows into a valid key: the first 26 characters of the lowercase base32 of the name's SHA-256, 130 bits of the hash. Pack keys end in a TID, so a scan of packs walks them newest first, in reverse push order.

Details:

• A page asks for 100 records. A short page, or one that reaches another repository's key, ends the scan.
• A record that does not parse fails the scan rather than being skipped.

The snapshot

Every operation starts by reading a snapshot: the repo record, all its refs and the newest page of packs. With the repository's key cached locally, the three reads go out at once, one round trip. The repo record still comes along, to catch a name that now means a different repository: if its key differs, refs and packs are reread under the new one. Without a cached key, the repo record comes first, two round trips. A name with no repo record gives an empty snapshot, which a first push can fill.

Push

A push has to move refs safely against concurrent pushes, without locking the account, in as few round trips as possible. A warm push takes two: the snapshot, then one applyWrites.

1. Read the snapshot.
2. Check each refspec. A ref already at its new target writes nothing. An unforced update is refused "fetch first" if its old target is not present locally, and "non-fast-forward" if that target is not an ancestor of the new one. A forced update, a new ref and a delete skip these checks.
3. For a first push, mint the key and create the repo record, with main as default branch if it was pushed, else the first branch pushed.
4. Run git pack-objects with the new targets as tips and every live remote ref target the local repository has as bases. Place the pack inline or over blobs.
5. For each accepted ref, create its claim at the generation read and update its ref record to generation + 1; for a new ref, create its record at generation 1, with no claim.
6. Refuse the whole push if the refs were read more than 12 hours ago.
7. Send the writes in applyWrites calls of at most 200, the first carrying the repo and pack records, each claim beside its ref.

A small push costs 8 rate-limit points (pack create 3, claim create 3, ref update 2), about 600 an hour per account, and a new ref 3. A push of more than about 99 moved refs, or 199 new ones, spans several calls, which are not atomic together: an earlier one can land while a later one loses. One the rate limit refuses (429) ends the push there: the refs of the calls that landed are pushed, and the rest say when the limit's RateLimit-Reset lets writes in again. Pushing them again then writes only those, since a ref already at its target writes nothing. A delete, or a new ref at a commit the remote already has, writes no pack.

Compare-and-swap through claims

Two writers that both read generation n of a ref both try to create claim(<key>.<h>.<n>). A create on an existing key fails, so the PDS accepts only the first. The ref update travels in the same applyWrites as its claim, so it lands only if the claim does. Each ref thus has its own compare-and-swap, with no contention against the account's other writes and no extra round trip. A new ref record is a create, so two racing creates of one new ref fail the same way.

sequenceDiagram
    participant A as Pusher A
    participant P as Owner's PDS
    participant B as Pusher B
    A->>P: snapshot: main at generation 7
    B->>P: snapshot: main at generation 7
    A->>P: applyWrites: create claim main.7, update ref main to 8
    P-->>A: ok
    B->>P: applyWrites: create claim main.7, update ref main to 8
    P-->>B: refused: claim main.7 exists
    B->>P: reread refs: main at generation 8
    Note over B: main moved, so "fetch first"

A refusal says nothing reliable about its cause: the reference PDS answers a duplicate create with a 500. So after any refused batch the client rereads the refs, one more round trip, and compares each pending ref's generation with the one it read. A ref that moved is reported "fetch first". The other refs of that batch and later ones are reported "not pushed: another ref in the push moved concurrently", and refs in batches that landed are ok. If no ref moved, the push fails with the original error, such as a bad token. On a first push, a repo record that now exists means a racing first push won.

A claim guards its generation only while it exists, and compaction deletes claims a day old (KNOTLESS_CLAIM_GRACE_SECS, 86400 by default). A push therefore refuses to write against refs read more than half that grace ago. No writer can hold a generation whose claim is gone, so an old claim can never be replayed.

Of several writers pushing straight to one branch, one lands per round, and the reference PDS also serializes an account's writes. Many agents work best on a branch each, or by committing through a gateway, which queues commits per branch (see gateway.md).

Large packs

A pack up to 192 KiB is stored inline in its record's data. Small pushes then cost no blob uploads, which hosts limit far more tightly than record writes, and a fetch gets the pack with the listing. 192 KiB is 256 KiB of base64, the lexicon's cap on data, well inside the 1 MB body, and small enough for the firehose, which carries every record to every subscriber.

A larger pack is cut into consecutive parts and uploaded with uploadBlob. Parts are 4,000,000 bytes, never above the blob limit the PDS advertises. A PDS's advertised limit is not always what is in front of it, a proxy for instance. So a part refused as too large halves the part size and the pack is uploaded again, until parts are no larger than 64 KiB. From the default size, the last try uses parts of 62,500 bytes.

Details:

• Parts upload four at a time, in order, as application/x-git-packed-objects.
• Too large means HTTP 413, BlobTooLarge or PayloadTooLargeError.

Fetch

A fetch has to find the packs holding what it wants without reading every pack, and has to survive compaction replacing packs while it reads.

If every wanted object is already present locally, there is nothing to do. Otherwise the fetch walks the pack pages newest first, starting from the snapshot's page. On each page it downloads and indexes the packs with any tip missing locally. Their bases join the wanted set, and the set is checked against the local repository again. The walk stops once everything wanted is present.

A compaction can delete packs between two pages, after the walk has passed the newest page where their replacement lands. So when the pages run out with something still missing, the fetch walks once more from the newest page. A compaction can also delete a pack the walk has listed but not yet downloaded, and its blobs with it. A download that fails restarts the walk from the newest page the same way. The two cases share the one restart. If something is still missing after it, the fetch fails: "the remote has no pack holding <sha>"; a download that fails again fails the fetch.

A fetch of a small push is one round trip, because its inline pack arrives with the listing. Larger packs add parallel blob reads.

Details:

• Up to 8 packs download and index at once; a pack's parts download 8 at a time and are concatenated in order.
• Each pack is indexed with git index-pack --stdin --fix-thin.

Compaction

Every push adds a pack, and a clone reads them all. Compaction folds old packs into one. knotless compact runs it on demand, as do MCP's compact and the Compact button in a repository's settings on the pages, and the owner's git push through the helper starts it once the repository holds 32 packs (KNOTLESS_COMPACT_THRESHOLD). The count comes from the snapshot the push read: its newest page of packs, plus the push's own, with a further page counted as 100 more. KNOTLESS_COMPACT runs the compaction in the background (the default, a detached knotless compact), inline before the push returns, or off. It writes and deletes the owner's records, so only the owner runs it.

No other write starts it: not a push through the gateway, nor knotless commit, nor a commit over MCP. A repository written only those ways gains a pack with every write until its owner runs knotless compact.

Compaction must not pull packs out from under a fetch in progress. Only packs older than ten minutes, aged by the TID in their key, are folded. A reader partway through a walk still finds the young packs it listed, and one that loses an old pack restarts its walk once, as above. A fold needs at least two old packs.

The fold syncs the local mirror, then packs every live ref target with no bases, so the new pack holds everything the live refs reach. It also holds the bases the young packs that stay name. A push names every tip the repository had as a base, including branches deleted since, such as a landed pull's. A fetch of a young pack wants its bases too, and without them in the fold a base whose branch is gone would be in no pack, and every fresh clone would fail with "the remote has no pack holding <sha>". A base the fold cannot fetch stops it, with nothing deleted, rather than being left out. It folds later, once the pack naming it is old enough to go too. It is placed like a push's pack, keyed with a fresh TID, its supersedes naming the packs it replaces. The same writes delete the old packs and the repository's claims older than a day, in chunks of 200 with the new pack in the first.

Compaction drops objects that no live ref or young pack's base reaches and that were only in old packs, the old pack records, their blobs (which the PDS removes once nothing references them), and claims a day old. The PDS thus holds about a day of reflog; knotless-serve's index keeps every claim it sees (see serve.md). Packs younger than the grace stay, as do the repo record and every ref record, including deleted refs' empty ones, whose generations must never repeat.

Commits without a checkout

An agent that changes three files should not need a working copy. knotless commit builds the commit in the local mirror and pushes it. Its changes are puts and deletes of named paths.

Each attempt reads the snapshot, syncs the mirror and picks the parent: the branch's tip, or for a new branch the given base or the default branch's tip. It builds a tree from the parent's with a temporary index, writes the commit, and pushes it. A tree equal to the parent's pushes nothing. The push is forced, because the claim pins the parent's generation, which already guarantees what git's fast-forward check would.

If the branch moved, the push is refused "fetch first". Puts and deletes are defined by path, so applying the same changes to the new tip is a rebase that cannot conflict. The commit replays, up to five more times by default (--retries), after a random pause that grows with each lost round, so that losers do not return together. Each lost race costs three round trips: the reread, a snapshot and an applyWrites. Given a base commit, a change instead fails "conflict" if a path it touches changed between the base and the current tip.

A logged-in reader who is not the owner commits through the repository's gateway (see gateway.md); a repository without one refuses non-owner writes.

Details:

• Puts have mode 100644 or 100755. A path must be relative, with no empty, . or .. segment, no NUL or newline, and no .git segment in any case.
• The pause before a replay is under 20 ms × 2^attempts, at most 1.28 s.
• With no author given, the author is the committer's handle with email <did>@knotless.invalid, which needs no email claim. The committer is the author.

The local cache

A warm operation depends on what the client remembers: a handle's DID and PDS, the session, and the repository's key. The cache lives in KNOTLESS_CACHE_DIR, else $XDG_CACHE_HOME/knotless, else ~/.cache/knotless. Sessions and space credentials live in the config directory instead.

File	Maps	Kept for
identity.json	handle or DID to DID and PDS	an hour, or until that PDS says the account is not there
handles.json	DID to verified handle	an hour
keys.json	<did>/<name> (or #space) to key	always; each snapshot checks it
servers.json	PDS to advertised blob limit	an hour
emails.json	DIDs whose email claim was looked up	always

Several processes share the cache, so every JSON file is written to a temporary file and renamed, and a reader never sees half a file. A missing or unreadable file reads as empty.

A mirror is a bare copy of a repository at mirrors/<did>/<key>.git in the cache, with each : of the DID written _, kept in step with the records. Compaction, checkout-less commits, branches, merges, the gateway and knotless-serve all work from one, shared by every process using the cache. A sync fetches what the snapshot's live refs need, then sets the mirror's refs exactly to the snapshot's in one git update-ref --stdin transaction. Packs are indexed without a lock, because each lands under its own name. The ref transaction runs under the operating system's lock on a file beside the mirror. The system lets go of it when its process ends, so a crashed process cannot block the mirror. A new mirror is created under a temporary name and renamed into place, so racing processes end up with the same one.

A mirror is created with git's own gc off (gc.auto=0). Instead, a sync that leaves it holding more than 50 packs repacks them into one, keeping objects no ref reaches.

A sync that moves the refs, or finds a mirror with no commit-graph, then writes one (git commit-graph write --reachable --split): each commit's parents and date in one file beside the packs, which walks of history read rather than each commit from the packs. On the rust repository's 345,000 commits, a walk that takes 2.4 seconds without one takes 0.17 with it. Split, the write adds only the commits the graph does not hold yet, so a push costs it milliseconds. Only the first, of a mirror's whole history, takes long.

The git remote helper

git runs git-remote-knotless for knotless:// URLs. It lists live refs, with the default branch as HEAD, fetches in one walk, and sends all of a push's refs as one push, answering per ref. It refuses what git refuses, so git push behaves as it does against any remote.

An unforced update whose old remote target is not in the local repository is refused "fetch first": it would overwrite commits the pusher has never seen. An update that is not a descendant is refused "non-fast-forward". A forced push skips both checks but still goes through the claim's compare-and-swap, so it loses to a concurrent move rather than overwriting it. A delete keeps the ref record with no target and the next generation.

A handle can change hands while a remote's URL keeps naming it, and the remote would then fetch someone else's refs. So on first contact through a named remote, the helper stores the account's DID in remote.<name>.knotlessdid, and refuses a later run whose URL resolves to another DID. A URL given straight on the command line is not pinned. A repository that was renamed or moved is followed, and the helper suggests the git remote set-url for its new place.

A logged-in reader who is not the owner pushes through the repository's gateway. ../reference/performance.md has round-trip timings, and ../reference/cli.md the commands and the environment variables that set the limits on this page.