~ knotless Search/
↑↓ move↵ open⇧↵ all results
0
0
438 lines · 25.5 KB Raw History

The store: what is kept, the backfill window, and the bounds

What the server keeps, seen from outside: which history survives and which may go, how far back the firehose replays, and how the data directory's size and the time to start follow what the server holds now rather than everything it was ever sent. The server may choose how it lays its data out on disk. What must survive a kill is in DURABILITY.md; what every method shares, and how the server is checked, is in PDS.md.

Environment

Besides those PDS.md lists:

VariableIsDefault
PDS_REPO_BACKFILL_LIMIT_MSthe backfill window W: how far back, in milliseconds, subscribeRepos replays86400000 (a day, as the reference's)
PDS_CACHE_BYTESthe most the server holds in memory of accounts' state and data read from disk, beyond what each account costs (below)268435456 (256 MiB)

The reference takes the first, with the same default.

What survives

Some parts below are of pages a server may not serve yet (LIFECYCLE.md, EMAIL.md, PROXY.md, SPACES-SYNC.md, SPACES-EXPORT.md); they hold of a server once it serves them.

Across restarts, and however the server keeps it, every read answers as it did before, and so:

  • getRepo answers the latest commit and its tree, and with since the blocks added after that rev (SYNC.md); getBlocks and the sync getRecord answer the blocks of the current tree (FEDERATION.md);
  • listBlobs, with since too, lists what it did, and getBlob serves every blob the account holds, a record's or an upload within PDS_BLOB_TTL_SECS (BLOBS.md, LIMITS.md);
  • the next commit's rev, and the next TID the server makes for a record key or a space repo, is later than every one before (DURABILITY.md);
  • a handle change in flight is settled as IDENTITY.md says;
  • each space repo's listRecords, getLatestCommit, listRepoOps from its start (every operation since the repo began, or since its last clearRecords or import), and listBlobs with since (SPACES-REPO.md, SPACES-EXPORT.md); each space's policy, members and writer set, as getSpace, listMembers and exportSpaceMeta answer them, and that a space was deleted (SPACES-POLICIES.md);
  • a write notification failed for now is still sent again when due; a registration is honoured until it expires, and a deletion notice owed is still owed (SPACES-SYNC.md);
  • each account's state, deactivated or taken down, a record's or a blob's takedown, as getSubjectStatus answers them, and a deletion (LIFECYCLE.md);
  • each account's email, whether it is confirmed, its email sign-in code, and each token mailed to it while it is neither used nor past its time (EMAIL.md);
  • each account's preferences, as getPreferences answers them (PROXY.md);
  • the sessions, app passwords, OAuth's state, invite codes and single-use values, as their pages say;
  • the firehose's events in the backfill window (below), and the seq of the last event ever made.

What may go, and is not answered once gone:

  • Blocks no longer in the current tree: earlier commits, and nodes and records a later commit took out of the tree. getBlocks answers such a block as one the repository does not hold, 400 InvalidRequest, as the reference does, whether or not the server still has it (FEDERATION.md).
  • Events out of the window (below).
  • Blobs not held, with their takedowns (BLOBS.md, LIMITS.md).
  • Takedowns lifted, and a record's once a commit deletes it.
  • Notifications settled: delivered, refused, replaced by a later commit's, or past their time; registrations withdrawn or expired.
  • Mail tokens used, replaced or past their time.
  • Preferences an account has since replaced, and sessions and single-use values past their time.
  • A deleted account: everything of it but its deletion's event, while that is in the window (LIFECYCLE.md).

The backfill window

Every event carries its time (subscribeRepos' lexicon), when the server made it. The window's start, at any moment, is the first event, by seq, whose time is no more than W before now. The server keeps every event from the window's start on, across restarts; an event before it may be dropped.

A subscribeRepos with cursor:

  • later than the last event's seq is refused, FutureCursor (SYNC.md);
  • whose next event, the first after cursor, is at or after the window's start, or that is the last event's seq, replays every event after cursor, then goes on live, as SYNC.md says;
  • otherwise, 0 included, sends first an #info message, header {op: 1, t: "#info"} and body {name: "OutdatedCursor", message: "Requested cursor exceeded limit. Possibly missing events"}; then every event from the window's start on, in order; then goes on live. When no event is in the window, the #info message alone, then live.

This is what the reference does. A subscription without cursor is live only, as before. Events out of the window are never sent, whether or not the server still has them. Seqs are never reused: after a restart, new events are numbered after the last event ever made, though it has gone out of the window and been dropped, and though its account has been deleted.

Bounds

The server compacts what it keeps, in the background, so that these hold. Compacting never stops reads, and is crash-safe at every moment, as DURABILITY.md says of every write. It holds up an account's writes no longer than another write to it would, but for an account written faster than it is compacted: once what has been written to it since its last compaction counts for more than a quarter of what the cache gives all accounts' state (an eighth of PDS_CACHE_BYTES), or holds 2 MiB more than what has its log folded (4 MiB, or its table's size if that is less), its next write waits for the compaction under way to end. So what the server holds of an account being written, and what its first use after a start reads, do not follow how fast the writes come.

The live state of an account is the sum of:

  • the bytes of its getRepo without since;
  • the sizes of the blobs it holds, referenced or uploads within PDS_BLOB_TTL_SECS;
  • for each space repo it has, the bytes of the JSON bodies of com.atproto.space.listRecords and of listRepoOps with excludeValues: true, read whole at limit 1000;
  • for each space it is the authority of and has not deleted, the bytes exportSpaceMeta answers.

The data directory's size is the sum of the sizes of the files in PDS_DATA_DIR, at any depth. With blobs in a bucket (BLOBS.md), it holds none of their bytes, and the live state counts them all the same; the bucket holds what BLOBS.md says.

  • Settled. When no write has come for 2 W, whether or not the server was started again meanwhile, and then one comes to each account, within 10 seconds the data directory's size is at most three times the accounts' live states, plus 128 KiB (131072 bytes) for each account, for what the server keeps of it besides (its sessions, app passwords, mail, preferences and OAuth sessions), plus 1 MiB (1048576 bytes) for what it keeps of no one account (its keys, invite codes, OAuth's clients and the like), plus 128 bytes and the DID's length for each issuer's key it keeps (SPACES-CREDENTIALS.md, "Issuers' keys"). However many times the records were rewritten, the blobs replaced, the space records written and the sessions refreshed before, what is left is what the server holds, the space repos' operation logs, and the last writes' events.
  • Starting. The time to start does not grow with the number of accounts, with what they hold, or with how many writes they have had: started on any data directory, the server answers describeServer within twice the time it takes to start on an empty one, on the same machine and under the same load, plus 250 ms. A start that a passing load slowed more than the starts beside it breaks no bound when the server, started again on the same data, meets it: what does not hold is a start that is slow each time. An account's state may be read from disk on its first use after that (below).

Scale

The server serves as many accounts, and as much data, as its disk holds, on one machine: what it holds in memory follows a cache of a size it is given, not the accounts or their data.

  • Memory. The server's resident memory (its RSS, as ps reports it) is at every moment at most its base, plus PDS_CACHE_BYTES, plus 4 KiB (4096 bytes) for each account, plus 256 bytes and the DID's length for each issuer's key it keeps (SPACES-CREDENTIALS.md, "Issuers' keys", at most one for each 1024 bytes of PDS_CACHE_BYTES). The base is its resident memory answering describeServer just after starting on an empty data directory, plus 32 MiB (33554432 bytes) for the requests under way. So an account no request touches costs at most 4 KiB of memory, its repository nothing; an account's state, read from disk on first use, is held in the cache, which the server empties of what was least recently used to stay within its size; and holding, reading or writing a repository of any size stays within the bound, a commit of LIMITS.md's largest too, and so does a space repo of any size (SPACES-REPO.md).
  • The first request to a cold account. After a start, an account's first request, a read or a write, is answered within 1 second more than it would be once the account's state is in memory, whatever the account holds. To answer it the server reads at most 6 MiB of the account's files besides what the request itself reads, whatever the account holds: an account's log is folded into its table before it holds 4 MiB of writes, so that a first use replays little however large the table, and the events the account keeps for the firehose are not read for it. Many accounts' first requests at once, as when a server busy before its start is started again, share its processors, each first use's work so bounded: they take together what that work costs on them, not what their accounts hold. After a start that followed a shutdown (SIGTERM) the server syncs nothing to the disk for it (OPERATIONS.md, pds_fsync_duration_seconds): what the account's files hold was made durable as the server shut down. After a kill or a power cut an account's log is synced as it is first used, before anything is made of what it holds. A server that started after a kill makes what it found durable in the background, the whole data directory at once, within seconds; a shutdown before that is done is as a kill to the next start.
  • Streaming. getRepo, the firehose's replay from a cursor, and listBlobs are read from disk as they are sent: what the server holds to send them does not grow with the size of the answer, a getRepo of any size within the memory bound. A subscriber that reads slowly is sent the events as it reads them, not buffered ahead of it.
  • The firehose after a start. After any start, one that followed a shutdown (SIGTERM), a kill or a power cut, a subscribeRepos with a cursor is sent its first message within 1 second of asking, the first subscription after the start too, whatever the number of accounts: which events were made, and whose, the server keeps as it makes them, apart from the accounts' own files, and reads no account's files at a start to learn it. What it keeps of an event is written before the write the event is of, and both are on disk before the write is answered and the event sent: after a kill every write found has its event kept. After a power cut a write that was never answered may be found whole with nothing kept of its event. That event is never sent, and nothing of the write is undone: as its account is next used the server makes, for an active account, the events an activation makes (LIFECYCLE.md), the #sync of its repository as it is among them, once, so that those who read the firehose learn what it holds.

Upgrades

A server starts on a data directory the previous version of itself wrote, the last release, tagged v<version>, whether that version shut down or was killed, and serves all of it, as after a restart: every read above answers as it did before, and its writes carry on from that version's, the next commit's rev later than every one before and its since the last, and the next event's seq after the last ever made. Its first start on such a directory reads every account's files once, for the events made before, and may take that much longer to serve. A data directory a later version wrote it may refuse, exiting 1 with pds: cannot recover <dir>: and why, as DURABILITY.md says of data it cannot recover; it never serves part of one.

Spaces. An account's spaces state (SPACES.md: its space repos, the spaces it is the authority of, and their notifications) is carried from a mark on, the entry {"t":"spaces","v":1}, which begins every account's log from 0.2.0 on. What a version wrote of it before the mark, in a log with none or in a table of the format before, is let go as the account's store is first loaded, which writes the mark then: nothing is made of it and nothing of it is refused, and a blob that only those spaces' records referenced is held no longer. Nothing else of the account changes. So data that 0.2.0 or a later release wrote keeps its spaces across an upgrade, and 0.1.0's does not.

How it is checked

node compare.mjs --store --crash-every 40 ... is a run of its own. It starts both sides with PDS_REPO_BACKFILL_LIMIT_MS=5000, the reference's repoBackfillLimitMs too, makes four accounts, each with a memberspace, and fills each with records of 8 to 24 KB, at sixteen keys; then kills and starts the server again three times, each start held to the bound above as a crash's is (below). Then it churns, on random accounts:

  • rewrites of those keys, alone and in batches, and deletes of them, each compared with the reference as REPO.md says;
  • records with blob refs replacing each other, and uploads, so that blobs stop being held (BLOBS.md), compared as BLOBS.md says;
  • writes to the account's space repo and its member list, and to its repo in a space whose authority no PLC directory knows, so that each write's notification fails for now, on the server alone (the reference serves no spaces), held to what the harness wrote: listRecords, listRepoOps from the start, getLatestCommit and listMembers, and each repo's rev only ever later;
  • sessions made and refreshed;
  • where the server serves them, checked once it does: takedowns of a record kept for them and of blobs records reference, applied and lifted by the admin's updateSubjectStatus on both sides, and getSubjectStatus of each compared, after every restart too (LIFECYCLE.md), a ref to a blob taken down being a write's one ref that may be wrong (BLOBS.md); and putPreferences on both sides, compared, whose list getPreferences must answer on both after every restart (PROXY.md);
  • reads compared as everywhere: getRecord, listRecords, getBlocks, the sync getRecord, listBlobs, getBlob;
  • getBlocks of blocks the repository held before and no longer does, an earlier commit of each side's own and nodes and records the reference's tree had, which both must answer 400 InvalidRequest;
  • subscribeRepos from a cursor, on both sides, held each to the rule above: a cursor whose next event is more than W + 1.5 s old must have the #info first, then a run of events with no gap, ending at the last, holding every event less than W − 1.5 s old and none more than W + 1.5 s old; one whose next event is less than W − 1.5 s old, every event after it and no #info; the two sides must agree on whether there is an #info. Events nearer the window's edge than 1.5 s may be on either side of it.

Every --crash-every steps it rewrites a random account's records until it sees the server reorganizing its data directory, a file made, removed or made shorter by a write that answered, and kills the server a few milliseconds later; if none comes in 200 writes, it kills it as DURABILITY.md's crashes do, sometimes with a write on its way. Then it checks what those crashes check, the space repos, takedowns and preferences too, with the firehose carrying on from its last event; and the start must answer within the bound above. The harness times it among starts of the same server on empty data directories, each of its own and on a port of its own, so that the machine's load at that moment slows them alike: two while the server is down before its start, and two after, the server killed again as soon as it answers and then started once more on its data; the start must take at most twice the median of those four, plus 250 ms. A start over that bound is taken again, up to twice more: the server killed and started on the same data among four new starts on empty data directories, as before. The start fails only if every take is over its own bound, as a start whose time grows with the data is each time.

At the end it rewrites every key of every account eight times, at once, then waits 2 W and a second, kills the server and starts it again, and, reading nothing first, writes once to each account; within 10 seconds the data directory's size, but for the server's log the harness keeps there, must be within the settled bound. Then it waits 2 W and a second again, writes once to each account, and holds the size to the bound the same way, the server not started again. Then, where the machine counts what a process reads (Linux's /proc/<pid>/io, rchar), it makes an account with a space of its own and writes it 48 records of 250 KB at keys of their own; waits 2 W and a second and writes once more, which folds its log; at once rewrites the records 16 times, well within 2 W; lets the data directory settle, shuts the server down and starts it again, and, nothing else asked of the server, sends that account a getRecord: the server must read no more than 6 MiB to answer it. It deletes those records after. Then the server is killed and started once more, the start held to the bound so, and every account checked again. The run reports the data directory's size against the accounts' live states, and the starts' times.

node compare.mjs --scale ... is a run of its own, of the server alone, as the reference says nothing of its memory, with PDS_CACHE_BYTES 16 MiB. It starts the server on an empty data directory three times, the slowest the empty start's time and its resident memory then the base, and then:

  • makes 100 accounts, without passwords, and starts the server again; then 2000, and starts it again: each start within the starting bound, and its resident memory, just started, at most the empty start's, 8 MiB, and 4 KiB for each account;
  • reads every account's latest commit, 8 at a time, sampling the server's memory every 20 ms, within the memory bound throughout; an account whose first read takes more than a second must be no more than a second slower than its next;
  • writes about 64 MiB of records of 120 KB to one account, by applyWrites, within the memory bound throughout; starts the server again, within the starting bound and the idle memory; reads a record of that account, its first request, within a second of its next, and as written;
  • reads that account's getRepo as it streams, and then the firehose from cursor 0 as fast as it is sent, within the memory bound throughout: the repository must hold every record as written, and the replay every event, at least the accounts' first four each and the large writes', in order;
  • gives one account much besides its repository: 4000 blobs of 512 bytes, each referenced by a record of its own, and a space of its own with 40000 records of about 1 KB in its repo there, by applyWrites, within the memory bound throughout; starts the server again, within the starting bound and the idle memory; reads a record of that space repo, its first request, within a second of the same request again, and as written; reads two pages of the space repo's listRecords and of the account's listBlobs; and reads the space repo whole (com.atproto.space.getRepo) as it is sent, which must hold every record's bytes: all within the memory bound;
  • makes accounts until there are 100000, the next 10000 each with an email and a password, and starts the server again, within the starting bound and the idle memory. Then, within the memory bound throughout, it looks for one account in 50 in every way an account is found: getLatestCommit of its DID; resolveHandle of its handle; getSession with the token it had before the start; and, with a password, createSession with its email and with its handle. Each account is looked for another way first, so that each way finds accounts nothing has asked for since the start, and that first request must be within a second of the same request again. An account made with the handle, or with the email, of an account with a password that nothing has asked for since the start must be refused, 400; and listRepos, three pages of 100, must list accounts made, none twice;
  • refreshes the session of every account with a password, four times over, 8 accounts at a time, so that what the server keeps of its sessions is written several times over and compacted as it is, within the memory bound throughout; starts the server again, within the starting bound and the idle memory; and looks for the same accounts again, each session by its last refresh's token;
  • asks for the firehose from cursor 0, as a relay would, on that start, which followed a kill: its first message must come within a second, and be the first event ever made. Then it writes a record to one account, with a subscription open to learn its commit's seq; shuts the server down with SIGTERM and starts it again, within the starting bound and the idle memory; and, the first request it is sent, asks for the firehose from cursor 0, then from the seq before that commit's: each one's first message must come within a second, and be the first event ever made, and that commit. Then it writes another record, kills the server and starts it, and, the first request it is sent, asks for the firehose from the seq before that record's commit: it must be sent that commit, within a second, so that a kill loses the firehose nothing of a write. Before that write, on the start that followed the shutdown, it lists a record of each of 200 accounts nothing has asked for since that start: the server's count of syncs rises by fewer than 20 meanwhile, its own housekeeping's, none of them an account's first use's. And after the kill that ends the run, 200 other such accounts: the count rises by at least 200, each account's log synced as it is first used.

node compare.mjs --upgrade ... is a run of its own, of the server alone, against harness/upgrade/: what the previous version made, with node compare.mjs --upgrade-write run with its binary as the release is tagged. That is two data directories, one it shut down with SIGTERM and one it was killed on with SIGKILL; the variables they are bound to (the server's host name, public URL, handle domains, rotation and recovery keys and the admin's password); each DID's operations at the PLC directory; and every read below as that version answered it. It made four accounts: records written, rewritten and deleted by createRecord, putRecord, deleteRecord and applyWrites; a blob in a profile; preferences; an app password; an email confirmed; a handle changed; a memberspace with records, one deleted, and a member; an account deactivated, a record taken down and an account taken down. Both versions run with a backfill window of ten years, so that every event stays in it.

The run gives its PLC directory each DID's operations, then starts the server on each data directory in turn, with those variables, and holds it to what the previous version answered: describeServer and listRepos; for each account getLatestCommit, getRepoStatus, getRepo (its root and the CIDs of its blocks), describeRepo, listRecords of its collections, listBlobs and each getBlob (its type and bytes), and getSubjectStatus of it and of the record taken down; createSession with its password, but for its tokens, and with that session getSession, checkAccountStatus, getPreferences and listAppPasswords; the firehose from cursor 0, every frame's bytes; and the spaces (SPACES.md), carried as the rest is: the first account's listSpaces, its memberspace's getSpace and listMembers, and the space's listRecords, listRepoOps and getLatestCommit. Then a record written to the first account must commit at a later rev, its #commit event after the last seq the previous version made and its since that version's last rev, and an account must be made.

Not checked by the harness: what survives a power cut, which the server's own tests check, as DURABILITY.md says; spaces state before the mark let go ("Upgrades", "Spaces"), which the server's own tests check, as no data the harness keeps was written before it; how long compacting holds up writes; that a notification queued, a registration or a deletion notice owed, an import, deactivation, an account's takedown or deletion, or a mail token survive compacting (the runs of their own pages exercise them); of an upgrade, the sessions and OAuth's state made before it, mail tokens, invite codes and a handle change in flight; a subscriber that reads slowly; listBlobs, whose pages are bounded by its limit; and more accounts or data than the scale run makes.