~ knotless Search/
↑↓ move↵ open⇧↵ all results
0
0
214 lines · 12.3 KB Raw History

Durability

What survives the server being killed and started again. What the server serves, and how it is checked, is in PDS.md.

The server keeps all its state in PDS_DATA_DIR, but blobs' bytes when they are kept in a bucket (BLOBS.md), and started again on the same directory carries on where it stopped: its accounts, each with its DID, handle, email, password, signing key and KEM key (SPACES.md); whether its email is confirmed, its email factor and the tokens it mailed (EMAIL.md); every account's repository and blobs (BLOBS.md); the firehose events of every account in the backfill window; the sessions it gave out, password sessions and OAuth's (OAUTH.md, OAUTH-CLIENTS.md); and each account's passkeys (PASSKEYS.md). What of its history it keeps and lets go (earlier commits, blocks out of the tree, events out of the window), and how large its data may grow, is in STORE.md; the server may choose how it lays its data out.

  • A write the server has answered must survive the server being killed at any moment after, and so must an account createAccount has answered.
  • A write it is killed during must land whole or not at all. An account it is killed while creating either exists whole when it serves again, with its repository's first commit and that commit's events, or does not exist.
  • A handle change is a write too, in the server and in the PLC directory, which the server settles with after it starts (IDENTITY.md).
  • An importRepo is a write: answered, it survives; killed during, the repository is the one before it or the one it makes, whole, with its records and the blobs they reference as missing or held (MIGRATION.md).
  • The first commit after a restart has a rev later than every one before.
  • An event the firehose has sent keeps its seq, and its place among every account's events: after a restart, a subscription from any cursor in the backfill window (STORE.md) gets the same events in the same order, and new events are numbered after every one before, those that left the window too; but for the events of an account deleted since, which are dropped (LIFECYCLE.md).
  • An account's state, deactivated, taken down or deleted, survives as a write does, and a deletion the server is killed during is whole or not at all (LIFECYCLE.md).
  • A restart answers describeServer within a second at this size.
  • Invite codes and their uses survive as accounts do (LIMITS.md).
  • The server never aborts for a failed write: a write it cannot make whole, as on a full disk or with no file descriptor left, is 500 and leaves nothing of itself, and below PDS_MIN_FREE_BYTES of free space a write is 507 InsufficientStorage before it starts (LIMITS.md).
  • Data the server cannot recover is never served. Entries that do not parse, past a last write cut short (which recovery drops), a rebuilt tree that is not the one the head commit names, or two accounts kept with one handle or one email, make pds serve exit 1, with pds: cannot recover <dir>: and why on standard error: as it starts, for what it reads then, and for what is one account's, which it reads on that account's first use (STORE.md, "Scale"), when it reads it. Nothing of that account is answered first.
  • All of this holds at every moment, whatever the server is doing in the background to keep its data small (STORE.md), and on a data directory the previous version of the server wrote.

node compare.mjs --crash-every 25 ... kills the server with SIGKILL every 25 steps, often while a write is on its way, starts it again, and checks all of this: the accounts and the sessions it gave out, the write it was killed during, every account's repository against the reference's, and, with --firehose, the stream from the last event it had. --store kills it while it reorganizes its data directory (STORE.md).

SIGKILL loses what the process held, not what it had handed to the operating system, so this does not check what survives a power cut. Those writes are checked by the server's own tests: the files it recovers from, cut short at every byte of its last write, must recover to the state before that write or after it, but for the firehose's event of a write found whole with nothing kept of that event, which is never sent (STORE.md, "The firehose after a start"); and so must its data cut short at every point of whatever it does in the background to keep its data small, with the writes of every file it had not made durable lost.

Writes waiting at once

A write is on disk before it is answered, and writes waiting on the disk at once share its syncs: one sync makes durable every write it covers that was waiting when it began, and a write that came after it began waits for the next. So what the server answers in a second is not held to the syncs its disk makes in one.

VariableIsDefault
PDS_SYNChow waiting writes are made durable: together, every account's by one sync; or each, each account's by a sync of its own, which only the writes waiting on that account sharetogether

Any other value makes pds serve exit 1, pds: PDS_SYNC: and why. Which way the server syncs is in the start line of its log (OPERATIONS.md), as sync.

With together the server writes each write twice: where it keeps it, and in a journal for all, in PDS_DATA_DIR. The journal's sync is what a write waits for, and one covers every write in the journal when it began, whichever accounts they are to. What the journal holds the server makes durable where it keeps it, in the background, and then lets go of: after a kill or a power cut a start puts back from the journal whatever is missing where it is kept, before it serves, in a time that follows what was written in the last seconds before, not what the server holds. A shutdown leaves no journal. The server syncs fewer times than it answers writes: over 300 writes, sixty at once and each of those to an account of its own, fewer than 150 times (OPERATIONS.md, pds_fsync_duration_seconds), on a disk whose sync takes a millisecond or more; with each it syncs 300 times at least. What survives a kill or a power cut is the same either way, and a server started with one on a data directory the other left carries on from it.

A sync of the journal that fails is not tried again over the same writes: the disk may have dropped what it was given and say nothing the next time. From then on every write is 500 and nothing of the journal is let go, until the server is started again, which puts back from the journal what it holds. The server's log says so once, store failure (OPERATIONS.md). Where it is what the journal holds that could not be made durable where it is kept, writes go on and the journal is kept whole for the next start.

node compare.mjs --limits ... checks this, of the server alone (harness/checks/limits.mjs): sixty accounts each written to once, all at once, five times over, on a server started after a shutdown. With PDS_SYNC together the server's count of syncs rises by fewer than 150 over those 300 writes; with each, by at least 300; and with sometimes it exits as above. The crash runs (above) run with together, so that every kill is recovered through the journal; the server's own tests check it against a power cut, the journal cut short at every byte of its last write and every file not made durable lost.

Blobs

A blob's bytes are a file of their own, not in a log, or an object in a bucket (BLOBS.md), so what survives of them is said apart:

  • An upload uploadBlob has answered survives the server being killed: its file is written, and its entry handed to the operating system, before the answer. Neither need be synced then.
  • Against a power cut, a blob is durable once a committed record references it. The commit that first references a blob, public or in a space repo, syncs the blob's file and the directory it is in before anything of the commit can reach the disk: with PDS_SYNC together, before the commit's frame is written, since the journal's next sync, any account's, takes it; with each, before the commit's own frame is synced, within the log's group commit, so commits waiting at once share their flushes, and before a compaction syncs the log it seals (STORE.md). A commit on disk never has a blob it references that is not.
  • An upload no record references yet may be lost to a power cut. Started again on data where its file did not survive, the server holds it no more, as if it had never been uploaded: a ref to it is BlobNotFound until it is uploaded again.

Note: syncing an upload when it is answered would cost each upload flushes of its own, which no other request shares; a blob no record references is held only for PDS_BLOB_TTL_SECS (LIMITS.md), and a client that lost one uploads it again.

The server's own tests model a power cut as the loss of every file not synced, and check that a referenced blob survives wherever its commit does.

In a bucket, the store keeps an upload's bytes before uploadBlob answers, and a commit has no file of a blob to sync: what a power cut may lose of an upload no record references is the server's entry of it, which its log holds as it holds a write's, and the store then keeps an object the server no longer holds (BLOBS.md).

Backups

A copy of PDS_DATA_DIR taken at one instant while the server runs, as a filesystem or volume snapshot takes it, is a data directory the server starts on. A snapshot that holds what the process had handed to the operating system is as the disk after a kill; one of the disk alone, as a block device's, is as the disk after a power cut. Started on the copy, the server holds what this page says survives the one or the other at that instant: every write answered before it, and every account createAccount answered; of a write under way, all or nothing; and of each account's writes after the last answered, those in the copy are the first of them in order. The sessions it gave out before still work on it. The copy holds the secrets the server signs and seals its tokens with, which it keeps in the directory, and none of what is in its environment: the PLC rotation key (ACCOUNTS.md), the admin's password and the bucket's credentials are kept apart from it.

With blobs in a bucket, the copy holds which blobs each account holds, but not their bytes, which stay in the bucket. A blob the server stops holding after the copy has its object deleted (BLOBS.md), and the server started on the copy, which holds it still, answers getBlob of it as of an object the store does not have, though listBlobs lists it. Keeping the bucket's earlier versions (S3's versioning, R2's and others' alike), or copying the bucket at the same instant, keeps those. A server started on a copy while the one it was copied from runs needs a bucket of its own (BLOBS.md).

A copy made file by file over a span of time, as cp or rsync of a running server's directory makes it, is not such a copy, and nothing is said of it. A server stopped first, with SIGTERM or SIGKILL, may be copied any way.

A server started on a copy is a rollback, and does nothing of its own about it, as the reference does nothing for a database brought back. What was written after the copy is lost. The firehose numbers its next events from the copy's last seq, so seqs it sent before are sent again for other events, and those who read it, relays and AppViews, hold commits and a cursor the server no longer has. Handle changes after the copy stay at the PLC directory.

With --crash-every N, before every second kill, the harness puts a run of eight records to one account, one after another. Once some are answered it stops the server with SIGSTOP, which freezes it between two of its system calls as a snapshot freezes its files, with the next put on its way; it copies the data directory and lets the server go on with SIGCONT. A second server, started on the copy on a port of its own, with no crawlers, no metrics or traces, SSRF protection on (PROXY.md), and, with blobs in a bucket, a bucket of its own, must answer every put answered before the stop; of the rest, a prefix of the run, each record as the put answered it; the latest commit, the last put's in it; and getSession with the account's token. What a block device's snapshot holds is what a power cut leaves, which the server's own tests check (above).