# Durability
What survives the server being killed and started again. What the server
serves, and how it is checked, is in [`PDS.md`](PDS.md).
The server keeps all its state in `PDS_DATA_DIR`, but blobs' bytes when
they are kept in a bucket (`BLOBS.md`), and started again on the same
directory carries on where it stopped: its accounts, each with its DID,
handle, email, password, signing key and KEM key (`SPACES.md`); whether its
email is confirmed, its email factor and the tokens it mailed (`EMAIL.md`);
every account's repository and blobs (`BLOBS.md`); the firehose events of
every account in the backfill window; the sessions it gave out, password
sessions and OAuth's (`OAUTH.md`, `OAUTH-CLIENTS.md`); and each account's
passkeys (`PASSKEYS.md`). What of its history it
keeps and lets go (earlier commits, blocks out of the tree, events out of the
window), and how large its data may grow, is in [`STORE.md`](STORE.md); the
server may choose how it lays its data out.
- A write the server has answered must survive the server being killed at
any moment after, and so must an account `createAccount` has answered.
- A write it is killed during must land whole or not at all. An account it
is killed while creating either exists whole when it serves again, with
its repository's first commit and that commit's events, or does not
exist.
- A handle change is a write too, in the server and in the PLC directory,
which the server settles with after it starts (`IDENTITY.md`).
- An `importRepo` is a write: answered, it survives; killed during, the
repository is the one before it or the one it makes, whole, with its
records and the blobs they reference as missing or held
(`MIGRATION.md`).
- The first commit after a restart has a `rev` later than every one before.
- An event the firehose has sent keeps its `seq`, and its place among every
account's events: after a restart, a subscription from any cursor in the
backfill window (`STORE.md`) gets the same events in the same order, and
new events are numbered after every one before, those that left the
window too; but for the events of an account deleted since, which are
dropped (`LIFECYCLE.md`).
- An account's state, deactivated, taken down or deleted, survives as a
write does, and a deletion the server is killed during is whole or not at
all (`LIFECYCLE.md`).
- A restart answers `describeServer` within a second at this size.
- Invite codes and their uses survive as accounts do (`LIMITS.md`).
- The server never aborts for a failed write: a write it cannot make whole,
as on a full disk or with no file descriptor left, is `500` and leaves
nothing of itself, and below `PDS_MIN_FREE_BYTES` of free space a write
is `507 InsufficientStorage` before it starts (`LIMITS.md`).
- Data the server cannot recover is never served. Entries that do not parse,
past a last write cut short (which recovery drops), a rebuilt tree that
is not the one the head commit names, or two accounts kept with one
handle or one email, make `pds serve` exit 1, with
`pds: cannot recover
: ` and why on standard error: as it starts,
for what it reads then, and for what is one account's, which it reads on
that account's first use (`STORE.md`, "Scale"), when it reads it. Nothing
of that account is answered first.
- All of this holds at every moment, whatever the server is doing in the
background to keep its data small (`STORE.md`), and on a data directory
the previous version of the server wrote.
`node compare.mjs --crash-every 25 ...` kills the server with SIGKILL every
25 steps, often while a write is on its way, starts it again, and checks all
of this: the accounts and the sessions it gave out, the write it was killed
during, every account's repository against the reference's, and, with
`--firehose`, the stream from the last event it had. `--store` kills it
while it reorganizes its data directory (`STORE.md`).
SIGKILL loses what the process held, not what it had handed to the operating
system, so this does not check what survives a power cut. Those writes are
checked by the server's own tests: the files it recovers from, cut short at
every byte of its last write, must recover to the state before that write
or after it, but for the firehose's event of a write found whole with
nothing kept of that event, which is never sent (`STORE.md`, "The firehose
after a start"); and so must its data cut short at every point of whatever
it does in the background to keep its data small, with the writes of every
file it had not made durable lost.
## Writes waiting at once
A write is on disk before it is answered, and writes waiting on the disk
at once share its syncs: one sync makes durable every write it covers that
was waiting when it began, and a write that came after it began waits for
the next. So what the server answers in a second is not held to the syncs
its disk makes in one.
| Variable | Is | Default |
| --- | --- | --- |
| `PDS_SYNC` | how waiting writes are made durable: `together`, every account's by one sync; or `each`, each account's by a sync of its own, which only the writes waiting on that account share | `together` |
Any other value makes `pds serve` exit 1, `pds: PDS_SYNC: ` and why. Which
way the server syncs is in the `start` line of its log (`OPERATIONS.md`),
as `sync`.
With `together` the server writes each write twice: where it keeps it, and
in a journal for all, in `PDS_DATA_DIR`. The journal's sync is what a write
waits for, and one covers every write in the journal when it began,
whichever accounts they are to. What the journal holds the server makes
durable where it keeps it, in the background, and then lets go of: after a
kill or a power cut a start puts back from the journal whatever is missing
where it is kept, before it serves, in a time that follows what was written
in the last seconds before, not what the server holds. A shutdown leaves no
journal. The server syncs fewer times than it answers writes: over 300
writes, sixty at once and each of those to an account of its own, fewer
than 150 times (`OPERATIONS.md`, `pds_fsync_duration_seconds`), on a disk
whose sync takes a millisecond or more; with `each` it syncs 300 times at
least. What survives a kill or a power cut is the same either way, and a
server started with one on a data directory the other left carries on from
it.
A sync of the journal that fails is not tried again over the same writes:
the disk may have dropped what it was given and say nothing the next time.
From then on every write is `500` and nothing of the journal is let go,
until the server is started again, which puts back from the journal what
it holds. The server's log says so once, `store failure`
(`OPERATIONS.md`). Where it is what the journal holds that could not be
made durable where it is kept, writes go on and the journal is kept whole
for the next start.
`node compare.mjs --limits ...` checks this, of the server alone
(`harness/checks/limits.mjs`): sixty accounts each written to once, all at
once, five times over, on a server started after a shutdown. With
`PDS_SYNC` `together` the server's count of syncs rises by fewer than 150
over those 300 writes; with `each`, by at least 300; and with `sometimes`
it exits as above. The crash runs (above) run with `together`, so that
every kill is recovered through the journal; the server's own tests check
it against a power cut, the journal cut short at every byte of its last
write and every file not made durable lost.
## Blobs
A blob's bytes are a file of their own, not in a log, or an object in a
bucket (`BLOBS.md`), so what survives of them is said apart:
- An upload `uploadBlob` has answered survives the server being killed:
its file is written, and its entry handed to the operating system,
before the answer. Neither need be synced then.
- Against a power cut, a blob is durable once a committed record
references it. The commit that first references a blob, public or in a
space repo, syncs the blob's file and the directory it is in before
anything of the commit can reach the disk: with `PDS_SYNC` `together`,
before the commit's frame is written, since the journal's next sync, any
account's, takes it; with `each`, before the commit's own frame is
synced, within the log's group commit, so commits waiting at once share
their flushes, and before a compaction syncs the log it seals
(`STORE.md`). A commit on disk never has a blob it references that is
not.
- An upload no record references yet may be lost to a power cut. Started
again on data where its file did not survive, the server holds it no
more, as if it had never been uploaded: a ref to it is `BlobNotFound`
until it is uploaded again.
*Note:* syncing an upload when it is answered would cost each upload
flushes of its own, which no other request shares; a blob no record
references is held only for `PDS_BLOB_TTL_SECS` (`LIMITS.md`), and a client
that lost one uploads it again.
The server's own tests model a power cut as the loss of every file not
synced, and check that a referenced blob survives wherever its commit does.
In a bucket, the store keeps an upload's bytes before `uploadBlob`
answers, and a commit has no file of a blob to sync: what a power cut may
lose of an upload no record references is the server's entry of it, which
its log holds as it holds a write's, and the store then keeps an object the
server no longer holds (`BLOBS.md`).
## Backups
A copy of `PDS_DATA_DIR` taken at one instant while the server runs, as a
filesystem or volume snapshot takes it, is a data directory the server
starts on. A snapshot that holds what the process had handed to the
operating system is as the disk after a kill; one of the disk alone, as a
block device's, is as the disk after a power cut. Started on the copy, the
server holds what this page says survives the one or the other at that
instant: every write answered before it, and every account `createAccount`
answered; of a write under way, all or nothing; and of each account's
writes after the last answered, those in the copy are the first of them in
order. The sessions it gave out before still work on it. The copy holds the
secrets the server signs and seals its tokens with, which it keeps in the
directory, and none of what is in its environment: the PLC rotation key
(`ACCOUNTS.md`), the admin's password and the bucket's credentials are kept
apart from it.
With blobs in a bucket, the copy holds which blobs each account holds, but
not their bytes, which stay in the bucket. A blob the server stops holding
after the copy has its object deleted (`BLOBS.md`), and the server started
on the copy, which holds it still, answers `getBlob` of it as of an object
the store does not have, though `listBlobs` lists it. Keeping the bucket's
earlier versions (S3's versioning, R2's and others' alike), or copying the
bucket at the same instant, keeps those. A server started on a copy while
the one it was copied from runs needs a bucket of its own (`BLOBS.md`).
A copy made file by file over a span of time, as `cp` or `rsync` of a
running server's directory makes it, is not such a copy, and nothing is
said of it. A server stopped first, with SIGTERM or SIGKILL, may be copied
any way.
A server started on a copy is a rollback, and does nothing of its own
about it, as the reference does nothing for a database brought back.
What was written after the copy is lost. The firehose numbers its next
events from the copy's last `seq`, so `seq`s it sent before are sent again
for other events, and those who read it, relays and AppViews, hold commits
and a cursor the server no longer has. Handle changes after the copy stay
at the PLC directory.
With `--crash-every N`, before every second kill, the harness puts a run of
eight records to one account, one after another. Once some are answered it
stops the server with SIGSTOP, which freezes it between two of its system
calls as a snapshot freezes its files, with the next put on its way; it
copies the data directory and lets the server go on with SIGCONT. A second
server, started on the copy on a port of its own, with no crawlers, no
metrics or traces, SSRF protection on (`PROXY.md`), and, with blobs in a
bucket, a bucket of its own, must answer every put answered before the
stop; of the rest, a prefix of the run, each record as the put answered
it; the latest commit, the last put's in it; and `getSession` with the
account's token. What a block device's snapshot holds is what a power cut
leaves, which the server's own tests check (above).