~ knotless Search/
↑↓ move↵ open⇧↵ all results
0
0
2209 lines · 89.5 KB Raw History
//! What the server holds in memory of its accounts (`spec/STORE.md`,
//! "Scale"), within `PDS_CACHE_BYTES`.
//!
//! **Held once used**, for each account: its DID, handle, keys, state, mail
//! and DID document (`Account`), with its sessions, app passwords and
//! preferences, a few KiB, made of its row of the service checkpoint and
//! its entries of the service log on its first use (`directory`), and held
//! from then on. **Held while used**: its store (`Store`), loaded from its
//! checkpoint and log on its first use, a read or a write (`Slot`), and let
//! go when it has not been used for `IDLE`, or when the stores held count
//! more than the cache's share of `PDS_CACHE_BYTES`, least recently used
//! first. A store counts what it holds in memory (`Store::cost`): its
//! table's index, where in the log each entry the log changed is, its
//! blobs, and of each space repo its commit and hash, not its records, which
//! are read from the table and the log as they are asked for.
//!
//! **A store being written** holds where each entry of its log after its
//! table is (its memtable), which grows with the log. Past an eighth of
//! the stores' share of the cache (`Cache::memtable`), its log is
//! compacted, whatever the log's size, and what the table then holds is
//! let go; past twice that while the compaction runs, a writer waits for
//! it (`Slot::write_lock`): written faster than its log can be compacted,
//! a store holds no more for it. So too once its log holds `store::BEHIND`
//! past what compacts it by size (`store::ACCOUNT_CAP` at most, past its
//! table): its first use after a start replays no more of it, however
//! large its table. The compaction holds as much again of
//! the logs it folds, with the account's state and a few blocks
//! (`Cache::fold_blocks`), counted with the store while it runs
//! (`Slot::folding`). A store is let go only with nothing of it in flight: no write
//! waiting for its log to be on disk, no compaction of its log, no upload
//! moving into place. Everything it holds is on disk by then, so letting it
//! go is dropping it; its log gives back the room in its file as it goes
//! (`pds_store`), and loading it again syncs nothing: its slot keeps that
//! its log's frames are all on disk (`Slot::durable`), as every account's
//! are from the start where the last server shut down. A log found with
//! zeros after its frames all the same, as a kill of a run before left
//! one it had open, is cut to its frames as it is loaded, with no sync.
//! No store is loaded once a shutdown has closed the accounts' logs
//! (`ops::settle`): what asks for one not held then is unwound
//! (`ShuttingDown`), and the account's files stay as the shutdown's mark
//! says they are.
//!
//! **A start** loads no account but those marked in the service log
//! (`Eager`): an account whose handle change is in flight, that has write
//! notifications or notices to syncers to send, or, with blobs in a bucket,
//! an upload on its way or objects to delete (`blobstore`). Each is marked,
//! durably, before the entry that needs it reaches its own log, and
//! unmarked, with the handle it has then, once nothing needs it. So the
//! service log holds each account's handle but while a mark says to look in
//! its own log.

use crate::events::Firehose;
use crate::{Account, Locked, Store};
use k256::ecdsa::SigningKey;
use pds_store::{Files, Policy, Wal};
use serde_json::{Value as Json, json};
use std::collections::{HashMap, HashSet};
use std::io;
use std::ops::{Deref, DerefMut};
use std::path::PathBuf;
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::sync::{Arc, Mutex, PoisonError, RwLock, RwLockReadGuard, RwLockWriteGuard, Weak};
use std::time::{Duration, Instant};

/// How long a store not used is held.
pub(crate) const IDLE: Duration = Duration::from_secs(120);

/// The service log's entry of an account's mark and handle.
pub(crate) const DIR: &str = "dir";

/// The bytes a record taken down is counted as in the cache: its path and
/// its takedown's reference.
const TAKEDOWN_COST: u64 = 192;

/// What every account's slot shares: where the files are, and what loading
/// a store needs.
pub(crate) struct Ctx {
    pub dir: PathBuf,
    /// The accounts' logs, of which at most a share are open at once
    /// (`Config::share`).
    pub files: Arc<Files>,
    pub policy: Policy,
    /// The backfill window, in milliseconds.
    pub window: u64,
    pub events: Arc<Firehose>,
    pub eager: Arc<Eager>,
    pub cache: Cache,
    /// The blocks of the stores' tables and logs read from disk, and of
    /// the service checkpoint's (`directory`), shared, and the tables'
    /// files, of which at most a share are open at once.
    pub blocks: Arc<pds_store::table::BlockCache>,
    /// While the server starts: a store loaded then with no commit gets
    /// its first from the start, once it knows where seqs go on from.
    pub starting: std::sync::atomic::AtomicBool,
    /// Whether the last server shut down, in this very directory, with
    /// every frame of every log on disk (`ops::durable`): an account's log
    /// is then synced by no load of its store, its first in this run
    /// neither (`Slot::durable`).
    pub durable: bool,
    /// Whether every frame the logs held as this run started is on disk:
    /// from the start where `durable`, or where the directory held no data;
    /// else once the run has made durable what it found (`ops::sync_found`).
    /// Its shutdown leaves the word in its mark only then (`ops::mark`). It
    /// changes nothing for this run's own loads.
    pub found: AtomicBool,
    /// Whether a load of a store failed in this run. What it had mended
    /// of its log by then nothing has synced, and the server stops for it
    /// (`Slot::ensure`); a shutdown that got there first leaves no word
    /// in its mark (`ops::mark`).
    pub failed: AtomicBool,
    /// The bucket blobs are kept in, if they are not kept in the data
    /// directory (`blobstore`).
    pub bucket: Option<Arc<crate::blobstore::Bucket>>,
}

fn lock<T>(m: &Mutex<T>) -> std::sync::MutexGuard<'_, T> {
    m.lock().unwrap_or_else(PoisonError::into_inner)
}

/// Milliseconds since the server started: when a store was last used.
fn clock() -> u64 {
    static START: std::sync::LazyLock<Instant> = std::sync::LazyLock::new(Instant::now);
    START.elapsed().as_millis() as u64 + 1
}

// ---- The marks ----------------------------------------------------------------

/// The accounts a start loads, as the service log marks them.
pub(crate) struct Eager {
    log: Arc<Wal>,
    marked: Mutex<HashSet<String>>,
}

/// The service log's entry: the account's mark, and its handle if given.
pub(crate) fn dir_entry(did: &str, handle: Option<&str>, eager: bool) -> Json {
    let mut e = json!({ "t": DIR, "did": did, "eager": eager });
    if let Some(h) = handle {
        e["handle"] = json!(h);
    }
    e
}

impl Eager {
    pub fn new(log: Arc<Wal>, marked: HashSet<String>) -> Eager {
        Eager {
            log,
            marked: Mutex::new(marked),
        }
    }

    pub fn is_marked(&self, did: &str) -> bool {
        lock(&self.marked).contains(did)
    }

    /// Mark `did`, on disk before this returns, unless it is already.
    pub fn mark(&self, did: &str) -> io::Result<()> {
        if self.is_marked(did) {
            return Ok(());
        }
        let n = self.log.append(&[dir_entry(did, None, true)])?;
        self.log.wait(n)?;
        lock(&self.marked).insert(did.to_string());
        Ok(())
    }

    /// Nothing of `did` needs a start to load it now: unmark it, with the
    /// handle it has, `handle`, if given. Not waited for: a mark lost
    /// only makes a start load the account.
    pub fn relax(&self, did: &str, handle: Option<&str>) {
        let marked = lock(&self.marked).remove(did);
        if !marked && handle.is_none() {
            return;
        }
        if let Err(e) = self.log.append(&[dir_entry(did, handle, false)]) {
            crate::ops::failed_for(
                crate::ops::Level::Error,
                "unmarking an account failed",
                did,
                e,
            );
            if marked {
                lock(&self.marked).insert(did.to_string());
            }
        }
    }

    /// Note the handle `did` has now, its mark kept. Not waited for.
    pub fn relax_handle(&self, did: &str, handle: &str) {
        if let Err(e) = self.log.append(&[dir_entry(did, Some(handle), true)]) {
            crate::ops::failed_for(
                crate::ops::Level::Error,
                "noting an account's handle failed",
                did,
                e,
            );
        }
    }

    /// Forget a deleted account's mark.
    pub fn forget(&self, did: &str) {
        lock(&self.marked).remove(did);
    }
}

impl Store {
    /// Whether a start must load it: a handle change in flight, sends due,
    /// or with blobs in a bucket, an upload on its way or a stray.
    pub(crate) fn needs_start(&self) -> bool {
        self.pending.is_some()
            || self.notify.next_due().is_some()
            || (self.ctx.bucket.is_some() && self.blobs.objects_unsettled())
    }

    /// Whether nothing of it is in flight, so that it may be let go: every
    /// frame its log was given is on disk, with the logs a compaction
    /// sealed, and no compaction runs (`pds_store`, "An open that syncs
    /// nothing").
    fn idle(&self) -> bool {
        Arc::strong_count(&self.wal) == 1
            && self.wal.is_durable(self.wal.written())
            && !self.wal.compacting()
            && self.blobs.settled()
    }

    /// The bytes it is counted as in the cache: what it holds in memory,
    /// not the blocks it reads, which the block cache counts, nor what a
    /// compaction of its log holds, which its slot counts with it. Its table's
    /// index and what the memtable keeps of the log after it, its blobs,
    /// each space repo's commit and hash, the spaces it is the authority
    /// of, its write notifications and its records taken down: each a
    /// total kept as it changes, but the records' blob refs, which are
    /// walked.
    pub(crate) fn cost(&self) -> u64 {
        4096 + self.repo.head_bytes().len() as u64
            + self.disk.cost()
            + self.blobs.cost()
            + self.spaces.cost()
            + self.authority.cost()
            + self.notify.cost()
            + self.takedowns.len() as u64 * TAKEDOWN_COST
    }
}

// ---- The cache --------------------------------------------------------------------

/// The stores held, and the bytes they count, within `budget`.
pub(crate) struct Cache {
    budget: u64,
    held: Mutex<Held>,
}

/// The least a store's memtable may count before its log is compacted for
/// what it counts, however small the cache.
const MEMTABLE_FLOOR: u64 = 256 << 10;

#[derive(Default)]
struct Held {
    slots: HashMap<String, Weak<Account>>,
    bytes: u64,
}

impl Cache {
    pub fn new(budget: u64) -> Cache {
        Cache {
            budget,
            held: Mutex::default(),
        }
    }

    /// What one store's memtable may count before its log is compacted,
    /// whatever the log's size: an eighth of the stores' share of the cache.
    /// A store being written holds where each entry of its log after its
    /// table is, and its log grows to its table's size, or
    /// `store::ACCOUNT_CAP` if that is less, before it is compacted for that
    /// (`store::policy`): past this, it is compacted for what it holds in
    /// memory.
    pub fn memtable(&self) -> u64 {
        (self.budget / 8).max(MEMTABLE_FLOOR)
    }

    /// What one store's memtable may count while its log is compacted
    /// before a writer waits for the compaction: twice `memtable`. So a
    /// store written faster than its log is compacted holds no more than
    /// this and one write of its log, and the compaction as much again of
    /// the logs it folds: half the stores' share in all.
    pub fn memtable_most(&self) -> u64 {
        self.memtable() * 2
    }

    /// The bytes a fold of an account's log may keep of what it reads to
    /// read it again: the places of its frames' items, and the table's
    /// blocks it looks records up in.
    pub fn fold_blocks(&self) -> u64 {
        (self.budget / 8).clamp(1 << 20, 4 << 20)
    }

    /// The bytes the stores held count.
    #[cfg(test)]
    pub fn bytes(&self) -> u64 {
        lock(&self.held).bytes
    }

    /// How many stores are held.
    #[cfg(test)]
    pub fn len(&self) -> usize {
        lock(&self.held).slots.len()
    }

    /// A store held now, or its cost changed by `delta`; let go of others,
    /// least recently used first, if the stores held count too much.
    fn count(&self, a: &Arc<Account>, delta: i64) {
        let over = {
            let mut h = lock(&self.held);
            h.slots
                .entry(a.did.clone())
                .or_insert_with(|| Arc::downgrade(a));
            h.bytes = h.bytes.saturating_add_signed(delta);
            h.bytes > self.budget
        };
        if over {
            self.shrink(Some(&a.did), self.budget - self.budget / 8, 0);
        }
    }

    /// A store let go, which counted `cost`.
    fn forget(&self, did: &str, cost: u64) {
        let mut h = lock(&self.held);
        h.slots.remove(did);
        h.bytes = h.bytes.saturating_sub(cost);
    }

    /// Let go of stores, least recently used first, but `keep`'s, until they
    /// count at most `target`; and of every store not used since `before`.
    fn shrink(&self, keep: Option<&str>, target: u64, before: u64) {
        let mut held: Vec<(u64, Arc<Account>)> = {
            let mut h = lock(&self.held);
            h.slots.retain(|_, w| w.strong_count() > 0);
            h.slots
                .iter()
                .filter(|(did, _)| Some(did.as_str()) != keep)
                .filter_map(|(_, w)| w.upgrade())
                .map(|a| (a.store.used.load(Ordering::Relaxed), a))
                .collect()
        };
        held.sort_by_key(|(used, _)| *used);
        for (used, a) in held {
            if used >= before && lock(&self.held).bytes <= target {
                break;
            }
            a.store.evict();
        }
    }

    /// Let go of every store not used for `IDLE`.
    pub fn sweep(&self) {
        let before = clock().saturating_sub(IDLE.as_millis() as u64);
        self.shrink(None, u64::MAX, before);
    }
}

/// Let go of idle stores every so often while the server runs.
pub(crate) fn sweep_every(svc: &crate::Shared) {
    let weak = Arc::downgrade(svc);
    tokio::spawn(async move {
        loop {
            tokio::time::sleep(IDLE / 4).await;
            let Some(svc) = weak.upgrade() else { return };
            let _ = tokio::task::spawn_blocking(move || svc.ctx.cache.sweep()).await;
        }
    });
}

// ---- An account's slot -------------------------------------------------------------

/// An account's signing key, made on first use. Making one derives its
/// public key, which is most of what making an account of its entries
/// would cost, and many an account is loaded to answer what signs nothing:
/// until then it is the secret alone, checked to be one.
pub(crate) struct Key {
    secret: k256::SecretKey,
    made: std::sync::OnceLock<SigningKey>,
}

impl Key {
    /// The key of these bytes, if they are one: as `SigningKey::from_slice`
    /// takes them.
    pub fn from_slice(bytes: &[u8]) -> Option<Key> {
        let secret = k256::SecretKey::from_slice(bytes).ok()?;
        Some(Key {
            secret,
            made: std::sync::OnceLock::new(),
        })
    }

    /// A key made already.
    pub fn made(key: SigningKey) -> Key {
        Key {
            secret: k256::SecretKey::from(&key),
            made: std::sync::OnceLock::from(key),
        }
    }

    pub fn get(&self) -> &SigningKey {
        self.made.get_or_init(|| SigningKey::from(&self.secret))
    }

    /// The secret's bytes, as an account's entry keeps them.
    pub fn to_bytes(&self) -> k256::FieldBytes {
        self.secret.to_bytes()
    }
}

/// An account's store: loaded on first use, let go when the cache needs
/// the room or it is idle.
pub(crate) struct Slot {
    account: Weak<Account>,
    key: Key,
    ctx: Arc<Ctx>,
    store: RwLock<Option<Box<Store>>>,
    /// When it was last used (`clock`).
    used: AtomicU64,
    /// The bytes the store held counts, `0` if none is.
    cost: AtomicU64,
    /// The bytes a compaction of its log under way holds, counted with its
    /// store's: `0` with none.
    folding: AtomicU64,
    /// Whether every frame its log's files hold is on disk, so that the
    /// load that next opens the log syncs nothing unless it mends it
    /// (`store`, "Starting"): from the start, where the last server left
    /// them so (`Ctx::durable`); and from when its store is let go, which
    /// it is only with all it wrote on disk (`Store::idle`). Not so while
    /// its store is held or being loaded, nor once a load has failed.
    durable: AtomicBool,
    /// Whether its events were looked into for void ones in this run
    /// (`events::Firehose::loaded`): as its store is first loaded, or from
    /// the start for an account made in it.
    looked: AtomicBool,
}

/// An account's store, read.
pub(crate) struct StoreRead<'a>(RwLockReadGuard<'a, Option<Box<Store>>>);

impl Deref for StoreRead<'_> {
    type Target = Store;
    fn deref(&self) -> &Store {
        self.0.as_ref().expect("a store loaded")
    }
}

/// An account's store, written: its cost counted again once let go.
pub(crate) struct StoreWrite<'a> {
    guard: Option<RwLockWriteGuard<'a, Option<Box<Store>>>>,
    slot: &'a Slot,
}

impl Deref for StoreWrite<'_> {
    type Target = Store;
    fn deref(&self) -> &Store {
        self.guard
            .as_ref()
            .and_then(|g| g.as_ref())
            .expect("a store loaded")
    }
}

impl DerefMut for StoreWrite<'_> {
    fn deref_mut(&mut self) -> &mut Store {
        self.guard
            .as_mut()
            .and_then(|g| g.as_mut())
            .expect("a store loaded")
    }
}

impl Drop for StoreWrite<'_> {
    fn drop(&mut self) {
        let Some(g) = self.guard.take() else { return };
        let cost = g.as_ref().map_or(0, |st| {
            // More of its log held in memory than its share of the cache:
            // compacted now, whatever the log's size.
            if st.disk.mem.cost() > self.slot.ctx.cache.memtable() {
                st.wal.compact_soon();
            }
            st.cost()
        });
        drop(g);
        self.slot.recount(cost);
    }
}

/// Data the server cannot recover is never served: the server stops.
pub(crate) fn unrecoverable(dir: &std::path::Path, why: &str) -> ! {
    if cfg!(test) {
        panic!("cannot recover {}: {why}", dir.display());
    }
    eprintln!("pds: cannot recover {}: {why}", dir.display());
    std::process::exit(1)
}

/// What unwinds whatever asks for a store that is not held once the
/// shutdown has closed the accounts' logs (`Slot::load`): a request is
/// answered an internal error (`limits::serve`), and the server's own
/// housekeeping ends there. It is no panic to the log, and no data is
/// unrecoverable: the next start loads the store.
pub(crate) struct ShuttingDown;

/// What a load refused because the accounts' logs are closed answers.
const SHUTTING_DOWN: &str = "the server is shutting down";

impl Slot {
    /// The slot of `account`, signing with `key`, holding `store` if given.
    pub fn new(account: Weak<Account>, key: Key, ctx: Arc<Ctx>, store: Option<Store>) -> Slot {
        let slot = Slot {
            account,
            key,
            store: RwLock::new(None),
            used: AtomicU64::new(clock()),
            cost: AtomicU64::new(0),
            folding: AtomicU64::new(0),
            durable: AtomicBool::new(ctx.durable),
            looked: AtomicBool::new(store.is_some()),
            ctx,
        };
        if let Some(st) = store {
            slot.compacted(&st);
            *crate::write(&slot.store) = Some(Box::new(st));
        }
        slot
    }

    /// The account's signing key, its store loaded or not.
    pub fn signing_key(&self) -> &SigningKey {
        self.key.get()
    }

    /// Compact its store's log from now on, the table put in place under its lock.
    fn compacted(&self, st: &Store) {
        let fold = crate::checkpoint::AccountFold {
            did: st.repo.did.clone(),
            key: self.key.get().clone(),
            window: self.ctx.window,
            blocks: self.ctx.cache.fold_blocks(),
            account: self.account.clone(),
        };
        st.wal.set_fold(Arc::new(fold));
    }

    /// A compaction of its log holds `bytes` in memory now: counted with
    /// its store's when that is next counted.
    pub fn folding(&self, bytes: u64) {
        self.folding.store(bytes, Ordering::Relaxed);
    }

    /// Count a store made with the account in the cache, once the account is in its `Arc`.
    pub fn counted(&self) {
        let cost = crate::read(&self.store).as_ref().map_or(0, |st| st.cost());
        self.recount(cost);
    }

    /// Whether its store is held.
    pub fn is_loaded(&self) -> bool {
        crate::read(&self.store).is_some()
    }

    /// Its events, each one's seq and when it was made, in order of seq,
    /// the greatest seq any event of its has had, and the greatest of those
    /// its log's compaction let go: as its store holds them, or, while it is not loaded, as its files do, read without
    /// loading it, which would make its log durable first and count it in
    /// the cache (`checkpoint::events_on_disk`). Only where its events are
    /// is learnt so: each is sent as its store reads it.
    pub fn events(&self) -> (Vec<(u64, u64)>, u64, u64) {
        {
            // Held while its files are read: nothing loads it meanwhile, so
            // nothing appends to its log or compacts it.
            let g = crate::read(&self.store);
            if let Some(st) = g.as_deref() {
                return (st.event_seqs(), st.last_seq, st.let_go());
            }
            let found = self
                .account
                .upgrade()
                .map(|a| crate::checkpoint::events_on_disk(&self.ctx, &a.did));
            if let Some(Ok(found)) = found {
                return found;
            }
        }
        // Files a start must mend, or the server stop for: as a load finds them.
        let st = self.read_lock();
        (st.event_seqs(), st.last_seq, st.let_go())
    }

    /// Its store's log, if its store is held: nothing is loaded to answer.
    /// A store not held has nothing written that is not on disk (`idle`).
    pub fn loaded_wal(&self) -> Option<Arc<Wal>> {
        crate::read(&self.store).as_ref().map(|st| st.wal.clone())
    }

    /// The bytes of its logs and its checkpoint (`ops::metrics`): as its
    /// store's log counts them, or, while it is not loaded, as its files
    /// hold them (`directory::files_bytes`), read without loading it.
    pub fn log_bytes(&self) -> u64 {
        // Held while its files are read: nothing loads it meanwhile.
        let g = crate::read(&self.store);
        if let Some(st) = g.as_deref() {
            return st.wal.log_bytes() + st.wal.checkpoint_bytes();
        }
        match self.account.upgrade() {
            Some(a) => crate::directory::files_bytes(&self.ctx.dir, &a.did),
            None => 0,
        }
    }

    /// Load its store if it is not held: an error if the data directory
    /// holds what cannot be recovered. A store loaded after the start that
    /// recovery cut bytes off says so: the start's `recovered` line counts
    /// only the logs the start read (`ops::recovered`).
    ///
    /// None is loaded once the shutdown has closed the accounts' logs
    /// (`ops::settle`): a load is a write where it finds something to
    /// mend, and nothing would sync it before the shutdown's mark says
    /// that every log is durable. It is refused under the slot's lock,
    /// before anything of the account's files is read or changed, and the
    /// shutdown closes each store held under that lock too, after it has
    /// closed the logs: so a load either began before, and its log is
    /// closed, and synced, with the others, or is refused.
    pub fn load(&self) -> Result<(), String> {
        self.used.store(clock(), Ordering::Relaxed);
        let mut g = crate::write(&self.store);
        if g.is_some() {
            return Ok(());
        }
        if self.ctx.files.is_closed() {
            return Err(SHUTTING_DOWN.into());
        }
        // Its log is this load's from here on, whatever becomes of it.
        let durable = self.durable.swap(false, Ordering::SeqCst);
        let st = self.open(durable).inspect_err(|_| {
            // Noted under the slot's lock, which a shutdown waits for.
            self.ctx.failed.store(true, Ordering::SeqCst);
        })?;
        let dropped = st.wal.dropped();
        // First loaded in this run: its log may hold a write a power cut
        // kept whose event the index lost, whatever start this is, as a
        // start that followed a shutdown may follow a run that never used it.
        if !self.looked.swap(true, Ordering::SeqCst) {
            self.ctx.events.loaded(&self.account, || st.event_seqs());
        }
        if dropped > 0 && !self.ctx.starting.load(Ordering::SeqCst) {
            crate::ops::warn(
                "recovered an account's log",
                &[
                    ("did", serde_json::json!(st.repo.did)),
                    ("dropped_bytes", serde_json::json!(dropped)),
                ],
            );
        }
        let cost = st.cost();
        *g = Some(Box::new(st));
        drop(g);
        self.recount(cost);
        Ok(())
    }

    /// Its store as its checkpoint and log have it, its first commit made
    /// if it has none; `durable` if every frame its log's files hold is on
    /// disk already.
    fn open(&self, durable: bool) -> Result<Store, String> {
        let a = self.account.upgrade().ok_or("an account no longer held")?;
        let c = &self.ctx;
        let mut r = crate::log::open_repo(c, &a.did, self.key.get(), durable)?;
        self.compacted(&r.store);
        if r.fresh && !c.starting.load(Ordering::SeqCst) {
            // With no events if it was made deactivated, moving here.
            let quiet = crate::read(&a.state).deactivated;
            let landing =
                crate::log::first_commit(&c.events, &a.did, &a.handle(), &mut r.store, quiet)
                    .map_err(|e| format!("{}: {e}", a.did))?;
            landing
                .as_ref()
                .map_or(Ok(()), crate::events::Landing::wait_entry)
                .and_then(|()| r.store.wal.wait(r.store.wal.written()))
                .map_err(|e| format!("{}: {e}", a.did))?;
            if let Some(l) = landing {
                c.events.land(l);
            }
        }
        Ok(r.store)
    }

    /// Load its store, for a caller with no error to give. A load refused
    /// because the server is shutting down unwinds the caller
    /// (`ShuttingDown`), with nothing of the account's files touched. Any
    /// other that fails stops the server: one the shutdown overtook too,
    /// which may have mended part of its log, so that the shutdown leaves
    /// no mark.
    fn ensure(&self) {
        if let Err(why) = self.load() {
            if why == SHUTTING_DOWN {
                std::panic::resume_unwind(Box::new(ShuttingDown));
            }
            let did = self
                .account
                .upgrade()
                .map(|a| a.did.clone())
                .unwrap_or_default();
            unrecoverable(&self.ctx.dir.join("repos").join(did), &why);
        }
    }

    /// The store's cost now `cost`: counted in the cache, with what a
    /// compaction of its log holds.
    fn recount(&self, cost: u64) {
        let cost = match cost {
            0 => 0,
            held => held + self.folding.load(Ordering::Relaxed),
        };
        let before = self.cost.swap(cost, Ordering::Relaxed);
        if before == cost {
            return;
        }
        let Some(a) = self.account.upgrade() else {
            return;
        };
        if cost == 0 {
            self.ctx.cache.forget(&a.did, before);
        } else {
            self.ctx.cache.count(&a, cost as i64 - before as i64);
        }
    }

    /// Let go of its store if nothing of it is in flight: whether it did.
    pub fn evict(&self) -> bool {
        let mut g = match self.store.try_write() {
            Ok(g) => g,
            Err(std::sync::TryLockError::Poisoned(p)) => p.into_inner(),
            Err(std::sync::TryLockError::WouldBlock) => return false,
        };
        if !g.as_ref().is_some_and(|st| st.idle()) {
            return false;
        }
        *g = None;
        // Let go with all it wrote on disk (`idle`), under the lock a load takes.
        self.durable.store(true, Ordering::SeqCst);
        drop(g);
        self.recount(0);
        true
    }

    /// Put a compaction's table in place under the store's lock: `f` is
    /// given the store, if it is held, to read the table from then on.
    pub fn installing(
        &self,
        f: impl FnOnce(Option<&mut Store>) -> Result<(), String>,
    ) -> Result<(), String> {
        let mut g = crate::write(&self.store);
        let done = f(g.as_deref_mut());
        let cost = g.as_ref().map_or(0, |st| st.cost());
        drop(g);
        self.recount(cost);
        done
    }
}

impl Locked for Slot {
    type Read<'a> = StoreRead<'a>;
    type Write<'a> = StoreWrite<'a>;

    fn read_lock(&self) -> StoreRead<'_> {
        loop {
            self.used.store(clock(), Ordering::Relaxed);
            let g = crate::read(&self.store);
            if g.is_some() {
                return StoreRead(g);
            }
            drop(g);
            self.ensure();
        }
    }

    /// A writer of a store whose log is compacted waits for the compaction,
    /// which lets go of what it folded, once the store's memtable counts
    /// more than `Cache::memtable_most`, or its log holds more than
    /// `store::BEHIND` past what compacts it by size (`store::policy`): what
    /// a store being written holds, and what its first use after a start
    /// replays of its log, do not follow how fast the writes come
    /// (`spec/STORE.md`, "Bounds", "The first request to a cold account").
    /// The wait gives up its place among the runtime's workers, so that
    /// other requests go on meanwhile.
    fn write_lock(&self) -> StoreWrite<'_> {
        let most = self.ctx.cache.memtable_most();
        let behind = |wal: &Wal| {
            wal.log_bytes()
                > self
                    .ctx
                    .policy
                    .limit(wal.checkpoint_bytes())
                    .saturating_add(crate::store::BEHIND)
        };
        loop {
            self.used.store(clock(), Ordering::Relaxed);
            let g = crate::write(&self.store);
            match g.as_deref() {
                Some(st)
                    if st.wal.compacting() && (st.disk.mem.cost() > most || behind(&st.wal)) =>
                {
                    let wal = st.wal.clone();
                    drop(g);
                    off_worker(|| wal.wait_compacted());
                }
                Some(_) => {
                    return StoreWrite {
                        guard: Some(g),
                        slot: self,
                    };
                }
                None => {
                    drop(g);
                    self.ensure();
                }
            }
        }
    }
}

/// Run `wait`, which may block for long, without holding one of a
/// multi-threaded runtime's workers through it.
fn off_worker(wait: impl FnOnce()) {
    use tokio::runtime::{Handle, RuntimeFlavor};
    match Handle::try_current() {
        Ok(h) if h.runtime_flavor() == RuntimeFlavor::MultiThread => {
            tokio::task::block_in_place(wait)
        }
        _ => wait(),
    }
}

#[cfg(test)]
mod tests {
    use super::*;
    use crate::testing::*;
    use serde_json::json;

    impl Cache {
        /// Let go of every store nothing of which is in flight.
        fn empty(&self) {
            self.shrink(None, 0, u64::MAX);
        }
    }

    /// A server that loads each account on first use, with a cache of `bytes`.
    fn lazy(dir: &std::path::Path, plc: &str, bytes: u64) -> crate::Shared {
        open_with(dir, plc, |c| {
            c.load_all = false;
            c.cache_bytes = bytes;
        })
    }

    fn loaded(s: &crate::Service) -> usize {
        crate::read(&s.directory)
            .by_did
            .values()
            .filter(|a| a.store.is_loaded())
            .count()
    }

    async fn records(s: &crate::Shared, did: &str) -> Vec<Json> {
        let p = [
            ("repo".to_string(), did.to_string()),
            ("collection".to_string(), "a.b.c".to_string()),
            ("limit".to_string(), "100".to_string()),
        ];
        let got = call_with(s, "com.atproto.repo.listRecords", None, &p, Json::Null)
            .await
            .unwrap();
        let mut v: Vec<Json> = got["records"]
            .as_array()
            .unwrap()
            .iter()
            .map(|r| r["value"].clone())
            .collect();
        v.sort_by_key(|r| r["n"].as_i64());
        v
    }

    /// Stores let go while writes to them go on side by side, as often as
    /// the cache allows (it holds nothing): every write answered is there,
    /// read back, after a restart too, and on the firehose in order.
    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
    async fn evicted_under_concurrent_writes() {
        let plc = plc().await;
        let dir = scratch("evict-writes");
        let s = lazy(&dir, &plc, 0);
        let mut accounts = Vec::new();
        for i in 0..6 {
            accounts.push(create(&s, &format!("evict{i}")).await);
        }
        let stop = Arc::new(std::sync::atomic::AtomicBool::new(false));
        let evicting = {
            let (s, stop) = (s.clone(), stop.clone());
            std::thread::spawn(move || {
                let mut n = 0;
                while !stop.load(Ordering::SeqCst) {
                    s.ctx.cache.empty();
                    n += 1;
                }
                n
            })
        };
        let mut tasks = Vec::new();
        for (did, token) in accounts.clone() {
            let s = s.clone();
            tasks.push(tokio::spawn(async move {
                for n in 0..15 {
                    let body = json!({ "repo": did, "collection": "a.b.c", "rkey": format!("r{n}"), "record": record(n) });
                    call(&s, "com.atproto.repo.putRecord", Some(&token), body).await.unwrap();
                    // Read back at once, loaded again if it went.
                    let p = [("repo".to_string(), did.clone()), ("collection".to_string(), "a.b.c".to_string()), ("rkey".to_string(), format!("r{n}"))];
                    let got = call_with(&s, "com.atproto.repo.getRecord", None, &p, Json::Null).await.unwrap();
                    assert_eq!(got["value"]["n"], json!(n));
                }
            }));
        }
        for t in tasks {
            t.await.unwrap();
        }
        stop.store(true, Ordering::SeqCst);
        assert!(evicting.join().unwrap() > 0);
        s.ctx.cache.empty();
        assert_eq!(loaded(&s), 0, "every store let go once idle");
        let want: Vec<Json> = (0..15).map(record).collect();
        for (did, _) in &accounts {
            assert_eq!(records(&s, did).await, want);
        }
        let events = s.events.get(&s, 0, u64::MAX - 1, 0);
        assert_eq!(events.len(), 6 * (4 + 15));
        drop(s);
        let s = lazy(&dir, &plc, 0);
        assert_eq!(loaded(&s), 0, "a start loads no account");
        for (did, _) in &accounts {
            assert_eq!(records(&s, did).await, want);
        }
        assert_eq!(s.events.get(&s, 0, u64::MAX - 1, 0), events);
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// Killed with a write cut short in an account's log, the server starts
    /// without reading it; the account's first request reads it, as before
    /// the write; and the cache holds no more than it may.
    #[tokio::test]
    async fn cold_loads_after_a_crash() {
        let plc = plc().await;
        let dir = scratch("cold-crash");
        let s = lazy(&dir, &plc, 1 << 30);
        let (alice, ta) = create(&s, "alice").await;
        let (bob, tb) = create(&s, "bob").await;
        for n in 0..5 {
            for (did, t) in [(&alice, &ta), (&bob, &tb)] {
                let body = json!({ "repo": did, "collection": "a.b.c", "rkey": format!("r{n}"), "record": record(n) });
                call(&s, "com.atproto.repo.putRecord", Some(t), body)
                    .await
                    .unwrap();
            }
        }
        let before = records(&s, &alice).await;
        let body =
            json!({ "repo": alice, "collection": "a.b.c", "rkey": "late", "record": record(9) });
        call(&s, "com.atproto.repo.putRecord", Some(&ta), body)
            .await
            .unwrap();
        let events = s.events.get(&s, 0, u64::MAX - 1, 0);
        drop(s);
        redone(&dir);
        // The last write cut short, as a kill during it leaves it.
        let log = crate::log::repo_log(&dir, &alice);
        let bytes = crate::log::log_bytes(&log);
        let last = bytes.len() - last_frame_at(&bytes);
        std::fs::write(&log, &bytes[..bytes.len() - last / 2]).unwrap();
        let s = lazy(&dir, &plc, 1 << 30);
        assert_eq!(loaded(&s), 0);
        assert_eq!(records(&s, &alice).await, before);
        assert_eq!(loaded(&s), 1);
        assert_eq!(
            records(&s, &bob).await,
            (0..5).map(record).collect::<Vec<_>>()
        );
        // The events before the start are found in the stores; the lost one is not among them.
        let now = s.events.get(&s, 0, u64::MAX - 1, 0);
        assert_eq!(now[..], events[..events.len() - 1]);
        // New events go on after every seq reserved.
        let body =
            json!({ "repo": bob, "collection": "a.b.c", "rkey": "after", "record": record(7) });
        call(&s, "com.atproto.repo.putRecord", Some(&tb), body)
            .await
            .unwrap();
        assert_eq!(s.events.get(&s, 0, u64::MAX - 1, 0).len(), now.len() + 1);
        assert!(s.ctx.cache.bytes() > 0 && s.ctx.cache.len() == 2);
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// An account's events are read from its files, no store loaded for
    /// it: those a table holds and those of the log after it, as the store
    /// answers them once loaded, the event of a write cut short left out.
    /// The firehose learns them from its index, no store loaded either, and
    /// sends each as its store reads it; the entry of the write cut short
    /// is let go as its frame is found missing, its seq the last numbered
    /// still, whatever seqs were reserved past it.
    #[tokio::test]
    async fn events_found_without_loading() {
        let plc = plc().await;
        let dir = scratch("cold-events");
        let s = lazy(&dir, &plc, 1 << 30);
        let mut accounts = Vec::new();
        for i in 0..12 {
            let (did, token) = create(&s, &format!("found{i}")).await;
            for n in 0..3 {
                let body = json!({ "repo": did, "collection": "a.b.c", "rkey": format!("r{n}"), "record": record(n) });
                call(&s, "com.atproto.repo.putRecord", Some(&token), body)
                    .await
                    .unwrap();
                if (i, n) == (0, 1) {
                    // The first has a table, and a log after it.
                    let wal = crate::read(&s.account(&did).unwrap().store).wal.clone();
                    wal.wait_compacted();
                    wal.compact().unwrap();
                }
            }
            accounts.push(did);
        }
        let sent = s.events.get(&s, 0, u64::MAX - 1, 0);
        assert_eq!(sent.len(), 12 * 7);
        let held: Vec<Vec<(u64, u64)>> = accounts
            .iter()
            .map(|did| s.account(did).unwrap().store.events().0)
            .collect();
        assert!(held.iter().all(|e| e.len() == 7));
        drop(s);
        redone(&dir);
        // The last account's last write cut short, as a kill during it leaves it.
        let log = crate::log::repo_log(&dir, &accounts[11]);
        let bytes = crate::log::log_bytes(&log);
        std::fs::write(&log, &bytes[..bytes.len() - 100]).unwrap();
        let s = lazy(&dir, &plc, 1 << 30);
        for (did, held) in accounts.iter().zip(&held) {
            let (found, last, _) = s.account(did).unwrap().store.events();
            assert_eq!(found[..], held[..if *did == accounts[11] { 6 } else { 7 }]);
            assert_eq!(Some(last), found.last().map(|e| e.0));
        }
        let (published, last) = (*s.events.subscribe().borrow(), held[11][5].0);
        assert!(
            published > last + 1,
            "seqs are reserved past the last event"
        );
        // The index has the write cut short: numbered, never to be sent.
        s.events.find_old();
        assert_eq!((loaded(&s), s.events.held_len()), (0, sent.len()));
        assert_eq!(s.events.last_made(published), held[11][6].0);
        assert_eq!(
            std::fs::read(&log).unwrap().len(),
            bytes.len() - 100,
            "only read"
        );
        assert_eq!(
            s.events.get(&s, 0, u64::MAX - 1, 0)[..],
            sent[..sent.len() - 1]
        );
        assert_eq!(loaded(&s), 12);
        assert_eq!(s.events.held_len(), sent.len() - 1);
        // Loaded, a store answers the same.
        for (did, held) in accounts.iter().zip(&held).take(11) {
            assert_eq!(
                s.account(did).unwrap().store.events(),
                (held.clone(), held[6].0, 0)
            );
        }
        // An event numbered since the start is the last made.
        let (did, token) = create(&s, "found12").await;
        let published = *s.events.subscribe().borrow();
        assert_eq!(
            (
                s.events.last_made(published),
                s.account(&did).unwrap().store.events().1
            ),
            (published, published)
        );
        drop(token);
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// `spec/STORE.md`, "The first request to a cold account": an account's
    /// log is compacted once it holds the cap (`Config::account_cap`) past
    /// its table, however large the table: written past it twice over since
    /// its last compaction, with a table several times it, it is left
    /// holding no more, and its first load after a start replays no more of
    /// it, and reads little of the table. At a small cap, so that its data
    /// directory stays within a couple of MB.
    #[tokio::test]
    async fn a_first_load_replays_a_capped_log() {
        const CAP: u64 = 32 << 10;
        let plc = plc().await;
        let dir = scratch("cold-capped");
        let start = || {
            open_with(&dir, &plc, |c| {
                (c.load_all, c.cache_bytes) = (false, 1 << 30);
                (c.log_floor, c.account_cap) = (2048, CAP);
            })
        };
        let s = start();
        let (alice, ta) = create(&s, "alice").await;
        let big =
            |n: usize| json!({ "$type": "a.b.c", "text": format!("{n}{}", "e".repeat(1_000)) });
        let write = async |s: &crate::Shared, n: usize| {
            let body = json!({ "repo": alice, "collection": "a.b.c", "rkey": format!("r{n}"), "record": big(n) });
            call(s, "com.atproto.repo.putRecord", Some(&ta), body)
                .await
                .unwrap();
        };
        // A table of several times the cap: the log compacted as it passes
        // the table's size, while that is less than the cap.
        let wal = crate::read(&s.account(&alice).unwrap().store).wal.clone();
        let mut n = 0;
        while wal.checkpoint_bytes() <= 4 * CAP {
            write(&s, n).await;
            n += 1;
        }
        compact(&s, &alice);
        wal.wait_compacted();
        let generation = wal.checkpoint_generation();
        // Written past the cap twice over since: each write's frame holds
        // its record, of 1000 bytes and more.
        for n in n..n + (2 * CAP / 1000 + 1) as usize {
            write(&s, n).await;
        }
        // Once no compaction runs: one that ends begins the next if due.
        loop {
            wal.wait_compacted();
            tokio::time::sleep(Duration::from_millis(20)).await;
            if !wal.compacting() {
                break;
            }
        }
        let log = wal.log_bytes();
        assert!(
            wal.checkpoint_generation() > generation && log <= CAP,
            "a log of {log} bytes past a table of {}",
            wal.checkpoint_bytes()
        );
        drop(wal);
        shut_down(s);
        let s = start();
        s.events.find_old();
        let a = s.account(&alice).unwrap();
        let checkpoint = std::fs::metadata(crate::store::account_layout(&dir, &alice).checkpoint())
            .unwrap()
            .len();
        let replayed = crate::directory::files_bytes(&dir, &alice) - checkpoint;
        let before = s.ctx.blocks.read_bytes();
        a.store.load().unwrap();
        let read = s.ctx.blocks.read_bytes() - before;
        assert!(
            replayed <= CAP && read < checkpoint / 4,
            "its first load replayed {replayed} bytes of log, read {read} of a table of {checkpoint}"
        );
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// After a start with every event out of the backfill window, a
    /// subscriber at the last event made has missed none, though seqs are
    /// reserved past it; one before it has.
    #[tokio::test]
    async fn the_last_event_is_not_outdated() {
        let plc = plc().await;
        let dir = scratch("cold-window");
        let open = || {
            open_with(&dir, &plc, |c| {
                c.load_all = false;
                c.backfill = std::time::Duration::from_millis(300);
            })
        };
        let s = open();
        create(&s, "alice").await;
        let last = *s.events.subscribe().borrow();
        drop(s);
        tokio::time::sleep(std::time::Duration::from_millis(400)).await;
        let s = open();
        let published = *s.events.subscribe().borrow();
        s.events.find_old();
        assert!(published > last);
        assert_eq!(
            (s.events.last_made(published), s.events.held_len()),
            (last, 0)
        );
        assert!(!s.events.outdated(last, published));
        assert!(s.events.outdated(last - 1, published));
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// The events let go are real ones, not the seqs before the first kept:
    /// after a start, seqs are reserved past the last event, and a write
    /// since is numbered after them. A subscriber at the last event before
    /// that start has missed none; one before it has missed that event,
    /// which its log's compaction let go, and which only the table's state
    /// still tells of.
    #[tokio::test]
    async fn seqs_reserved_are_no_events_let_go() {
        let plc = plc().await;
        let dir = scratch("cold-holes");
        let open = || {
            open_with(&dir, &plc, |c| {
                c.load_all = false;
                c.backfill = std::time::Duration::from_millis(1500);
            })
        };
        let s = open();
        let (did, token) = create(&s, "alice").await;
        let old = *s.events.subscribe().borrow();
        drop(s);
        // Out of the window, and of what a compaction keeps past it.
        tokio::time::sleep(std::time::Duration::from_millis(2700)).await;
        let s = open();
        let body = json!({ "repo": did, "collection": "a.b.c", "rkey": "r", "record": record(1) });
        call(&s, "com.atproto.repo.putRecord", Some(&token), body)
            .await
            .unwrap();
        let new = *s.events.subscribe().borrow();
        assert!(new > old + 2, "numbered after the seqs reserved");
        let wal = crate::read(&s.account(&did).unwrap().store).wal.clone();
        wal.wait_compacted();
        wal.compact().unwrap();
        assert_eq!(
            s.account(&did).unwrap().store.events(),
            (
                vec![(new, s.account(&did).unwrap().store.events().0[0].1)],
                new,
                old
            )
        );
        drop(s);
        let s = open();
        let published = *s.events.subscribe().borrow();
        assert_eq!(s.account(&did).unwrap().store.events().2, old);
        s.events.find_old();
        assert_eq!(
            (
                loaded(&s),
                s.events.held_len(),
                s.events.last_made(published)
            ),
            (0, 1, new)
        );
        assert!(
            !s.events.outdated(old, published),
            "the next event after it is in the window"
        );
        assert!(
            s.events.outdated(old - 1, published),
            "the event after it was let go"
        );
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// An upload that is its account's first request after a start is
    /// stored: its store, loaded while the upload's bytes are on their way
    /// in, removes the uploads a process before left on theirs, not this one.
    #[tokio::test]
    async fn an_upload_loads_its_store() {
        let plc = plc().await;
        let dir = scratch("cold-upload");
        let s = lazy(&dir, &plc, 1 << 30);
        let (did, token) = create(&s, "alice").await;
        drop(s);
        let blobs = crate::blobs::blob_dir(&dir, &did);
        std::fs::create_dir_all(&blobs).unwrap();
        std::fs::write(blobs.join(".tmp-left"), b"half").unwrap();
        let s = lazy(&dir, &plc, 1 << 30);
        assert_eq!(loaded(&s), 0);
        let body = b"an upload to an account not loaded".to_vec();
        let got =
            crate::blobs::test_upload(&s, Some(&token), Some("text/plain"), body.clone(), false)
                .await
                .unwrap();
        assert_eq!((loaded(&s), &got["blob"]["size"]), (1, &json!(body.len())));
        let files: Vec<String> = std::fs::read_dir(&blobs)
            .unwrap()
            .map(|e| e.unwrap().file_name().into_string().unwrap())
            .collect();
        assert_eq!(files, [got["blob"]["ref"]["$link"].as_str().unwrap()]);
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// A start loads the accounts marked: one whose handle change is in
    /// flight is loaded, its change held and settled; one settled is not.
    #[tokio::test]
    async fn a_start_loads_the_accounts_marked() {
        let plc = crate::identity::stub::Plc::start().await;
        let dir = scratch("cold-marked");
        let s = lazy(&dir, &plc.url, 1 << 30);
        let (alice, ta) = create(&s, "alice").await;
        create(&s, "bob").await;
        let body = json!({ "handle": "carol.test" });
        call(&s, "com.atproto.identity.updateHandle", Some(&ta), body)
            .await
            .unwrap();
        drop(s);
        let s = lazy(&dir, &plc.url, 1 << 30);
        assert_eq!(loaded(&s), 0, "a change settled leaves no mark");
        assert_eq!(s.account("carol.test").unwrap().did, alice);
        drop(s);
        crate::identity::tests::stop_before_settling(&dir, &alice);
        let s = lazy(&dir, &plc.url, 1 << 30);
        assert_eq!(loaded(&s), 1);
        assert!(
            crate::read(&s.directory)
                .claimed_handles
                .contains("carol.test")
        );
        crate::identity::settle_unfinished(&s);
        for _ in 0..100 {
            if s.account("carol.test").is_some() {
                break;
            }
            tokio::time::sleep(std::time::Duration::from_millis(20)).await;
        }
        assert_eq!(s.account("carol.test").unwrap().did, alice);
        drop(s);
        let s = lazy(&dir, &plc.url, 1 << 30);
        assert_eq!(
            (loaded(&s), s.account("carol.test").unwrap().did.clone()),
            (0, alice)
        );
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// The memtable of `did`'s store: what it counts, and whether its log
    /// is being compacted.
    fn memtable(s: &crate::Service, did: &str) -> (u64, bool) {
        store_of(s, did, |st| (st.disk.mem.cost(), st.wal.compacting()))
    }

    /// A batch of space records written to `did`'s repo in a space of
    /// another's, each of a key no other has.
    async fn space_batch(s: &crate::Shared, did: &str, token: &str, batch: usize) {
        let space = "at://did:plc:ewvi7nxzyoun6zhxrhs64oiz/space/com.example.forum/main";
        let writes: Vec<Json> = (0..50)
            .map(|i| json!({ "$type": "com.atproto.space.applyWrites#create", "collection": "a.b.c", "rkey": format!("b{batch:04}r{i:02}"), "value": record(i) }))
            .collect();
        let body = json!({ "space": space, "repo": did, "writes": writes });
        call(s, "com.atproto.space.applyWrites", Some(token), body)
            .await
            .unwrap();
    }

    /// A batch of records written to `did`'s repository, each of a key no
    /// other has.
    async fn repo_batch(s: &crate::Shared, did: &str, token: &str, batch: usize) {
        let writes: Vec<Json> = (0..50)
            .map(|i| json!({ "$type": "com.atproto.repo.applyWrites#create", "collection": "a.b.c", "rkey": format!("b{batch:04}r{i:02}"), "value": record(i) }))
            .collect();
        let body = json!({ "repo": did, "writes": writes });
        call(s, "com.atproto.repo.applyWrites", Some(token), body)
            .await
            .unwrap();
    }

    /// A store's log is compacted for what its memtable counts, whatever
    /// the log's size: written without a pause, the memtable never counts
    /// more than `memtable_most` and one write, the store with a
    /// compaction's share no more than the cache gives a store being
    /// written, and every record is read back, after a start too.
    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
    async fn a_store_being_written_stays_within_its_share() {
        let plc = plc().await;
        let dir = scratch("memtable-share");
        // Never compacted for its log's size.
        let opened = || {
            open_with(&dir, &plc, |c| {
                (c.load_all, c.cache_bytes, c.log_floor) = (false, 2 << 20, 1 << 30)
            })
        };
        let s = opened();
        let (did, token) = create(&s, "alice").await;
        let (soft, most) = (s.ctx.cache.memtable(), s.ctx.cache.memtable_most());
        assert_eq!((soft, most), (256 << 10, 512 << 10));
        space_batch(&s, &did, &token, 0).await;
        let one = memtable(&s, &did).0;
        let (mut peak, mut compactions, mut last) = (0, 0, 0);
        for batch in 1..120 {
            space_batch(&s, &did, &token, batch).await;
            let (cost, _) = memtable(&s, &did);
            peak = peak.max(cost);
            assert!(cost <= most + one, "batch {batch}: {cost}");
            let generation = store_of(&s, &did, |st| st.wal.checkpoint_generation());
            compactions += u64::from(generation != last);
            last = generation;
        }
        assert!(peak > soft, "{peak}");
        assert!(compactions >= 4, "{compactions}");
        let wal = store_of(&s, &did, |st| st.wal.clone());
        wal.wait_compacted();
        assert!(wal.log_bytes() < 1 << 30);
        drop(wal);
        let listed = |s: &crate::Shared| {
            let (s, did, token) = (s.clone(), did.clone(), token.clone());
            async move {
                let space = "at://did:plc:ewvi7nxzyoun6zhxrhs64oiz/space/com.example.forum/main";
                let mut n = 0;
                let mut cursor: Option<String> = None;
                loop {
                    let mut p = vec![
                        ("space".to_string(), space.to_string()),
                        ("repo".to_string(), did.clone()),
                        ("limit".to_string(), "1000".to_string()),
                    ];
                    p.extend(cursor.take().map(|c| ("cursor".to_string(), c)));
                    let got = call_with(
                        &s,
                        "com.atproto.space.listRecords",
                        Some(&token),
                        &p,
                        Json::Null,
                    )
                    .await
                    .unwrap();
                    n += got["records"].as_array().unwrap().len();
                    match got["cursor"].as_str() {
                        Some(c) => cursor = Some(c.to_string()),
                        None => return n,
                    }
                }
            }
        };
        assert_eq!(listed(&s).await, 120 * 50);
        crate::space_notify::tests::idle(&s).await;
        drop(s);
        assert_eq!(listed(&opened()).await, 120 * 50);
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// An account's fold that waits for word before it folds, the first
    /// time; then folds as the account's does.
    struct Waits {
        fold: crate::checkpoint::AccountFold,
        word: Mutex<std::sync::mpsc::Receiver<()>>,
    }

    impl pds_store::Fold for Waits {
        fn fold_files(
            &self,
            checkpoint: Option<&std::path::Path>,
            logs: &[PathBuf],
            generation: u64,
            out: &mut dyn std::io::Write,
        ) -> Result<(), String> {
            let _ = lock(&self.word).recv();
            self.fold.fold_files(checkpoint, logs, generation, out)
        }

        fn install(
            &self,
            generation: u64,
            rename: &mut dyn FnMut() -> Result<(), String>,
        ) -> Result<(), String> {
            self.fold.install(generation, rename)
        }
    }

    /// A compaction that cannot keep up: while it runs, writes go on until
    /// the memtable counts more than `memtable_most`; the next writer waits
    /// for the compaction, and writes once it has let go of what it folded.
    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
    async fn a_writer_waits_for_a_compaction_that_cannot_keep_up() {
        let plc = plc().await;
        let dir = scratch("memtable-waits");
        let s = open_with(&dir, &plc, |c| {
            (c.load_all, c.cache_bytes, c.log_floor) = (false, 2 << 20, 1 << 30)
        });
        let (did, token) = create(&s, "alice").await;
        let a = s.account(&did).unwrap();
        let (go, word) = std::sync::mpsc::channel();
        store_of(&s, &did, |st| {
            let fold = crate::checkpoint::AccountFold {
                did: did.clone(),
                key: a.store.key.get().clone(),
                window: s.ctx.window,
                blocks: s.ctx.cache.fold_blocks(),
                account: Arc::downgrade(&a),
            };
            st.wal.set_fold(Arc::new(Waits {
                fold,
                word: Mutex::new(word),
            }));
        });
        let most = s.ctx.cache.memtable_most();
        // Past its share, a compaction starts, and waits; writes go on.
        let mut batch = 0;
        while memtable(&s, &did).0 <= most {
            repo_batch(&s, &did, &token, batch).await;
            batch += 1;
        }
        assert!(memtable(&s, &did).1, "no compaction started");
        let held = memtable(&s, &did).0;
        // The next writer waits.
        let writing = {
            let (s, did, token) = (s.clone(), did.clone(), token.clone());
            tokio::spawn(async move { repo_batch(&s, &did, &token, batch).await })
        };
        tokio::time::sleep(Duration::from_millis(300)).await;
        assert!(!writing.is_finished(), "a writer did not wait");
        assert_eq!(memtable(&s, &did), (held, true));
        // A read does not.
        let p = [("did".to_string(), did.clone())];
        call_with(&s, "com.atproto.sync.getLatestCommit", None, &p, Json::Null)
            .await
            .unwrap();
        go.send(()).unwrap();
        drop(go);
        writing.await.unwrap();
        // What the compaction folded is let go, and the write is there.
        assert!(memtable(&s, &did).0 < held);
        assert!(store_of(&s, &did, |st| st.wal.checkpoint_generation()) > 0);
        let _ = std::fs::remove_dir_all(&dir);
    }

    async fn put(s: &crate::Shared, did: &str, token: &str, rkey: &str, n: i64) {
        let body = json!({ "repo": did, "collection": "a.b.c", "rkey": rkey, "record": record(n) });
        call(s, "com.atproto.repo.putRecord", Some(token), body)
            .await
            .unwrap();
    }

    /// The syncs this thread makes loading the store of `did`, not held.
    fn load_syncs(s: &crate::Shared, did: &str) -> u64 {
        let a = s.account(did).unwrap();
        assert!(!a.store.is_loaded());
        let before = syncs();
        a.store.load().unwrap();
        syncs() - before
    }

    /// Compact the log of `did` now.
    fn compact(s: &crate::Shared, did: &str) {
        let wal = crate::read(&s.account(did).unwrap().store).wal.clone();
        wal.wait_compacted();
        wal.compact().unwrap();
    }

    /// Let go of the store of `did`, once nothing of it is in flight.
    async fn let_go(s: &crate::Shared, did: &str) {
        for _ in 0..500 {
            if s.account(did).unwrap().store.evict() {
                return;
            }
            tokio::time::sleep(Duration::from_millis(10)).await;
        }
        panic!("the store of {did} is never idle");
    }

    /// Where the last frame of a log's frames `bytes` begins.
    fn last_frame_at(bytes: &[u8]) -> usize {
        let mut last = 0;
        let end = pds_store::scan_each(bytes, |at, _| {
            last = at;
            Ok(())
        });
        assert_eq!(end, Ok(bytes.len()));
        last
    }

    fn numbered(n: std::ops::Range<i64>) -> Vec<Json> {
        n.map(record).collect()
    }

    /// A seal, as a compaction ends a log with.
    fn seal() -> Vec<u8> {
        pds_store::frame(&[json!({ "t": "seal" })])
    }

    /// After a kill, an account's log is synced as its store is first
    /// loaded. After a shutdown of a run that had made durable what it
    /// found it is not, whether the account has a table or a log alone:
    /// what it reads is what was written, the firehose's events too, and
    /// its next write is kept. The shutdown's word does not outlive the run
    /// that read it: killed, the next start syncs every account's log
    /// again, one that run never used too.
    #[tokio::test]
    async fn after_a_shutdown_first_loads_sync_nothing() {
        let plc = plc().await;
        let dir = scratch("clean-first");
        let start = || lazy(&dir, &plc, 1 << 30);
        let s = start();
        let (alice, ta) = create(&s, "alice").await;
        let (bob, tb) = create(&s, "bob").await;
        let (carol, tc) = create(&s, "carol").await;
        for n in 0..3 {
            for (did, t) in [(&alice, &ta), (&bob, &tb), (&carol, &tc)] {
                put(&s, did, t, &format!("r{n}"), n).await;
            }
        }
        // One has a table, and a log after it.
        compact(&s, &alice);
        put(&s, &alice, &ta, "r3", 3).await;
        let events = s.events.get(&s, 0, u64::MAX - 1, 0);
        let all = [(&alice, 4), (&bob, 3), (&carol, 3)];
        drop(s);
        // Killed: each log is synced as it is first used.
        assert!(crate::ops::unclean(&dir));
        let s = start();
        assert_eq!(loaded(&s), 0);
        for (did, n) in all {
            assert!(load_syncs(&s, did) >= 1, "{did} after a kill");
            assert_eq!(records(&s, did).await, numbered(0..n));
        }
        crate::ops::sync_found(&s).unwrap();
        shut_down(s);
        // Shut down: none is, and the word is out of the mark.
        assert!(!crate::ops::unclean(&dir));
        let s = start();
        assert_eq!(std::fs::read(dir.join(crate::ops::CLEAN)).unwrap(), b"");
        assert_eq!(loaded(&s), 0);
        for (did, n) in [(&alice, 4), (&bob, 3)] {
            assert_eq!(load_syncs(&s, did), 0, "{did} after a shutdown");
            assert_eq!(records(&s, did).await, numbered(0..n));
        }
        // Carol's store is not loaded in this run.
        assert_eq!(loaded(&s), 2);
        put(&s, &alice, &ta, "r4", 4).await;
        put(&s, &bob, &tb, "r3", 3).await;
        assert_eq!(records(&s, &alice).await, numbered(0..5));
        drop(s);
        // Killed since: every log is synced again, and the writes are there.
        let s = start();
        for (did, n) in [(&alice, 5), (&bob, 4), (&carol, 3)] {
            assert!(load_syncs(&s, did) >= 1, "{did} after a kill once more");
            assert_eq!(records(&s, did).await, numbered(0..n));
        }
        assert_eq!(s.events.get(&s, 0, u64::MAX - 1, 0).len(), events.len() + 2);
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// A store let go is loaded again with no sync, in a run that followed
    /// a kill too: it was let go with all it wrote on disk. Its first load
    /// of that run syncs, and what was written in between is read, and kept.
    #[tokio::test]
    async fn a_store_let_go_is_loaded_again_with_no_sync() {
        let plc = plc().await;
        let dir = scratch("clean-again");
        let s = lazy(&dir, &plc, 1 << 30);
        let (alice, ta) = create(&s, "alice").await;
        for n in 0..3 {
            put(&s, &alice, &ta, &format!("r{n}"), n).await;
        }
        drop(s);
        let s = lazy(&dir, &plc, 1 << 30);
        assert!(load_syncs(&s, &alice) >= 1);
        let_go(&s, &alice).await;
        assert_eq!(load_syncs(&s, &alice), 0);
        assert_eq!(records(&s, &alice).await, numbered(0..3));
        // Written, let go, loaded: and again once its log is compacted.
        put(&s, &alice, &ta, "r3", 3).await;
        let_go(&s, &alice).await;
        assert_eq!(load_syncs(&s, &alice), 0);
        assert_eq!(records(&s, &alice).await, numbered(0..4));
        compact(&s, &alice);
        put(&s, &alice, &ta, "r4", 4).await;
        let_go(&s, &alice).await;
        assert_eq!(load_syncs(&s, &alice), 0);
        assert_eq!(records(&s, &alice).await, numbered(0..5));
        // An account made in this run, its store never loaded from disk yet.
        let (bob, tb) = create(&s, "bob").await;
        put(&s, &bob, &tb, "r0", 0).await;
        let_go(&s, &bob).await;
        assert_eq!(load_syncs(&s, &bob), 0);
        assert_eq!(records(&s, &bob).await, numbered(0..1));
        drop(s);
        let s = lazy(&dir, &plc, 1 << 30);
        assert!(load_syncs(&s, &alice) >= 1 && load_syncs(&s, &bob) >= 1);
        assert_eq!(records(&s, &alice).await, numbered(0..5));
        assert_eq!(records(&s, &bob).await, numbered(0..1));
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// What a start that followed a shutdown finds to mend of an account's
    /// log, it mends, and syncs: a torn tail, the log a compaction cut
    /// short left under its table, the `.tmp` of one, a log that ends with
    /// its seal and none after it. Room is nothing to mend: a log with
    /// zeros after its frames, as a kill of a run before leaves one no run
    /// has loaded since, is cut to its frames with no sync, as an account
    /// with nothing to mend is loaded with none.
    #[tokio::test]
    async fn after_a_shutdown_what_is_mended_is_synced() {
        let plc = plc().await;
        let dir = scratch("clean-mend");
        let s = lazy(&dir, &plc, 1 << 30);
        let mut made = Vec::new();
        for name in ["alice", "bob", "carol", "dave", "erin", "frank"] {
            let (did, token) = create(&s, name).await;
            for n in 0..3 {
                put(&s, &did, &token, &format!("r{n}"), n).await;
            }
            made.push((did, token));
        }
        let [alice, bob, carol, dave, erin, frank] = [0, 1, 2, 3, 4, 5].map(|i| made[i].0.clone());
        let layout = |did: &str| crate::store::account_layout(&dir, did);
        // Bob's log folded into a table, and a write after it.
        let folded = std::fs::read(layout(&bob).log(0)).unwrap();
        compact(&s, &bob);
        put(&s, &bob, &made[1].1, "r3", 3).await;
        assert!(!layout(&bob).log(0).exists());
        shut_down(s);
        // Alice's last write cut short.
        let log = layout(&alice).log(0);
        let bytes = std::fs::read(&log).unwrap();
        let last = bytes.len() - last_frame_at(&bytes);
        std::fs::write(&log, &bytes[..bytes.len() - last / 2]).unwrap();
        // Bob's compaction cut short before it deleted the log it folded.
        std::fs::write(layout(&bob).log(0), [folded, seal()].concat()).unwrap();
        // Carol's cut short while it wrote its table.
        std::fs::write(layout(&carol).tmp(), b"part of a table").unwrap();
        // Erin's cut short once it had sealed her log.
        let sealed = [std::fs::read(layout(&erin).log(0)).unwrap(), seal()].concat();
        std::fs::write(layout(&erin).log(0), sealed).unwrap();
        // Frank's log with zeros after its frames: room, as a kill leaves it.
        let roomy = layout(&frank).log(0);
        let frames = std::fs::read(&roomy).unwrap();
        assert_eq!(pds_store::log_len(&roomy).unwrap(), frames.len() as u64);
        std::fs::write(&roomy, [&frames[..], &[0u8; 300]].concat()).unwrap();
        let s = lazy(&dir, &plc, 1 << 30);
        assert_eq!(load_syncs(&s, &dave), 0);
        assert!(load_syncs(&s, &alice) >= 1);
        assert_eq!(
            std::fs::metadata(&log).unwrap().len(),
            (bytes.len() - last) as u64
        );
        assert_eq!(records(&s, &alice).await, numbered(0..2));
        assert!(load_syncs(&s, &bob) >= 2);
        assert!(!layout(&bob).log(0).exists());
        assert_eq!(records(&s, &bob).await, numbered(0..4));
        assert!(load_syncs(&s, &carol) >= 1);
        assert!(!layout(&carol).tmp().exists());
        assert!(load_syncs(&s, &erin) >= 2);
        assert!(layout(&erin).log(1).exists());
        assert_eq!(load_syncs(&s, &frank), 0);
        assert_eq!(std::fs::read(&roomy).unwrap(), frames);
        // Each goes on, and what it then holds is kept.
        for (i, (did, token)) in made.iter().enumerate() {
            put(&s, did, token, "late", 9).await;
            let before = match i {
                0 => 2,
                1 => 4,
                _ => 3,
            };
            let mut want = numbered(0..before);
            want.push(record(9));
            assert_eq!(records(&s, did).await, want);
        }
        drop(s);
        let s = lazy(&dir, &plc, 1 << 30);
        assert_eq!(records(&s, &alice).await.len(), 3);
        assert_eq!(records(&s, &bob).await.len(), 5);
        assert_eq!(records(&s, &erin).await.len(), 4);
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// A shutdown that did not finish is as a kill to the next start, which
    /// syncs each log as it is first used: killed once its logs were synced,
    /// before its mark; one that could not sync a log, which leaves no
    /// mark; and a mark with no word, as a version before left.
    #[tokio::test]
    async fn a_shutdown_not_finished_is_as_a_kill() {
        let plc = plc().await;
        let dir = scratch("clean-unfinished");
        let start = || lazy(&dir, &plc, 1 << 30);
        let mark = dir.join(crate::ops::CLEAN);
        let s = start();
        let (alice, ta) = create(&s, "alice").await;
        put(&s, &alice, &ta, "r0", 0).await;
        let events = s.events.get(&s, 0, u64::MAX - 1, 0);
        // Killed before the mark.
        s.events.close();
        crate::ops::settle(&s).unwrap();
        drop(s);
        assert!(!mark.exists() && crate::ops::unclean(&dir));
        let s = start();
        s.events.find_old();
        assert!(load_syncs(&s, &alice) >= 1);
        assert_eq!(s.events.get(&s, 0, u64::MAX - 1, 0), events);
        // A log that cannot be synced: it holds an upload's entry no one
        // waited for, and is to sync first a file that cannot be opened.
        crate::blobs::test_upload(&s, Some(&ta), Some("text/plain"), b"kept".to_vec(), false)
            .await
            .unwrap();
        store_of(&s, &alice, |st| {
            assert!(!st.wal.is_durable(st.wal.written()));
            st.wal.sync_first([dir.join("jwt.key").join("x")]);
        });
        s.events.close();
        assert!(crate::ops::settle(&s).is_err());
        drop(s);
        assert!(!mark.exists());
        let s = start();
        assert!(load_syncs(&s, &alice) >= 1);
        assert_eq!(records(&s, &alice).await, numbered(0..1));
        drop(s);
        // The mark of a version before.
        std::fs::write(&mark, b"").unwrap();
        assert!(!crate::ops::unclean(&dir));
        let s = start();
        assert!(load_syncs(&s, &alice) >= 1);
        assert_eq!(records(&s, &alice).await, numbered(0..1));
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// Once a shutdown has synced the logs none takes a write: one to a
    /// store held and one to the service log are each refused, and a store
    /// not held is not loaded; the logs are left as they were. So the
    /// start after it reads what the shutdown synced, and syncs nothing.
    #[tokio::test]
    async fn a_shutdown_closes_the_logs() {
        let plc = plc().await;
        let dir = scratch("clean-closed");
        let s = lazy(&dir, &plc, 1 << 30);
        let (alice, ta) = create(&s, "alice").await;
        let (bob, _) = create(&s, "bob").await;
        put(&s, &alice, &ta, "r0", 0).await;
        // An upload's entry is not waited for: the shutdown syncs it.
        let upload = |token: &str, body: &[u8]| {
            let (s, token, body) = (s.clone(), token.to_string(), body.to_vec());
            async move {
                crate::blobs::test_upload(&s, Some(&token), Some("text/plain"), body, false).await
            }
        };
        upload(&ta, b"before").await.unwrap();
        let_go(&s, &bob).await;
        s.events.close();
        crate::ops::settle(&s).unwrap();
        store_of(&s, &alice, |st| {
            assert!(st.wal.is_durable(st.wal.written()))
        });
        let logs = || {
            [
                crate::store::service_layout(&dir).log(0),
                crate::log::repo_log(&dir, &alice),
                crate::log::repo_log(&dir, &bob),
            ]
            .map(|p| std::fs::read(p).unwrap())
        };
        let left = logs();
        assert!(upload(&ta, b"after, to a store held").await.is_err());
        let b = s.account(&bob).unwrap();
        assert!(!b.store.is_loaded() && b.store.load().is_err());
        drop(b);
        let body = json!({ "identifier": "alice.test", "password": "alice-pw" });
        assert!(
            call(&s, "com.atproto.server.createSession", None, body)
                .await
                .is_err()
        );
        assert_eq!(logs(), left);
        crate::ops::mark(&s).unwrap();
        drop(s);
        let s = lazy(&dir, &plc, 1 << 30);
        assert_eq!((load_syncs(&s, &alice), load_syncs(&s, &bob)), (0, 0));
        assert_eq!(records(&s, &alice).await, numbered(0..1));
        session(&s, "alice").await;
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// Once a shutdown has closed the logs no store is loaded: a load of
    /// an account not held is refused, whatever asks for its store unwinds,
    /// a request for it is answered an internal error, and nothing of its
    /// files is made, mended or synced, though its log has something to
    /// mend, a seal and no log after it. A store held is read as before.
    /// The start after the shutdown mends the log, and syncs.
    #[tokio::test]
    async fn a_shutdown_loads_no_store() {
        let plc = plc().await;
        let dir = scratch("clean-no-load");
        let s = lazy(&dir, &plc, 1 << 30);
        let (alice, ta) = create(&s, "alice").await;
        let (bob, tb) = create(&s, "bob").await;
        put(&s, &alice, &ta, "r0", 0).await;
        put(&s, &bob, &tb, "r0", 0).await;
        shut_down(s);
        // Bob's compaction was cut short once it had sealed his log.
        let layout = crate::store::account_layout(&dir, &bob);
        let sealed = [std::fs::read(layout.log(0)).unwrap(), seal()].concat();
        std::fs::write(layout.log(0), sealed).unwrap();
        let files = || {
            let mut files: Vec<(String, Vec<u8>)> = std::fs::read_dir(&layout.dir)
                .unwrap()
                .map(|e| e.unwrap())
                .filter(|e| e.file_type().unwrap().is_file())
                .map(|e| {
                    (
                        e.file_name().into_string().unwrap(),
                        std::fs::read(e.path()).unwrap(),
                    )
                })
                .collect();
            files.sort();
            files
        };
        let found = files();
        let s = lazy(&dir, &plc, 1 << 30);
        assert_eq!(load_syncs(&s, &alice), 0);
        let listener = tokio::net::TcpListener::bind(("127.0.0.1", 0))
            .await
            .unwrap();
        let addr = listener.local_addr().unwrap();
        tokio::spawn(crate::limits::serve(
            listener,
            crate::router(s.clone()),
            std::future::pending::<()>(),
        ));
        let latest = |did: &str| {
            let url = format!("http://{addr}/xrpc/com.atproto.sync.getLatestCommit?did={did}");
            async move { reqwest::get(url).await.unwrap().status().as_u16() }
        };
        assert_eq!(latest(&alice).await, 200);
        s.events.close();
        crate::ops::settle(&s).unwrap();
        let b = s.account(&bob).unwrap();
        let before = syncs();
        assert_eq!(b.store.load().unwrap_err(), SHUTTING_DOWN);
        let unwound = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
            let _ = crate::read(&b.store);
        }))
        .unwrap_err();
        assert!(unwound.is::<ShuttingDown>());
        assert_eq!((latest(&bob).await, latest(&alice).await), (500, 200));
        assert!(!b.store.is_loaded());
        assert_eq!((syncs(), files()), (before, found));
        drop(b);
        crate::ops::mark(&s).unwrap();
        drop(s);
        let s = lazy(&dir, &plc, 1 << 30);
        assert!(load_syncs(&s, &bob) >= 2);
        assert!(layout.log(1).exists());
        assert_eq!(records(&s, &bob).await, numbered(0..1));
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// A run in which a load of a store failed leaves no word at its
    /// shutdown: what that load had mended by then, nothing synced. The
    /// start after it syncs each log as it is first loaded.
    #[tokio::test]
    async fn a_load_that_failed_leaves_no_word() {
        let plc = plc().await;
        let dir = scratch("clean-failed-load");
        let mark = || std::fs::read(dir.join(crate::ops::CLEAN)).unwrap();
        let s = lazy(&dir, &plc, 1 << 30);
        let (alice, ta) = create(&s, "alice").await;
        let (bob, tb) = create(&s, "bob").await;
        put(&s, &alice, &ta, "r0", 0).await;
        put(&s, &bob, &tb, "r0", 0).await;
        shut_down(s);
        assert!(mark().starts_with(crate::ops::DURABLE));
        // A log after one that is missing: nothing a load can mend.
        let stray = crate::store::account_layout(&dir, &alice).log(2);
        std::fs::write(&stray, b"").unwrap();
        let s = lazy(&dir, &plc, 1 << 30);
        assert!(s.account(&alice).unwrap().store.load().is_err());
        assert_eq!(load_syncs(&s, &bob), 0);
        shut_down(s);
        assert_eq!(mark(), b"");
        std::fs::remove_file(&stray).unwrap();
        let s = lazy(&dir, &plc, 1 << 30);
        assert!(load_syncs(&s, &alice) >= 1 && load_syncs(&s, &bob) >= 1);
        assert_eq!(records(&s, &alice).await, numbered(0..1));
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// Copy every file of the directory `from` into `to`, made new.
    fn copy_all(from: &std::path::Path, to: &std::path::Path) {
        std::fs::create_dir_all(to).unwrap();
        for e in std::fs::read_dir(from).unwrap() {
            let e = e.unwrap();
            let to = to.join(e.file_name());
            if e.file_type().unwrap().is_dir() {
                copy_all(&e.path(), &to);
            } else {
                std::fs::copy(e.path(), to).unwrap();
            }
        }
    }

    /// The word is its directory's. A copy of a stopped server's directory,
    /// every file of it in a new one, holds the mark byte for byte, with
    /// its time, and not the word: a start on it syncs a log as it is
    /// first loaded, and makes durable what it found, as after a kill,
    /// though it is not said to have recovered. No word either in a mark
    /// written again since its shutdown wrote it, in one with a number
    /// altered, in the bare word, as nothing wrote it, or once `repos` has
    /// changed since the mark was left; and whatever a mark held is taken
    /// out of it. The directory the shutdown left has the word, and its
    /// first loads sync nothing.
    #[tokio::test]
    async fn the_word_is_its_directorys() {
        use crate::ops::{CLEAN, DURABLE, durable};
        let plc = plc().await;
        let dir = scratch("clean-bound");
        let copy = scratch("clean-bound-copy");
        let found = |s: &crate::Shared| s.ctx.found.load(Ordering::SeqCst);
        let s = lazy(&dir, &plc, 1 << 30);
        let (alice, ta) = create(&s, "alice").await;
        put(&s, &alice, &ta, "r0", 0).await;
        shut_down(s);
        let mark = dir.join(CLEAN);
        let word = std::fs::read(&mark).unwrap();
        let written = std::fs::metadata(&mark).unwrap().modified().unwrap();
        let text = String::from_utf8(word.clone()).unwrap();
        let lines: Vec<&str> = text.lines().collect();
        assert_eq!(lines.len(), 4, "{text}");
        assert!(word.starts_with(DURABLE) && text.ends_with('\n'));
        // Write a mark, as of `at` if given.
        let write = |mark: &std::path::Path, held: &[u8], at: Option<std::time::SystemTime>| {
            std::fs::write(mark, held).unwrap();
            if let Some(at) = at {
                let f = std::fs::File::options().write(true).open(mark).unwrap();
                f.set_modified(at).unwrap();
            }
        };
        // A copy, which keeps the mark's time.
        copy_all(&dir, &copy);
        write(&copy.join(CLEAN), &word, Some(written));
        assert!(!crate::ops::unclean(&copy));
        let c = lazy(&copy, &plc, 1 << 30);
        assert_eq!(std::fs::read(copy.join(CLEAN)).unwrap(), b"");
        assert!(!found(&c));
        assert!(load_syncs(&c, &alice) >= 1, "a first load on a copy");
        assert_eq!(records(&c, &alice).await, numbered(0..1));
        crate::ops::sync_found(&c).unwrap();
        shut_down(c);
        // Left by its own shutdown, the copy has a word of its own.
        let c = lazy(&copy, &plc, 1 << 30);
        assert!(found(&c));
        assert_eq!(load_syncs(&c, &alice), 0);
        drop(c);
        // The mark written over, in place: the directory is as it was left.
        let ask = |held: &[u8], at: Option<std::time::SystemTime>| {
            write(&mark, held, at);
            let word = durable(&dir).unwrap();
            assert_eq!(std::fs::read(&mark).unwrap(), b"", "taken out");
            word
        };
        let asked = |held: &[u8]| ask(held, Some(written));
        assert!(asked(&word), "the mark as it was left");
        assert!(!ask(&word, None), "written since");
        assert!(!asked(DURABLE), "the bare word");
        assert!(!asked(b""));
        assert!(!asked(&[&word[..], b"x"].concat()));
        assert!(!asked(&word[..word.len() - 1]));
        // Each number, altered.
        let mut altered = 0;
        for (l, line) in lines.iter().enumerate().skip(1) {
            let fields: Vec<&str> = line.split(' ').collect();
            for f in 1..fields.len() {
                let mut other = fields.clone();
                let last = if fields[f].ends_with('1') { "2" } else { "1" };
                let changed = format!("{}{last}", &fields[f][..fields[f].len() - 1]);
                other[f] = &changed;
                let mut all = lines.clone();
                let line = other.join(" ");
                all[l] = &line;
                assert!(!asked(format!("{}\n", all.join("\n")).as_bytes()), "{line}");
                altered += 1;
            }
        }
        assert_eq!(altered, 7, "{text}");
        assert!(asked(&word));
        // `repos` changed since: its mode set, to what it was.
        tokio::time::sleep(Duration::from_millis(50)).await;
        let repos = dir.join("repos");
        let mode = std::fs::metadata(&repos).unwrap().permissions();
        std::fs::set_permissions(&repos, mode).unwrap();
        assert!(!asked(&word));
        // A start takes the word of its own directory, and no other's.
        let s = lazy(&dir, &plc, 1 << 30);
        assert!(!found(&s));
        assert!(load_syncs(&s, &alice) >= 1);
        crate::ops::sync_found(&s).unwrap();
        shut_down(s);
        let left = std::fs::read(&mark).unwrap();
        let left_at = std::fs::metadata(&mark).unwrap().modified().unwrap();
        assert!(left.starts_with(DURABLE) && left != word);
        // A start comes later than its shutdown by more than the grain of
        // the file system's times, a few milliseconds at most.
        tokio::time::sleep(Duration::from_millis(50)).await;
        let s = lazy(&dir, &plc, 1 << 30);
        assert_eq!(std::fs::read(&mark).unwrap(), b"");
        assert!(found(&s));
        assert_eq!(load_syncs(&s, &alice), 0);
        assert_eq!(records(&s, &alice).await, numbered(0..1));
        // The word a start took names the directory no more, put back.
        drop(s);
        assert!(!ask(&left, Some(left_at)) && !asked(&word));
        for dir in [&dir, &copy] {
            let _ = std::fs::remove_dir_all(dir);
        }
    }

    /// A log no run has loaded since a kill is synced by no load and by no
    /// shutdown. So a run that started after a kill leaves the word at its
    /// shutdown only once it has made durable what it found, in syncs
    /// counted as any are. Shut down before that, it leaves its mark
    /// without the word, and the next start syncs such a log as it is first
    /// loaded. Shut down after it, the next start syncs nothing for it, and
    /// reads what was written. That pass changes nothing for its own run's
    /// first loads, which sync as after a kill.
    #[tokio::test]
    async fn what_a_start_found_is_durable_before_the_word_is_left() {
        let plc = plc().await;
        let dir = scratch("clean-found");
        let start = || lazy(&dir, &plc, 1 << 30);
        let mark = || std::fs::read(dir.join(crate::ops::CLEAN)).unwrap();
        let found = |s: &crate::Shared| s.ctx.found.load(Ordering::SeqCst);
        let s = start();
        assert!(found(&s), "a directory with no data");
        let mut made = Vec::new();
        for name in ["alice", "bob", "carol"] {
            let (did, token) = create(&s, name).await;
            put(&s, &did, &token, "r0", 0).await;
            made.push((did, token));
        }
        let [alice, bob, carol] = [0, 1, 2].map(|i| made[i].0.clone());
        // Bob has a table, and a log after it.
        compact(&s, &bob);
        put(&s, &bob, &made[1].1, "r1", 1).await;
        drop(s);
        // Killed. A run that loads alice and never bob, shut down before
        // it has made durable what it found, leaves no word.
        let s = start();
        assert!(!found(&s));
        assert!(load_syncs(&s, &alice) >= 1);
        shut_down(s);
        assert_eq!(mark(), b"");
        assert!(!crate::ops::unclean(&dir));
        // So bob's log, which nothing has synced since the kill, is synced
        // as it is first loaded.
        let s = start();
        assert!(!found(&s));
        assert!(load_syncs(&s, &bob) >= 1);
        assert_eq!(records(&s, &bob).await, numbered(0..2));
        // On a thread of its own, as a server that serves makes it; and
        // killed all the same, which leaves no word.
        crate::ops::sync_found_soon(&s);
        for _ in 0..500 {
            if found(&s) {
                break;
            }
            tokio::time::sleep(Duration::from_millis(10)).await;
        }
        assert!(found(&s));
        drop(s);
        // Killed again. This run makes durable what it found, bob's log
        // among it, which it never loads.
        let s = start();
        assert!(load_syncs(&s, &alice) >= 1);
        let before = syncs();
        crate::ops::sync_found(&s).unwrap();
        assert!(syncs() > before, "counted as any sync is");
        assert!(found(&s));
        let after = syncs();
        crate::ops::sync_found(&s).unwrap();
        assert_eq!(syncs(), after, "made durable once");
        assert!(load_syncs(&s, &carol) >= 1, "its own run's first load");
        shut_down(s);
        assert!(mark().starts_with(crate::ops::DURABLE));
        let s = start();
        assert_eq!(mark(), b"");
        assert!(found(&s));
        assert_eq!(load_syncs(&s, &bob), 0, "not loaded since the kill");
        assert_eq!(records(&s, &bob).await, numbered(0..2));
        assert_eq!(load_syncs(&s, &alice), 0);
        assert_eq!(records(&s, &alice).await, numbered(0..1));
        assert_eq!(records(&s, &carol).await, numbered(0..1));
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// File by file, what a start found is each account's logs, its table
    /// and its directory handed to the drive, and then one sync of `repos`:
    /// each counted, nothing loaded, on one thread or several. What is no
    /// account's directory is passed over, and a directory with no
    /// accounts needs none.
    #[tokio::test]
    async fn what_a_start_found_is_synced_file_by_file() {
        let plc = plc().await;
        let dir = scratch("clean-walk");
        let s = lazy(&dir, &plc, 1 << 30);
        let (alice, ta) = create(&s, "alice").await;
        let (bob, tb) = create(&s, "bob").await;
        put(&s, &alice, &ta, "r0", 0).await;
        compact(&s, &bob);
        put(&s, &bob, &tb, "r0", 0).await;
        drop(s);
        std::fs::write(dir.join("repos").join("stray"), b"no account").unwrap();
        let before = syncs();
        crate::store::flush_found(&dir, 1).unwrap();
        // Alice's log and directory; bob's log, table and directory; `repos`.
        assert_eq!(syncs() - before, 6);
        crate::store::flush_found(&dir, 4).unwrap();
        let s = lazy(&dir, &plc, 1 << 30);
        assert_eq!(loaded(&s), 0);
        assert_eq!(records(&s, &bob).await, numbered(0..1));
        let before = syncs();
        crate::store::flush_found(&scratch("clean-walk-none"), 4).unwrap();
        assert_eq!(syncs(), before);
        let _ = std::fs::remove_dir_all(&dir);
    }
}