~ knotless Search/
↑↓ move↵ open⇧↵ all results
0
0
2990 lines · 118.9 KB Raw History
//! Generations of a log, its checkpoint, and compacting it (`spec/STORE.md`).
//!
//! A log with generations is the files `<dir>/<stem>.<g>.log`, one for each
//! generation `g` from its checkpoint's on, and `<dir>/<stem>.checkpoint`, the
//! state as of the start of the checkpoint's generation, which the caller's
//! `Fold` makes of the checkpoint before and the logs after it. Every log but
//! the last ends with its seal, a frame of `[{"t":"seal"}]` alone, which no
//! caller ever sees. Frames are numbered on across generations, seals too.
//!
//! Compacting from generation `g` (`compact`, an append that makes it due,
//! or the end of one begun so, if what was appended meanwhile makes the
//! next due):
//!
//! 1. under the log's lock, with a torn tail cut back first, the room after
//!    log `g`'s last frame is given back, its seal written there, and log
//!    `g + 1` made, empty, for appends to go to: no fsync, nothing read;
//! 2. on the compaction's own thread, log `g` and the directory are made
//!    durable;
//! 3. the checkpoint before and the logs after it, `c..=g`, are read from disk
//!    and folded into `<checkpoint>.tmp`, which is fsynced;
//! 4. that is renamed over the checkpoint, and the directory fsynced;
//! 5. every log before `g + 1` is deleted.
//!
//! No frame of log `g + 1` is durable before step 2 is done: the first group
//! commit after the seal does it itself if the compaction has not yet, with
//! the file of log `g` kept open until then. A compaction that fails says so on
//! standard error, deletes the `.tmp` and leaves the logs as they are, for the
//! next to fold them all, and none is made due for 10 seconds after.
//!
//! A generation's **age**, for compacting by age (`Policy::max_age`), runs
//! from when its log was made, as its files tell when the log is opened
//! (`began`), not from the open: a log older than its policy's age when it
//! is opened, as after a start or a log let go and opened again, is
//! compacted on its first append, as it would have been had this process
//! held it all along. A start reads no file for it but the logs it opens.
//!
//! A sealed log holds no room, which goes before its seal is written. One
//! whose room could not be given back has zeros after its seal: it is
//! sealed and whole all the same, to a start, which cuts them off, and to
//! what reads it before one does (`read_sealed`, `read_log`).
//!
//! **Opening** (`Wal::open_log_with`) mends what a kill or a power cut at any
//! point of that left: the `.tmp` and the logs the checkpoint covers are
//! deleted; what follows a log's last whole frame is cut off, room (the
//! zeros a log open when its process was killed keeps after its frames), a
//! torn tail, or a frame torn in room; the logs after one cut short or not
//! sealed are deleted; and a log is made where the last ends with its seal.
//! It then makes every log kept durable, and the directory if its entries
//! changed, before it answers. A process killed may have left frames the
//! system had not written out, and what is made of a frame, an answer, an
//! entry of another file or a frame after it, must not outlive the frame.
//! The log's last file is locked before anything is changed, and let go;
//! each file is locked while it is cut and synced, and the one appended
//! to from then on (`lib`).
//!
//! **An open that syncs nothing.** The caller may say that every frame the
//! files hold is on disk (`durable`). An open that then finds nothing to
//! mend, a log to append to and no more, syncs nothing and holds no file
//! open; one that finds anything to mend does all of the above, as if it
//! had not been told. Room is nothing to mend. Zeros after a log's frames
//! may be there though its frames are all on disk: a kill left them, in a
//! run before the one whose shutdown the caller knows of, which never
//! opened the log; or the cut that gave them back did not outlive a power
//! cut. They are cut off, each file locked while it is, by a cut that
//! nothing syncs, as the one that gives room back is not: no frame's being
//! on disk rests on it, zeros are never read as a frame, and a frame
//! written where they were is synced with the file's size as any that
//! grows it. Part of a frame after a log's whole ones, in room or past the
//! frames, is a torn tail, and is mended and synced for. An open that
//! syncs nothing tries the last log's lock as every open does, so that a
//! log another handle writes to is an error then; it holds no lock after,
//! and needs none while nothing is appended, since nothing of the log is
//! written and what reads it takes no lock. The first append opens the
//! file and locks it, as it does a file the log's pool had closed: only if
//! it is the size it was found, so that a frame another handle wrote
//! meanwhile is an error, never written over.
//!
//! Every frame is on disk in three cases, and the caller is to know that
//! it is in one of them:
//!
//! - The log was closed (`Wal::close`), and nothing has written to its
//!   files since. Closing refuses every later append and every later seal
//!   under the log's lock, and gives back the current log's room there;
//!   then syncs as a group commit does: the files frames rest on, each
//!   sealed log not yet durable with the directory, and the current log;
//!   then lets the files go, under the log's lock, which are another's to
//!   open from then on. A compaction under way makes no `.tmp`, puts none
//!   in place and deletes no log and no `.tmp` from then on: each of those
//!   it does under the log's lock, having found there that the files are
//!   not let go. What it may still do changes no entry of the directory:
//!   write on in a `.tmp` it had made, which nothing puts in place, and
//!   sync the directory after a rename it had made. So the files are as a
//!   kill at the step it had reached leaves them. Before its `.tmp` is
//!   made, or once the logs a checkpoint put in place before the close
//!   covers are deleted, nothing to mend, and every frame on disk, the
//!   sealed log's by the close; else a `.tmp` or logs the checkpoint
//!   covers, which the open mends, and syncs for.
//! - This process opened the log before, and dropped it with every frame it
//!   was given waited for (`is_durable` of `written`) and no compaction
//!   running. What the open before found was durable, or it made it so. A
//!   wait that covers a seal syncs its log and the directory first, so every
//!   sealed log is durable; a checkpoint is synced before it is renamed
//!   into place, and the logs it covers are deleted only once the directory
//!   is synced. What may then not be on disk is no frame: the entries of
//!   logs deleted, which a power cut brings back for the next open to
//!   delete; and the bytes of an append that failed, cut off the end or
//!   still there, a torn tail to the next open if a power cut brings them
//!   back or the cut could not be made.
//! - Nothing has written to the log's files since a sync that reaches the
//!   disk took in all of them, whoever wrote them and whether or not
//!   anything opened the log: a sync of their file system
//!   (`sync_filesystem`), or each file and then their directory handed to
//!   the drive (`flush_file`) before a sync of anything on that drive
//!   (`sync_file`, `sync_dir`). The frames a killed process left in the
//!   system's memory are on disk from then on. What the kill left to mend
//!   is there still, a torn tail or a compaction cut short, for the open
//!   that finds it to mend and sync for, and so is its room.
//!
//! In the first two the room goes as the log does, by a cut of the file
//! that nothing syncs. The file is its frames alone to whatever opens it
//! next in a system still running; where a power cut undid the cut, or it
//! could not be made, the next open finds zeros after the frames and cuts
//! them off again. So a log said to be durable holds no room once it is
//! opened, and the first frame written to it then grows its file, by the
//! frame and as much again, under that frame's own sync.
//!
//! A log in none of these, as one of a process killed that nothing has
//! synced since, is not durable: its last frames may be in the system's
//! memory alone, and its file may hold room, or part of a frame in it.
//! Opened not said to be durable, its room is cut off and every log synced,
//! before anything is made of a frame.
//!
//! A frame of a log given a journal (`journal`) is on disk once the
//! journal's record of it is, until a checkpoint of the journal makes the
//! log's own file durable; a start puts back from the journal what the
//! files lack before any log is opened (`journal::redo`), so that an open
//! finds every such frame in its file, as on disk, and the three cases
//! above hold of it as they are said.

use super::*;
use std::time::Duration;

/// `L` past which a log is compacted by size, however small its checkpoint:
/// 1 MiB.
pub const FLOOR: u64 = 1 << 20;

/// After a compaction fails, how long until one is made due again.
const QUIET: Duration = Duration::from_secs(10);

/// The files of a log with generations.
#[derive(Clone, Debug)]
pub struct Layout {
    pub dir: PathBuf,
    pub stem: String,
}

impl Layout {
    pub fn new(dir: impl Into<PathBuf>, stem: &str) -> Layout {
        Layout {
            dir: dir.into(),
            stem: stem.to_string(),
        }
    }

    /// The log of generation `g`: `<dir>/<stem>.<g>.log`.
    pub fn log(&self, g: u64) -> PathBuf {
        self.dir.join(format!("{}.{g}.log", self.stem))
    }

    /// `<dir>/<stem>.checkpoint`.
    pub fn checkpoint(&self) -> PathBuf {
        self.dir.join(format!("{}.checkpoint", self.stem))
    }

    /// The checkpoint being written: `<checkpoint>.tmp`.
    pub fn tmp(&self) -> PathBuf {
        self.dir.join(format!("{}.checkpoint.tmp", self.stem))
    }

    /// The generations of the logs in the directory, in order.
    pub fn generations(&self) -> io::Result<Vec<u64>> {
        let mut gens = Vec::new();
        for e in std::fs::read_dir(&self.dir)? {
            if let Some(g) = e?.file_name().to_str().and_then(|n| self.generation_of(n)) {
                gens.push(g);
            }
        }
        gens.sort_unstable();
        Ok(gens)
    }

    /// The generation a file's name makes it the log of: decimal, with no
    /// leading zeros.
    fn generation_of(&self, name: &str) -> Option<u64> {
        let g = name
            .strip_prefix(self.stem.as_str())?
            .strip_prefix('.')?
            .strip_suffix(".log")?;
        let decimal = !g.is_empty()
            && g.bytes().all(|b| b.is_ascii_digit())
            && (g == "0" || !g.starts_with('0'));
        if decimal { g.parse().ok() } else { None }
    }
}

/// When a log is compacted, so that its size and the time to replay it
/// follow what is kept (`spec/STORE.md`, "Bounds").
#[derive(Clone, Copy, Debug)]
pub struct Policy {
    /// By size, once `L` is more than this and than `C`, or than `cap` if
    /// `C` is more than that.
    pub floor: u64,
    /// The most of `C` that `L` may grow to before it is compacted by size:
    /// once `L` is more than `max(min(C, cap), floor)`. `None` for no cap,
    /// `L` growing to `C` whatever that is.
    pub cap: Option<u64>,
    /// By age, once its generation began more than this ago; never if
    /// `None`. When it began is told by its files when the log is opened
    /// (`Wal::open_log_with`), so its age runs on across a start, or a log
    /// let go and opened again: one older than this when opened is
    /// compacted on its first append.
    pub max_age: Option<Duration>,
}

impl Policy {
    /// The policy of a backfill window `w`: 1 MiB, no cap, and `2 w`.
    pub fn window(w: Duration) -> Policy {
        Policy {
            floor: FLOOR,
            cap: None,
            max_age: Some(w.saturating_mul(2)),
        }
    }

    /// `L` past which a log whose checkpoint is `checkpoint` bytes is
    /// compacted by size.
    pub fn limit(&self, checkpoint: u64) -> u64 {
        self.cap
            .map_or(checkpoint, |cap| checkpoint.min(cap))
            .max(self.floor)
    }
}

impl Default for Policy {
    /// By size from 1 MiB, with no cap, never by age.
    fn default() -> Policy {
        Policy {
            floor: FLOOR,
            cap: None,
            max_age: None,
        }
    }
}

/// What makes a log's checkpoints.
pub trait Fold: Send + Sync {
    /// The checkpoint of generation `generation`, made from the checkpoint
    /// before (its bytes, `None` if there is none) and every frame of the
    /// sealed logs after it, in order, seals stripped. Runs on the
    /// compaction's own thread, outside every lock.
    fn fold(
        &self,
        _checkpoint: Option<Vec<u8>>,
        _frames: Vec<Vec<Json>>,
        _generation: u64,
    ) -> Result<Vec<u8>, String> {
        Err("a fold of files only".into())
    }

    /// The same, read from the files, the checkpoint at `checkpoint` if
    /// there is one and the sealed logs `logs` in order (each must be whole
    /// and sealed: `read_sealed`), written to `out` as it is made. By
    /// default, all read into memory and `fold`ed.
    fn fold_files(
        &self,
        checkpoint: Option<&Path>,
        logs: &[PathBuf],
        generation: u64,
        out: &mut dyn Write,
    ) -> Result<(), String> {
        let before = match checkpoint {
            Some(p) => Some(std::fs::read(p).map_err(at(p))?),
            None => None,
        };
        let mut frames = Vec::new();
        for p in logs {
            read_sealed(p, |_, _, f| {
                frames.push(f);
                Ok(())
            })?;
        }
        let folded = self.fold(before, frames, generation)?;
        out.write_all(&folded).map_err(|e| e.to_string())
    }

    /// Put the checkpoint written in place, by `rename`, which renames it
    /// over the one before: what reads the one before stops before, and
    /// reads the new one after. By default, `rename`.
    fn install(
        &self,
        _generation: u64,
        rename: &mut dyn FnMut() -> Result<(), String>,
    ) -> Result<(), String> {
        rename()
    }
}

/// Read a sealed log frame by frame, giving `f` each frame's offset, length
/// and entries but the seal's: an error if it is not whole and sealed.
pub fn read_sealed(
    path: &Path,
    mut f: impl FnMut(u64, u64, Vec<Json>) -> Result<(), String>,
) -> Result<(), String> {
    let mut sealed = false;
    let (good, held) = scan_file(path, |at, len, entries| {
        if sealed {
            return Err(format!("{}: a frame after its seal", path.display()));
        }
        if is_seal(&entries) {
            sealed = true;
            return Ok(());
        }
        f(at, len, entries)
    })
    .map_err(at(path))??;
    if good != held || !sealed {
        return Err(format!("{}: not whole and sealed", path.display()));
    }
    Ok(())
}

/// A log `replay` kept.
#[derive(Clone, Copy)]
struct Kept {
    generation: u64,
    /// The bytes of its whole frames.
    len: u64,
    /// Whether it ends with its seal.
    sealed: bool,
    /// Whether its file is longer than its frames: by room, or by part of a
    /// frame.
    long: bool,
}

/// What `replay` found of the logs it read.
struct Replayed {
    kept: Vec<Kept>,
    /// The bytes of the part of a frame the last of them ends in, if it
    /// does: a torn tail's.
    dropped: u64,
}

/// Replay the logs `gens` of `layout`, those from its checkpoint's
/// generation `c` on, in order and none missing (else `InvalidData`), up to
/// the first cut short. Each frame but a seal is given to `visit`, whose
/// error is `InvalidData`.
fn replay(
    layout: &Layout,
    c: u64,
    gens: &[u64],
    visit: &mut dyn FnMut(Pos, Vec<Json>) -> Result<(), String>,
) -> io::Result<Replayed> {
    if let Some((i, g)) = gens.iter().enumerate().find(|&(i, &g)| g != c + i as u64) {
        let missing = layout.log(c + i as u64);
        return Err(invalid(format!(
            "{}: missing, though {} is there",
            missing.display(),
            layout.log(*g).display()
        )));
    }
    if gens.is_empty() && c > 0 {
        return Err(invalid(format!(
            "{}: missing, though the checkpoint is of generation {c}",
            layout.log(c).display()
        )));
    }
    let mut kept: Vec<Kept> = Vec::new();
    let mut dropped = 0u64;
    for &g in gens {
        let path = layout.log(g);
        let mut sealed = false;
        let scanned = scan_log(&path, |offset, len, entries| {
            if sealed {
                return Err(format!("{}: a frame after its seal", path.display()));
            }
            if is_seal(&entries) {
                sealed = true;
                return Ok(());
            }
            visit(
                Pos {
                    generation: g,
                    offset,
                    len,
                },
                entries,
            )
        })
        .map_err(named(&path))?;
        let Scanned { whole, held, size } = scanned.map_err(invalid)?;
        kept.push(Kept {
            generation: g,
            len: whole,
            sealed,
            long: size > whole,
        });
        dropped += held - whole;
        if held > whole || !sealed {
            break;
        }
    }
    Ok(Replayed { kept, dropped })
}

/// Give `visit` each frame a log holds after its checkpoint, of generation
/// `c`, as a start would replay them (`Wal::open_log_with`), and only read:
/// nothing is deleted, cut back or made durable, and no log is open after.
/// For a reader of a log nothing appends to meanwhile, which makes nothing
/// of what it reads that must outlive a power cut.
pub fn read_log(
    layout: &Layout,
    c: u64,
    visit: &mut dyn FnMut(Pos, Vec<Json>) -> Result<(), String>,
) -> io::Result<()> {
    let mut gens = layout.generations().map_err(named(&layout.dir))?;
    gens.retain(|&g| g >= c);
    replay(layout, c, &gens, visit).map(|_| ())
}

/// Whether a frame's entries are a seal's: `[{"t":"seal"}]`.
pub fn is_seal(entries: &[Json]) -> bool {
    matches!(entries, [Json::Object(m)] if m.len() == 1 && m.get("t").and_then(Json::as_str) == Some("seal"))
}

fn seal() -> Vec<u8> {
    frame(&[serde_json::json!({ "t": "seal" })])
}

fn at(path: &Path) -> impl Fn(io::Error) -> String + '_ {
    move |e| named(path)(e).to_string()
}

/// An error with the file it is of named first, unless it names one
/// already, as that of a log another handle holds does.
fn named(path: &Path) -> impl Fn(io::Error) -> io::Error + '_ {
    move |e| {
        let (name, said) = (path.display().to_string(), e.to_string());
        if said.starts_with(&name) {
            e
        } else {
            io::Error::new(e.kind(), format!("{name}: {said}"))
        }
    }
}

fn invalid(why: String) -> io::Error {
    io::Error::new(io::ErrorKind::InvalidData, why)
}

/// Delete a file if it is there: whether it was.
fn remove(path: &Path) -> io::Result<bool> {
    match std::fs::remove_file(path) {
        Ok(()) => Ok(true),
        Err(e) if e.kind() == io::ErrorKind::NotFound => Ok(false),
        Err(e) => Err(named(path)(e)),
    }
}

/// How long the current generation of a log being opened has run, as its
/// files tell, given the last log `replay` kept: for its age to go on from
/// there (`Policy::max_age`), across a start or a log let go and opened
/// again. A generation begins as its log is made, by the compaction that
/// sealed the one before or by the log's first open. So of a log appended
/// to, it began at the earlier of its file's birth, where the file system
/// keeps one, and its last change: its birth, unless its times were set
/// or copied, and never later than its last frame. Of a log the open makes
/// after one ending with its seal, at that one's last change, its seal; of
/// one it makes with none before, now. Nothing is written for it, and no
/// frame rests on it: a time a power cut or a clock set back made wrong
/// starts a compaction sooner or later than due, never more. A time that
/// cannot be read, or is after now, is now.
fn began(layout: &Layout, last: Option<Kept>) -> Duration {
    let Some(k) = last else {
        return Duration::ZERO;
    };
    let Ok(meta) = std::fs::metadata(layout.log(k.generation)) else {
        return Duration::ZERO;
    };
    let changed = meta.modified().ok();
    let at = if k.sealed {
        changed
    } else {
        match (meta.created().ok(), changed) {
            (Some(a), Some(b)) => Some(a.min(b)),
            (a, b) => a.or(b),
        }
    };
    at.and_then(|t| std::time::SystemTime::now().duration_since(t).ok())
        .unwrap_or(Duration::ZERO)
}

impl Wal {
    /// Start a log with generations, given its checkpoint's generation `c`
    /// (`0` if there is none) and its size in bytes (`0` if none), which the
    /// caller has read:
    /// `<checkpoint>.tmp` and the logs before `c` are deleted; the rest must
    /// be `c`, `c + 1`, ... with none missing, else `InvalidData` naming the
    /// one missing; they are replayed in order, and a log other than the last
    /// that does not end with its seal, or any that ends in part of a frame,
    /// is cut back to its whole frames and every later log deleted. Every
    /// log kept is cut back to its frames, its room given back. Appends go
    /// to the last log, or if it ends with its seal (or there is none), to a
    /// new one of the next generation. Every log kept, and the directory, is
    /// durable before this returns. Answers every frame after the checkpoint,
    /// seals stripped. The current generation's age, for compacting by age,
    /// goes on from what its files tell (`began`). Given `pool`, the log is
    /// one of the pool's.
    pub fn open_log(
        layout: &Layout,
        c: u64,
        checkpoint_bytes: u64,
        pool: Option<Arc<Files>>,
        policy: Policy,
    ) -> io::Result<(Wal, Vec<Vec<Json>>)> {
        let mut frames = Vec::new();
        let wal = Wal::open_log_with(
            layout,
            c,
            checkpoint_bytes,
            pool,
            policy,
            false,
            &mut |_, f| {
                frames.push(f);
                Ok(())
            },
        )?;
        Ok((wal, frames))
    }

    /// The same, but each frame after the checkpoint is given to `visit` as
    /// it is read, with where it is, rather than all of them answered: a
    /// start holds one frame at a time. An error `visit` answers is the
    /// start's, `InvalidData`.
    ///
    /// With `durable`, the caller says that every frame the files hold is
    /// on disk already (the module's "An open that syncs nothing" says when
    /// that is so): if there is then nothing to mend, a log to append to
    /// and nothing deleted, cut back or made, nothing is synced, room is
    /// cut off where a log holds any, and the log's file is opened, and
    /// locked, only when a frame is first appended. With anything to mend,
    /// all is synced as without it.
    pub fn open_log_with(
        layout: &Layout,
        c: u64,
        checkpoint_bytes: u64,
        pool: Option<Arc<Files>>,
        policy: Policy,
        durable: bool,
        visit: &mut dyn FnMut(Pos, Vec<Json>) -> Result<(), String>,
    ) -> io::Result<Wal> {
        let dir = &layout.dir;
        let made = make_dirs(dir).map_err(named(dir))?;

        let mut gens = layout.generations().map_err(named(dir))?;
        // Nothing of the log is changed while another handle writes to it:
        // its last file, which that handle holds, is locked first, and let
        // go to be opened as what the replay finds it to be.
        if let Some(&g) = gens.last() {
            let path = layout.log(g);
            drop(open_locked(&path, OpenOptions::new().write(true)).map_err(named(&path))?);
        }
        // What a compaction cut short while it wrote its checkpoint left.
        let stale = remove(&layout.tmp())?;
        // Whether the directory's entries changed, to be made durable.
        let mut moved = false;
        let covered = gens.iter().take_while(|&&g| g < c).count();
        if covered > 0 {
            // The checkpoint that covers them is in place durably before they go.
            sync_dir(dir).map_err(named(dir))?;
            for &g in &gens[..covered] {
                remove(&layout.log(g))?;
            }
            moved = true;
        }
        gens.drain(..covered);
        // A log an append of this run left a frame in that it could not
        // cut back (`Wal::let_go`): cut now, and synced as anything mended.
        let mut cut = Vec::new();
        for &g in &gens {
            let path = layout.log(g);
            if cut_uncut(&path).map_err(named(&path))? {
                cut.push(path);
            }
        }
        let Replayed { kept, mut dropped } = replay(layout, c, &gens, visit)?;
        // Part of a frame after a log's whole ones: a torn tail, to cut off
        // durably before a frame goes in its place. Zeros alone are room.
        let torn = dropped > 0;
        for &g in &gens[kept.len()..] {
            let path = layout.log(g);
            dropped += log_len(&path).unwrap_or(0);
            remove(&path)?;
            moved = true;
        }
        // The log appended to: the last kept, unless it ends with its seal.
        let last = kept.last().copied();
        let appending = last.filter(|k| !k.sealed);
        // How long its generation has run, read before anything mended
        // changes a file's times.
        let before = began(layout, last);
        // Files said to be durable, found as a log at rest: no byte is
        // written to them here, and so nothing is synced.
        let at_rest = durable
            && made.is_empty()
            && !stale
            && !moved
            && !torn
            && cut.is_empty()
            && appending.is_some();
        // Every other log kept is durable, cut back, before a frame after it
        // is. Of a log at rest only room is cut off, as it is when given
        // back: no frame's being on disk rests on that cut, and nothing
        // syncs it.
        let others = &kept[..kept.len() - appending.map_or(0, |_| 1)];
        for k in if at_rest { &kept[..] } else { others } {
            if at_rest && !k.long {
                continue;
            }
            let path = layout.log(k.generation);
            (|| {
                let f = open_locked(&path, OpenOptions::new().write(true))?;
                if k.long {
                    f.set_len(k.len)?;
                }
                if at_rest { Ok(()) } else { sync_file(&f) }
            })()
            .map_err(named(&path))?;
        }
        let mut inner = match appending {
            Some(k) => {
                let mut inner = Inner::new(layout.log(k.generation), k.len);
                inner.generation = k.generation;
                inner.sealed = kept[..kept.len() - 1].iter().map(|k| k.len).sum();
                inner
            }
            None => {
                let g = last.map_or(c, |k| k.generation + 1);
                let mut inner = Inner::new(layout.log(g), 0);
                inner.generation = g;
                inner.sealed = kept.iter().map(|k| k.len).sum();
                inner
            }
        };
        inner.checkpoint = c;
        inner.checkpoint_bytes = checkpoint_bytes;
        inner.before = before;
        if at_rest {
            // No file is held: opened and locked as a frame is first
            // appended, if it is then as it was found.
            inner.found = Some(inner.len);
        }
        let shared = Shared::new(
            dir.clone(),
            Some(layout.clone()),
            pool,
            policy,
            inner,
            dropped,
        );
        if !at_rest {
            let mut inner = lock(&shared.inner);
            let path = inner.path.clone();
            let file = match appending {
                Some(_) => shared.file(&mut inner),
                None => {
                    moved = true;
                    shared.create(&mut inner)
                }
            }
            .map_err(named(&path))?;
            if appending.is_some_and(|k| k.long) {
                file.set_len(inner.len).map_err(named(&path))?;
            }
            sync_file(&file).map_err(named(&path))?;
        }
        if moved {
            sync_dir(dir).map_err(named(dir))?;
        }
        for x in made.iter() {
            if let Some(parent) = x.parent().filter(|p| !p.as_os_str().is_empty()) {
                sync_dir(parent).map_err(named(parent))?;
            }
        }
        for path in &cut {
            cut_noted(path);
        }
        Ok(Wal(shared))
    }

    /// Turn compaction on, with this fold to make the checkpoints. It is off
    /// until set, and always for a log of `Wal::open`.
    pub fn set_fold(&self, fold: Arc<dyn Fold>) {
        lock(&self.0.inner).fold = Some(fold);
    }

    /// Compact now, whatever the triggers: step 1 under the lock, then steps 2
    /// to 5 on this thread. Fails if one is running, there is no fold, or a
    /// step fails.
    pub fn compact(&self) -> Result<(), String> {
        self.compact_until(5)
    }

    /// Start compacting now, whatever the log's size and age, as an append
    /// that makes a compaction due does: step 1 under the lock, steps 2 to
    /// 5 on a thread of its own. For a caller that holds more of the log in
    /// memory than it would. Whether one started: none does while one
    /// runs, for a while after one failed, with no fold set, or with
    /// nothing in the logs after the checkpoint.
    pub fn compact_soon(&self) -> bool {
        let s = &self.0;
        let mut inner = lock(&s.inner);
        if !s.ready(&inner) || inner.sealed + inner.len == 0 {
            return false;
        }
        let failed = s.begin(&mut inner);
        let started = inner.compacting;
        drop(inner);
        if let Some(why) = failed {
            crate::report(&why);
        }
        started
    }

    /// Compact, stopping after step `last`, as if killed then.
    pub(crate) fn compact_until(&self, last: u8) -> Result<(), String> {
        let s = &self.0;
        let (g, fold) = {
            let mut inner = lock(&s.inner);
            let fold = inner.fold.clone().ok_or("no fold is set")?;
            (s.start(&mut inner)?, fold)
        };
        let done = s.settle(g, &*fold, last);
        s.end(g, done, false)
    }

    /// The current log's generation.
    pub fn generation(&self) -> u64 {
        lock(&self.0.inner).generation
    }

    /// The checkpoint's generation, `0` if there is none.
    pub fn checkpoint_generation(&self) -> u64 {
        lock(&self.0.inner).checkpoint
    }

    /// `L`: the bytes of the logs from the checkpoint's generation on.
    pub fn log_bytes(&self) -> u64 {
        let inner = lock(&self.0.inner);
        inner.sealed + inner.len
    }

    /// `C`: the bytes of the checkpoint, `0` if there is none.
    pub fn checkpoint_bytes(&self) -> u64 {
        lock(&self.0.inner).checkpoint_bytes
    }

    /// Whether a compaction of the log runs.
    pub fn compacting(&self) -> bool {
        lock(&self.0.inner).compacting
    }

    /// Return once no compaction of the log runs.
    pub fn wait_compacted(&self) {
        let mut inner = lock(&self.0.inner);
        while inner.compacting {
            inner = self
                .0
                .compacted
                .wait(inner)
                .unwrap_or_else(PoisonError::into_inner);
        }
    }
}

impl Shared {
    /// How much more the log may hold, once a frame of `len` bytes is in,
    /// before it is compacted by size; no bound for a log that is not
    /// compacted. Room is made within it, so that a log's files reach no
    /// further with room than its frames may without, and no zeros are
    /// written that its seal would give back.
    pub(crate) fn left(&self, inner: &Inner, len: u64) -> u64 {
        if self.layout.is_none() || inner.fold.is_none() {
            return u64::MAX;
        }
        self.policy
            .limit(inner.checkpoint_bytes)
            .saturating_sub(inner.sealed + inner.len + len)
    }

    /// Whether a compaction may start now: a fold is set, none runs,
    /// none failed a moment ago, and the log is not closed.
    fn ready(&self, inner: &Inner) -> bool {
        inner.fold.is_some()
            && self.layout.is_some()
            && !inner.compacting
            && !inner.quiet_until.is_some_and(|t| Instant::now() < t)
            && !self.closed(inner)
    }

    /// After an append, or a compaction that ended (`again`), under the
    /// lock: start a compaction if one is due, steps 2 to 5 on a thread of
    /// its own. Answers what to say if one could not start.
    pub(crate) fn trigger(self: &Arc<Self>, inner: &mut Inner) -> Option<String> {
        if !self.ready(inner) {
            return None;
        }
        let by_size = inner.sealed + inner.len > self.policy.limit(inner.checkpoint_bytes);
        let by_age = self
            .policy
            .max_age
            .is_some_and(|age| inner.before + inner.began.elapsed() > age);
        if !by_size && !by_age {
            return None;
        }
        self.begin(inner)
    }

    /// Start a compaction, under the lock: step 1 here, steps 2 to 5 on a
    /// thread of its own. Answers what to say if it could not start.
    fn begin(self: &Arc<Self>, inner: &mut Inner) -> Option<String> {
        let fold = inner.fold.clone()?;
        let g = inner.generation;
        if let Err(why) = self.start(inner) {
            return Some(why);
        }
        let s = self.clone();
        let spawned = std::thread::Builder::new()
            .name("pds-compact".into())
            .spawn(move || {
                let done = s.settle(g, &*fold, 5);
                if s.end(g, done, true).is_ok() {
                    s.again();
                }
            });
        match spawned {
            Ok(_) => None,
            Err(e) => {
                inner.compacting = false;
                inner.quiet_until = Some(Instant::now() + QUIET);
                Some(format!("{}: {e}", inner.path.display()))
            }
        }
    }

    /// After a compaction begun by a trigger has ended well, start the next
    /// if one is due now, as the appends made while it ran would have had
    /// it not run (`trigger`): so that, once none runs, a log holds no more
    /// than its policy lets it, however much came while one did.
    fn again(self: &Arc<Self>) {
        let mut inner = lock(&self.inner);
        let failed = self.trigger(&mut inner);
        drop(inner);
        if let Some(why) = failed {
            crate::report(&why);
        }
    }

    /// Step 1, under the lock, if no compaction runs: seal the current log,
    /// `g`, and make log `g + 1` for appends to go to. Answers `g`.
    fn start(self: &Arc<Self>, inner: &mut Inner) -> Result<u64, String> {
        let layout = self
            .layout
            .as_ref()
            .ok_or("a log of one file has no generations")?;
        if inner.compacting {
            return Err("a compaction is running".into());
        }
        // Nothing would sync a seal written once the log is closed.
        if self.closed(inner) {
            return Err(self.refused(inner).to_string());
        }
        let g = inner.generation;
        match self.rotate(inner, layout) {
            Ok(()) => {
                inner.compacting = true;
                Ok(g)
            }
            Err(why) => {
                inner.quiet_until = Some(Instant::now() + QUIET);
                Err(why)
            }
        }
    }

    fn rotate(self: &Arc<Self>, inner: &mut Inner, layout: &Layout) -> Result<(), String> {
        let path = inner.path.clone();
        self.mend(inner).map_err(at(&path))?;
        let file = self.file(inner).map_err(at(&path))?;
        // Nothing is written after a seal: the room goes before it is
        // written. Room that cannot be given back stays, as zeros after the
        // seal, until a start cuts it off.
        inner.give_back();
        let seal = seal();
        let sealed = inner.len + seal.len() as u64;
        if let Err(e) = file.write_all_at(&seal, inner.len) {
            self.cut_back(inner);
            return Err(at(&path)(e));
        }
        let next = layout.log(inner.generation + 1);
        let new = match create_log(&next) {
            Ok(f) => Arc::new(f),
            Err(e) => {
                // The seal off again: appends go on in this log.
                self.cut_back(inner);
                return Err(at(&next)(e));
            }
        };
        // With a journal, the seal is recorded there once the next log is
        // made, so that a start never puts back a seal with no log after
        // it; if it cannot be, the seal is cut off again and the next log
        // goes.
        if let Some(j) = self.journal.get() {
            match j.append(&path, inner.len, &seal) {
                Ok(n) => inner.journaled = n,
                Err(e) => {
                    drop(new);
                    let _ = std::fs::remove_file(&next);
                    self.cut_back(inner);
                    return Err(at(&path)(e));
                }
            }
        }
        // Kept open until it is durable, outside the pool's count: in its
        // place, the new log.
        let old = inner.file.replace(new).unwrap_or(file);
        inner.unsynced.push((inner.generation, old));
        inner.sealed += sealed;
        inner.len = 0;
        inner.size = 0;
        inner.since = 0;
        inner.path = next;
        inner.generation += 1;
        inner.began = Instant::now();
        inner.before = Duration::ZERO;
        self.written.fetch_add(1, Ordering::SeqCst);
        Ok(())
    }

    /// Steps 2 to 5 of compacting from `g`, outside the lock, stopping after
    /// step `last`. What changes the directory's entries, the `.tmp` made,
    /// the rename and each log deleted, is done under the log's lock, with
    /// the files found there not to be let go (`Wal::let_go`): once they
    /// are, they are another's, and none of these is done.
    fn settle(&self, g: u64, fold: &dyn Fold, last: u8) -> Result<(), String> {
        let layout = self
            .layout
            .as_ref()
            .ok_or("a log of one file has no generations")?;
        let dir = &layout.dir;
        if last < 2 {
            return Ok(());
        }
        // 2. Log `g`, sealed, and the directory with log `g + 1`, durable:
        // after every file a frame in it rests on. The sealed logs' files
        // are held, and locked, through their sync and no longer.
        {
            let unsynced = lock(&self.inner).unsynced.clone();
            self.sync_files_first().map_err(at(dir))?;
            self.sync_sealed(&unsynced).map_err(at(dir))?;
            if !unsynced.is_empty() {
                lock(&self.inner)
                    .unsynced
                    .retain(|(h, _)| !unsynced.iter().any(|(k, _)| k == h));
            }
        }
        if last < 3 {
            return Ok(());
        }
        // 3. The checkpoint before and the sealed logs after it, folded into
        // the `.tmp`, durable: not of a log whose files are let go, which
        // are another's to open, and to compact, from then on.
        let closed = || format!("{}: the log is closed", dir.display());
        let tmp = layout.tmp();
        let (c, made) = {
            let inner = lock(&self.inner);
            if inner.shut {
                return Err(closed());
            }
            let made = open_private(
                &tmp,
                OpenOptions::new().create(true).write(true).truncate(true),
            )
            .map_err(at(&tmp))?;
            (inner.checkpoint, made)
        };
        let checkpoint = layout.checkpoint();
        let before = match std::fs::metadata(&checkpoint) {
            Ok(_) => Some(checkpoint.as_path()),
            Err(e) if e.kind() == io::ErrorKind::NotFound && c == 0 => None,
            Err(e) => return Err(at(&checkpoint)(e)),
        };
        let logs: Vec<PathBuf> = (c..=g).map(|h| layout.log(h)).collect();
        let mut out = io::BufWriter::new(made);
        std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
            fold.fold_files(before, &logs, g + 1, &mut out)
        }))
        .unwrap_or_else(|_| Err("the fold panicked".into()))?;
        let file = out.into_inner().map_err(|e| at(&tmp)(e.into_error()))?;
        sync_file(&file).map_err(at(&tmp))?;
        let folded = file.metadata().map_err(at(&tmp))?.len();
        drop(file);
        if last < 4 {
            return Ok(());
        }
        // 4. In the checkpoint's place, durably.
        fold.install(g + 1, &mut || {
            let mut inner = lock(&self.inner);
            if inner.shut {
                return Err(closed());
            }
            std::fs::rename(&tmp, layout.checkpoint()).map_err(at(&tmp))?;
            inner.checkpoint = g + 1;
            inner.checkpoint_bytes = folded;
            inner.sealed = 0;
            Ok(())
        })?;
        sync_dir(dir).map_err(at(dir))?;
        if last < 5 {
            return Ok(());
        }
        // 5. The logs it covers deleted.
        for h in layout.generations().map_err(at(dir))? {
            if h <= g {
                let inner = lock(&self.inner);
                if inner.shut {
                    return Err(closed());
                }
                remove(&layout.log(h)).map_err(|e| e.to_string())?;
            }
        }
        Ok(())
    }

    /// The end of a compaction from `g`, done or not: if not, the `.tmp`
    /// deleted, said (`report`) if `report`, and none due for a while. Of a
    /// log whose files are let go nothing is deleted or said: a `.tmp` is
    /// then as a kill leaves it, for the next open to delete. Which of the
    /// two is found, and the `.tmp` deleted, under the log's lock.
    fn end(&self, g: u64, done: Result<(), String>, report: bool) -> Result<(), String> {
        let held = {
            let inner = lock(&self.inner);
            if let (Err(_), Some(layout), false) = (&done, &self.layout, inner.shut) {
                let _ = std::fs::remove_file(layout.tmp());
            }
            !inner.shut
        };
        if let (Err(why), Some(layout), true, true) = (&done, &self.layout, held, report) {
            crate::report(&format!("{}: {why}", layout.log(g).display()));
        }
        let mut inner = lock(&self.inner);
        inner.compacting = false;
        if done.is_err() {
            inner.quiet_until = Some(Instant::now() + QUIET);
        }
        drop(inner);
        self.compacted.notify_all();
        done
    }
}

#[cfg(test)]
mod tests {
    use super::*;
    use serde_json::json;
    use std::collections::BTreeMap;
    use std::sync::mpsc;

    fn scratch(name: &str) -> PathBuf {
        let d = std::env::temp_dir().join(format!("pds-store-gen-{}-{name}", std::process::id()));
        let _ = std::fs::remove_dir_all(&d);
        d
    }

    /// Compacting only when asked.
    const NEVER: Policy = Policy {
        floor: u64::MAX,
        cap: None,
        max_age: None,
    };

    /// A checkpoint as a JSON array: its generation, then every entry folded
    /// into it.
    struct Entries;

    impl Fold for Entries {
        fn fold(
            &self,
            checkpoint: Option<Vec<u8>>,
            frames: Vec<Vec<Json>>,
            generation: u64,
        ) -> Result<Vec<u8>, String> {
            let mut out = vec![json!(generation)];
            if let Some(bytes) = checkpoint {
                let before: Vec<Json> =
                    serde_json::from_slice(&bytes).map_err(|e| e.to_string())?;
                out.extend(before.into_iter().skip(1));
            }
            out.extend(frames.into_iter().flatten());
            Ok(serde_json::to_vec(&out).unwrap())
        }
    }

    struct Refuse;

    impl Fold for Refuse {
        fn fold(&self, _: Option<Vec<u8>>, _: Vec<Vec<Json>>, _: u64) -> Result<Vec<u8>, String> {
            Err("refused".into())
        }
    }

    /// Folds as `Entries`, then puts a directory where the checkpoint goes,
    /// so that the `.tmp` is written and cannot be renamed.
    struct Blocked(PathBuf);

    impl Fold for Blocked {
        fn fold(
            &self,
            checkpoint: Option<Vec<u8>>,
            frames: Vec<Vec<Json>>,
            generation: u64,
        ) -> Result<Vec<u8>, String> {
            std::fs::create_dir_all(self.0.join("x")).unwrap();
            Entries.fold(checkpoint, frames, generation)
        }
    }

    /// Folds as `Entries` once let: says it is reached, then waits for word.
    struct Held {
        reached: Mutex<mpsc::Sender<()>>,
        go: Mutex<mpsc::Receiver<()>>,
    }

    impl Fold for Held {
        fn fold(
            &self,
            checkpoint: Option<Vec<u8>>,
            frames: Vec<Vec<Json>>,
            generation: u64,
        ) -> Result<Vec<u8>, String> {
            let _ = lock(&self.reached).send(());
            let _ = lock(&self.go).recv();
            Entries.fold(checkpoint, frames, generation)
        }
    }

    /// A `Held` fold, where it says it is reached, and the word to go on.
    fn held() -> (Arc<Held>, mpsc::Receiver<()>, mpsc::Sender<()>) {
        let (reached, at) = mpsc::channel();
        let (go, word) = mpsc::channel();
        (
            Arc::new(Held {
                reached: Mutex::new(reached),
                go: Mutex::new(word),
            }),
            at,
            go,
        )
    }

    /// The checkpoint as a start reads it: its generation, its bytes and
    /// its entries; none of them with no checkpoint.
    fn checkpointed(layout: &Layout) -> (u64, u64, Vec<Json>) {
        match std::fs::read(layout.checkpoint()) {
            Ok(b) => {
                let v: Vec<Json> = serde_json::from_slice(&b).unwrap();
                (v[0].as_u64().unwrap(), b.len() as u64, v[1..].to_vec())
            }
            Err(_) => (0, 0, Vec::new()),
        }
    }

    /// Start as the server does, reading the checkpoint, then the logs:
    /// answers the state, the checkpoint's entries and then every frame's.
    fn start(layout: &Layout, policy: Policy) -> io::Result<(Wal, Vec<Json>)> {
        let (c, bytes, mut state) = checkpointed(layout);
        let (wal, frames) = Wal::open_log(layout, c, bytes, None, policy)?;
        state.extend(frames.into_iter().flatten());
        Ok((wal, state))
    }

    /// Start so, the files said to be durable or not: the log, the state,
    /// and how many syncs the start made.
    fn start_as(layout: &Layout, pool: Option<Arc<Files>>, durable: bool) -> (Wal, Vec<Json>, u64) {
        let (c, bytes, mut state) = checkpointed(layout);
        let before = crate::syncs();
        let wal = Wal::open_log_with(layout, c, bytes, pool, NEVER, durable, &mut |_, f| {
            state.extend(f);
            Ok(())
        })
        .unwrap();
        (wal, state, crate::syncs() - before)
    }

    /// Append each value as a frame of its own; answers the last's number.
    fn append(wal: &Wal, values: &[Json]) -> u64 {
        values
            .iter()
            .fold(0, |_, v| wal.append(std::slice::from_ref(v)).unwrap())
    }

    fn nums(r: std::ops::Range<i64>) -> Vec<Json> {
        r.map(|i| json!(i)).collect()
    }

    fn frames_of(values: Vec<Json>) -> Vec<Vec<Json>> {
        values.into_iter().map(|v| vec![v]).collect()
    }

    fn names(dir: &Path) -> Vec<String> {
        let mut names: Vec<String> = std::fs::read_dir(dir)
            .unwrap()
            .map(|e| e.unwrap().file_name().into_string().unwrap())
            .collect();
        names.sort();
        names
    }

    /// The bytes a file holds: a log's less its room, which is no part of
    /// the log.
    fn size(path: &Path) -> u64 {
        if path.extension().is_some_and(|e| e == "log") {
            log_len(path).unwrap()
        } else {
            file_size(path)
        }
    }

    /// A file's size, room and all.
    fn file_size(path: &Path) -> u64 {
        std::fs::metadata(path).unwrap().len()
    }

    fn checkpoint(layout: &Layout) -> Vec<Json> {
        serde_json::from_slice(&std::fs::read(layout.checkpoint()).unwrap()).unwrap()
    }

    /// Write log `g` of these frames, sealed or not.
    fn put_log(layout: &Layout, g: u64, values: &[Json], sealed: bool) {
        let mut bytes: Vec<u8> = values
            .iter()
            .flat_map(|v| frame(std::slice::from_ref(v)))
            .collect();
        if sealed {
            bytes.extend(seal());
        }
        std::fs::write(layout.log(g), bytes).unwrap();
    }

    /// Compacting folds the logs into a checkpoint of the next generation and
    /// deletes them; started again, the log answers only what came after it.
    #[test]
    fn compacting_round_trips() {
        let dir = scratch("round");
        let layout = Layout::new(&dir, "repo");
        let (wal, state) = start(&layout, NEVER).unwrap();
        assert!(state.is_empty());
        assert_eq!(names(&dir), ["repo.0.log"]);
        assert_eq!(wal.compact(), Err("no fold is set".into()));
        wal.set_fold(Arc::new(Entries));
        let n = append(&wal, &nums(0..5));
        wal.wait(n).unwrap();
        assert_eq!(
            (wal.log_bytes(), wal.checkpoint_bytes()),
            (size(&layout.log(0)), 0)
        );
        wal.compact().unwrap();
        assert_eq!((wal.checkpoint_generation(), wal.generation()), (1, 1));
        assert_eq!(checkpoint(&layout), [vec![json!(1)], nums(0..5)].concat());
        assert_eq!(names(&dir), ["repo.1.log", "repo.checkpoint"]);
        assert_eq!(
            (wal.log_bytes(), wal.checkpoint_bytes()),
            (0, size(&layout.checkpoint()))
        );
        // Frames are numbered on, the seal too.
        let m = append(&wal, &nums(5..8));
        assert_eq!(m, n + 4);
        wal.wait(m).unwrap();
        assert_eq!(wal.log_bytes(), size(&layout.log(1)));
        assert_eq!(wal.log_bytes(), wal.len());
        wal.compact().unwrap();
        assert_eq!((wal.checkpoint_generation(), wal.generation()), (2, 2));
        assert_eq!(checkpoint(&layout), [vec![json!(2)], nums(0..8)].concat());
        assert_eq!(names(&dir), ["repo.2.log", "repo.checkpoint"]);
        wal.wait(append(&wal, &nums(8..10))).unwrap();
        drop(wal);
        let (wal, state) = start(&layout, NEVER).unwrap();
        assert_eq!(state, nums(0..10));
        assert_eq!((wal.checkpoint_generation(), wal.generation()), (2, 2));
        assert_eq!(
            (wal.log_bytes(), wal.checkpoint_bytes()),
            (size(&layout.log(2)), size(&layout.checkpoint()))
        );
        drop(wal);
        let (wal, frames) =
            Wal::open_log(&layout, 2, size(&layout.checkpoint()), None, NEVER).unwrap();
        assert_eq!(frames, frames_of(nums(8..10)));
        // Every file it makes is its owner's alone.
        wal.set_fold(Arc::new(Entries));
        wal.compact_until(3).unwrap();
        for path in [
            layout.log(2),
            layout.log(3),
            layout.tmp(),
            layout.checkpoint(),
        ] {
            assert_eq!(
                std::fs::metadata(&path).unwrap().permissions().mode() & 0o777,
                FILE_MODE,
                "{}",
                path.display()
            );
        }
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// By size: once `L` is more than `C` and the floor, and never before;
    /// after it, the checkpoint and one log.
    #[test]
    fn compacts_by_size() {
        let dir = scratch("size");
        let layout = Layout::new(&dir, "repo");
        let floor = 200;
        let (wal, _) = start(
            &layout,
            Policy {
                floor,
                cap: None,
                max_age: None,
            },
        )
        .unwrap();
        wal.set_fold(Arc::new(Entries));
        let mut compactions = 0;
        for i in 0..300 {
            let g = wal.generation();
            let due = wal.log_bytes() + frame(&[json!(i)]).len() as u64
                > wal.checkpoint_bytes().max(floor);
            wal.append(&[json!(i)]).unwrap();
            wal.wait_compacted();
            assert_eq!(wal.generation() != g, due, "at {i}");
            compactions += due as u32;
            assert!(wal.log_bytes() <= wal.checkpoint_bytes().max(floor));
            assert_eq!(
                names(&dir).len(),
                if wal.checkpoint_generation() > 0 {
                    2
                } else {
                    1
                }
            );
        }
        assert!(compactions >= 4, "{compactions}");
        drop(wal);
        assert_eq!(start(&layout, NEVER).unwrap().1, nums(0..300));
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// Asked for: a compaction starts whatever the log's size, on a thread
    /// of its own, unless one runs, no fold is set, or the logs hold
    /// nothing; what it folds is kept, and writes go on meanwhile.
    #[test]
    fn compacts_when_asked() {
        let dir = scratch("asked");
        let layout = Layout::new(&dir, "repo");
        let (wal, _) = start(&layout, NEVER).unwrap();
        wal.append(&[json!(0)]).unwrap();
        assert!(!wal.compact_soon(), "with no fold");
        let (fold, reached, go) = held();
        wal.set_fold(fold);
        assert!(wal.compact_soon());
        reached.recv().unwrap();
        assert!(wal.compacting() && wal.generation() == 1);
        let n = wal.append(&[json!(1)]).unwrap();
        wal.wait(n).unwrap();
        assert!(!wal.compact_soon(), "while one runs");
        go.send(()).unwrap();
        wal.wait_compacted();
        assert_eq!(checkpoint(&layout), [json!(1), json!(0)]);
        assert_eq!(names(&dir), ["repo.1.log", "repo.checkpoint"]);
        // The log after the checkpoint is folded in turn; an empty one is not.
        assert!(wal.compact_soon());
        reached.recv().unwrap();
        go.send(()).unwrap();
        wal.wait_compacted();
        assert_eq!(checkpoint(&layout), [json!(2), json!(0), json!(1)]);
        assert!(!wal.compact_soon(), "with nothing to fold");
        assert_eq!(wal.generation(), 2);
        drop(wal);
        assert_eq!(start(&layout, NEVER).unwrap().1, nums(0..2));
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// With a cap: by size once `L` is more than `max(min(C, cap), floor)`,
    /// so that a checkpoint grown past the cap leaves the log no more than
    /// the cap, where without one the log grows to the checkpoint's size.
    #[test]
    fn compacts_by_size_up_to_a_cap() {
        let p = |cap| Policy {
            floor: 200,
            cap,
            max_age: None,
        };
        assert_eq!(p(None).limit(0), 200);
        assert_eq!(p(None).limit(5000), 5000);
        assert_eq!(p(Some(400)).limit(300), 300);
        assert_eq!(p(Some(400)).limit(5000), 400);
        assert_eq!(p(Some(100)).limit(5000), 200);
        let dir = scratch("cap");
        let layout = Layout::new(&dir, "repo");
        let (floor, cap) = (200, 400);
        let (wal, _) = start(&layout, p(Some(cap))).unwrap();
        wal.set_fold(Arc::new(Entries));
        let mut past = 0;
        for i in 0..400 {
            let g = wal.generation();
            let limit = wal.checkpoint_bytes().min(cap).max(floor);
            let due = wal.log_bytes() + frame(&[json!(i)]).len() as u64 > limit;
            wal.append(&[json!(i)]).unwrap();
            wal.wait_compacted();
            assert_eq!(wal.generation() != g, due, "at {i}");
            assert!(wal.log_bytes() <= cap);
            // Compacted with the log smaller than the checkpoint: by the cap.
            past += (due && limit == cap && wal.checkpoint_bytes() > cap) as u32;
        }
        assert!(wal.checkpoint_bytes() > 2 * cap);
        assert!(past >= 2, "{past}");
        drop(wal);
        assert_eq!(start(&layout, NEVER).unwrap().1, nums(0..400));
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// What is appended while a compaction runs, past what makes one due,
    /// is compacted as that one ends, with no append after it: once none
    /// runs, the log holds no more than its policy lets it.
    #[test]
    fn compacts_again_as_one_ends_if_due() {
        let dir = scratch("again");
        let layout = Layout::new(&dir, "repo");
        let limit = 200;
        let policy = Policy {
            floor: limit,
            cap: Some(limit),
            max_age: None,
        };
        let (wal, _) = start(&layout, policy).unwrap();
        let (fold, at, go) = held();
        wal.set_fold(fold);
        // Past the limit: a compaction begins, and waits in its fold.
        let mut n = 0;
        while !wal.compacting() {
            wal.append(&[json!(n)]).unwrap();
            n += 1;
        }
        at.recv().unwrap();
        let g = wal.generation();
        // Past it again while that one runs.
        let sealed = wal.log_bytes();
        while wal.log_bytes() - sealed <= limit {
            wal.append(&[json!(n)]).unwrap();
            n += 1;
        }
        assert_eq!(wal.generation(), g, "none begins while one runs");
        go.send(()).unwrap();
        // The next begins as it ends.
        at.recv_timeout(Duration::from_secs(10))
            .expect("a compaction begun as the one before ended");
        assert_eq!(wal.generation(), g + 1);
        go.send(()).unwrap();
        wal.wait_compacted();
        assert!(wal.log_bytes() <= limit, "{}", wal.log_bytes());
        assert_eq!(wal.checkpoint_generation(), g + 1);
        drop(wal);
        assert_eq!(start(&layout, NEVER).unwrap().1, nums(0..n));
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// By age: once its generation began more than `max_age` ago, when a
    /// compaction made its log or the log was first made, whether or not
    /// it was started again meanwhile.
    #[test]
    fn compacts_by_age() {
        let dir = scratch("age");
        let layout = Layout::new(&dir, "repo");
        let age = Duration::from_millis(300);
        let older = || std::thread::sleep(age + Duration::from_millis(100));
        let policy = Policy {
            floor: u64::MAX,
            cap: None,
            max_age: Some(age),
        };
        let (wal, _) = start(&layout, policy).unwrap();
        wal.set_fold(Arc::new(Entries));
        wal.append(&[json!(0)]).unwrap();
        wal.wait_compacted();
        assert_eq!(wal.generation(), 0);
        older();
        wal.append(&[json!(1)]).unwrap();
        wal.wait_compacted();
        assert_eq!((wal.checkpoint_generation(), wal.generation()), (1, 1));
        wal.append(&[json!(2)]).unwrap();
        wal.wait_compacted();
        assert_eq!(wal.generation(), 1);
        drop(wal);
        // Started again at once, the generation young still: no compaction.
        let (wal, state) = start(&layout, policy).unwrap();
        assert_eq!(state, nums(0..3));
        wal.set_fold(Arc::new(Entries));
        wal.append(&[json!(3)]).unwrap();
        wal.wait_compacted();
        assert_eq!(wal.generation(), 1);
        drop(wal);
        // Started again once it is older, though nothing held it meanwhile:
        // its first append compacts.
        older();
        let (wal, state) = start(&layout, policy).unwrap();
        assert_eq!(state, nums(0..4));
        wal.set_fold(Arc::new(Entries));
        wal.append(&[json!(4)]).unwrap();
        wal.wait_compacted();
        assert_eq!((wal.checkpoint_generation(), wal.generation()), (2, 2));
        wal.append(&[json!(5)]).unwrap();
        wal.wait_compacted();
        assert_eq!(wal.generation(), 2);
        drop(wal);
        assert_eq!(start(&layout, NEVER).unwrap().1, nums(0..6));
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// A log opened older than `max_age`, as its files tell, is compacted
    /// on its first append, and one younger is not: of a log appended to,
    /// by its file's birth or last change, whichever is earlier; of one the
    /// open makes after a sealed one, by that one's last change, its seal.
    #[test]
    fn compacts_by_age_its_files_tell() {
        let policy = Policy {
            floor: u64::MAX,
            cap: None,
            max_age: Some(Duration::from_secs(60)),
        };
        let hour_ago = std::time::SystemTime::now() - Duration::from_secs(3600);
        for (sealed, old) in [(false, false), (false, true), (true, false), (true, true)] {
            let dir = scratch(&format!("age-files-{sealed}-{old}"));
            std::fs::create_dir_all(&dir).unwrap();
            let layout = Layout::new(&dir, "repo");
            put_log(&layout, 0, &nums(0..2), sealed);
            if old {
                let f = OpenOptions::new().write(true).open(layout.log(0)).unwrap();
                f.set_times(std::fs::FileTimes::new().set_modified(hour_ago))
                    .unwrap();
            }
            let (wal, state) = start(&layout, policy).unwrap();
            assert_eq!(state, nums(0..2));
            let g = u64::from(sealed);
            assert_eq!(wal.generation(), g);
            wal.set_fold(Arc::new(Entries));
            wal.append(&[json!(2)]).unwrap();
            wal.wait_compacted();
            let due = u64::from(old);
            assert_eq!(
                (wal.checkpoint_generation(), wal.generation()),
                (if old { g + 1 } else { 0 }, g + due),
                "sealed {sealed}, old {old}"
            );
            drop(wal);
            assert_eq!(start(&layout, NEVER).unwrap().1, nums(0..3));
            let _ = std::fs::remove_dir_all(&dir);
        }
    }

    /// Writes go on while a compaction folds, to the next log, each durable;
    /// and no other compaction starts meanwhile, though one is due: the
    /// next begins as it ends, and folds them.
    #[test]
    fn writes_go_on_while_it_compacts() {
        let dir = scratch("during");
        let layout = Layout::new(&dir, "repo");
        let (wal, _) = start(
            &layout,
            Policy {
                floor: 64,
                cap: None,
                max_age: None,
            },
        )
        .unwrap();
        let (fold, reached, go) = held();
        wal.set_fold(fold);
        let mut i = 0;
        while !wal.compacting() {
            wal.append(&[json!(i)]).unwrap();
            i += 1;
        }
        reached.recv().unwrap();
        for j in i..i + 50 {
            let n = wal.append(&[json!(j)]).unwrap();
            wal.wait(n).unwrap();
            assert!(wal.compacting());
            assert_eq!(wal.generation(), 1);
        }
        assert_eq!(
            scan(&std::fs::read(layout.log(1)).unwrap()).unwrap().0,
            frames_of(nums(i..i + 50))
        );
        go.send(()).unwrap();
        reached.recv().unwrap();
        assert_eq!(wal.generation(), 2);
        go.send(()).unwrap();
        wal.wait_compacted();
        assert_eq!(
            checkpoint(&layout),
            [vec![json!(2)], nums(0..i + 50)].concat()
        );
        assert_eq!(names(&dir), ["repo.2.log", "repo.checkpoint"]);
        assert_eq!(wal.log_bytes(), size(&layout.log(2)));
        drop(wal);
        assert_eq!(start(&layout, NEVER).unwrap().1, nums(0..i + 50));
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// Writers at once, each waiting for its frames, while compactions come
    /// and go: every frame is durable once waited for, and kept, in each
    /// writer's order.
    #[test]
    fn writers_at_once_through_compactions() {
        let dir = scratch("writers");
        let layout = Layout::new(&dir, "repo");
        let (wal, _) = start(
            &layout,
            Policy {
                floor: 256,
                cap: None,
                max_age: None,
            },
        )
        .unwrap();
        wal.set_fold(Arc::new(Entries));
        std::thread::scope(|s| {
            for t in 0..4 {
                let wal = &wal;
                s.spawn(move || {
                    for i in 0..40 {
                        let n = wal.append(&[json!([t, i])]).unwrap();
                        wal.wait(n).unwrap();
                        assert!(wal.is_durable(n));
                    }
                });
            }
        });
        wal.wait_compacted();
        assert!(wal.checkpoint_generation() > 1);
        drop(wal);
        let state = start(&layout, NEVER).unwrap().1;
        for t in 0..4 {
            let mine: Vec<Json> = state.iter().filter(|e| e[0] == json!(t)).cloned().collect();
            assert_eq!(mine, (0..40).map(|i| json!([t, i])).collect::<Vec<_>>());
        }
        assert_eq!(state.len(), 160);
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// No frame of a new generation is durable before the log before it, with
    /// its seal, and the directory are: the first group commit after the seal
    /// makes them so if the compaction has not yet.
    #[test]
    fn a_generation_is_durable_after_the_last() {
        let dir = scratch("barrier");
        let layout = Layout::new(&dir, "repo");
        let (wal, _) = start(&layout, NEVER).unwrap();
        let trace = || std::mem::take(&mut *lock(&wal.0.trace));
        wal.set_fold(Arc::new(Entries));
        wal.wait(wal.append(&[json!(0)]).unwrap()).unwrap();
        let a = wal.append(&[json!(1)]).unwrap();
        trace();
        wal.compact_until(1).unwrap();
        let b = wal.append(&[json!(2)]).unwrap();
        assert_eq!(b, a + 2);
        assert!(trace().is_empty(), "step 1 syncs nothing");
        assert!(!wal.is_durable(a) && !wal.is_durable(b));
        wal.wait(b).unwrap();
        assert_eq!(trace(), ["log 0", "dir", "current 1"]);
        assert!(wal.is_durable(a) && wal.is_durable(b));
        wal.wait(wal.append(&[json!(3)]).unwrap()).unwrap();
        assert_eq!(trace(), ["current 1"]);
        // A compaction makes them durable itself, before it folds.
        let (fold, reached, go) = held();
        wal.set_fold(fold);
        std::thread::scope(|s| {
            let compaction = s.spawn(|| wal.compact());
            reached.recv().unwrap();
            assert_eq!(trace(), ["log 1", "dir"]);
            let c = wal.append(&[json!(4)]).unwrap();
            assert!(!wal.is_durable(c));
            wal.wait(c).unwrap();
            assert_eq!(trace(), ["current 2"]);
            go.send(()).unwrap();
            compaction.join().unwrap().unwrap();
        });
        assert_eq!(checkpoint(&layout), [vec![json!(2)], nums(0..4)].concat());
        drop(wal);
        assert_eq!(start(&layout, NEVER).unwrap().1, nums(0..5));
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// A compaction that fails leaves the logs as they were and its `.tmp`
    /// gone, and none is due for a while after; the next folds them all.
    #[test]
    fn a_failed_compaction_leaves_the_logs() {
        let dir = scratch("failed");
        let layout = Layout::new(&dir, "repo");
        let (wal, _) = start(
            &layout,
            Policy {
                floor: 64,
                cap: None,
                max_age: None,
            },
        )
        .unwrap();
        wal.set_fold(Arc::new(Refuse));
        let mut i = 0;
        while wal.generation() == 0 {
            wal.append(&[json!(i)]).unwrap();
            i += 1;
            wal.wait_compacted();
        }
        assert!(!wal.compacting());
        assert_eq!(names(&dir), ["repo.0.log", "repo.1.log"]);
        assert_eq!(wal.checkpoint_generation(), 0);
        // None due, though `L` is past the floor.
        for _ in 0..20 {
            wal.append(&[json!(i)]).unwrap();
            i += 1;
            wal.wait_compacted();
        }
        assert_eq!(wal.generation(), 1);
        // A checkpoint written and not put in place: its `.tmp` goes.
        wal.set_fold(Arc::new(Blocked(layout.checkpoint())));
        assert!(wal.compact().is_err());
        assert!(!layout.tmp().exists());
        assert_eq!(
            names(&dir),
            ["repo.0.log", "repo.1.log", "repo.2.log", "repo.checkpoint"]
        );
        std::fs::remove_dir_all(layout.checkpoint()).unwrap();
        wal.set_fold(Arc::new(Entries));
        wal.compact().unwrap();
        assert_eq!(checkpoint(&layout), [vec![json!(3)], nums(0..i)].concat());
        assert_eq!(names(&dir), ["repo.3.log", "repo.checkpoint"]);
        drop(wal);
        assert_eq!(start(&layout, NEVER).unwrap().1, nums(0..i));
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// A log `g + 1` that cannot be made: the seal comes off log `g`, which
    /// appends go on in, and no frame number is taken.
    #[test]
    fn a_seal_with_no_log_after_comes_off() {
        let dir = scratch("unmade");
        let layout = Layout::new(&dir, "repo");
        let (wal, _) = start(&layout, NEVER).unwrap();
        wal.set_fold(Arc::new(Entries));
        wal.wait(append(&wal, &nums(0..3))).unwrap();
        let len = size(&layout.log(0));
        std::fs::create_dir(layout.log(1)).unwrap();
        assert!(wal.compact().is_err());
        assert_eq!(
            (wal.generation(), wal.log_bytes(), size(&layout.log(0))),
            (0, len, len)
        );
        let n = append(&wal, &nums(3..5));
        assert_eq!(n, 5);
        wal.wait(n).unwrap();
        std::fs::remove_dir(layout.log(1)).unwrap();
        wal.compact().unwrap();
        drop(wal);
        let (wal, state) = start(&layout, NEVER).unwrap();
        assert_eq!(state, nums(0..5));
        assert_eq!((wal.checkpoint_generation(), wal.generation()), (1, 1));
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// A generation missing between the checkpoint and the last log is data
    /// that cannot be recovered.
    #[test]
    fn a_missing_generation_is_unrecoverable() {
        let dir = scratch("gap");
        let layout = Layout::new(&dir, "repo");
        make_dirs(&dir).unwrap();
        let missing = |c: u64, bytes: u64, name: &str| {
            let e = Wal::open_log(&layout, c, bytes, None, NEVER)
                .err()
                .expect("an error");
            assert_eq!(e.kind(), io::ErrorKind::InvalidData);
            assert!(e.to_string().contains(name), "{e}");
        };
        put_log(&layout, 0, &nums(0..2), true);
        put_log(&layout, 2, &nums(2..4), false);
        missing(0, 0, "repo.1.log");
        // The checkpoint's own.
        put_log(&layout, 4, &nums(0..1), false);
        missing(3, 10, "repo.3.log");
        assert_eq!(names(&dir), ["repo.4.log"], "those before it deleted");
        std::fs::remove_file(layout.log(4)).unwrap();
        missing(3, 10, "repo.3.log");
        // With no checkpoint and no log, a log is begun.
        let (wal, frames) = Wal::open_log(&layout, 0, 0, None, NEVER).unwrap();
        assert!(frames.is_empty());
        assert_eq!(
            (wal.generation(), names(&dir)),
            (0, vec!["repo.0.log".to_string()])
        );
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// What follows a log cut short goes: every log after one not sealed or
    /// ending in part of a frame; and so do the logs the checkpoint covers,
    /// and its `.tmp`.
    #[test]
    fn what_follows_a_log_cut_short_goes() {
        let dir = scratch("stray");
        let layout = Layout::new(&dir, "repo");
        let fresh = || {
            let _ = std::fs::remove_dir_all(&dir);
            make_dirs(&dir).unwrap();
        };
        // Not sealed, with a log after it.
        fresh();
        put_log(&layout, 0, &nums(0..2), false);
        put_log(&layout, 1, &nums(2..3), false);
        let (wal, frames) = Wal::open_log(&layout, 0, 0, None, NEVER).unwrap();
        assert_eq!(
            (frames, wal.generation(), names(&dir)),
            (frames_of(nums(0..2)), 0, vec!["repo.0.log".to_string()])
        );
        wal.wait(wal.append(&[json!(2)]).unwrap()).unwrap();
        drop(wal);
        assert_eq!(start(&layout, NEVER).unwrap().1, nums(0..3));
        // Sealed, then one ending in part of a frame, then another: cut back
        // there, the last deleted.
        fresh();
        put_log(&layout, 0, &nums(0..2), true);
        put_log(&layout, 1, &nums(2..4), false);
        let whole = size(&layout.log(1));
        let mut torn = std::fs::read(layout.log(1)).unwrap();
        torn.extend(&frame(&[json!(4)])[..7]);
        std::fs::write(layout.log(1), torn).unwrap();
        put_log(&layout, 2, &nums(5..6), false);
        let (wal, frames) = Wal::open_log(&layout, 0, 0, None, NEVER).unwrap();
        assert_eq!(frames, frames_of(nums(0..4)));
        assert_eq!(
            (wal.generation(), size(&layout.log(1)), wal.len()),
            (1, whole, whole)
        );
        assert_eq!(names(&dir), ["repo.0.log", "repo.1.log"]);
        drop(wal);
        // Sealed, then part of a frame: cut back to the seal, and a log after it.
        fresh();
        put_log(&layout, 0, &nums(0..2), true);
        let sealed = size(&layout.log(0));
        let mut torn = std::fs::read(layout.log(0)).unwrap();
        torn.extend(&frame(&[json!(2)])[..13]);
        std::fs::write(layout.log(0), torn).unwrap();
        let (wal, frames) = Wal::open_log(&layout, 0, 0, None, NEVER).unwrap();
        assert_eq!(frames, frames_of(nums(0..2)));
        assert_eq!(
            (wal.generation(), wal.log_bytes(), size(&layout.log(0))),
            (1, sealed, sealed)
        );
        assert_eq!(names(&dir), ["repo.0.log", "repo.1.log"]);
        drop(wal);
        // Logs before the checkpoint, and its `.tmp`.
        fresh();
        put_log(&layout, 0, &nums(0..1), true);
        put_log(&layout, 1, &nums(1..2), true);
        put_log(&layout, 2, &nums(2..3), false);
        std::fs::write(layout.checkpoint(), b"[2,0,1]").unwrap();
        std::fs::write(layout.tmp(), b"[3,").unwrap();
        let (wal, state) = start(&layout, NEVER).unwrap();
        assert_eq!(state, nums(0..3));
        assert_eq!(
            (
                wal.checkpoint_generation(),
                wal.generation(),
                wal.checkpoint_bytes()
            ),
            (2, 2, 7)
        );
        assert_eq!(names(&dir), ["repo.2.log", "repo.checkpoint"]);
        drop(wal);
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// A log only read gives the frames a start would replay, and nothing
    /// is changed: the logs before the checkpoint, its `.tmp` and part of a
    /// frame at the end are left as they are, for the start to mend.
    #[test]
    fn reading_changes_nothing() {
        let dir = scratch("read");
        let layout = Layout::new(&dir, "repo");
        std::fs::create_dir_all(&dir).unwrap();
        put_log(&layout, 0, &nums(0..2), true);
        put_log(&layout, 1, &nums(2..4), true);
        put_log(&layout, 2, &nums(4..6), false);
        let mut torn = std::fs::read(layout.log(2)).unwrap();
        torn.extend(&frame(&[json!(6)])[..9]);
        std::fs::write(layout.log(2), torn).unwrap();
        std::fs::write(layout.checkpoint(), b"[1,0,1]").unwrap();
        std::fs::write(layout.tmp(), b"[3,").unwrap();
        let before = snapshot(&dir);
        let read = |c: u64| {
            let mut frames = Vec::new();
            read_log(&layout, c, &mut |pos, f| {
                frames.push((pos.generation, f));
                Ok(())
            })
            .map(|()| frames)
        };
        let want: Vec<(u64, Vec<Json>)> = frames_of(nums(2..6))
            .into_iter()
            .enumerate()
            .map(|(i, f)| (1 + i as u64 / 2, f))
            .collect();
        assert_eq!(read(1).unwrap(), want);
        assert_eq!(snapshot(&dir), before);
        // What a start refuses, it refuses: the checkpoint's log missing.
        assert_eq!(read(4).unwrap_err().kind(), io::ErrorKind::InvalidData);
        // And a start replays the same.
        let (_, frames) = Wal::open_log(&layout, 1, 7, None, NEVER).unwrap();
        assert_eq!(frames, frames_of(nums(2..6)));
        assert_ne!(snapshot(&dir), before);
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// Each log reopened by the pool is its current generation's.
    #[test]
    fn the_pool_reopens_the_current_generation() {
        let dir = scratch("pool");
        let pool = Files::new(1);
        let (a, b) = (
            Layout::new(dir.join("a"), "repo"),
            Layout::new(dir.join("b"), "repo"),
        );
        let (wa, _) = Wal::open_log(&a, 0, 0, Some(pool.clone()), NEVER).unwrap();
        let (wb, _) = Wal::open_log(&b, 0, 0, Some(pool.clone()), NEVER).unwrap();
        wa.set_fold(Arc::new(Entries));
        wa.wait(wa.append(&[json!(0)]).unwrap()).unwrap();
        wa.compact().unwrap();
        wb.wait(wb.append(&[json!("b")]).unwrap()).unwrap();
        assert_eq!(pool.open_count(), 1);
        wa.wait(wa.append(&[json!(1)]).unwrap()).unwrap();
        assert_eq!(pool.open_count(), 1);
        assert_eq!(
            scan(&std::fs::read(a.log(1)).unwrap()).unwrap().0,
            frames_of(nums(1..2))
        );
        drop((wa, wb));
        assert_eq!(start(&a, NEVER).unwrap().1, nums(0..2));
        assert_eq!(start(&b, NEVER).unwrap().1, vec![json!("b")]);
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// What is on disk, by name.
    type Disk = BTreeMap<String, Vec<u8>>;

    fn snapshot(dir: &Path) -> Disk {
        std::fs::read_dir(dir)
            .unwrap()
            .map(|e| {
                let e = e.unwrap();
                (
                    e.file_name().into_string().unwrap(),
                    std::fs::read(e.path()).unwrap(),
                )
            })
            .collect()
    }

    /// The log a compaction under test starts from: checkpoint 1, log 1
    /// sealed by a compaction that failed, and log 2; every frame durable.
    fn before(dir: &Path) -> Wal {
        let layout = Layout::new(dir, "repo");
        let (wal, _) = start(&layout, NEVER).unwrap();
        wal.set_fold(Arc::new(Entries));
        wal.wait(append(&wal, &nums(0..3))).unwrap();
        wal.compact().unwrap();
        wal.wait(append(&wal, &nums(3..6))).unwrap();
        wal.set_fold(Arc::new(Refuse));
        assert!(wal.compact().is_err());
        wal.set_fold(Arc::new(Entries));
        wal.wait(append(&wal, &nums(6..9))).unwrap();
        assert_eq!((wal.checkpoint_generation(), wal.generation()), (1, 2));
        wal
    }

    /// Appended to log 3 while the compaction runs, never waited for.
    fn later() -> Vec<Json> {
        vec![json!("x"), json!("y")]
    }

    /// A disk with each log cut to its frames, without the room a kill
    /// leaves after them: as the allocation not yet on disk leaves it.
    fn bare(mut disk: Disk) -> Disk {
        for (name, bytes) in disk.iter_mut() {
            if name.ends_with(".log") {
                bytes.truncate(torn(bytes));
            }
        }
        disk
    }

    /// `bytes`, a log or the first of one, and `n` zeros of room after.
    fn with_room(bytes: &[u8], n: usize) -> Vec<u8> {
        [bytes, &vec![0u8; n]].concat()
    }

    /// The disk before the compaction, and as it was left killed after step
    /// `k`, with `later` appended meanwhile; each log without its room.
    fn killed_after(k: u8) -> (Disk, Disk) {
        killed(&format!("killed-{k}"), k)
    }

    /// The same, in the scratch directory `name`.
    fn killed(name: &str, k: u8) -> (Disk, Disk) {
        let dir = scratch(name);
        let wal = before(&dir);
        let was = bare(snapshot(&dir));
        wal.compact_until(k).unwrap();
        append(&wal, &later());
        let is = bare(snapshot(&dir));
        drop(wal);
        let _ = std::fs::remove_dir_all(&dir);
        (was, is)
    }

    /// A disk to start from; how many of `later` it must keep; and the
    /// checkpoint's generation and the current one, once started.
    type Case = (Disk, usize, (u64, u64));

    /// Start from each disk as the server does, in a directory of its own: the
    /// state must be every frame before the compaction, then the first so many
    /// of `later`; the logs `c..=g`, and the checkpoint. And for the first and
    /// last disk of each outcome (a dozen fsyncs each, too slow for all), the
    /// log goes on, and compacts.
    fn recovers(cases: Vec<Case>, name: &str) {
        let outcomes: Vec<(usize, (u64, u64))> =
            cases.iter().map(|(_, kept, gens)| (*kept, *gens)).collect();
        let go_on = |i: usize| {
            let first = outcomes.iter().position(|o| *o == outcomes[i]);
            let last = outcomes.iter().rposition(|o| *o == outcomes[i]);
            first == Some(i) || last == Some(i)
        };
        let cases = Mutex::new(cases.into_iter().enumerate());
        std::thread::scope(|s| {
            for _ in 0..8 {
                s.spawn(|| {
                    while let Some((i, (disk, kept, (c, g)))) = lock(&cases).next() {
                        let name = format!("{name}-{i}");
                        let dir = scratch(&name);
                        make_dirs(&dir).unwrap();
                        for (file, bytes) in &disk {
                            let path = dir.join(file);
                            make_dirs(path.parent().unwrap()).unwrap();
                            std::fs::write(path, bytes).unwrap();
                        }
                        // A start redoes the journal, if one is there, before
                        // it opens a log; it leaves its directory, empty.
                        crate::journal::redo(&dir, crate::journal::Redo::Eager).unwrap();
                        let _ = std::fs::remove_dir(dir.join(crate::journal::DIR));
                        let layout = Layout::new(&dir, "repo");
                        let (wal, state) = start(&layout, NEVER).unwrap();
                        let mut want = [nums(0..9), later()[..kept].to_vec()].concat();
                        assert_eq!(state, want, "{name}");
                        assert_eq!(
                            (wal.checkpoint_generation(), wal.generation()),
                            (c, g),
                            "{name}"
                        );
                        let mut files: Vec<String> =
                            (c..=g).map(|h| format!("repo.{h}.log")).collect();
                        files.push("repo.checkpoint".into());
                        assert_eq!(names(&dir), files, "{name}");
                        // No log a start keeps is longer than its frames.
                        for h in c..=g {
                            let log = layout.log(h);
                            assert_eq!(file_size(&log), size(&log), "{name}");
                        }
                        if go_on(i) {
                            wal.wait(wal.append(&[json!("z")]).unwrap()).unwrap();
                            want.push(json!("z"));
                            wal.set_fold(Arc::new(Entries));
                            wal.compact().unwrap();
                            drop(wal);
                            assert_eq!(start(&layout, NEVER).unwrap().1, want, "{name}");
                        }
                        let _ = std::fs::remove_dir_all(&dir);
                    }
                });
            }
        });
    }

    /// The journal's name in a `Disk`.
    const JOURNAL: &str = "journal/1.log";

    /// Each case again with a journal beside it, as `PDS_SYNC` `together`
    /// leaves one, its records in the order the frames were written, of
    /// the logs before the compaction (`was`) and after it (`is`): as a
    /// power cut leaves it after the syncs it had, its records of the
    /// frames before the compaction alone, every one of them durable when
    /// it began, so that the case recovers as it does with none; and whole,
    /// the seal and what was appended after it too, all of which the start
    /// puts back: every log sealed, and the next holding all of `later`
    /// where the case has it at all.
    fn journaled(cases: &[Case], was: &Disk, is: &Disk) -> Vec<Case> {
        let logs = |disk: &Disk| -> Vec<(String, Vec<u8>)> {
            let mut logs: Vec<(String, Vec<u8>)> = disk
                .iter()
                .filter(|(n, _)| n.ends_with(".log"))
                .map(|(n, b)| (n.clone(), b.clone()))
                .collect();
            logs.sort_by_key(|(n, _)| n[5..n.len() - 4].parse::<u64>().unwrap());
            logs
        };
        let segment = |logs: &[(String, Vec<u8>)]| {
            let mut frames = Vec::new();
            for (name, bytes) in logs {
                let mut at = 0;
                while at < bytes.len() {
                    let len =
                        HEADER + u32::from_le_bytes(bytes[at..at + 4].try_into().unwrap()) as usize;
                    frames.push((name.as_str(), at as u64, &bytes[at..at + len]));
                    at += len;
                }
            }
            crate::journal::segment_of(&frames)
        };
        let before = logs(was);
        // Every log as long as it grew: one the compaction deleted as it
        // was before.
        let mut after = before.clone();
        for (name, bytes) in logs(is) {
            match after.iter_mut().find(|(n, _)| *n == name) {
                Some(log) => log.1 = bytes,
                None => after.push((name, bytes)),
            }
        }
        let (durable, whole) = (segment(&before), segment(&after));
        let mut out = Vec::new();
        for (disk, kept, (c, g)) in cases {
            let mut d = disk.clone();
            d.insert(JOURNAL.into(), durable.clone());
            out.push((d, *kept, (*c, *g)));
            let mut d = disk.clone();
            d.insert(JOURNAL.into(), whole.clone());
            let kept = if disk.contains_key("repo.3.log") {
                later().len()
            } else {
                0
            };
            out.push((d, kept, (*c, (*g).max(3))));
        }
        out
    }

    /// Killed in step 1 or 2, before log 2 with its seal, and the directory
    /// with log 3, are durable: the seal cut at every byte; log 3 not made,
    /// or made, with what was appended to it cut at every byte. A log 3 after
    /// log 2 unsealed goes: with none of the seal, log 3 is not made, empty or
    /// whole; with part of it, whole, the most it could hold.
    #[test]
    fn killed_in_steps_1_and_2() {
        let (was, is) = killed_after(1);
        assert_eq!(killed_after(2).1, is, "step 2 only makes it durable");
        let durable = was["repo.2.log"].len();
        let log2 = &is["repo.2.log"];
        assert_eq!(*log2, [was["repo.2.log"].clone(), seal()].concat());
        let log3 = &is["repo.3.log"];
        assert_eq!(scan(log3), Ok((frames_of(later()), log3.len())));
        let mut cases: Vec<Case> = Vec::new();
        for cut in durable..=log2.len() {
            let sealed = cut == log2.len();
            let gens = if sealed { (1, 3) } else { (1, 2) };
            let mut disk = is.clone();
            disk.get_mut("repo.2.log").unwrap().truncate(cut);
            disk.remove("repo.3.log");
            if sealed || cut == durable {
                cases.push((disk.clone(), 0, gens));
            }
            for end in 0..=log3.len() {
                if sealed || end == log3.len() || (cut == durable && end == 0) {
                    disk.insert("repo.3.log".into(), log3[..end].to_vec());
                    cases.push((
                        disk.clone(),
                        if sealed {
                            scan(&log3[..end]).unwrap().0.len()
                        } else {
                            0
                        },
                        gens,
                    ));
                }
            }
        }
        let more = journaled(&cases, &was, &is);
        cases.extend(more);
        recovers(cases, "step-1");
    }

    /// Killed in step 3, the `.tmp` not yet durable: at every length, or not
    /// made; and in step 4 before the rename, whole.
    #[test]
    fn killed_in_step_3() {
        let (was, is) = killed_after(3);
        let tmp = is["repo.checkpoint.tmp"].clone();
        assert_eq!(
            serde_json::from_slice::<Vec<Json>>(&tmp).unwrap(),
            [vec![json!(3)], nums(0..9)].concat()
        );
        let mut disk = is.clone();
        disk.remove("repo.checkpoint.tmp");
        let mut cases: Vec<Case> = vec![(disk.clone(), 2, (1, 3))];
        for end in 0..=tmp.len() {
            disk.insert("repo.checkpoint.tmp".into(), tmp[..end].to_vec());
            cases.push((disk.clone(), 2, (1, 3)));
        }
        let more = journaled(&cases, &was, &is);
        cases.extend(more);
        recovers(cases, "step-3");
    }

    /// Killed in step 4 after the rename, before the directory is durable:
    /// the rename made or lost; and in step 5, with any of the logs it
    /// deletes gone.
    #[test]
    fn killed_in_steps_4_and_5() {
        let (was, renamed) = killed_after(4);
        let (_, deleted) = killed_after(5);
        assert_eq!(
            serde_json::from_slice::<Vec<Json>>(&renamed["repo.checkpoint"]).unwrap(),
            [vec![json!(3)], nums(0..9)].concat()
        );
        assert!(!renamed.contains_key("repo.checkpoint.tmp"));
        let mut lost = renamed.clone();
        lost.insert(
            "repo.checkpoint.tmp".into(),
            renamed["repo.checkpoint"].clone(),
        );
        lost.insert("repo.checkpoint".into(), was["repo.checkpoint"].clone());
        let mut cases: Vec<Case> = vec![(renamed.clone(), 2, (3, 3)), (lost, 2, (1, 3))];
        assert_eq!(
            deleted.keys().collect::<Vec<_>>(),
            ["repo.3.log", "repo.checkpoint"]
        );
        for kept in [
            &[][..],
            &["repo.1.log"],
            &["repo.2.log"],
            &["repo.1.log", "repo.2.log"],
        ] {
            let mut disk = deleted.clone();
            for &log in kept {
                disk.insert(log.into(), renamed[log].clone());
            }
            cases.push((disk, 2, (3, 3)));
        }
        let more = journaled(&cases, &was, &renamed);
        cases.extend(more);
        recovers(cases, "step-4");
    }

    /// Killed in step 1 with room in the logs: the seal written over log
    /// 2's room and cut at every byte, the room after it as it was; log 3
    /// not made, or made, what was appended to it cut at every byte, with
    /// room after that or none. Each recovers as the same disk without room
    /// does (`killed_in_steps_1_and_2`), every log cut back to its frames.
    #[test]
    fn killed_in_step_1_with_room() {
        let (was, is) = killed("killed-room", 1);
        let durable = was["repo.2.log"].len();
        let log2 = &is["repo.2.log"];
        let log3 = &is["repo.3.log"];
        let mut cases: Vec<Case> = Vec::new();
        for cut in durable..=log2.len() {
            let sealed = cut == log2.len();
            let gens = if sealed { (1, 3) } else { (1, 2) };
            let mut disk = is.clone();
            let room = log2.len() - cut + 20;
            disk.insert("repo.2.log".into(), with_room(&log2[..cut], room));
            disk.remove("repo.3.log");
            if sealed || cut == durable {
                cases.push((disk.clone(), 0, gens));
            }
            for end in 0..=log3.len() {
                if sealed || end == log3.len() || (cut == durable && end == 0) {
                    let kept = if sealed {
                        scan(&log3[..end]).unwrap().0.len()
                    } else {
                        0
                    };
                    for room in [0, log3.len() - end + 7] {
                        disk.insert("repo.3.log".into(), with_room(&log3[..end], room));
                        cases.push((disk.clone(), kept, gens));
                    }
                }
            }
        }
        let more = journaled(&cases, &was, &is);
        cases.extend(more);
        recovers(cases, "step-1-room");
    }

    /// Room follows a log through its generations: made in the current log
    /// as it is written to, counted in no size, given back as the log is
    /// sealed, and made again in the next. A seal that comes off takes the
    /// room with it.
    #[test]
    fn room_is_given_back_as_a_log_is_sealed() {
        let dir = scratch("room");
        let layout = Layout::new(&dir, "repo");
        let (wal, _) = start(&layout, NEVER).unwrap();
        wal.set_fold(Arc::new(Entries));
        append(&wal, &nums(0..1));
        assert_eq!(file_size(&layout.log(0)), 2 * wal.len());
        append(&wal, &nums(1..3));
        let len = wal.len();
        assert!(file_size(&layout.log(0)) > len);
        assert_eq!((wal.log_bytes(), size(&layout.log(0))), (len, len));
        wal.compact_until(1).unwrap();
        let sealed = len + seal().len() as u64;
        assert_eq!(
            (file_size(&layout.log(0)), wal.log_bytes()),
            (sealed, sealed)
        );
        assert_eq!((wal.len(), file_size(&layout.log(1))), (0, 0));
        append(&wal, &nums(3..4));
        assert_eq!(file_size(&layout.log(1)), 2 * wal.len());
        let n = append(&wal, &nums(4..6));
        assert!(file_size(&layout.log(1)) > wal.len());
        assert_eq!(wal.log_bytes(), sealed + wal.len());
        wal.wait(n).unwrap();
        wal.compact().unwrap();
        assert_eq!(checkpoint(&layout), [vec![json!(2)], nums(0..6)].concat());
        assert_eq!(names(&dir), ["repo.2.log", "repo.checkpoint"]);
        append(&wal, &nums(6..9));
        let len = wal.len();
        assert!(file_size(&layout.log(2)) > len);
        std::fs::create_dir(layout.log(3)).unwrap();
        assert!(wal.compact().is_err());
        assert_eq!((wal.len(), file_size(&layout.log(2))), (len, len));
        std::fs::remove_dir(layout.log(3)).unwrap();
        wal.wait(append(&wal, &nums(9..10))).unwrap();
        assert_eq!(file_size(&layout.log(2)) - wal.len(), wal.len() - len);
        drop(wal);
        assert_eq!(start(&layout, NEVER).unwrap().1, nums(0..10));
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// A log sealed with room after its seal, as one whose room could not
    /// be given back is left, is sealed and whole: to what reads it, to a
    /// start, which cuts the room off and keeps the logs after it, and to
    /// the compaction that folds it. Anything but zeros after a seal is a
    /// torn tail, as without room.
    #[test]
    fn a_sealed_log_with_room_is_sealed() {
        let dir = scratch("sealed-room");
        let layout = Layout::new(&dir, "repo");
        let fresh = || {
            let _ = std::fs::remove_dir_all(&dir);
            make_dirs(&dir).unwrap();
            put_log(&layout, 0, &nums(0..3), true);
            put_log(&layout, 1, &nums(3..5), false);
            (file_size(&layout.log(0)), file_size(&layout.log(1)))
        };
        let pad = |g: u64, more: &[u8]| {
            let mut bytes = std::fs::read(layout.log(g)).unwrap();
            bytes.extend_from_slice(more);
            std::fs::write(layout.log(g), bytes).unwrap();
        };
        let (sealed, open) = fresh();
        pad(0, &[0u8; 40]);
        pad(1, &[0u8; 100]);
        let before = snapshot(&dir);
        let mut read = Vec::new();
        read_sealed(&layout.log(0), |_, _, f| {
            read.push(f);
            Ok(())
        })
        .unwrap();
        assert_eq!(read, frames_of(nums(0..3)));
        assert!(read_sealed(&layout.log(1), |_, _, _| Ok(())).is_err());
        let mut read = Vec::new();
        read_log(&layout, 0, &mut |pos, f| {
            read.push((pos.generation, pos.offset, f));
            Ok(())
        })
        .unwrap();
        let (generations, entries): (Vec<u64>, Vec<Vec<Json>>) =
            read.iter().map(|(g, _, f)| (*g, f.clone())).unzip();
        assert_eq!(generations, [0, 0, 0, 1, 1]);
        assert_eq!(entries, frames_of(nums(0..5)));
        assert_eq!(snapshot(&dir), before);
        let (wal, frames) = Wal::open_log(&layout, 0, 0, None, NEVER).unwrap();
        assert_eq!(frames, frames_of(nums(0..5)));
        assert_eq!((wal.generation(), wal.dropped()), (1, 0));
        assert_eq!(
            (file_size(&layout.log(0)), file_size(&layout.log(1))),
            (sealed, open)
        );
        assert_eq!((wal.log_bytes(), wal.len()), (sealed + open, open));
        // The room there again when the compaction reads the log.
        pad(0, &[0u8; 40]);
        wal.set_fold(Arc::new(Entries));
        wal.wait(append(&wal, &nums(5..6))).unwrap();
        wal.compact().unwrap();
        assert_eq!(checkpoint(&layout), [vec![json!(2)], nums(0..6)].concat());
        drop(wal);
        let (sealed, open) = fresh();
        let stray = frame(&[json!(9)]);
        pad(0, &[0u8; 7]);
        pad(0, &stray);
        assert!(read_sealed(&layout.log(0), |_, _, _| Ok(())).is_err());
        let (wal, frames) = Wal::open_log(&layout, 0, 0, None, NEVER).unwrap();
        assert_eq!(frames, frames_of(nums(0..3)));
        assert_eq!(
            (wal.generation(), wal.len(), file_size(&layout.log(0))),
            (1, 0, sealed)
        );
        assert_eq!(wal.dropped(), 7 + stray.len() as u64 + open);
        assert_eq!(names(&dir), ["repo.0.log", "repo.1.log"]);
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// A directory written before logs had room, each log its frames alone,
    /// is started on and carried on from: its logs are given room as they
    /// are written to, and compacted.
    #[test]
    fn logs_with_no_room_are_carried_on() {
        let dir = scratch("no-room");
        let layout = Layout::new(&dir, "repo");
        make_dirs(&dir).unwrap();
        put_log(&layout, 0, &nums(0..2), true);
        put_log(&layout, 1, &nums(2..4), true);
        put_log(&layout, 2, &nums(4..6), false);
        std::fs::write(layout.checkpoint(), b"[1,0,1]").unwrap();
        let (wal, state) = start(&layout, NEVER).unwrap();
        assert_eq!(state, nums(0..6));
        assert_eq!((wal.checkpoint_generation(), wal.generation()), (1, 2));
        wal.set_fold(Arc::new(Entries));
        wal.wait(append(&wal, &nums(6..9))).unwrap();
        assert!(file_size(&layout.log(2)) > size(&layout.log(2)));
        assert_eq!(
            scan(&std::fs::read(layout.log(2)).unwrap()).unwrap().0,
            frames_of(nums(4..9))
        );
        wal.compact().unwrap();
        assert_eq!(checkpoint(&layout), [vec![json!(3)], nums(0..9)].concat());
        wal.wait(append(&wal, &nums(9..11))).unwrap();
        drop(wal);
        assert_eq!(start(&layout, NEVER).unwrap().1, nums(0..11));
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// A log whose files are said to be durable starts with no sync and no
    /// file opened, where there is nothing of them to mend: a checkpoint,
    /// sealed logs after it and a log to append to, or an empty one, as a
    /// compaction leaves it. It goes on as any other: its appends are
    /// synced, and it compacts. Not said to be, every log is synced.
    #[test]
    fn a_durable_log_starts_with_no_sync() {
        let dir = scratch("durable");
        let layout = Layout::new(&dir, "repo");
        drop(before(&dir));
        let was = snapshot(&dir);
        assert_eq!(names(&dir), ["repo.1.log", "repo.2.log", "repo.checkpoint"]);
        let (wal, state, syncs) = start_as(&layout, None, false);
        assert_eq!((state, syncs), (nums(0..9), 2), "each log synced");
        drop(wal);
        let pool = Files::new(4);
        let (wal, state, syncs) = start_as(&layout, Some(pool.clone()), true);
        assert_eq!((state, syncs), (nums(0..9), 0));
        assert_eq!((pool.open_count(), snapshot(&dir)), (0, was));
        assert_eq!(
            (
                wal.checkpoint_generation(),
                wal.generation(),
                wal.dropped(),
                wal.len()
            ),
            (1, 2, 0, size(&layout.log(2)))
        );
        let synced = crate::syncs();
        wal.wait(wal.append(&[json!(9)]).unwrap()).unwrap();
        assert_eq!((crate::syncs() - synced, pool.open_count()), (1, 1));
        wal.set_fold(Arc::new(Entries));
        wal.compact().unwrap();
        assert_eq!(checkpoint(&layout), [vec![json!(3)], nums(0..10)].concat());
        drop(wal);
        let (wal, state, syncs) = start_as(&layout, None, true);
        assert_eq!((state, syncs, wal.generation()), (nums(0..10), 0, 3));
        wal.wait(wal.append(&[json!(10)]).unwrap()).unwrap();
        drop(wal);
        assert_eq!(start(&layout, NEVER).unwrap().1, nums(0..11));
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// Room is made within what the log may grow to before it is compacted
    /// by size: with room, a log's file is never longer than its frames
    /// may be without.
    #[test]
    fn room_stays_within_what_the_log_may_grow_to() {
        let dir = scratch("room-limit");
        let layout = Layout::new(&dir, "repo");
        let floor = 400;
        let (wal, _) = start(
            &layout,
            Policy {
                floor,
                cap: None,
                max_age: None,
            },
        )
        .unwrap();
        wal.set_fold(Arc::new(Entries));
        let (mut roomy, mut full) = (0, 0);
        for i in 0..300 {
            wal.append(&[json!(i)]).unwrap();
            wal.wait_compacted();
            assert_eq!(wal.log_bytes(), wal.len(), "at {i}");
            let limit = wal.checkpoint_bytes().max(floor);
            let file = file_size(&layout.log(wal.generation()));
            assert!(file <= limit.max(wal.len()), "{file} of {limit} at {i}");
            roomy += (file > wal.len()) as u32;
            full += (file == limit) as u32;
        }
        assert!(roomy > 150, "{roomy} of 300 with room");
        assert!(full > 10, "{full} of 300 with all the room there is");
        assert!(wal.checkpoint_generation() > 3);
        drop(wal);
        assert_eq!(start(&layout, NEVER).unwrap().1, nums(0..300));
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// Files said to be durable with something to mend are mended, and
    /// synced as files not said to be: as many syncs, and the same files
    /// after. A torn tail; a frame torn in room; a `.tmp`; logs the
    /// checkpoint covers; a last log that ends with its seal; a log not
    /// sealed with one after it; no log.
    #[test]
    fn a_durable_log_with_something_to_mend_is_synced() {
        type Make = fn(&Layout);
        let cases: [(&str, Make, &[&str]); 7] = [
            (
                "a torn tail",
                |l| {
                    put_log(l, 0, &nums(0..2), false);
                    let mut torn = std::fs::read(l.log(0)).unwrap();
                    torn.extend(&frame(&[json!(2)])[..7]);
                    std::fs::write(l.log(0), torn).unwrap();
                },
                &["repo.0.log"],
            ),
            (
                "a frame torn in room",
                |l| {
                    put_log(l, 0, &nums(0..2), false);
                    let mut torn = std::fs::read(l.log(0)).unwrap();
                    torn.extend(&frame(&[json!(2)])[..7]);
                    std::fs::write(l.log(0), with_room(&torn, 40)).unwrap();
                },
                &["repo.0.log"],
            ),
            (
                "a .tmp",
                |l| {
                    put_log(l, 0, &nums(0..2), false);
                    std::fs::write(l.tmp(), b"[1,").unwrap();
                },
                &["repo.0.log"],
            ),
            (
                "logs the checkpoint covers",
                |l| {
                    put_log(l, 0, &nums(0..1), true);
                    put_log(l, 1, &nums(1..2), true);
                    put_log(l, 2, &[], false);
                    std::fs::write(l.checkpoint(), b"[2,0,1]").unwrap();
                },
                &["repo.2.log", "repo.checkpoint"],
            ),
            (
                "a last log that ends with its seal",
                |l| put_log(l, 0, &nums(0..2), true),
                &["repo.0.log", "repo.1.log"],
            ),
            (
                "a log not sealed with one after it",
                |l| {
                    put_log(l, 0, &nums(0..2), false);
                    put_log(l, 1, &[], false);
                },
                &["repo.0.log"],
            ),
            ("no log", |_| {}, &["repo.0.log"]),
        ];
        for (i, (what, make, left)) in cases.into_iter().enumerate() {
            let started = [false, true].map(|durable| {
                let dir = scratch(&format!("mend-{i}-{durable}"));
                let layout = Layout::new(&dir, "repo");
                make_dirs(&dir).unwrap();
                make(&layout);
                let (wal, state, syncs) = start_as(&layout, None, durable);
                assert_eq!(names(&dir), left, "{what}");
                // What is left is a log at rest.
                wal.close().unwrap();
                drop(wal);
                let (_, again, none) = start_as(&layout, None, true);
                assert_eq!((&again, none), (&state, 0), "{what}");
                let disk = snapshot(&dir);
                let _ = std::fs::remove_dir_all(&dir);
                (state, syncs, disk)
            });
            assert!(started[0].1 > 0, "{what}");
            assert_eq!(started[0], started[1], "{what}");
        }
    }

    /// A log closed takes no frame, starts no compaction and is not
    /// rewritten; every frame it took is on disk, with the log sealed
    /// before and the directory, synced in a group commit's order; and its
    /// files, said to be durable, start with no sync.
    #[test]
    fn a_closed_log_takes_no_more() {
        let dir = scratch("closed");
        let layout = Layout::new(&dir, "repo");
        let (wal, _) = start(&layout, NEVER).unwrap();
        let trace = || std::mem::take(&mut *lock(&wal.0.trace));
        wal.set_fold(Arc::new(Entries));
        wal.wait(append(&wal, &nums(0..2))).unwrap();
        // A seal and frames no one has waited for.
        wal.compact_until(1).unwrap();
        let n = append(&wal, &nums(2..4));
        assert!(!wal.is_durable(n));
        trace();
        wal.close().unwrap();
        assert_eq!(trace(), ["log 0", "dir", "current 1"]);
        assert!(wal.is_durable(wal.written()));
        let was = snapshot(&dir);
        let refused = wal.append(&[json!(4)]).unwrap_err();
        assert!(
            refused.to_string().ends_with("the log is closed"),
            "{refused}"
        );
        assert!(wal.compact().is_err() && !wal.compact_soon());
        assert!(wal.rewrite(Ok).is_err());
        wal.close().unwrap();
        assert!(trace().is_empty(), "closed again, nothing to sync");
        assert_eq!(snapshot(&dir), was);
        drop(wal);
        let (wal, state, syncs) = start_as(&layout, None, true);
        assert_eq!((state, syncs, wal.generation()), (nums(0..4), 0, 1));
        drop(wal);
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// A pool closed: none of its logs takes a frame, one opened after
    /// neither, and a frame one took before is durable once its log is
    /// closed. A log of no pool goes on.
    #[test]
    fn a_closed_pool_takes_no_more() {
        let dir = scratch("closed-pool");
        let pool = Files::new(4);
        let (a, b) = (
            Layout::new(dir.join("a"), "repo"),
            Layout::new(dir.join("b"), "repo"),
        );
        let (wa, _) = Wal::open_log(&a, 0, 0, Some(pool.clone()), NEVER).unwrap();
        let n = wa.append(&[json!(0)]).unwrap();
        pool.close();
        assert!(wa.append(&[json!(1)]).is_err());
        assert!(!wa.is_durable(n));
        wa.close().unwrap();
        assert!(wa.is_durable(n));
        let (wb, frames) = Wal::open_log(&b, 0, 0, Some(pool.clone()), NEVER).unwrap();
        assert!(frames.is_empty());
        wb.set_fold(Arc::new(Entries));
        assert!(wb.append(&[json!(0)]).is_err() && wb.compact().is_err());
        drop((wa, wb));
        let (wa, frames) = Wal::open_log(&a, 0, 0, None, NEVER).unwrap();
        assert_eq!(frames, frames_of(nums(0..1)));
        wa.wait(wa.append(&[json!(1)]).unwrap()).unwrap();
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// A log closed while it is compacted, the process ending after each
    /// step of the compaction: its files, said to be durable, start with
    /// every frame it took, and with no sync but where the compaction left
    /// something to mend, its `.tmp` or the logs its checkpoint covers.
    #[test]
    fn closed_while_compacting_then_started_as_durable() {
        for (k, mends, gens) in [
            (1, false, (1, 3)),
            (2, false, (1, 3)),
            (3, true, (1, 3)),
            (4, true, (3, 3)),
            (5, false, (3, 3)),
        ] {
            let dir = scratch(&format!("closed-at-{k}"));
            let layout = Layout::new(&dir, "repo");
            let wal = before(&dir);
            wal.compact_until(k).unwrap();
            append(&wal, &later());
            wal.close().unwrap();
            drop(wal);
            let (wal, state, syncs) = start_as(&layout, None, true);
            let mut want = [nums(0..9), later()].concat();
            assert_eq!(state, want, "step {k}");
            assert_eq!(
                (wal.checkpoint_generation(), wal.generation()),
                gens,
                "step {k}"
            );
            assert_eq!(syncs > 0, mends, "step {k}: {syncs} syncs");
            wal.wait(wal.append(&[json!("z")]).unwrap()).unwrap();
            want.push(json!("z"));
            wal.set_fold(Arc::new(Entries));
            wal.compact().unwrap();
            drop(wal);
            assert_eq!(start(&layout, NEVER).unwrap().1, want, "step {k}");
            let _ = std::fs::remove_dir_all(&dir);
        }
    }

    /// A compaction under way when its log is closed puts no checkpoint in
    /// place: the log's files are another's to open from then on. Its
    /// `.tmp` stays, as the process ending there leaves it, for the next
    /// start to delete, which syncs; the start after that syncs nothing.
    #[test]
    fn a_compaction_puts_nothing_in_place_past_a_close() {
        let dir = scratch("close-during");
        let layout = Layout::new(&dir, "repo");
        let (wal, _) = start(&layout, NEVER).unwrap();
        wal.wait(append(&wal, &nums(0..3))).unwrap();
        let (fold, reached, go) = held();
        wal.set_fold(fold);
        std::thread::scope(|s| {
            let compaction = s.spawn(|| wal.compact());
            reached.recv().unwrap();
            let n = wal.append(&[json!(3)]).unwrap();
            wal.close().unwrap();
            assert!(wal.is_durable(n));
            // As the process ending here leaves the files.
            let ended = scratch("close-during-ended");
            make_dirs(&ended).unwrap();
            for (file, bytes) in snapshot(&dir) {
                std::fs::write(ended.join(file), bytes).unwrap();
            }
            assert!(Layout::new(&ended, "repo").tmp().exists());
            let (_, state, syncs) = start_as(&Layout::new(&ended, "repo"), None, true);
            assert_eq!(state, nums(0..4));
            assert!(syncs > 0);
            let _ = std::fs::remove_dir_all(&ended);
            go.send(()).unwrap();
            let refused = compaction.join().unwrap().unwrap_err();
            assert!(refused.ends_with("the log is closed"), "{refused}");
        });
        assert!(layout.tmp().exists() && !layout.checkpoint().exists());
        drop(wal);
        let (wal, state, syncs) = start_as(&layout, None, true);
        assert_eq!(state, nums(0..4));
        assert!(syncs > 0 && !layout.tmp().exists());
        wal.close().unwrap();
        drop(wal);
        let (_, state, syncs) = start_as(&layout, None, true);
        assert_eq!((state, syncs), (nums(0..4), 0));
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// Folds as `Entries`; once its checkpoint is renamed into place it
    /// says so, and waits for word before it answers.
    struct Placed {
        reached: Mutex<mpsc::Sender<()>>,
        go: Mutex<mpsc::Receiver<()>>,
    }

    impl Fold for Placed {
        fn fold(
            &self,
            checkpoint: Option<Vec<u8>>,
            frames: Vec<Vec<Json>>,
            generation: u64,
        ) -> Result<Vec<u8>, String> {
            Entries.fold(checkpoint, frames, generation)
        }

        fn install(
            &self,
            _generation: u64,
            rename: &mut dyn FnMut() -> Result<(), String>,
        ) -> Result<(), String> {
            rename()?;
            let _ = lock(&self.reached).send(());
            let _ = lock(&self.go).recv();
            Ok(())
        }
    }

    /// A compaction whose log is closed once its checkpoint is in place
    /// deletes no log: the files are as the process ending after step 4
    /// leaves them, the log the checkpoint covers there for the next start
    /// to delete, which syncs; the start after that syncs nothing.
    #[test]
    fn a_compaction_deletes_nothing_past_a_close() {
        let dir = scratch("close-placed");
        let layout = Layout::new(&dir, "repo");
        let (wal, _) = start(&layout, NEVER).unwrap();
        wal.wait(append(&wal, &nums(0..3))).unwrap();
        let (reached, at) = mpsc::channel();
        let (go, word) = mpsc::channel();
        wal.set_fold(Arc::new(Placed {
            reached: Mutex::new(reached),
            go: Mutex::new(word),
        }));
        std::thread::scope(|s| {
            let compaction = s.spawn(|| wal.compact());
            at.recv().unwrap();
            let n = wal.append(&[json!(3)]).unwrap();
            wal.close().unwrap();
            assert!(wal.is_durable(n));
            go.send(()).unwrap();
            let refused = compaction.join().unwrap().unwrap_err();
            assert!(refused.ends_with("the log is closed"), "{refused}");
        });
        assert_eq!(names(&dir), ["repo.0.log", "repo.1.log", "repo.checkpoint"]);
        drop(wal);
        let (wal, state, syncs) = start_as(&layout, None, true);
        assert_eq!(
            (state, wal.checkpoint_generation(), wal.generation()),
            (nums(0..4), 1, 1)
        );
        assert!(syncs > 0);
        assert_eq!(names(&dir), ["repo.1.log", "repo.checkpoint"]);
        wal.close().unwrap();
        drop(wal);
        let (_, state, syncs) = start_as(&layout, None, true);
        assert_eq!((state, syncs), (nums(0..4), 0));
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// Room and an open that syncs nothing, together. A log dropped with
    /// its frames waited for, or closed, holds no room: said to be durable
    /// it starts with no sync, and the first frame written to it then grows
    /// its file by itself and as much again, under one sync. Zeros found
    /// after the frames of a log said to be durable, as a kill of a run
    /// before leaves them, are room: cut off, with no sync; not said to be
    /// durable, the log is synced. It holds no lock until its first frame:
    /// another handle that takes the log meanwhile is found out then, by
    /// its lock or by what it wrote, and nothing is written over its
    /// frames.
    #[test]
    fn a_durable_log_has_no_room_and_no_lock_until_it_is_written() {
        let dir = scratch("durable-room");
        let layout = Layout::new(&dir, "repo");
        let log = layout.log(0);
        let (wal, _) = start(&layout, NEVER).unwrap();
        wal.wait(append(&wal, &nums(0..3))).unwrap();
        let len = wal.len();
        assert!(file_size(&log) > len);
        drop(wal);
        assert_eq!(file_size(&log), len);
        let (wal, state, syncs) = start_as(&layout, None, true);
        assert_eq!((state, syncs), (nums(0..3), 0));
        let synced = crate::syncs();
        let n = wal.append(&[json!(3)]).unwrap();
        let one = wal.len() - len;
        assert_eq!(file_size(&log), wal.len() + one, "and as much again");
        wal.wait(n).unwrap();
        assert_eq!(crate::syncs() - synced, 1);
        // Closed with room, and a frame no one waited for.
        let n = append(&wal, &nums(4..6));
        assert!(file_size(&log) > wal.len() && !wal.is_durable(n));
        wal.close().unwrap();
        assert!(file_size(&log) == wal.len() && wal.is_durable(n));
        drop(wal);
        // Zeros there again: cut off, with a sync only if the frames need one.
        let frames = std::fs::read(&log).unwrap();
        for durable in [true, false] {
            std::fs::write(&log, with_room(&frames, 4096)).unwrap();
            let (wal, state, syncs) = start_as(&layout, None, durable);
            assert_eq!((state, wal.dropped()), (nums(0..6), 0));
            assert_eq!(syncs == 0, durable, "{syncs} syncs");
            assert_eq!(std::fs::read(&log).unwrap(), frames);
            drop(wal);
        }
        // Two handles started so: the one that writes first has the log.
        let (a, state, syncs) = start_as(&layout, None, true);
        assert_eq!((state, syncs), (nums(0..6), 0));
        let (b, _, syncs) = start_as(&layout, None, true);
        assert_eq!(syncs, 0);
        b.wait(b.append(&[json!("b")]).unwrap()).unwrap();
        let held = std::fs::read(&log).unwrap();
        let refused = a.append(&[json!("a")]).unwrap_err();
        assert_eq!(refused.kind(), io::ErrorKind::ResourceBusy);
        assert_eq!(std::fs::read(&log).unwrap(), held);
        drop(b);
        let refused = a.append(&[json!("a")]).unwrap_err();
        assert_eq!(refused.kind(), io::ErrorKind::InvalidData, "{refused}");
        assert_eq!(std::fs::read(&log).unwrap(), held[..torn(&held)]);
        drop(a);
        let all = [nums(0..6), vec![json!("b")]].concat();
        assert_eq!(start(&layout, NEVER).unwrap().1, all);
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// A frame whose journal append failed, its cut back failing too: the
    /// log takes no frame from then on. Let go with the cut failing still,
    /// the next open of it in the run, as durable, is not at rest: it cuts
    /// the frame off and syncs, and the next frame is journaled where the
    /// last whole one ends, as a start after a power cut finds.
    #[test]
    fn a_log_whose_cut_failed_is_not_opened_at_rest() {
        use crate::journal::{Fault, Journal, Redo, redo};
        use std::sync::atomic::AtomicBool;
        let dir = scratch("uncut");
        let layout = Layout::new(&dir, "repo");
        let j = Journal::open_with(&dir, None, false).unwrap();
        let pool = Files::with_journal(4, j.clone());
        let (wal, _, _) = start_as(&layout, Some(pool.clone()), false);
        wal.wait(wal.append(&[json!(0)]).unwrap()).unwrap();
        let refuse = Arc::new(AtomicBool::new(true));
        let refusing = refuse.clone();
        *lock(&j.test.fault) = Some(Arc::new(move |f: &Fault<'_>| match f {
            Fault::Write { .. } if refusing.load(Ordering::SeqCst) => {
                Err(io::Error::other("the journal took no record"))
            }
            _ => Ok(()),
        }));
        lock(&wal.0.inner).cut_fails = true;
        assert!(wal.append(&[json!(1)]).is_err());
        refuse.store(false, Ordering::SeqCst);
        assert!(
            wal.append(&[json!(2)]).is_err(),
            "a log not cut back took a frame"
        );
        drop(wal);
        let (wal, state, syncs) = start_as(&layout, Some(pool.clone()), true);
        assert_eq!(state, nums(0..1));
        assert!(syncs > 0, "opened at rest");
        wal.wait(wal.append(&[json!(3)]).unwrap()).unwrap();
        j.let_go();
        drop(wal);
        // A power cut: the log's own file as its first frame left it.
        std::fs::write(layout.log(0), frame(&[json!(0)])).unwrap();
        redo(&dir, Redo::Eager).unwrap();
        let (_, state, _) = start_as(&layout, None, false);
        assert_eq!(state, vec![json!(0), json!(3)]);
        let _ = std::fs::remove_dir_all(&dir);
    }

    /// A read that fails while a log is replayed is an error, not a torn
    /// tail: the open fails, the log is not cut, and no later generation is
    /// deleted.
    #[test]
    fn a_failed_read_is_no_torn_tail() {
        let dir = scratch("read-fails");
        let layout = Layout::new(&dir, "repo");
        make_dirs(&dir).unwrap();
        put_log(&layout, 0, &nums(0..3), true);
        put_log(&layout, 1, &nums(3..5), false);
        let before = snapshot(&dir);
        let second = frame(&[json!(0)]).len() as u64;
        crate::READ_FAILS.set(Some(second));
        let opened = Wal::open_log(&layout, 0, 0, None, NEVER);
        crate::READ_FAILS.set(None);
        assert!(opened.is_err(), "a failed read taken as the end");
        assert_eq!(snapshot(&dir), before);
        assert_eq!(start(&layout, NEVER).unwrap().1, nums(0..5));
        let _ = std::fs::remove_dir_all(&dir);
    }
}