hum.delve.town/at_mcp
hum's fork of GroveResearch/at_mcp. hum/next is what releases are built from: every change offered upstream, merged. listeners is the listener work in progress (docs/listeners.md). Work is tracked in this repository's issues; upstream cherry-picks what it wants.
at_mcp 0.3.3-hum.4 Pre-release 1 package
Updates: bounded references and a cursor per listener
Closeddisk_log can't hold saved cursors. Checked in OTP's source, kernel-10.6.3.4, disk_log.hrl/disk_log.erl:
- A chunk continuation is the internal record
#continuation{pid, pos, b}. It carries the log process's pid, so it means nothing after a restart. - Rebuilding one from a saved
posmeans building an undocumented record by hand. - Its own docs say a wrap log's continuation "might not be valid in the next call" once the log wraps and deletes the file it points into.
- There's no reading from a saved position other than by continuation, and wrap logs rotate by size and file count, not age.
So it fails three of the six tests above (crash safety, per-listener cursors, retention) unless we lean on its internals.
Embedded stores on Hex, 22 Nov:
| Store | Latest release | Downloads, last 90 days | Fit |
|---|---|---|---|
exqlite (SQLite) | 0.43.0, 20 Oct 2026 | ~639k | One transaction covers the new updates and the stream cursor, the outbox's guarantee kept by the database. Cursor = rowid; dedupe = UNIQUE(account, listener, record); retention = DELETE ... WHERE; an append costs the same however big the table gets. It's native code, built from SQLite's single source file or a precompiled binary. at_mcp already builds native code (NativeLock, elixir_make), so the toolchain is there. |
cubdb | 2.0.2, 1 Jan 2023 | ~52k | Pure Elixir, ordered keys and atomic transactions, so it could work. But there's been no release in nearly three years, and less of it is proven than SQLite. |
khepri | 0.19.3 | ~556k | A replicated database built on Raft: far more than one process on one machine needs. |
rocksdb | 3.2.1 | ~23k | A heavy native build for no gain here. |
| DETS | in OTP | — | No ordered traversal, a 2 GB limit, slow repair. |
Recommendation: exqlite, with the updates in one SQLite file in the state folder under the existing state lock. Later, the outbox and checkpoints could move into the same file. That would also settle "serializes on every accept", but it's not in this issue.
This adds a native dependency, which is a maintenance choice, so it's put to deepfates before I build it.
Correction to my comment above: the recommendation to add SQLite through exqlite is withdrawn. Checked against all eleven stories, updates only need to be references (a URI, the listener, the source, when seen), with the records fetched fresh at read time. At bounded sizes that fits one small state file like at_mcp's others, so no database and no native dependency (deepfates, Q24, 22 Nov). The issue body now says this. The disk_log finding stands.
Landed on listeners: commit db75894af27ae727efaf08b8d2d7e16e24cb82ea, "Keep listener updates as bounded references, with a cursor per listener".
What changed
AtMcp.Updates, one writing process.- Each account has a private file in the state folder,
listeners/<did>.json, holding its listeners, their updates and cursors, and receipts. It's replaced whole underAtMcp.PrivateFile's lock, fsynced and renamed. - Streams' cursors are in
listeners/streams.json, written after the updates. - An update is
seq,uri,listener,source,fromandseen_at, and nothing else.
- Each account has a private file in the state folder,
- Reading is oldest first. Acknowledging names how far (
through) for one listener or all, and never moves back. - Bounds default to 200 updates and 7 days per listener, settable per listener.
- Started with the runtime. Nothing calls it yet; the pipeline issue will.
One change from the issue as written: one file per account rather than one file for everything, so one account's busy listener doesn't make every account's writes heavier. The guarantee is kept by ordering instead: updates first, cursor after, receipts for the replay. docs/listeners.md item 9 and decisions.md now say this.
Tests (test/at_mcp/updates_test.exs, 9 tests), by the issue's numbers:
- Stopped between the two writes, a new process finds the updates with the cursor behind them. The replay adds nothing and moves the cursor.
- Per-listener cursors.
- An update arriving between the read and the acknowledgement stays unread. A cursor never moves back, nor past the newest update.
- Dedupe, within a listener and across two.
- Count and age bounds drop the oldest and count the unread drops; acknowledging clears the count; a dropped record replayed isn't new.
- Only the reference half is here: no other field of what arrived is kept, checked in the file itself. Reading fresh (deleted reads as deleted, blocked since reads as blocked) is how the updates are read, so it moves to the ways-out issue.
- Write cost: one more update with 20 full listeners took 44 ms on a 671 KiB file. The test fails above 500 ms.
Plus: bounds are checked when saved, and removing a listener removes its updates.
Checked against breakage:
- saving the stream cursor before the updates fails test 1;
- ignoring receipts fails 3 tests.
Full suite: 1122 tests, 0 failures. Format is clean.
Closing. Next: the pipeline issue.
Design: docs/listeners.md, "The model", items 5, 6 and 9.
Decided 22 Nov (Q24): updates are references, not content.
Kept in one small file in the state folder, with the listeners and the sources' stream cursors, replaced whole under a lock as
read-positions.jsonis (AtMcp.PrivateFile). One write holds new updates and the stream cursor together: the outbox's guarantee, kept.Not chosen:
disk_log: its read position holds the log's pid and can go stale when the log wraps (comment below).exqlite): not needed while updates are bounded references. It would be if a listener had to keep thousands of matches.The outbox (
AtMcp.Inbound.Store) is unchanged and keeps serving account delivery.Tests that would show it wrong