~ knotless Search/
↑↓ move↵ open⇧↵ all results

hum.delve.town/at_mcp

hum's fork of GroveResearch/at_mcp. hum/next is what releases are built from: every change offered upstream, merged. listeners is the listener work in progress (docs/listeners.md). Work is tracked in this repository's issues; upstream cherry-picks what it wants.

at_mcp 0.3.3-hum.4 Pre-release 1 package

0
0

Updates: bounded references and a cursor per listener

Closed
Closed hum.delve.town opened this issue 3 comments

Design: docs/listeners.md, "The model", items 5, 6 and 9.

Decided 22 Nov (Q24): updates are references, not content.

  • An update is the record's URI, the listener that matched, the source and when it was seen.
  • Reading fetches the records fresh, through the reads the tools already use. So other people's text reaches the model only on a read and marked as theirs, moderation is current, and a deleted record reads as deleted.
  • Each listener keeps its newest updates up to a bound by count and by age, set per listener, with defaults. An organiser can keep every reply to a casting call; a digest is capped. What a bound drops is counted and shown.
  • Each listener has its own cursor. Acknowledging names the listener and how far.

Kept in one small file in the state folder, with the listeners and the sources' stream cursors, replaced whole under a lock as read-positions.json is (AtMcp.PrivateFile). One write holds new updates and the stream cursor together: the outbox's guarantee, kept.

Not chosen:

  • OTP's disk_log: its read position holds the log's pid and can go stale when the log wraps (comment below).
  • A database (exqlite): not needed while updates are bounded references. It would be if a listener had to keep thousands of matches.

The outbox (AtMcp.Inbound.Store) is unchanged and keeps serving account delivery.

Tests that would show it wrong

  1. Crash safety: kill the VM between saving updates and moving the stream cursor. On restart every update is there, or the stream replays it.
  2. Per-listener cursors: acknowledging listener A up to update 5 leaves B's unread.
  3. Acknowledgement: a newer update arriving between a read and an acknowledgement stays unread.
  4. Dedupe: the same record for the same listener, arriving twice (at-least-once streams, catch-up overlap), is one update. For two listeners it's two.
  5. Bounds: past a listener's count or age bound, the oldest go and the listener reports how many were dropped. A cursor pointing into dropped updates reads from the oldest kept and says so.
  6. Reading is fresh: a record deleted since it matched reads as deleted. An author blocked since reads as blocked. No update holds record text.
  7. Write cost: saving stays within a measured budget at the default bounds with 20 listeners, and the test records the file size it ran at.

disk_log can't hold saved cursors. Checked in OTP's source, kernel-10.6.3.4, disk_log.hrl/disk_log.erl:

  • A chunk continuation is the internal record #continuation{pid, pos, b}. It carries the log process's pid, so it means nothing after a restart.
  • Rebuilding one from a saved pos means building an undocumented record by hand.
  • Its own docs say a wrap log's continuation "might not be valid in the next call" once the log wraps and deletes the file it points into.
  • There's no reading from a saved position other than by continuation, and wrap logs rotate by size and file count, not age.

So it fails three of the six tests above (crash safety, per-listener cursors, retention) unless we lean on its internals.

Embedded stores on Hex, 22 Nov:

StoreLatest releaseDownloads, last 90 daysFit
exqlite (SQLite)0.43.0, 20 Oct 2026~639kOne transaction covers the new updates and the stream cursor, the outbox's guarantee kept by the database. Cursor = rowid; dedupe = UNIQUE(account, listener, record); retention = DELETE ... WHERE; an append costs the same however big the table gets. It's native code, built from SQLite's single source file or a precompiled binary. at_mcp already builds native code (NativeLock, elixir_make), so the toolchain is there.
cubdb2.0.2, 1 Jan 2023~52kPure Elixir, ordered keys and atomic transactions, so it could work. But there's been no release in nearly three years, and less of it is proven than SQLite.
khepri0.19.3~556kA replicated database built on Raft: far more than one process on one machine needs.
rocksdb3.2.1~23kA heavy native build for no gain here.
DETSin OTP—No ordered traversal, a 2 GB limit, slow repair.

Recommendation: exqlite, with the updates in one SQLite file in the state folder under the existing state lock. Later, the outbox and checkpoints could move into the same file. That would also settle "serializes on every accept", but it's not in this issue.

This adds a native dependency, which is a maintenance choice, so it's put to deepfates before I build it.

Correction to my comment above: the recommendation to add SQLite through exqlite is withdrawn. Checked against all eleven stories, updates only need to be references (a URI, the listener, the source, when seen), with the records fetched fresh at read time. At bounded sizes that fits one small state file like at_mcp's others, so no database and no native dependency (deepfates, Q24, 22 Nov). The issue body now says this. The disk_log finding stands.

Landed on listeners: commit db75894af27ae727efaf08b8d2d7e16e24cb82ea, "Keep listener updates as bounded references, with a cursor per listener".

What changed

  • AtMcp.Updates, one writing process.
    • Each account has a private file in the state folder, listeners/<did>.json, holding its listeners, their updates and cursors, and receipts. It's replaced whole under AtMcp.PrivateFile's lock, fsynced and renamed.
    • Streams' cursors are in listeners/streams.json, written after the updates.
    • An update is seq, uri, listener, source, from and seen_at, and nothing else.
  • Reading is oldest first. Acknowledging names how far (through) for one listener or all, and never moves back.
  • Bounds default to 200 updates and 7 days per listener, settable per listener.
  • Started with the runtime. Nothing calls it yet; the pipeline issue will.

One change from the issue as written: one file per account rather than one file for everything, so one account's busy listener doesn't make every account's writes heavier. The guarantee is kept by ordering instead: updates first, cursor after, receipts for the replay. docs/listeners.md item 9 and decisions.md now say this.

Tests (test/at_mcp/updates_test.exs, 9 tests), by the issue's numbers:

  1. Stopped between the two writes, a new process finds the updates with the cursor behind them. The replay adds nothing and moves the cursor.
  2. Per-listener cursors.
  3. An update arriving between the read and the acknowledgement stays unread. A cursor never moves back, nor past the newest update.
  4. Dedupe, within a listener and across two.
  5. Count and age bounds drop the oldest and count the unread drops; acknowledging clears the count; a dropped record replayed isn't new.
  6. Only the reference half is here: no other field of what arrived is kept, checked in the file itself. Reading fresh (deleted reads as deleted, blocked since reads as blocked) is how the updates are read, so it moves to the ways-out issue.
  7. Write cost: one more update with 20 full listeners took 44 ms on a 671 KiB file. The test fails above 500 ms.

Plus: bounds are checked when saved, and removing a listener removes its updates.

Checked against breakage:

  • saving the stream cursor before the updates fails test 1;
  • ignoring receipts fails 3 tests.

Full suite: 1122 tests, 0 failures. Format is clean.

Closing. Next: the pipeline issue.