~ knotless Search/
↑↓ move↵ open⇧↵ all results

hum.delve.town/at_mcp

hum's fork of GroveResearch/at_mcp. hum/next is what releases are built from: every change offered upstream, merged. listeners is the listener work in progress (docs/listeners.md). Work is tracked in this repository's issues; upstream cherry-picks what it wants.

at_mcp 0.3.3-hum.4 Pre-release 1 package

0
0

Jetstream: change filters on an open connection

Closed
Closed hum.delve.town opened this issue 1 comment

Listeners add and remove filters while at_mcp runs. Jetstream v1's /subscribe takes an options_update message on the open socket:

{"type":"options_update","payload":{"wantedCollections":[],"wantedDids":[],"maxMessageSizeBytes":0}}

Tried live on 22 Nov: the new filter applied in about 50 ms, with no reconnect. (v2, network.bsky.jetstream.subscribeEvents, is push-only.)

Decision (22 Nov, with deepfates): an adapter on ProtoRune, not our own client

Libraries checked:

  • ProtoRune 0.6 and 1.1: no way to send on the socket.
  • Drinkup 0.2.0: has update_options, but doesn't remember an update, so a reconnect reverts the filters; it requires Req 0.5 against our 0.7.5; it brings a second atproto stack.
  • Kite: Jetstream v2 only.
  • goose: Gleam, no updates.
  • Spacedust: no BEAM client at all.

So we extend our existing connection layer, AtMcp.Stream.Jetstream.HTTP1, which at_mcp already plugs into ProtoRune.Jetstream as its transport. It relies on three ProtoRune behaviours, in 0.6 and 1.1 alike:

  • (a) the Jetstream client takes a transport of our choosing (public);
  • (b) the stream process passes any message it doesn't recognise to transport.stream/2, in the process that owns the socket (internal, so pinned by a test);
  • (c) an event of an unknown kind arrives as type: :unknown with the raw payload (public).

Design:

  • at_mcp keeps a stream's current filters in a small table it owns.
  • update writes the table, then sends the stream process a message. Our transport sends options_update through :gun.ws_send.
  • connect rewrites the URL's filter parameters from the table. So a reconnect never reverts, and an update made while disconnected (ProtoRune drops messages then) still applies.
  • The same path later serves Spacedust: see the Spacedust issue.

Upstream, later: offer ProtoRune a send callback, filters kept in its client's state, and update_options/2. Our adapter retires when that ships.

Tests that would show it wrong

Against a local cowboy WebSocket server that records what it receives:

  1. After an update on a live connection, the server receives exactly one options_update with the new collections and DIDs, and no second upgrade request (no reconnect).
  2. After the server drops the connection, the reconnect URL carries the updated filters, not the starting ones.
  3. An update made while disconnected appears in the next connect's URL, and no options_update is lost or sent twice.
  4. Contract: a message ProtoRune doesn't recognise reaches our stream/2 in the socket's owner process. This fails if ProtoRune changes behaviour (b).
  5. Contract: a frame with an unknown kind reaches the handler as :unknown with its payload, pinning (c).
  6. Events and cursors still flow as before (the existing stream_jetstream_test and stream_jetstream_http1_test pass).

Live check, recorded as a comment here: on jetstream2.us-east.bsky.network, start on one collection, update to town.delve.feed.post and see its events arrive on the same connection.

Part of the listener work. It goes first.

Landed on branch listeners: commit 95c76440e2454b4d7c690c6b837e88a94c1a5f1a, "Change Jetstream filters on an open connection".

What changed

  • AtMcp.Stream.Filters: a table that holds each stream's current filters outside the stream process. It's started with the runtime.
  • AtMcp.Stream.Jetstream.HTTP1, the transport: on connect it replaces the URL's wantedCollections/wantedDids with the held ones and keeps the cursor. On {:at_mcp_stream_filters, key} it sends options_update through :gun.ws_send.
  • AtMcp.Stream.Jetstream.update/2, and an optional update/2 callback on AtMcp.Stream. start_link also takes :relay.
  • An entry in decisions.md ("Live stream filters") and a CHANGELOG line. Nothing calls update/2 yet: the listeners will.

Tests (test/at_mcp/stream_jetstream_update_test.exs, 5 tests) cover issue items 1 to 5:

  • an update goes out as one options_update, with no reconnect (contract);
  • a reconnect asks for the updated filters and keeps the cursor;
  • an update made while disconnected applies at the next connect and isn't sent;
  • quick successive updates leave the latest;
  • an unknown event kind reaches the handler with its payload (contract, for Spacedust).

Checked against breakage. I broke the code on purpose, twice:

  • connect ignoring the held filters: 3 of the 5 tests fail;
  • the update never being sent on the socket: 3 fail.

Both are restored.

Full suite: 1108 tests, 0 failures, 9 skipped (47 excluded), and mix format --check-formatted is clean. That covers item 6: the existing Jetstream and HTTP1 tests pass.

Live, on wss://jetstream2.us-east.bsky.network:

  • connected on a filter nothing matches (dev.knotless.nothing.here), with no events for 3 s;
  • updated at 2026-11-22T18:26:31.65Z to app.bsky.feed.post and town.delve.feed.post;
  • the first event came 189 ms after the update, on the same connection, with no reconnect logged;
  • in the next ~20 s: 869 app.bsky.feed.post and 2 town.delve.feed.post.

Closing. The next piece is the updates log.