johnandersen777.bsky.social/asyncai

Switch repository

Loading…

# 0004 — Code mode: MCP tools driven through Monty

Status: accepted

Date: 2026-11-03

## Context

`OpenAiCompatibleBackend` exposes every MCP tool to the model as an OpenAI
function. The model calls one tool per round, the full result enters the
conversation, and the loop allows 8 rounds.

That shape fails on Kubernetes data:

- `kubectl get ... -o yaml` for a busy namespace is tens to hundreds of
  kilobytes. The answer the user wants is usually a handful of names or a
  count.
- The MCP client capped every result at 64k characters. `get_api_resources`
  produced ~21 KB of YAML, a 12k cap cut it mid-document, and the model and the
  verbose logs both received invalid YAML. Raising the cap only moves the
  failure.
- A dependent sequence (list workspaces, then list objects in each, then get
  one) costs one model round trip per step, and every intermediate result is
  paid for in context.

Pydantic AI solves this with
[`CodeMode`](https://pydantic.dev/docs/ai/harness/code-mode/), backed by
[Monty](https://pydantic.dev/docs/monty/quickstart/rust/), a sandboxed Python
subset written in Rust:

> Runs Monty behind `CodeModeToolset`. Instead of making sequential tool calls,
> the model writes Python that calls your tools as functions, and Monty
> executes it: one round trip for a task that would otherwise take three.

The model writes a program that fans out tool calls with `asyncio.gather`,
filters and aggregates the results in Python, and returns only what matters.
Large intermediate results stay inside the sandbox.

Monty ships as Rust crates (`monty`, `monty-types`, 1.4.0 on crates.io, MIT,
MSRV 1.96). `MontyRepl` keeps globals between snippets and pauses execution at
every call to a host function, so the host can run the real call and resume.

## Decision

1. Every MCP tool call goes through one model-facing tool, `run_code`. The
   model never sees the MCP tools as OpenAI functions; it sees them as Python
   `async def` stubs in the `run_code` description.
2. `run_code` executes the model's Python in an in-process Monty REPL owned by
   a dedicated OS thread. MCP calls made from that Python are dispatched by the
   async host to the `rmcp` clients.
3. MCP tool output is unbounded. The client no longer truncates. The only
   bound is on what `run_code` returns to the model (see Limits).
4. kcp tools emit JSON (`kubectl -o json`) so the sandbox can parse and filter
   everything with the stdlib. `http_get` returns JSON
   `{status, content_type, body}` with the full body. MCP results that parse
   as JSON arrive in Python as `dict` / `list` values; anything else arrives
   as `str`.
5. `McpToolExecutor` holds its clients without a lock, so calls gathered in
   Python run concurrently against the MCP servers.

### `run_code`

Input schema:

```json
{
  "type": "object",
  "properties": {
    "code": {"type": "string", "description": "Python to run"},
    "restart": {"type": "boolean", "description": "reset REPL state first"}
  },
  "required": ["code"]
}
```

The description carries three parts: the sandbox rules, the return contract,
and the catalog of available functions.

Example program for "which socialweb.computer kinds exist in root:app?":

```python
resources = await get_api_resources(kcp=KCP, workspace="root:app")
sorted({r["kind"] for r in resources if r["api_version"].startswith("socialweb.computer/")})
```

Parallel calls:

```python
import asyncio
pods, events = await asyncio.gather(
    list_objects(kcp=KCP, workspace="root:app", resource="pods"),
    get_events(kcp=KCP, workspace="root:app"),
)
bad = [p["metadata"]["name"] for p in pods["items"]
       if p["status"].get("phase") != "Running"]
{"not_running": bad, "warnings": sum(1 for e in events["items"] if e["type"] == "Warning")}
```

### Flow

```mermaid
sequenceDiagram
    participant M as model
    participant B as OpenAiCompatibleBackend
    participant C as CodeModeRun (async)
    participant T as Monty thread
    participant S as MCP servers

    M->>B: tool_call run_code {code}
    B->>C: call_tool("run_code", {code})
    C->>T: Job {code, restart}
    T->>T: feed_start(code)
    T-->>C: Call {call_id: 1, list_objects, kwargs}
    T->>T: resume_pending (future)
    T-->>C: Call {call_id: 2, get_events, kwargs}
    T->>T: resume_pending (future)
    T-->>C: NeedResults
    par concurrent
        C->>S: tools/call list_objects
        C->>S: tools/call get_events
    end
    S-->>C: JSON text (unbounded)
    C->>T: (1, dict) / (2, dict)
    T->>T: resume futures, filter in Python
    T-->>C: Complete {value, prints}
    C-->>B: compact JSON of the filtered value
    B->>M: tool message
```

### Threads and channels

```mermaid
flowchart LR
    subgraph tokio
        B[backend start_run] -->|run_scope| R[CodeModeRun]
        R -->|FuturesUnordered| X[inner ToolExecutor<br/>McpToolExecutor]
    end
    subgraph monty thread
        E[engine loop<br/>Option&lt;MontyRepl&gt;]
    end
    R -- Job --> E
    E -- HostEvent::Call / NeedResults / Done --> R
    R -- call results --> E
```

`monty` uses `Rc` internally, so the REPL and its suspended snapshots never
cross threads. The engine thread owns them for its whole life and talks to the
async side over channels. CPU-bound Python never blocks a tokio worker.

### Run scope

`ToolExecutor` gains `fn run_scope(&self) -> Option<Arc<dyn ToolExecutor>>`,
which defaults to `None`. `OpenAiCompatibleBackend::start_run` calls it once
per run. `CodeModeToolExecutor` returns a fresh `CodeModeRun` with its own
engine thread and an empty REPL.

REPL state (variables, imports, functions) therefore persists across
`run_code` calls within one agent turn, so the model can store a large result
in a variable, inspect its shape, and filter it in a later call without
refetching it. `restart: true` clears it. When the turn ends, the scope is
dropped, the job channel closes, and the thread exits. Concurrent agents never
share a REPL.

### Engine loop

| Monty progress | Host answer |
|---|---|
| `NameLookup(name)` with a catalog name | `MontyObject::function(name, docstring)` |
| `NameLookup(name)` otherwise | `NotFound` → `NameError` |
| `FunctionCall` for a catalog function, keyword args only | send `Call` to async side, `resume_pending` |
| `FunctionCall` with positional args | `TypeError: use keyword arguments` |
| `FunctionCall` past the nested-call budget | `RuntimeError: nested tool call budget exceeded` |
| `ResolveFutures` | send `NeedResults`, block for ≥1 result, `resume` with every result received |
| `OsCall` | `PermissionError: os access disabled in asyncai code mode` |
| `Complete {repl, value}` | keep `repl`, send `Done(value, prints)` |
| `ReplStartError` | keep the preserved `repl`, send the traceback as an error |

Nested MCP results map into the sandbox as follows:

| MCP result | Python value |
|---|---|
| `is_error: true` | raises `RuntimeError(<text>)`; catchable |
| text (or `structured_content` when there is no text) that parses as JSON | the parsed value |
| other text | `str` |
| `rmcp` transport error | raises `RuntimeError("tool error: ...")` |

### Python stubs

The catalog renders each MCP tool's JSON schema as an `async def` stub with
keyword-only parameters:

```python
async def list_objects(*, kcp: str, workspace: str, resource: str, namespace: str | None = None, selector: str | None = None) -> Any:
    """List objects of one resource type in a workspace."""
```

| JSON schema | Python |
|---|---|
| `string` | `str` |
| `integer` | `int` |
| `number` | `float` |
| `boolean` | `bool` |
| `array` of T | `list[T]` |
| `object` / absent / boolean schema | `dict[str, Any]` / `Any` |
| `enum` | `Literal[...]` |
| type list with `null`, or optional property | `T \| None = None` |

Required parameters come first. A tool name that is not a Python identifier is
sanitized (`[^A-Za-z0-9_]` → `_`, leading digit gets a `_` prefix, collisions
get a numeric suffix). The engine maps the Python name back to the MCP name.

### Value conversion

| JSON | Python | back to JSON |
|---|---|---|
| `null` | `None` | `null` |
| bool | `bool` | bool |
| integer in i64 | `int` | number |
| other number | `float` | number (`NaN`/`inf` → string) |
| string | `str` | string |
| array | `list` | array |
| object | `dict` | object |
| — | `tuple`, `set`, `frozenset` | array |
| — | dict with non-`str` keys | object, keys via `repr` |
| — | anything else | `repr()` string |

### Return contract

The last expression in the snippet is its result. `print()` output is
collected separately.

| Snippet ends with | `run_code` returns |
|---|---|
| a non-`None` value, nothing printed | the value |
| an assignment or `None`, nothing printed | `(no output; the last expression was None)` (A11) |
| an assignment or `None`, something printed | `{"output": "<prints>"}` |
| a non-`None` value, something printed | `{"output": "<prints>", "result": <value>}` |
| an exception | error result with the Python traceback, plus a `Hint:` for known Monty gaps (A11) |
| an un-awaited coroutine or `gather` | error: add `await` (A11) |

A `str` result is returned as-is. Everything else is compact JSON.

### Limits

| Limit | Value | On breach |
|---|---|---|
| execution time per snippet (`max_feed_duration`) | 30 s; excludes time awaiting tools | error result; REPL kept |
| heap | not capped in-process (see below) | — |
| recursion depth | Monty default (1000) | `RecursionError` |
| suspensions per snippet (`max_suspensions`) | 10 000 | uncatchable `RuntimeError` |
| nested tool calls per snippet | 100 | `RuntimeError` at the call site |
| collected `print()` output | 1 MiB | `MemoryError` |
| MCP tool output | **unbounded** | — |
| `run_code` result to the model | 20 000 chars | error result with size, a 2 000-char preview, and "assign to a variable and return a filtered subset" |

The 20 000-char guard protects the context window, not the tools. The value is
still in the REPL if the model assigned it, so the next snippet filters
without refetching.

Monty enforces `max_memory` only when `monty-alloc` is the process's global
allocator. That allocator caps the whole process, so in an agent it would
count the tokio runtime, the MCP clients, and every buffer next to the
sandbox. asyncai therefore does not set `max_memory`. A hard heap ceiling
comes with the `monty-pool` follow-up, where each worker process installs
`monty-alloc` for itself.

Limits are fields of `CodeModeLimits`, so library users can tune them.

### Logging

With `--verbose`, NDJSON on stderr:

| Event | Fields |
|---|---|
| `model.tool_call` | `tool: "run_code"`, `arguments: {code, restart?}` |
| `model.tool_result` | `tool`, `is_error`, `content` (JSON-decoded when possible, otherwise truncated text) |
| `mcp.tool_call` | `via: "code_mode"`, `tool`, `arguments` |
| `mcp.tool_result` | `via: "code_mode"`, `tool`, `is_error`, `bytes`, `content` (JSON-decoded when possible, otherwise truncated text) |

### Session transcripts

Session harnesses record what the model saw: the `run_code` `tool_use` and its
`tool_result`. Nested MCP calls are not model turns. They are not written as
transcript records and do not enter the `parentUuid` chain; they appear in the
verbose log only. The transcript order rule from ADR 0002 is unchanged.

## Invariants

1. With `--mcp-stdio`, the model's tool list is exactly `[run_code]`.
2. Every MCP call made for the model goes through the Monty engine.
3. The MCP client returns full tool output. No component between the MCP
   server and the sandbox truncates it.
4. The sandbox has no filesystem, environment, network, or subprocess access.
   Its only way out is a catalog function.
5. A snippet cannot block a tokio worker: Python runs on the engine thread.
6. A runaway snippet ends with an error result within its time budget; it
   never hangs the agent.
7. REPL state is scoped to one backend run and never shared across runs or
   agents.
8. kcp object tools emit JSON; `get_api_resources` emits a JSON array.
9. Tests use a fake `ToolExecutor` or the bundled stdio servers with fake
   backends. No live model, cluster, or atirc server.

## Test plan

Unit (`src/code_mode/`):

- stub rendering: required/optional ordering, enum, array, nullable type list,
  boolean schema, name sanitizing and collision suffixes;
- JSON → Monty → JSON round trip, including non-`str` keys and tuples;
- return contract table;
- 20 000-char guard.

Integration (`tests/code_mode.rs`, fake in-memory executor):

- `asyncio.gather` of two tools plus filtering returns one small value; both
  inner calls recorded;
- a 1 MB JSON tool result is filtered down inside the sandbox;
- `is_error` tool result raises a catchable `RuntimeError`;
- unknown function → `NameError` in the error result;
- `open(...)` and `os.getenv(...)` → `PermissionError`;
- `while True: pass` → time-limit error, no hang;
- REPL state persists across two `run_code` calls in one scope; `restart`
  clears it; a new scope starts empty;
- positional arguments → `TypeError`;
- nested-call budget enforced.

Backend and CLI:

- mocked LLM requests list only `run_code`; a `run_code` tool call runs and
  the tool message carries the filtered result;
- `asyncai --mcp-stdio asyncai-mcp-simple` with a mocked model runs
  `await http_get(url=...)` against a loopback HTTP server; the Claude Code
  transcript has a `run_code` `tool_use` and `tool_result`.

kcp:

- fake `kubectl` sees `-o json` on object calls; `get_api_resources` returns a
  JSON array.

## Live verification

In tmux pane `0:1.1`, run the readwrite agent with `--verbose`, subscribe to
the channel with `atirc sub`, and publish prompts with `atirc pub`:

1. "which socialweb.computer kinds exist in root:app?" → one
   `model.tool_call run_code`, a nested `mcp.tool_call get_api_resources`, a
   short reply.
2. "list the names of all configmaps in every namespace" → a large
   `list_objects` result filtered in the sandbox; the reply lists names only.
3. apply → get → patch → delete of a test ConfigMap through `run_code`.

Repeat 1–2 with `socialweb-computer-kcp-readonly`. Confirm kubectl
invocations use `-o json`, and that the session transcript resumes with
`deepseek-claude -p --resume <uuid>`.

## Consequences

- One model round trip covers dependent and parallel tool sequences.
- Large tool output costs sandbox memory, not context.
- Models must write valid Monty Python. Errors return as tracebacks, and the
  model can retry within the 8-round loop. Small models explore: they print
  shapes, hit errors, and retry. The final round withdraws `run_code` and asks
  for an answer from the results so far, so exploration never ends a turn
  without a reply.
- Tools without an output schema render as `-> Any`. The description tells
  the model to inspect a small sample before relying on such a shape. The
  bundled servers declare schemas (Amendment A1).
- Tool results are JSON-shaped rather than YAML. Humans read the verbose logs,
  which were already JSON.
- In the in-process sandbox, a stack-overflow or allocator abort inside
  Monty would take the agent process down. The pool sandbox (Amendment A5)
  removes that risk.

## Alternatives considered

- **`monty-pool` only.** This was deferred in the first version and is now
  the preferred sandbox (Amendment A5). In-process stays available because
  the pool needs a separately installed worker binary, and tests must not.
- **Keep direct tool calls next to `run_code`.** This contradicts "all MCP
  tool calls driven through Monty" and invites the model to bypass filtering.
  Library users can still pass `McpToolExecutor` directly to
  `with_tool_executor`.
- **Raise the output cap.** It only moves the failure, and it still pays for
  every byte in context.
- **Dynamic catalog in system instructions** (Pydantic's `dynamic_catalog`).
  Our tool set is fixed per process, so the catalog lives in the `run_code`
  description.

## Amendments (2026-11-03): hardening

Live runs with a 27B local model showed the remaining failure modes:

- untyped results caused shape guessing (`kcps[0]` on a dict);
- every call was tried on every MCP server;
- unknown keyword arguments passed silently;
- a hung tool blocked for up to an hour;
- there was no crash isolation and no heap cap;
- writes left no trace in the session file;
- the model spent rounds discovering the kcp id.

These amendments supersede the matching parts above.

### A1. Typed results

`ToolDefinition` carries the MCP `outputSchema` and annotations
(`read_only`, `destructive`). Each stub's return annotation comes from the
output schema. Its docstring gains a `Returns:` line with a compact shape
sketch: at most 4 levels and 12 properties per object, then `...`.

```python
async def list_kcps(*, refresh: bool | None = None) -> dict[str, Any]:
    """
    Discover running socialweb-computer kcp servers...
    Returns: {"kcps": [{"id": str, "kubeconfig": str, "server": str, "running": bool, ...}], "errors": [str]}
    """
```

The kcp servers and `http_get` declare output schemas and return
`structuredContent`. MCP requires `structuredContent` to be an object, so
array results are wrapped: `list_workspaces` → `{"workspaces": [...]}`,
`get_api_resources` → `{"resources": [...]}`. `get_logs` and
`delete_object` stay plain text, with no schema. Each tool description ends
with a one-line Python example.

### A2. Routing

`McpToolExecutor` lists every server's tools at connect time and builds a
name → server table. A call goes to exactly one server. If two servers
declare the same name, the first wins; the duplicate is recorded and logged
as a `warn` at startup.

### A3. Call validation

A shared dispatcher checks every nested call before it reaches the host:

| Check | Python error |
|---|---|
| positional arguments | `TypeError: f() takes keyword arguments only` |
| unknown keyword | `TypeError: f() got an unexpected keyword argument 'x'; accepted: a, b` |
| missing required keyword | `TypeError: f() missing required keyword argument 'x'` |
| nested-call budget exhausted | `RuntimeError` |

Optional keywords passed as `None` are dropped, matching the stub default.

### A4. Timeouts

| Limit | Default | On breach |
|---|---|---|
| per nested tool call (`tool_timeout`) | 120 s | `RuntimeError: tool X timed out after 120s` inside the snippet |
| per `run_code`, wall clock (`max_wall`) | 300 s | in-flight calls dropped, snippet aborted, error result |

The engine's wait for tool results is bounded by `tool_timeout + 30 s`.

### A5. Sandboxes: in-process or monty-pool

`--code-sandbox auto|in-process|pool`, default `auto`.

| Sandbox | Isolation | Heap cap |
|---|---|---|
| `pool` | each REPL scope is a checkout of a `monty` worker subprocess (`monty-pool`); a crash or hang kills only the worker | `--code-max-memory`, default **4 GiB**, enforced by `monty-alloc` in the worker |
| `in-process` | engine thread inside the agent | none |

`auto` picks `pool` when a worker binary is found (`--monty-worker`, then
`MONTY_WORKER`, then `monty` on `PATH`). Otherwise it uses `in-process` and
logs a `warn` that there is no heap cap. Install the worker with
`cargo install monty-runtime --locked`.

Both sandboxes speak the same host protocol (`HostEvent::Call` / `Done` out,
call results in), so the async host loop, logging, and audit do not depend
on the sandbox. A pool worker crash or timeout becomes an error result that
says the REPL state was reset. The pool's `request_timeout` is the snippet
time limit plus 10 s.

### A6. Mutating-call audit

A nested call to a tool annotated `destructive_hint: true` or
`read_only_hint: false` is logged at `warn` as `mcp.tool_call` /
`mcp.tool_result` with `"mutating": true`. This is **always on**, without
`--verbose`. The kcp read tools are annotated read-only. `apply_object`,
`patch_object`, and `scale_object` are non-read-only and non-destructive;
`delete_object` is destructive.

### A7. Session audit

`ToolCallOutput` carries an `audit` value. For `run_code` it is
`{"nested_calls": [{tool, arguments, is_error, bytes, mutating}]}`. The
Claude Code harness writes it as `toolUseResult` on the `tool_result`
record. That field is UI metadata: the model never sees it on resume, and
record order and the `parentUuid` chain are unchanged.

### A8. Fewer exploratory rounds

- The kcp `kcp` argument is optional. When exactly one kcp is discovered it
  is used; otherwise the error lists the ids.
- Built-in helpers are fed into every new REPL scope (and after `restart`):
  `items(x)`, `names(x)`, `dig(obj, "a.b.c", default=None)`,
  `count_by(xs, "dotted.key")`. `asyncio`, `json`, and `re` are
  pre-imported; live runs showed the model forgetting `import asyncio`.

### Live results (local 27B model, real kcp)

| Prompt | Before amendments | After |
|---|---|---|
| socialweb kinds in root:app | KeyError, 8 rounds, no reply | 2 rounds, 1 `run_code`, 22 506 B in → 871 B out |
| configmaps + events gather | 4 rounds, `bytes_out > bytes_in` | 2 rounds, 1 `run_code`, 6 015 B in → 77 B out |
| apply → patch → get → delete | 4 failed applies before success | first apply succeeds; every write logged `warn` |
| 6 GiB allocation | (in-process: would grow the agent) | `MemoryError` at 4 GiB, worker reset, agent alive |
- `--code-preamble PATH` runs operator Python after the helpers in every new
  scope.
- The prompt and tool description say: plan, then write one complete
  `run_code` that fetches, filters, and returns the answer; inspect shapes
  only after a failure.

### A9. Turn metrics

With `--verbose`, the backend emits `model.turn_summary` at the end of each
run:

| Field | Meaning |
|---|---|
| `rounds` | model requests in the run |
| `run_code_calls` | model tool calls |
| `nested_calls` | MCP calls made from Python |
| `bytes_in` | MCP result bytes received in the sandbox |
| `bytes_out` | tool-result bytes sent to the model |
| `final_round_forced` | whether the tool-free final round was used |

`bytes_in` / `bytes_out` is the context saving code mode buys.

### A10. REPL per conversation thread

Studying the core (ADR 0001) showed that runs, not threads, owned the REPL:

- A coalesced follow-up run (`queued_run`) started with an empty REPL.
- Thread history keeps only user and assistant text, so that run also lost
  every earlier tool result.

`RunRequest` now carries `thread_id`, and `ToolExecutor::thread_scope`
returns one REPL per thread (LRU of 8). The runtime never overlaps runs on
one thread, so the scope needs no extra locking. The `run_code` description
tells the model that variables persist across the conversation and that
earlier tool results are not in its history.

### A11. Session review fixes (session `e4017aa9`)

A 42-minute live session (46 `run_code` calls, 8 errors) showed the model
re-fetching data it already held, shadowing helpers, and misreading empty
results.

- **Variables note.** After each snippet, a probe in the same scope
  describes the module-level names snippets bound (`ss: list[80]`).
  `ToolExecutor::context_note` appends `[run_code variables still in scope:
  ...]` to the newest user message of the next run. It never changes the
  tool description, so the cached prompt prefix stays stable. Names are
  found line by line (`code_mode::scope`); a miss only means a name is not
  listed. At most 30 names, 600 chars.
- **Reserved helpers.** A snippet that rebinds `items`, `names`, `dig`,
  `count_by`, or `to_json` at module level is rejected before it runs.
  Live: `names = []` in one turn broke `names(v)` three turns later.
- **`dig(obj, *path, default=None)`.** Segments may be dotted or split:
  `dig(i, "spec.payload", "requesterDid")`. A trailing non-`str`/`int`
  argument is still taken as the default (`dig(x, "a", {})`).
  `count_by(xs, *path)` likewise.
- **`to_json(x, indent=None)`.** `json.dumps` with `str()` for unknown
  types; Monty's `json.dumps` has no `default=`.
- **Hints.** Known Monty gaps in a traceback add one `Hint:` line
  (`json.dumps(default=)`, unhashable keys).
- **Un-awaited awaitables** (`<gather(4)>`) are errors, not results.
- **Empty results** are explicit text, not `{}`.

Live after the fixes: a follow-up question about data fetched the turn
before made 0 MCP calls (the model used `items_` from the note, avoiding
the reserved `items`).

### Amended invariants

10. A nested call reaches exactly one MCP server, or none if it fails
    validation.
11. Every mutating nested call produces a `warn` log line, regardless of
    flags.
12. With the pool sandbox, no Python runs in the agent process.
13. No nested call or `run_code` call waits forever: per-call and wall
    deadlines always apply.
14. Helper names are never rebound by model code; the variables note never
    changes the tool definitions.