johnandersen777.bsky.social/asyncai

Switch repository

Loading…

# 0003 — socialweb-computer kcp MCP servers

Status: accepted

Date: 2026-11-02

## Context

`asyncai` already runs stdio MCP servers through `rmcp`. The first server,
`asyncai-mcp-simple`, exposes one HTTP GET tool. The next tool surface is the
set of kcp instances that `socialweb-computer` starts beside each application
instance.

A `socialweb-computer` installation is not one cluster. Every application owns
a kcp. A bidder keeps its kcp under
`/var/lib/socialweb-computer/swc-bidder/handles/<hash>/kcp`; a production
gateway keeps one under `/var/lib/socialweb-computer/socialweb/kcp`; a dev
network keeps one under its state root. Each kcp exposes a Kubernetes API with
a root logical cluster and child workspaces. The state root also holds the
admin kubeconfig.

Discovery against `hat-7` and the production gateway found the same shape:

| Fact | hat-7 | gateway (`socialweb.computer`) |
|---|---|---|
| kcp front | `127.0.0.1:33059` to slot `127.0.0.1:40359` | kcp API on `127.0.0.1:6443` |
| admin kubeconfig | `/var/lib/socialweb-computer/swc-bidder/handles/<hash>/kcp/admin.kubeconfig` | `/var/lib/socialweb-computer/socialweb/kcp/slot-a/admin.kubeconfig` |
| identity | `shard-admin`, `system:masters` | `shard-admin` |
| workspaces | `root`, `root:app`, `root:socialweb-provider` | `root`, `root:app`, `root:socialweb-provider`, account workspaces |
| access | loopback only; root-owned files; agents used `ssh` + `sudo -n kubectl` | loopback only; root-owned files |
| existing read-only role | none | none |

Two facts constrain the design.

First, the admin kubeconfig is a bearer token in `system:masters`. Every agent
session inspected reached for that file directly, so every read ran with full
write authority. A read tool that accepts the admin kubeconfig is therefore not
read-only in practice.

Second, a kcp service account token is scoped to one logical cluster. A token
minted in `root` cannot read `root:app`; each workspace needs its own token and
its own binding. "Read-only credentials" are plural by construction.

The kcp API is reachable only on the host's loopback interface. Tests must not
require a live cluster, a fixed port, or a real state path, so all transport
and file access goes through a command runner that tests replace with a fake
`kubectl`.

## Decision

Add two stdio MCP binaries:

| Binary | Tools | Credential |
|---|---|---|
| `socialweb-computer-kcp-readonly` | discovery, workspaces, api-resources, list/get, logs, events, credential status | a freshly minted, workspace-scoped service account token |
| `socialweb-computer-kcp-readwrite` | the read tools plus server-side apply, delete, patch, scale | the admin kubeconfig |

Shared implementation lives in `src/mcp/servers/socialweb_kcp/`.

```
src/mcp/servers/socialweb_kcp/
  config.rs      clap flags, no baked-in endpoint
  runner.rs      local or `ssh <host> sudo -n` command execution
  discovery.rs   find kcp admin kubeconfigs, read context, walk workspaces
  auth.rs        ServiceAccount/RBAC bootstrap and TokenRequest
  service.rs     kcp resolution, credential cache, protected-host guard
  tools.rs       tool implementations
  readonly.rs    rmcp router for the read-only binary
  readwrite.rs   rmcp router for the read-write binary
src/mcp/servers/socialweb_kcp_readonly/main.rs
src/mcp/servers/socialweb_kcp_readwrite/main.rs
```

The read-only server never uses the admin kubeconfig for a read. On the first
tool call for a workspace it:

1. resolves the kcp from discovery;
2. applies one manifest containing a `socialweb-computer-kcp-readonly`
   `ServiceAccount`, a `ClusterRole` with `get`, `list`, and `watch`, and a
   `ClusterRoleBinding` in that workspace;
3. calls `kubectl create token ... --duration=900s`;
4. writes a token-only kubeconfig with the workspace URL and the admin
   kubeconfig's CA data, mode `0600`, no client certificate and no client key;
5. runs every read with `--kubeconfig <token-kubeconfig>`.

Tokens are cached per `(kcp, workspace)` and refreshed when less than three
minutes remain. The token itself is never returned by a tool; the credential
tool reports only the private kubeconfig path and lifetime.

Discovery searches state roots for `admin.kubeconfig` under a `kcp` directory,
excludes `shared`, groups slot directories under one kcp root, and prefers the
kcp-root file over a slot file. `--root` and `--kubeconfig` are repeatable.
Nothing is searched unless configured; the default roots are the documented
state locations.

`--ssh <host>` makes the runner execute every command on that host. The admin
kubeconfig and minted token files never leave the host. The kubeconfig's server
URL is read from the file at runtime; no host or port is compiled in.

Protected hosts (`socialweb.computer`, `hat-7`, `hat-9`) refuse every operation
until `--allow-prod` is passed. This includes the read-only server, because
minting its credential creates RBAC objects.

`--mcp-stdio` wires either binary into `asyncai`:

```bash
./target/debug/asyncai \
  --mcp-stdio "./target/debug/socialweb-computer-kcp-readonly --ssh hat-7 --sudo --allow-prod"
```

## Invariants

1. No endpoint or port is compiled in. A server URL comes from a kubeconfig
   read at runtime or from `--ssh` plus that kubeconfig.
2. The read-only binary never passes the admin kubeconfig to `get`, `logs`,
   `api-resources`, or any other read command.
3. A read-only token is minted per workspace. Tokens are never reused across
   workspaces.
4. A minted token has a 900 second lifetime and is refreshed before it expires.
5. A token kubeconfig contains a bearer token, CA data, and an optional TLS
   server name. It contains no client certificate and no client key, and is
   mode `0600`.
6. Tool output and command logs never contain the admin token, the minted
   token, or kubeconfig contents.
7. The read-only tool set contains no mutating verb. Its only writes are the
   RBAC bootstrap and the TokenRequest.
8. The read-write server uses the admin kubeconfig and no read-only token.
9. `apply_object` validates `apiVersion`, `kind`, and `metadata.name`, and
   rejects a `List` and `generateName`.
10. Discovery excludes `shared` admin kubeconfigs and collapses slot files
    under their kcp root.
11. Protected hosts require `--allow-prod`, in both binaries.
12. Tests use a fake `kubectl`, a temporary state directory, and an ephemeral
    loopback port. No test reaches a real kcp.
13. `apply_object` with `dry_run` performs server-side admission and stores
    nothing.
14. Object-returning kubectl calls use `-o json`, so code mode
    ([ADR 0004](0004-code-mode.md)) parses them into Python `dict`s and the
    model filters them with the stdlib. `get_api_resources` converts kubectl's
    wide table into a JSON array of `{name, short_names, api_version,
    namespaced, kind, verbs, categories}`. `delete_object` uses `-o name`;
    `get_logs` is plain text. Output is never truncated. A failed tool
    returns an MCP result with `is_error: true` and an `error: ...` text, so
    code mode raises it as `RuntimeError`.
15. JSON tools declare an MCP `outputSchema` and return `structuredContent`
    (plus the same JSON as text). Because `structuredContent` must be an
    object, array results are wrapped: `list_workspaces` returns
    `{"workspaces": [...]}` and `get_api_resources` returns
    `{"resources": [...]}`. Kubernetes objects and lists use a shared
    `KubeObject` / `KubeList` schema. Each description ends with a one-line
    Python example for code mode.
16. Tools carry MCP annotations. Read tools are `read_only_hint: true`.
    `apply_object`, `patch_object`, and `scale_object` are
    `read_only_hint: false`, `destructive_hint: false`. `delete_object` is
    `destructive_hint: true`. Code mode logs every non-read-only call at
    `warn`.
17. The `kcp` argument is optional. When omitted, the server uses the only
    discovered kcp, or fails with the list of kcp ids if there are several.
18. Watchable resources (ADR 0005):
    `kcp-events://watch/{workspace}/{resource}{?namespace,labelSelector,fieldSelector,kcp}`
    is an event-log resource. One watcher per URI seeds with a list, streams
    `kubectl get -w --output-watch-events -o json --request-timeout=0` with
    the read credential (minted token), dedupes on uid + resourceVersion,
    strips `managedFields`, keeps a 256-event ring with `seq`, pushes
    `notifications/resources/updated` via `subscriptions/listen`, and
    reconciles by re-listing after a stream ends. It gives up after 5
    consecutive failures with an `ERROR` event. No polling. `MODIFIED`
    events (streamed or reconciled) carry `old`, the previous known object.
19. `kcp-api://{workspace}/resources{?kcp}` is a readable, non-watchable
    resource with the same JSON as `get_api_resources`.
20. `list_objects` / `get_object` with an unknown resource type append
    `did you mean: ...` (up to 3 names from api-resources: plural, short
    name, kind, `name.group`; a `-`/`_` group typo maps to `.`).

## Test plan

Hermetic tests in `tests/socialweb_kcp_mcp_stdio.rs` drive the real binaries
through `McpToolExecutor`:

- the read-only server lists `list_kcps`, `list_workspaces`, `get_api_resources`,
  `list_objects`, `get_object`, `get_logs`, `get_events`, and
  `readonly_credentials`, and does not list `apply_object` or `delete_object`;
- discovery reads a context from the fake kubeconfig and reports the state path;
- a recursive workspace walk visits `root` and `root:app` and returns both;
- `list_objects` triggers the RBAC bootstrap and TokenRequest, then reads with
  `--kubeconfig <temp token file>`;
- the generated token kubeconfig contains the minted token and CA data, and
  contains no client certificate, client key, or admin token;
- the token kubeconfig is mode `0600`;
- the credential tool reports a lifetime and never echoes the token;
- the read-write server lists the write tools and `apply_object` runs
  `kubectl apply --server-side` with the admin kubeconfig and sends the
  manifest on stdin;
- object-returning calls pass `-o json`; `get_api_resources` returns a JSON
  array;
- `apply_object` rejects a manifest without `metadata.name` before running
  `kubectl`.

Unit tests cover kubeconfig context parsing, workspace URL derivation, remote
shell quoting, token kubeconfig construction, and API-resource table to JSON
conversion.

## Verification

```text
$ cargo fmt --check
$ cargo clippy --all-targets -- -D warnings
$ cargo test
running 3 tests
test apply_rejects_a_manifest_without_a_name ... ok
test readwrite_server_applies_with_admin_credentials ... ok
test readonly_server_discovers_and_reads_with_a_minted_token ... ok
```

All Rust tests pass without a kcp, a state directory, an SSH host, or a known
port.

Live verification is separate from `cargo test`. The read-only binary ran over
SSH with `--ssh hat-7 --sudo --allow-prod`:

| Call | Result |
|---|---|
| `list_kcps` | one kcp, discovery path from the host, `running: true` |
| `list_workspaces` recursive | `root:app` and `root:socialweb-provider`, both `Ready` |
| `get_api_resources` in `root:app` | served resource list |
| `list_objects` `computevms` in `root:app` | empty `List` |
| `readonly_credentials` in `root:app` | token kubeconfig written mode `0600`, lifetime 900 s |
| `list_objects` workspaces in `root` | two workspaces, read with the `root` token |

The read-write binary ran with `--ssh hat-7 --sudo --allow-prod`:

| Call | Result |
|---|---|
| `list_objects` `computevms` in `root:app` | empty `List` with the admin kubeconfig |
| `apply_object` dry run of a ConfigMap | object returned by server-side admission, not stored |

The dry-run object was checked afterwards and was absent:

```text
Error from server (NotFound): configmaps "mcp-dry-run-probe" not found
```

The minted identity on the host:

```text
Username  system:serviceaccount:default:socialweb-computer-kcp-readonly
Extra     authentication.kcp.io/scopes [cluster:1mrhuj6frnoznfpq]
```

Permission probe with that token, in `root:app`:

```text
create configmaps  no - access denied
list computevms    yes
list workspaces in root  no - access denied
```

The last line is the workspace scoping invariant in action: the `root:app`
token cannot read the `root` workspace. A later read in `root` minted a
separate `root` token and succeeded.

The read-only ServiceAccount, ClusterRole, and ClusterRoleBinding were present
in the workspace after the call.

## Consequences

Reads from the read-only server are genuinely read-only: the admin token is
used only to create three RBAC objects and a TokenRequest, and every read after
that carries a workspace-scoped token that the server did not possess before
the first call.

The bootstrap leaves one ServiceAccount, one ClusterRole, and one
ClusterRoleBinding per workspace. They are named deterministically and applying
them again is a no-op. Deleting them revokes future tokens but does not revoke
tokens already minted; those expire within fifteen minutes.

The server shells out to `kubectl` rather than embedding a Kubernetes client.
This keeps the implementation small and uses the kubeconfig and TLS behavior
operators already rely on. It requires `kubectl` on the target host, which
`hat-7` has and the production gateway obtains from its embedded k3s. A future
change can replace the runner with a native client behind the same tool
surface.

The read-only role grants `get`, `list`, and `watch` on every resource,
including Secrets. It is read-only, not least-privilege. A caller that needs a
narrower surface should replace the generated ClusterRole; the tool surface
does not change.

`--allow-prod` is a deliberate speed bump, not an authorization boundary. The
admin kubeconfig remains the real authority in the read-write binary.