0
0
# AGENTS.md - working rules for this repository
Read this file first. It is the one guide for agents in this repository. `CLAUDE.md` imports it.
This repository (socialweb-computer-kcp) is the monorepo of the socialweb-computer project. It is the
single source of truth for its Go code. The only TypeScript left is `scripts/*.ts` and the policy
actions under `internal/policyengine/bundled-actions/tangy/`.
`docs/ARCHITECTURE.md` holds the decisions, the measurements and the invariants. This file points at it
and states the working rules. Open a section of `docs/ARCHITECTURE.md` by its heading when a change
needs the argument behind it. Do not read it end to end.
## Orientation: which file answers which question
| Question | File |
| --- | --- |
| What is this, in one paragraph? | `README.md` |
| The objective, stated once | `docs/ARCHITECTURE.md` section 1 |
| Where a concept lives; the package layout | `docs/ARCHITECTURE.md` section 12, and Layout below |
| Rules a change must not break | `docs/ARCHITECTURE.md` section 13 |
| Why a decision was made, and what it closed | `docs/ARCHITECTURE.md` section 15 |
| The plans and the status of each item | `docs/plans/*.yaml` |
| The live backlog (what is open now) | `NEXT.yaml` |
| The TypeScript that is left | `scripts/*.ts`, and the policy actions in `internal/policyengine/bundled-actions/tangy/` |
| Env the clusterprovider reads | `internal/providermain/config.go`; the gateway's defaults for it in `internal/stack/gatewaykcp/env.go` |
| Env the gateway reads | `internal/gatewaymain/config.go`, `deploy/socialweb-gateway.env.example` |
| How prod is installed and upgraded | `scripts/deploy.sh` (gateway: `install gateway`, then `upgrade`), `socialweb-computer install bidder`, `socialweb-computer install requester` |
| What a host must provide | `socialweb-computer check-host -role gateway`, `-role bidder` or `-role requester`, `docs/host-contract.md` |
| The market network | `deploy/market-network/README.md` |
| The operator handoff | `handoff.yaml` |
| What CI runs on a push, and its runners | `.knotless/workflows/` (knotless only; GitHub CI is gone): the workflows `release`, `test` and `promote`, in series (each after the one before), and `pins` (kcp-small 4/8) runs only when asked. `release` (kcp-build 12/16; on pushes to main only, a branch when asked; no v* tag is built: plan 0040 step 15) is the gate's checks (`deploy/check.sh --fast`: gofmt, apigen, codegen drift, go vet on the host and every cross-OS target, architecture rules, the docs checker, the pins records, the policy engine tests; nothing is built), then the ONE build (the release binaries for every platform, the static test host, the wasm test modules and the static meshd of this commit's trees; a tree already published is skipped), the amd64 binary pushed to ttl.sh by digest for `test`, the node downloads and the guest image, and on main its last steps stage `staging/<sha>/`, copy it inside the bucket to `main/<sha>/` and `main/latest/` and check what was published from the public URLs (its key a sealed secret of main pushes; `superseded: finish` on main, a newer branch push cancels the older run). `test` (kcp-test 4/8, `after: [release]`) is the staging gate: it stands up ONE staging network (two bidders) from its CI VM (its guest requester) and `tests/compositions/staging/suite.sh` runs every entry of `tests/entries.txt` (46) on it (as the network's alice1 to alice4, four at once; `scripts/ci-test.sh` `input_default`), printing `gate: staging: N of 46 PASS` (about 2 h 40 min; `scripts/ci-staging-total.sh <commit>` reads it back); it takes release's binary (main: `main/<sha>/`) and release's test host and meshd. `promote` (the `release` executor binding, which asks for no guest requester: no `association`, `scripts/promote.sh prepare` exits 1 on a VM that has one and makes the gate's unix user r2, docs/ci-debug.md "The release binding"; `after: [release, test]`: it starts only once this commit's `test` passed, and is filed failure without running when it failed; on a push to main with test's paths, and when asked: plan 0040 step 15) is the release: a plain first step (`scripts/promote.sh test`) ends the run unless this commit's `test` ran green (`scripts/ci-check-wait.sh`; path-skipped is not green), before release-workload writes the rc's records and moves channel `rc`; then `scripts/promote.sh` writes the prod target (`scripts/ci-prod-target.sh`), deploys the droplets, waits for every rc operator's heartbeat, runs the prod gate (`tests/gate.sh <target> --account bob,alice` with `GATE_VMS=6`: the `both` entries; about 90 min), then copies `main/<sha>/` to `releases/<vT>/` in the bucket (`scripts/copy-spaces.ts` with the bucket key, checked from the public URLs; vT is release-workload's `RELEASE_STABLE_VERSION`); exit 0 moves channel stable to vT, anything else moves rc back to what it rolled forward from at once. Nothing but `release` compiles anything, and nothing rebuilds anything to compare (no reproducible-build verification in CI). The release pipeline: `docs/plans/0040-release-pipeline.md` Flow. Since 2026-11-19 the cycle is in series (owner: less compute, though slower): release, then test, then promote, so staging and the prod gate do not rent at once for one commit; a rerun of a red `test` does not start promote again (docs/ci-debug.md "After a red test"). |
| What the system does, end to end, and its tests | `docs/acceptance/` (the runbooks), `tests/system` (one test file per runbook), `tests/testkit/README.md` |
## Layout
Go:
| Directory | What it holds |
| --- | --- |
| `cmd/` | one directory per binary (`socialweb-computer`, `socialweb-node`, `socialweb-vmexec`, `socialweb-guest-agent`, `socialweb-apigen`, the two OpenBao auth plugins) |
| `internal/` | the Go packages, in the plan 0026 layers (`docs/ARCHITECTURE.md` section 12) |
| `pkg/` | the packages another module imports: `pkg/embedded`, the requester an app runs in its own in-process kcp (plan 0038); its `Config.OnSnapshot` bytes hold every vault secret in clear JSON: store them encrypted (Keychain, Data Protection Complete, no backup), never log them; `Config.StateDir` holds the renter PLC rotation key `components/Requester-requester/plc-rotation.key` (k256, 0600, made on first use; first rotation key of every guest DID it rents, so it tombstones them when the guest is gone): same protection, never in a backup |
| `api/` | the kcp API types; `go generate ./api/...` writes the schemas, the APIExports and the generated Go |
| `deploy/` | manifests, sets, the generated schemas and exports, `check.sh` |
| `docs/` | `ARCHITECTURE.md`, `plans/`, inventories, research notes |
| `rust/` | socialweb-meshd |
| `did-key-associator/` | the QR session-transfer service (qr.fedfork.com): static page and deno backend, vendored 2026-11-18 with its history from github.com/publicdomainrelay/did-key-associator (0a48e76; that repo is private since 2026-11-20, this tree is the source); deployed by `infra/did-key-associator` |
| `typescript-helpers/` | the deno libraries did-key-associator imports by relative path (`../typescript-helpers/lib/{oauth-scope,cli-args-env,logger,serve}`), vendored 2026-11-18 with its history from github.com/publicdomainrelay/typescript-helpers (3fae6da; private since 2026-11-20, this tree is the source) |
| `site/` | https://socialweb.computer/: the landing page (`site/index.md`, the owner's text; README sections spliced in at build) and the user docs rendered to static pages (`site/build.sh`; mermaid drawn at build), deployed by hand with `site/deploy.sh` (`site/README.md`); in no CI workflow's `paths:` |
| `tests/` | the Rust system integration tests: `testkit` (harness), `system` (runbook ports), `targets/`; `vmm` (the VMM wrapper on a KVM host, `-p swc-vmm-tests`) |
TypeScript:
| Directory | What it holds |
| --- | --- |
| `scripts/*.ts` | one-off operator scripts (`oauth-signin.ts`, `associate-bidder.ts`, `publish-spaces.ts`), run with `deno run` |
| `internal/policyengine/bundled-actions/tangy/<action>/` | one policy action: `index.ts` and its built `dist/index.js`, embedded by the policy engine |
There is no deno workspace and no vendored TypeScript repository here any more; the one lockfile is `site/deno.lock` (the site build: the markdown renderer, shiki, mermaid and puppeteer-core).
## The TypeScript that is left
- `scripts/*.ts` are one-off operator scripts, run with `deno run -A`. No Go code runs them; the
knotless `release` workflow and `tests/publish-test-host.sh` run `scripts/publish-spaces.ts`.
- The policy actions under `internal/policyengine/bundled-actions/tangy/<action>/` are gha-lite assets
that the policy engine embeds: `action.yml`, `index.ts` and its built `dist/index.js`. Edit them as
such assets, with the workflows beside them in `internal/policyengine/bundled-actions/workflows/`.
- No comments in code. ASCII only.
## The kcp paradigm, port 0, and the RFP flow
- kcp is part of each application, not infrastructure. An application instance is a kcp of its own (or,
in a market network, a workspace of the network's one kcp), a manifest set and a profile. State is kcp
objects. Components (Pds, Plc, Relay, Bidder, Requester and the other component kinds) run in
process in the core, in the component runtime of `internal/component`, and host processes are
HostProcess units; both are reconciled from those objects. Only the virtual Kubernetes clusters
project uses kcp in the traditional way.
- Which kcp, by profile: a prod profile (the gateway, `run bidder`, `run request` and `run requester`
without `--network`) runs a strict kcp per application. The market network profile (dev, CI, soak) runs every application
in one kcp, each in its own workspace `root:<app>` (components in `root:<app>:app`), from the same
manifest set: the applier rebases the set's workspaces under the application's workspace. Either way
applications talk only over the atproto market protocol; the shared kcp is composition and isolation,
never a data channel. Write service names in manifests as `<name>.default.{{.ServiceSuffix}}`.
- Port 0 behind the edge: every listener binds port 0 and reports its bound address into the status of
its kcp object. Peers find it by service name through kcp. Only the stable public entry points
(caddy on 443, an application's front) keep a known address.
- The RFP flow is the only way a guest comes up: the requester posts a `compute.vm` record and a signed
`market.rfp`, bidders bid, the requester accepts, and the winner boots the guest from cloud-init
`user_data` (`BuildUserData` and the module registry in `internal/atproto/cloudinit`).
The requester reaches the guest only through the guest transports below. Never start a container or VM by hand in
code or in a test, and never install a guest agent outside cloud-init. A live run that needs a
guest brings up its own market network,
and `socialweb-computer run request` or a tenant cluster of the network's gateway rents the guest.
`run request` composes a MarketRequester root (`internal/app/marketrequester`), and its Requester
component (`internal/services/requester` over `internal/market/requester`) runs the RFP, the
accept and the release; `run request list`, `run request release` and `run request resume` manage
the invocations it recorded.
- `run requester --name <name>` (plan 0039 A-RUN-REQUESTER, `cmd/socialweb-computer/requester.go`) is the
long-lived form: the same MarketRequester root and the same bring-up (standalone: bringUp profile quick,
role requester; `--network <market network>`: a child application), state under
`<state-dir>/requesters/<name>` (never `requests/`) with the owner flock of `internal/stack/request/owner.go`,
no contract of its own. Its application options (`deploy/apps/option-map.yaml`) are the MarketRequesterSpec part of
`run request`'s, parsed by one shared function (`parseRequesterSpec`); a contract option is refused. Once the root is
Ready it writes a workspace kubeconfig (`<dir>/kubeconfig`, 0600) and the ready file (`<dir>/ready.json`,
plus `--ready-file` when given; tmp then rename, 0600; `internal/stack/request/ready.go`) {schema 1, name,
pid, startedAt, kcpUrl, kubeconfig, workspace, did, handle, identityMode, pdsUrl, ingress {pds, requester},
sshPublicKey}: paths and public values only. It prints that as a JSON summary, rewrites the file when a
field changes (a watch on the MarketRequester), stays in the foreground (`--fg` is a no-op for the unit
form), and on Ctrl-C or SIGTERM removes the ready file and stops its kcp keeping its state (a network
child is stopped, not removed). `<dir>/requester.json` records the first start's network, `--identity` and
spec; a restart with no application options, or the same ones, reuses it (other options are refused),
adopts the units and keeps the DID, the vault and every contract (Held ones stay Held). `kubectl apply`
Infra, EphemeralInfra or ComputeContract against the ready file's kubeconfig. Helpers on a running one
read its ready file: `run requester env <name>` (KUBECONFIG, the kcp URL, the workspace, the DID, the ssh
public key; `--json`) and `run requester ssh <name> --contract <cc> [--transport webrtc] [--exec -- <command> [args...]]`
(the `run request ssh` code with the requester's own key `<dir>/ssh/id_ed25519`; put its `.pub` in the
contract's `sshAuthorizedKeys`); `run requester proxy` is its webrtc ProxyCommand. Every ssh the CLI runs gets `SHELL=/bin/sh`, because ssh runs its
ProxyCommand with `$SHELL -c` and an installed account's shell is nologin (sudo and the user manager set SHELL from passwd). With `--host-backend
systemd-user` (plan 0039 B-INSTALL-REQUESTER, the installed form; refused with `--network`) the process is the core
itself (`request.RunCore`, the `bidder.RunCore` pattern over `devcore.Run` with the bootstrap app
`table.Requester`, key `req-<sha256(<state dir>/requesters/<name>/dev/<name>)[:10]>` (plan 0046 step 6: by state dir and name), layout `install.RequesterLayout` under `<dir>/dev/<name>`): it places
kine, front and slot as user units, applies the root, signs in and sends READY=1 only after the ready file; a stop
leaves those units and the HostProcesses running for the next start to adopt. `install requester --name <n>`
(default account `swc-requester`) writes the unit template `socialweb-computer-requester@.service` (Type=notify,
TimeoutStartSec the ready budget plus `install.RequesterStartGrace`, 1 h for a sign-in) and, once per name,
`<dir>/requester.json` from its application options (default `--atproto-oauth-qr`). The account is a system account, so
journald puts its user units' lines in the system journal (only root reads them: `journalctl _UID=<uid>
_SYSTEMD_USER_UNIT=<unit>`); `run requester signin <name>` prints the MarketIdentity's `pendingAction.url` from the
requester's own kcp, and `install` prints both. A password never goes into requester.json:
`run requester password <name>` reads one from stdin into `identity/requester/keys` of the requester's vault.
With `--atproto-identity-mode pdsAccount`, `run request` and `run requester` (standalone or `--network`, where the
network child takes only that mode, `--atproto-handle` and `--atproto-pds-url`, default the network's bob PDS) put
`ATPROTO_PASSWORD` from their own environment into that path before the sign-in; a missing or unreadable keysRef keeps
the MarketIdentity Pending (`WaitingForCredential`, `VaultUnavailable`), never Failed; so does a transient sign-in failure
(`SignInRetrying`: a timeout, a connection error, a 5xx or a 429 of createSession, or of an oauthBrowser begin, by
`netretry.Transient`), retried by the controller's backoff; Failed (`SignInFailed`) is the PDS's own refusal (401
AuthenticationRequired, AccountTakedown, AuthFactorTokenRequired, a 400), a DNS not-found and a certificate
verification error. A PDS call of the controller is bounded by the
core's `identityResolverTimeout` (10 s) on the core's network client. The CLI waits for Ready up to its deadline
(`run request` and `run bidder` 5 min). A pdsAccount requester signs its
market callbacks with its PDS service auth (`signing: owner-identity`) and publishes its badge.blue signer key.
`upgrade` takes the role from the install record. The same-account form is a kubectl-applied HostProcess
(`examples/infra/requester-hostprocess.yaml`); no root prunes a HostProcess outside its set.
- `run requester --with-gateway` (plan 0039 C-GATEWAY-COMPOSE, `cmd/socialweb-computer/requestergateway.go`,
`internal/stack/gatewaykcp/requester.go`, `internal/app/gateway/requester.go`) applies a Gateway root `gateway` of
profile `requester` with `spec.requesterRef` (the MarketRequester `app` of the same workspace) and runs its
clusterprovider in the `run requester` process. With requesterRef the Gateway composes no PlanAccount, Pds,
Requester, PolicyEngine, root vault or web app: only the rbac phase of the gateway set (the NodeClaim agent) and a
nodes-only GatewayAPI (`spec.nodesOnly`, the NodeClaim binding and the downloads route) whose node bundles go to
the requester's vault (`nodeMarket.vaultMount` requester); the nodes' Infra is in the requester workspace and the
MarketRequester's Requester rents them as the user's DID. A Gateway without requesterRef composes what it did
before. Its clusterprovider serves only the clusters of the requester's DID (`SOCIALWEB_CLUSTERPROVIDER_ACCOUNT`:
the registry and the Cluster controller skip every other account), and `--with-gateway` is refused in a market
network that has a gateway child (that provider serves every Cluster of the network's kcp). The cluster's k3s,
kine and OpenBao are HostProcesses of the requester's core; in a market network the
child core runs the gateway's outbound trust and egress (`gatewaykcp.NetworkOutbound`) when
`<instance state>/gateway` exists, which `run requester --with-gateway` makes before the network places it.
`requester.json` records `withGateway`; the ready file is first written only once the Gateway is Ready too, with
`gateway {workspace, name, clusters}`;
`run requester cluster <name> --cluster <c>` makes the home workspace `root:<account>:<c>` and its Cluster, and
`run requester kubeconfig <name> --cluster <c>` prints its admin kubeconfig path once the Cluster is Ready (exit 1 before). A node fetches socialweb-meshd and
socialweb-node from `<base>/downloads/<sha256>/<name>` and checks the pinned sha256: the base is the nodes-only
GatewayAPI's `spec.publicOrigin`, set from `--node-download-base` (default the public bucket, where the CI
`publish` job puts the pinned socialweb-computer-k8s-downloads tree content-addressed under `downloads/`); with
`--network`, a dev build and the default it is the local Exposure (the requester's host only; a release build uses the bucket).
- Comms modes (`spec.comms.mode` of the MarketRequester and its Requester; `run request --comms-mode
XRPC|Firehose|Both`). XRPC takes bids and events on the requester ingress only. Firehose has no
inbound endpoint. Both (the requester default) is the ingress plus the same streams as Firehose. In
Firehose and Both the requester composer (`internal/app/compose/streams.go`, the same code in the
browser build) composes EventStreams from the network relay, `spec.comms.eventSourceRefs` and the
default public firehoses; it composes no per-bidder stream (no subscribeRepos of a bidder PDS: a
did:key bidder requestCrawls our relay and a bsky-hosted one is on the Jetstreams, and a shared
host replays every repo it holds), and it prunes the old `bidder-pds-*` sources by their label
`socialweb.computer/bidder-source`. The defaults (`api/v1alpha1` `DefaultFirehoseRelayURLs`,
`DefaultFirehoseJetstreamURLs`): our relay
`https://relay.mini-cloud-0002.chadig.com` (subscribeRepos; on a network with a relay host the network
relay replaces it) and the two Jetstream v2 instances Bluesky documents,
`wss://jetstream.us-east.bsky.network` and `wss://jetstream.us-west.bsky.network` at
`/xrpc/network.bsky.jetstream.subscribeEvents` (`collections`, `dids`, `kinds=commit`). bsky.network
is not a default. A Jetstream URL whose path ends in `/subscribe` is spoken as v1 (`wantedCollections`,
`wantedDids`); a bare host is v2. Every relay subscribe sends the stream's filter as repeated
`wantedCollections` and `wantedDids` (at most 100 and 10000, else that param is left out); our Go relay
(`internal/atproto/relay`, the network relay and prod) filters on them, other relays ignore them, and
the client filter (`events.Filter.Narrow`) always runs too. Each subscribe logs `relay_subscribe_dial`
or `jetstream_subscribe_dial` (endpoint without query, the filter params, the cursor), and the
EventStream's `status.serverFilter {collections, dids, supported: true|false|unknown}` is written on
connect only. Each is a RelayEventSource or JetstreamEventSource with its
EventSource (`firehose-relay-<host>`, `firehose-jetstream-<host>`, label
`socialweb.computer/market-source=public-requester`) and an EventStream with `spec.resume: Live`
(no stored cursor; a reconnect resumes from the cursor in memory). `spec.comms.relayUrls` and
`spec.comms.jetstreamUrls` replace a default list; an empty list turns it off. A requester stream
carries only `market.bid`, `market.receipt` and `market.event`; a bidder stream only `market.rfp`,
`market.accept` and `market.event` (`compose.MarketStreamCollections`); payload records
(`bids.free`, `compute.events.*`, ...) are read through the strongRef, never taken as stream input.
Every market EventStream of a role delivers to one handler sink (`internal/eventstream/market.go`),
which drops a `uri|cid` it took in the last 15 s (at most 10000 keys) and logs
`market_event_duplicate_dropped` once per key with the running count. The requester tracks only the
records of its own ComputeContracts (`internal/atproto/market/addressee.go`; the sink's index
`internal/eventstream/addressees.go` is fed by a ComputeContract watch, no kcp read per record, and
no DID allowlist): a bid whose `rfp` is the RFP of one of its contracts that has no accept yet, a
receipt whose `accept` is one of its accepts, an event whose `receipt` is one of its receipts or,
before that receipt is in the contract status, whose `receipt` resolves to a receipt of one of its
accepts. Before the fetch it drops a bid while no contract takes bids, and a receipt or event whose
repo is not the repo of an accepted bid of a contract that is not settled. Every StateSync and
inbound ContractEvent it creates has an ownerReference to that ComputeContract, so deleting the
contract (or the Infra or EphemeralInfra that owns it) deletes them. The bidder takes every RFP,
an accept whose `bid` is in its repo and an event whose `receipt` is in its repo (its MarketIdentity
DIDs). The rest is dropped before any object and counted in `market_foreign_dropped` (every 1000).
The roles prune their market inbox StateSyncs (Outbound, Retain: the role adapter is the target,
and a delete has no remote side): one that matches no own contract at once; the requester's of a
Released or Failed contract once Acked and 10 min old (`MarketInboxRetention`); the bidder's
every Acked one at 10 min (`requester_inbox_pruned`, `bidder_inbox_pruned`, every 100). No
inbox StateSync is a dedup fact. A bidder bids only inside an RFP's bid window
(`market.RFPBidWindow`, 10 min from its `createdAt`, else from its TID rkey; the lexicon has no
expiry), which the bidder's inbox retention equals: its sink drops a closed RFP before the record
fetch and again after it (`market_rfp_closed_dropped`, with the source, every 100), submitRfp
answers 409 `RFPClosed`, and a Deciding Bid of a closed RFP fails `BidWindowOver`
(`bidder_rfp_closed`). It never bids twice on one RFP: the handled fact is the Bid object (name
`market.BidRKey`) and, after its TTL, the bidder's own `market.bid` record at that rkey
(`bidder_rfp_already_bid`). A relay stream reconnects 1000 events back, a Jetstream 30 s back,
but never before the cursor its last connection started at, after a 1 s backoff that doubles to
30 s and resets only after a connection that stayed up 30 s; each close logs `event_stream_closed`
(the cursor, `openSeconds`, `retryIn`, `restored`, the error) and a failed dial `event_stream_subscribe_failed`.
A close the browser page fires after a memory snapshot restore (code 1006, reason
`wsconn.RestoredCloseReason` "restored from a memory snapshot", the page's `RESTORE_CLOSE_REASON`;
`wsconn.Restored`) redials at once from the cursor in memory and resets the backoff to 1 s; the
pdsclient subscribeRepos stream does the same. Any other close keeps the backoff. Our relay answers a subscribeRepos
without a cursor with live events only (the atproto sync spec); a cursor replays its backlog. The
js/wasm WebSocket (`internal/kit/wsconn/conn_js.go`) has no receive backpressure, so it queues
frames up to 16 MiB (`MaxBufferedBytes`), never by a frame count.
- The bidder's requester streams (any comms mode, `internal/app/compose/requesterstreams.go`): requester
events reach the bidder through submitEvent first; as a record-only fallback, while the bidder has
live ComputeVMs (not deleted, not Lost or Failed), it composes EventStreams with the bidder's market NSIDs
and `dids` = the requester DIDs of those ComputeVMs (at most 100, read at runtime, never configured):
sources `requesters-relay-<host>` (the network relay, else `spec.comms.relayUrls` or our relay; the
relay filters on the server) and `requesters-jetstream-<host>` (`spec.comms.jetstreamUrls`, else
`wss://jetstream.us-east.bsky.network` when the network has no relay). They deliver to the bidder's
market sink (the same intake as submitEvent, deduplicated by `uri|cid`). A ComputeVM watch
reconciles the MarketBidder only when that DID set changes; a new set is a spec change, so the stream
reconnects with the new filter. Their `spec.resume` is `Recent`: after a restart or a spec change the
stream resumes from its stored cursor only when it was saved in the last
`delivery.RecentResumeWindow` (10 min), else it starts live. No live ComputeVM, no stream.
## Guest user data: never reload systemd while a start job is in flight
- The guest image's systemd (255.4-1ubuntu8.17) PID 1 aborts in `service_sigchld_event()`
(`src/core/service.c:4054`, `assert_not_reached()` in the control-process-exited switch) when a
`daemon-reload` lands while a service start job is in flight (observed: `ssh.service` starting, a
`systemctl` of cloud-final reloading 2 ms later). `CrashAction=freeze` then freezes PID 1: every
later `systemctl` gets `Connection reset by peer` or a 25 s timeout, iroh never starts, and the
guest never reports on the network ("did not come on network within 2m0s"; the apport core pipe
can deadlock the same crash). Closest upstream issue: systemd/systemd#44705 (same assert, line 4064,
a shutdown race, closed). We do not patch systemd.
- Rule for the runcmd entries OUR modules contribute (RuncmdPrepend and Runcmd of the modules and the
overrides of `cloudinit.BuildUserData`, which also covers guestseed's seed modules): one
`systemctl daemon-reload`, standing where our first systemctl command stood; every `enable` is
`systemctl --no-reload enable`; no `enable ssh` (it runs systemd-sysv-install and update-rc.d, which
reload; the image has `ssh.socket` enabled, runcmd only starts `ssh.service`); all our `start`s come
last, after the base's entries and our appended ones. `assembleRuncmd` lays this out: our prepended
entries, the base's runcmd, our appended entries, our starts; it drops module-local reloads, rewrites
enables and splits `enable --now` into a no-reload enable plus a start. A module writes one
`systemctl` command per runcmd entry (a compound line with a reloading verb fails the build: our bug).
- The base's runcmd entries (a requester's user_data, or what an earlier BuildUserData wrote) pass through
byte for byte and never fail a rental: their VM, their user data. Each BuildUserData layer puts its
reload before its own starts, so layers stack safely. An older requester CLI builds its modules' reloads
and starts into the base it sends: the bidder cannot fix those; they need a requester release.
- A restart from the recorded state (`dev up`, a `run request release` of a killed owner) never adopts the units of a devsup that is
shutting down: its Hello says `stopping`, it refuses Start, `startOrAdoptDevsup` waits for it to exit (120 s) and starts a new one, and
`Observe` waits for a unit that is deactivating (60 s). Before that, a release started right after its owner was SIGKILLed mid-stop
adopted kine and front that devsup then stopped under the new kcp (not ready for 60 s). Testcase: composition 0017 entry `devsup` (the host runs a real devsup with no VM: `dev up --profile quick`, then `dev down` SIGKILLed mid-stop and `dev up` at once, 12 times; every kcp must be ready).
- Debugging a silent guest on a dev net: `SWC_DEV_GUEST_BOOT_ARGS_EXTRA` or the file
`$XDG_RUNTIME_DIR/swc-dev-guest-boot-args-extra` (read by the VM runtime, dev only) appends kernel
arguments; `sysrq_always_enabled=1 ignore_loglevel systemd.log_target=kmsg
systemd.journald.forward_to_console=1` puts PID 1's crash line on the serial console, and QMP
`send-key` alt+sysrq+w/t/s over the VM's `api.sock` shows what PID 1 waits for. With
`systemd.mask=apport.service sysctl.kernel.core_pattern=/var/tmp/core.%e.%p.%s` the core lands on the
guest disk (copy `rootfs.ext4` with `cp --reflink` and read it with gdb and debuginfod).
`systemd.log_level=debug` writes 7 MB to the serial console per boot and slows a boot past the
2 minute window by itself.
## Guest transports, guest events and the host-loopback fence
- Three guest transports, and no other. Never add one.
- iroh is the default and required transport: every guest gets the iroh module, the requester's
ssh and `status.contract.vmAddress` are iroh, and a contract is Held on the iroh address and the
host keys alone.
- opentunnel https, opt-in per entry of the ComputeContract's `spec.tunnels` with transport
`opentunnel` (the default): one https URL per tag for the guest's `127.0.0.1:<port>`, back as a
`vm.onNetwork` kind `https` in `status.contract.networks` (plans 0032, 0034).
- webrtc, opt-in per entry with transport `webrtc`: socialweb-guest-agent (`cmd/socialweb-guest-agent`,
`internal/guestagent`, pion and age) bridges each datachannel labelled with the tag to
`127.0.0.1:<port>`; its DTLS fingerprint comes back as a `vm.onNetwork` kind `webrtc`. The CLI
side is `run request proxy <name> --tag <tag>` (`internal/webrtcproxy`), and `run request ssh
--transport webrtc` uses it as the ssh ProxyCommand; `run request --tunnel <tag>:<port>[:<transport>]`
asks for tunnels. STUN is `stun:stun.cloudflare.com:3478` only; no TURN (plan 0034).
A tunnels.json entry maps its tag to `"port": <n>` (127.0.0.1) or `"unix": "<socket path>"`; each
datachannel is one connection to it. The guest agent's shell service (an MCP server:
`internal/guestagent/mcp.go`, `jobs.go`; tools exec, start, read, write, kill, list, as root) is
`/etc/socialweb-guest-agent/services.json` `{"shell": {"unix": "/run/socialweb-guest-agent/shell.sock"}}`
(the socket directly in the state dir): HTTP (POST /mcp, JSON, stateless) on that unix socket only,
root:root 0600 in a 0700 dir, never TCP; a services.json makes the bidder install the agent too. Reach
it over webrtc with `{"tag": "shell", "unix": "/run/socialweb-guest-agent/shell.sock"}`, over iroh with
`ssh root@guest socialweb-guest-agent relay /run/socialweb-guest-agent/shell.sock` (stdin and stdout
to any unix socket, raw bytes) or ssh streamlocal forwarding. Jobs outlive a connection.
- The browser shell: ssh over webrtc comes first; wootty is the fallback. An opentunnel entry with
`shell {envFile}` (default `/etc/wootty/wootty.env`) runs `wootty-<tag>.service` on
`127.0.0.1:<port>` behind that tag's https URL; a shell on transport webrtc is invalid. The shell
needs `spec.secretsRef`: its `WOOTTY_AUTH_TOKEN` (32 characters or more, else it does not start)
comes only in the secrets bundle, and the unit waits for `setup-guest-binaries.service` and the
secrets ready file. `run request --tunnel <tag>:<port>:opentunnel:shell[=<envFile>]` (or the
older `<tag>:<port>:<envFile>`) asks for one.
- The atirc agent (plan 0035; peer to peer, no token): `spec.agent {envFile}` (default `/etc/atirc/atirc.env`)
needs `spec.secretsRef`; with no `spec.association` the requester treats it as `{fromGuest: true}` plus
`guestTools: [socialweb-computer]` (`requester.DefaultAssociation`, every path: CLI, infra, kubectl, page, iOS). After the receipt is verified the requester (`internal/market/requester/agentpeer.go`)
generates the agent's iroh key (a 32-byte ed25519 seed) into the bundle at `/etc/atirc/iroh_key.hex` (hex, 0600),
writes envFile `ATP_USERNAME=<agent handle>`, `ATIRC_DID=<agent DID>` (0600) and `/etc/socialweb-computer/r2-agent.env`
`R2_COLOCATED=<label>-<rfp rkey>=<the key's ticket>`. The agent is a co-located account of the guest requester R2
(identity-only when nothing else asked for one; no atirc token is minted for R2 or the agent): R2 makes its did:plc
(renter rotation key first, `agent-of/<R2 DID>`), endorses and grants it in R2's repo and writes
`computer.socialweb.peer/<agent handle>` {iroh: that ticket} in the agent's repo. R2's handle is
`<label>-<rfp rkey>.<renter handle>` (the association request entry's `handle`, when the renter's handle verifies), the agent's
`<label>-<rfp rkey>.<R2 handle>`; the requester reads both from the DID documents after the nonce check and logs
`requester_agent_identity peer=<agent handle>@<agent DID>`, the owner node's `atirc mesh run --peer`. getSecrets
and the sealed fallback serve the bundle without the envFile and the key until the envFile names `ATP_USERNAME`,
and that write publishes guest.secretsUpdated so the guest fetches
again; user_data installs `atirc-agent.service`
and `/etc/socialweb-guest-agent/agent.json` but never starts the unit; it enables `atirc-agent-env.path`
on the envFile, which runs `systemctl try-restart --no-block atirc-agent.service` (never a start)
when the envFile's sha256 differs from the one the agent's run script read. A requester-originated
`guest.agentStart` makes socialweb-guest-agent run `systemctl start --no-block atirc-agent.service`; its run script
copies the key to the agent user's `~/.config/atirc/iroh_key` (0600) and runs `atirc mesh run --no-publish` beside
`atirc agent run`; with `/etc/atirc/config.toml` in the bundle (`run request --agent-config <file>`: an atirc client config, 0600,
its `[agent] harness` and `[agent.anthropic]` model, baseUrl, apiKey) it installs that as the agent user's `config.toml` (0600),
where `atirc agent run` takes its harness. `run request --agent[=<envFile>]` asks for one; `run request agent <name>` starts it. atirc is
linux/amd64 only on guests; it is the pinned source build `assets.atirc` (fetch kind cargosrc,
`scripts/build-cargosrc.sh`), also embedded for the host, where an `AtircNode` (export hostprocesses,
`internal/controllers/atircnode`, `examples/infra/atircnode.yaml`) runs `atirc mesh run` as a HostProcess (secrets
only in `spec.passwordEnvFile`, never argv; the key at `<configDir>/iroh_key`, 0600). The ssh session mints below sign with `atircSigning` of the `requester_service_auth` log (atirc only; the market
callbacks keep `signing`): the OAuth session, the owner MarketIdentity for pdsAccount (`owner-identity`), or for
localRepo its own Pds component (`local-pds`, a `pds.serviceauth` ComponentLink Requester -> Pds the MarketRequester
composes only when the Pds is public through its ingress (the DID may be on any PLC: atirc resolves it with its own `--plc-directory-url`),
`role.LocalPdsSignsAtirc`); else `none` with `atircReason`. Before the RFP of a spec.agent contract (no
`spec.atircParentRef`) the requester writes the placeholder bundle (one empty entry at envFile, 0600) when
the secretsRef path holds none (`requester_agent_bundle_placeholder`), then refuses the contract (Failed `refused:
... --atproto-handle ...`, no record) when its handle has a disallowed TLD (`.local`) or fails `identlink.HandleOK`
(it verifies both ways, or the DID document declares `at://<handle>` and a strict parent of it verifies both ways
to an ancestor in the DID's ident.link walk, so a guest requester's namespaced handle passes). An ssh gateway session (`internal/market/requester/sessionatirc.go`) whose
command starts with `atirc`, or whose client sends `LC_ATIRC=1` (`LC_ATIRC=0` turns it off), gets,
when it is not an `LC_SESSION` held one, the agent module in its user_data (so the bidder seeds
`/usr/local/bin/atirc`), and after the receipt the requester mints, with its own service auth
(lxm chat.atirc.mint, aud `did:web:irc.fedfork.com#atirc`; the ssh gateway's sign-in scope carries
`rpc:chat.atirc.mint` and `rpc:chat.atirc.revoke`), the child `ssh-<rfp rkey>.<its handle>` leased
on the session's receipt and vm.terminated, exp at most the session cap; the command runs with
`ATIRC_TOKEN=<child>` (sent as the ssh session variable `LC_ATIRC_TOKEN` and exported by the
command, never in the command line) and the child is revoked when the command ends; a held
`LC_SESSION` session gets none. A failed mint runs the
command without it and writes one stderr line. A command of exactly the argv `atirc agent run <sub>`
instead gets a token the requester mints itself the same way for the agent: `<sub>` when it is
the user's handle or ends in `.<handle>`, else `<sub>.<handle>` (DNS labels of letters, digits and
`-`, at most 253 chars; anything else is refused with one stderr line before provisioning), and
the guest runs plain `atirc agent run` with that token, so the agent joins atirc as
`<sub>.<handle>`; its mint failing fails the command, and the token is revoked at command end.
A session command with a minted token runs as the atirc agent user, not root (the guest's claude
refuses `--dangerously-skip-permissions` as root): the agent module writes
`/usr/local/bin/atirc-agent-user.sh` (creates the user named by `atirc nick` for the token, in
wheel with NOPASSWD sudo, lingering, and prints its name; `atirc-agent-run.sh` uses it too) and
`/usr/local/bin/atirc-agent-exec.sh '<command>'`, which runs the command with `runuser -u <nick>`
and `bash -c` from that user's HOME; the session waits for both scripts like the binary, and
ATIRC_TOKEN reaches only that process environment (never argv or a file).
A requester with no `vaultRef` (the per-user ssh requesters) mints into no bundle, so it skips the
atirc orphan sweep (`requester_atirc_orphan_sweep_skipped`, logged once).
The SshGateway's `spec.ssh.publicListen` (`run gateway ssh --ssh-public-listen host:port`,
standalone only) makes the SshServer bind that address itself (the edge; a fixed bind, reported as
listener `ssh-public`, next to the loopback Exposure); `spec.ssh.webListen` (`--ssh-web-listen
host:port`) binds the SshServer's web page and API the same way (listener `http-public`, the
address caddy proxies the public origin to). With the option `public-origin` (and no
owner sessions file) the SshServer serves the web sign-in at `/` over the same OpenBao atproto
plugin as GatewayAPI (`gatewayhttp.NewOAuth` over `internal/gateway/custody`, the session at
`auth/atproto session/<did>` of the ssh-gateway vault) and `/keys` (badgeBlueKeys at keyRkey in
the user's own repo, through the custody agent). A sign-in first runs the policy
`bidder-tangled-vouch` (scope lane, self `spec.ssh.operatorDid`, `--ssh-operator-did`; no operator
admits nobody); a refused or failed sign-in is signed out again and the page reads "this account
was not admitted". An admitted sign-in asks the SshServer-only handle `market.sessionowners`
(EnsureSessionOwner(did), built server side from the SshServer's vault and root labels, refused
past `spec.ssh.maxAccounts`, default 64) for the session owner MarketIdentity `owner-<stamp(did)>`
(oauthSessionProxy, `auth/atproto session/<did>`); sign-out deletes it. The SshGateway composer
applies each session owner (compose.ApplySessionOwner), prunes the kcp objects of the ones whose
session expired or past the cap (compose.PruneSessionOwners; never a repo record), and, once one
is Ready, composes a Requester `requester-<stamp(did)>` on it with its own ingress, policy, service
auth, repo links and the SshServer's stream link to it. The sign-in answers only once that link is
on the bus (`bus.AwaitLink` on the link change broadcast, the same SshServer to `requester-<stamp(did)>`
ssh run triple the session resolves; bounded by the request and `component.ReadyTimeout`, past which
it logs `sshserver_account_access_unready` and completes anyway), so a command right after it is never
told to sign in first; the ssh path still judges by the link alone. An ssh session of DID X runs on X's
requester (which refuses any other DID), so its RFP, accept and records are X's; with no link to
it the client gets "sign in at <public-origin> first", any other failure "ssh gateway error", exit
1 either way. The owner sessions file (market network seam) composes the same per-DID requesters,
one per account in the file, signed in with its password. In both modes the shared Requester
`requester` and its identity are not composed, the SshServer has `spec.accountRequesters`, and the
SshGateway is Ready without one. Every ssh gateway requester names its ingress Exposure in
`spec.marketExposureRef`, so its RFPs and accepts carry the submitBid and submitEvent callbacks,
and a per-DID requester has `spec.contractOwnerDid`: it runs only the ComputeContracts whose
`spec.ownerDid` is that DID and their StateSyncs and ContractEvents, so many requesters share one
workspace. Every ssh session, transient (`ssh-run-<hash>`, TTL 1 h, released when the command
ends unless `LC_KEEP_VM`) or held, is a ComputeContract, so bids and events reach it through the
addressee index and the ingress like any contract. A session ends with its ssh channel: when the
client closes it or the connection drops (the server sends keepalive@openssh.com every 15 s and
closes a connection after 3 missed replies), the command's context is cancelled, the guest
session closes, the session's atirc token is revoked and then the contract released (a release
is tried 6 times, 1 s doubling to 30 s). A requester's Shutdown ends its live transient sessions
the same way, and its first sync releases every `ssh-run-` contract it owns that is not Released,
Failed, already releasing, labelled `socialweb.computer/ssh-keep-vm` (`LC_KEEP_VM`), created
after its start or held by a live session (`requester_stale_session_release`); a VM whose token
could not be revoked then ends it through the token's vm.terminated lease.
A held `LC_SESSION` contract (`ssh-<hash>`) records `status.heldSession` {name, guestKeyPath
(in the requester state dir), until}: until is the session cap plus idle when a command starts
and the idle when it ends. Shutdown keeps held sessions (`requester_held_session_kept`); the first sync adopts
one whose recorded expiry is ahead, whose guest address and key are still there, with its idle
timer at the time left (`requester_held_session_adopted`), and releases the rest
(`requester_stale_session_release`), sparing `socialweb.computer/ssh-keep-vm`.
The iroh dialer socket lives in the SshServer
state dir, or in the instance runtime dir when that path is over 107 bytes; `run gateway ssh`
refuses a layout where both are. The set `ssh-gateway` binds marketidentities in root. A restart
keeps the path whole: every app root (gateway, bidder, requester, network, ssh gateway), when it
is not suspended, renders its set's HostProcesses and OpenBaoServers each pass
(compose.SetWorkloads, CheckWorkloads). A missing one, or a HostProcess Lost because its unit is
gone, the host rebooted, its manager or devsup restarted or its identity mismatched (the
supervisor already stopped a mismatched unit, and a unit name runs once), which the root deletes,
voids the applied-set shortcut and the set is applied again; one that is being deleted holds the
pass until it is gone. A per-DID account requester stays composed, with the repo and RemotePDS
links of its last applied PDS URL, while its session owner is Pending or unresolved; it is
dropped only when the session expired for a reason that is not one the plugin keeps active, the
owner signed out or the cap is reached. In per-user mode the SshServer has no requesterRef and
the shared requester no links. The SshServer's Start waits, without polling, until its vault's
OpenBaoServer is Ready (`sshserver_waits_for_vault`; woken by the vault provider's bus NotifyBound,
whose binding fingerprint carries the server's readiness, so a Starting to Ready change notifies)
instead of exiting. A requester that publishes its signer key binding (owner-session and OAuth
identities) retries that write on every Vault, MarketIdentity or AtprotoRepo bind
(`requester_signer_key_waits_for_vault`) instead of exiting while its identity's vault is
Starting, and fails the start only when no bind comes within 2 min; the MarketIdentity controller
sends a bus NotifyBound when an identity turns Ready, because the session it serves comes up
after the Vault bind that woke the requester. The SshServer's web sign-in follows its vault: a rebound Vault
(bus NotifyBound from the vault provider) rebuilds the custody sign-in on the new address, with
at most 6 attempts at 1 s doubling to 30 s when the open fails; the MarketIdentity controller
reconciles on the same signal and never drops it. The atproto plugin keeps a session active when
a refresh is refused with invalid_client, invalid_client_metadata, unauthorized_client,
use_dpop_nonce, server_error, temporarily_unavailable, or HTTP 429 or 5xx without a code, and
when the refresh never reaches the server; every other answer (invalid_grant after the rotated
token check, invalid_dpop_proof, invalid_request, any other code) expires it. A session an older
plugin expired for one of the kept codes is refreshed again on a read or a proof, at most once a
minute per DID. A release whose destroy record or submitEvent keeps failing settles as Released
once the bidder's vm.terminated for that receipt appears, whatever its reason.
- The requester association of a guest (plan 0036 A5): `spec.association {subject | fromGuest, refFile}`
(default refFile `/etc/socialweb-computer/association-ref`, under `/etc/socialweb-computer/` only)
needs `spec.secretsRef`. With a subject the requester writes a contract-scoped market.association
after the accept; with fromGuest, user_data writes `/etc/socialweb-guest-agent/association.json`,
the bundle carries `/etc/socialweb-computer/association-request.json` {authority, contract (the
accept uri), nonce}. The bundle files are root 0600 and a dev requester refuses root, so
socialweb-guest-agent makes the group `socialweb-r2` (user_data does too; the atirc agent user joins
it) and `/run/socialweb-r2` (root:socialweb-r2 2770) and mirrors `association-request.json`,
`r2-atirc.env` (when the bundle holds a non-empty one; an `--agent` contract has no token) and then
`association-ref` (empty until the ref is written) into it 0640 on start, on a bundle change and
after a refetch. The requester inside the guest (`run request --association-ref`, bare, as the
agent user; it reads `/run/socialweb-r2` when that dir exists, `/etc/socialweb-computer` otherwise),
finding the ref missing or empty, writes the request as a market.associationRequest record in its
own repo at rkey `requester_associate-<sha256(authority)[:32]>-<accept rkey>`, then its DID to
`requester-did` in that dir (`/run/socialweb-guest-agent/requester-did` in the root case);
socialweb-guest-agent posts guest.requesterDid (did:plc
only, never the requester's own DID; a did:web is refused because its resolution fetches a host the
guest names); the requester records the first DID as Pending in `status.contract.guestRequester`
before it proves it (a second DID is refused, a give-up turns it Refused); the requester reads that request record by one getRecord in the reported
repo and writes nothing unless authority, contract and nonce match exactly; then it writes the
record at rkey `requester_associate-<sha256(subject)[:32]>-<accept rkey>` (never over a record of another
contract, and release deletes only its own), the ref into the bundle, publishes
guest.secretsUpdated, and the guest agent fetches the bundle again (`SECRETS_REFETCH=1
setup-secrets.sh`, or the sealed fallback). A failed step retries with a bounded backoff and shows in
`status.associationMessage`. chat.atirc.mint never gets the association as `assoc` (atirc wants the
standing rkey and the minting account as subject); every token is minted with an explicit exp
(at most 24h, the parent token's exp and, for the guest requester, the association expiresAt). The
agent token is one token per subhandle (named after the RFP rkey) minted after the receipt. Once a fromGuest association is
written, the requester also mints the guest requester its own child token, subhandle
`r2-<accept rkey>.<the requester's handle or subhandle>`, into the bundle at
`/etc/socialweb-computer/r2-atirc.env` (`ATIRC_TOKEN`, `ATIRC_SUBHANDLE`, `ATIRC_EXP`, mode 0600);
`source` it to run `atirc sub` / `atirc pub` as that subhandle in the owner's channel. A requester
whose root names `spec.atircParentRef` (a vault secret {token, subhandle, exp}; the guest requester's
`run request` writes it from that file) mints its own agents' tokens below that subhandle with the
parent token as bearer instead of service auth. Release revokes the agent and the guest requester
tokens in the background with a backoff capped at 5 min until each is revoked or expired (24h at
most after release, from the release time kept in a marker in the bundle secret), starts only
once that marker is written, stops with the requester and resumes after a restart from the marker;
every mint is also recorded (path, subhandle, token, exp) under `atircMinted` in the bundle secret,
and a minted subhandle counts as revoked only on a 200, a 401 for its own token, or its exp; a 401
counts for an unrecorded (rc126) token only for a subhandle read from the env file. Every revoke
of an `atircMinted` entry (release, orphan sweep, a refused mint) and of an env-file token whose
subhandle the env file names is sent, when the subhandle equals the token's own `subhandle` claim
exactly (the server answers 200 for any name in the caller's scope), first with a service auth
for chat.atirc.revoke (lxm `chat.atirc.revoke`) when the requester identity is OAuth, the
subhandle is below the requester's handle and the session's granted scopes (the token response,
never the access token's scope claim) include `rpc:chat.atirc.revoke` (RequesterOAuthScope asks
for it, but it is in `oauth.OptionalScopes`, so the identity's scope check does not require it
and a session granted without it is not asked to sign in again; such a session never asks for
that service auth; the MarketIdentity stays Ready with `status.pendingAction {kind: signInAgain,
reason: ScopeUpgrade, message, missingScopes, handle}`, plus `url` and `expiresAt` for mode
oauthBrowser (an authorize URL for the full requested scope, reissued when it expires), cleared
once a session grants the full requested set), then, for an `atircMinted`
entry, with the minting parent's
bearer, the `spec.atircParentRef` long token, when the subhandle is below it; an entry revoked
by a 200 is stamped `revoked` there and not revoked again, while a 401 counts as Done but leaves
it unstamped: an unstamped entry stays in the bundle and its bundle in the orphan-sweep index
until its exp, the sweep revokes it again on each requester start and sweeps once more at the
latest such exp, and a bundle is finished for the sweep only when every entry of its contract is
stamped or expired (the release marker stays until then). A derived subhandle, a claim mismatch, no grant and no parentRef, or each of them
answered 401/403 uses the token itself; a leased token is revoked before
vm.delete, while its lease is alive, and a 401 saying the lease is not alive counts as revoked
(`requester_atirc_revoked` byLease). The first
attempt runs before vm.delete is recorded when the bundle holds a leased token (after the guest
requester's release), else before Released is written, or Failed when no guest can still run (no
receipt, an acknowledged release, or the release bound), and its outcome is `status.atircRevokes` (Done only
when every token path is revoked that way or never held a token; Pending while retries run;
Abandoned after 24 h); only that first write and the retries set it, a later status write never
does, and no token is minted once it is set. `run request` and `run request release` share one
settle step before they remove an invocation: they wait up to `cloudinit.RequestRevokeWait`
(15 min) while it is Pending or unset on a contract that names token paths; on that timeout they
exit non-zero and keep the invocation's state, and a rerun of `run request release` starts the kcp
so the requester resumes the revokes, from the marker or, with no marker, over every token path
again. The process that starts an invocation's kcp (`run request`, `resume`, or a `release` with no
other process) owns it through `requests/<name>/owner.lock` (flock) and is the only one that
stops it. A `run request release` that finds the lock held asks for the release through the
running kcp, waits for Released (exit non-zero otherwise), then for the owner to exit, and stops
nothing; a `--hold` owner whose contract leaves Held with `spec.release` set keeps its kcp until
the release settles, then stops (and removes) it as its own release would. A refFile, agent envFile or `--secrets` path equal to a token path is refused (CLI and
requester admission). Every release request stamps `spec.releaseRequestedAt` (SSA, the release
field manager) and the outcome it answers is `status.release.requestedAt`; a denied accept whose
receipt never arrives leaves the contract Failed with ReleaseFailed, and a new release request
restarts its runner, looks for the receipt and releases again; the CLI waits for an outcome of its
own request (with the receipt waits added to its release wait). Every read-modify-write of a
contract's bundle holds one lock per contract on the requester. Revoking a subhandle revokes every
token below it. `spec.atircParentRef` must name `identity/atirc-parent` in the requester's own
vault and mount.
`run request --associate <did>|guest` asks for one. Inside such a guest, the agent user starts the
guest requester with one command, `socialweb-computer run request --name <name> --guest-requester
[--agent --hold]`: it implies `--association-ref` on `/run/socialweb-r2`, `--skip-ssh` and the
parent atirc token, which it uses only to mint its own agents' tokens (the guest requester runs no
`atirc sub` of its own). The atirc agent run script enables linger for the agent user and runs the agent
with its `XDG_RUNTIME_DIR`, so a dev requester can run as that user. With no command in the guest,
`run request r2 <name> [--r2-name r2] [--agent] [--hold]` publishes guest.r2Start and the guest agent
starts `socialweb-r2.service` as the user `r2`; releasing the parent publishes guest.r2Release first (also when no guest requester DID was reported, once the guest is on the network: a guest requester that failed before its hand-off still retires, so its did:plc is tombstoned with its own PDS rotation key and an OAuth renter with no renter key leaves no orphan)
and waits for guest.r2Released after `socialweb-r2-release.service` released the guest requester's
contracts. The bounds nest, from the constants in `internal/atproto/cloudinit`: each guest
`run request release` gets RequestReleaseWait + RequestRevokeWait + 2 min, the release unit 1 min
more, the cascade (GuestRequesterReleaseTimeout) the unit stop plus 1 min more, and the parent's
`run request release` waits the cascade plus `--contract-timeout`. An identity-only guest requester (no r2Start
for the contract) rents nothing: the guest agent sends SIGTERM to the unit's main process only (the run
script execs `socialweb-computer`; a plain `systemctl stop` signals the whole cgroup, its kcp too, and the
kcp runs the retire), waits for it to exit (it retires its accounts on stop, bounded to 1 min, and its
summary says Released), then runs the release unit as for a renting one (it releases, and so retires, any invocation a crashed guest requester left) and answers guest.r2Released, all within
GuestRequesterStopTimeout (2 min), and the parent waits for that answer GuestRequesterIdentityReleaseTimeout
(the stop plus 1 min) before it goes on; the outcome is
`status.contract.r2Release` and the GuestRequesterReleased condition. A bidder releases a VM whose
requester's contract-scoped association expired 10 min ago (ComputeVM `spec.associationExpiresAt`,
at most 24 h after the accept, the earliest `expiresAt` of the proof's `parent` chain, only from a proof
in another repo than the requester's that names the requester as subject and an accept of a repo of the
chain as contract; each skip is logged), and, every `AssociationRecheck` (1 min, 20 s bound), a VM whose
association's contract is over on affirmative evidence only: for each contract recorded at accept (ComputeVM
`spec.associationContracts`: the grant's `contract` accept and its receipt; all or nothing, and nothing for a requester
with any standing grant) the outer bidder's `vm.terminated` or the
outer requester's `vm.delete` for that receipt is present at its deterministic rkey (a 404, a lookup error, a DID that
does not resolve or an empty walk keep the VM); `spec.release.reason` `association_ended`, the `vm.terminated`
reason. The guest requester's PDS requestCrawls our relay only
(`DefaultMarketRelayURL`, when its spec names no network relay). `run request --guest-requester
--colocated a,b` (the r2 unit passes `R2_COLOCATED` from `/run/socialweb-r2/r2-start.env`) makes
each `<label>.<its handle>` an account on its own PDS (`spec.peer.colocated` of the MarketRequester),
associated by a market.association in the guest requester's repo whose `parent` is the owner's
association of the guest requester (same contract and expiresAt), plus the account's keys mirror;
every verifier follows `parent` (maxDepth 1 to 100, default 100) with the expiry and contract checks at
each link, and `socialweb-computer internal association <did>` prints that verdict. It also writes
`computer.socialweb.peer/<subhandle>` {lease, iroh, webrtc, channels} in its own repo and each
co-located one (`docs/protocol/lexicons-market.md`). Its `run request release` sets `spec.peer.retire`
before its kcp goes: those records are deleted, every local account deactivated and its did:plc
tombstoned, best effort; trust never depends on that cleanup (vm.delete, vm.terminated and expiresAt
end every link).
- The bidder installs the guest binaries, never the requester and never the internet. The requester's
tunnel module carries config only; the bidder adds `setup-guest-binaries.service`
(`internal/guestseed`), which fetches opentunnel, wootty, atirc or socialweb-guest-agent from its own
GuestControlPlane `GET /v1/downloads/<sha256>/<name>` at `http://192.0.2.1:<port>` (a local
Exposure), checks the sha256 and installs to `/usr/local/bin`. The binaries are the release asset
`socialweb-guest-downloads` (`scripts/build-guest-downloads.sh`, which takes wootty from its pinned
ghcr.io image layer at build time). A guest never fetches from GitHub or GHCR. The did:key ingress cannot carry binaries (JSON string bodies, a frame cap).
The one opt-in exception to that asset: a contract whose `spec.guestTools` names `socialweb-computer`
(`run request --guest-tools socialweb-computer`) gets the bidder's own running binary
(`/proc/self/exe`, hashed once per process, linux/amd64 bidders only; any other bidder refuses at
provisioning), served streamed at `/v1/downloads/<sha256>/socialweb-computer-x86_64` and installed by
the same unit as `/usr/local/bin/socialweb-computer`. The requester marks it in user_data as
`/etc/socialweb-computer/guest-tools/<tool>`. Across a self-update a guest still gets what the old binary's user_data
names: a request the GuestControlPlane does not pin is served from what the install keeps, `<install>/bin/<sha256>/`
(its own binary) or an older `socialweb-guest-downloads` tree whose sidecar names that sha256, once the file hashes to
it (`internal/services/guestcontrolplane/downloads.go`; testcase `0007:upgrade`, staging). So any pruning rule must
keep binaries for the guest download window: `install.GC` keeps `bin/current`, `bin/previous` and every binary a
process or unit names, and releasefollow never prunes (docs/plans/0040-findings.md, "THE SELF-UPDATE FIX DEPENDS ON
KEPT FILES").
- An Exposure in mode Ingress is published through the IngressProxy its `spec.ingressRef` names (the bidder's market
endpoint over iroh: Exposure `svc-bidder-bidder-iroh` behind IngressProxy `ingress-iroh`; the requester's
`requester-iroh`). Each start of a component's service (a new generation, or a rebuild after it exited) gets a new
`component.Entry.Attempt` in the registry; the exposure publisher's fingerprint holds the serving IngressProxy's
workspace, name, UID and Attempt, so the registry's change event re-registers the Exposure on the new instance (the
old instance's registrations die with it); between the two the Exposure is not Ready (IngressNotReady), and it is
Ready only after the provider's Register returned (iroh: dumbpipe printed its ticket,
`ingressproxy_iroh_dumbpipe_listening`); `exposure_ingress_instance_changed` logs each move. The iroh provider runs
the dumbpipe of the running binary's own instance record (devcore: `componentcore.Config.DumbpipePath`); the compose
passes no longer write `spec.iroh.dumbpipePath`, and a stored one is used only by a core whose record names no dumbpipe
(a core started on another binary's stored spec ignores its path). Testcase `0007:ingress` (staging; docs/plans/0040-
findings.md, "A BIDDER GOES DEAF OVER IROH WHEN ITS STORED DUMBPIPE PATH CHANGES").
- Guest events live under `com.publicdomainrelay.temp.compute.events.guest.*` in the market.event
envelope. The guest holds no repo key: it POSTs `{acceptUri, acceptCid, record}` to GuestControlPlane
`POST /v1/events` (vm.onNetwork, guest.publicKey, guest.signalWebRTC, guest.secretsRequested,
guest.secretsApplied, guest.requesterDid, guest.memoryPressure, guest.oomKill; never guest.secrets, guest.agentStart or
guest.secretsUpdated) and the bidder signs and publishes them; requester events to the guest
(guest.publicKey, guest.signalWebRTC, guest.secrets, guest.agentStart, guest.secretsUpdated) reach it through the bidder's per-ComputeVM queue
and the XRPC subscription `com.publicdomainrelay.temp.compute.guest.subscribeEvents` (WebSocket,
`cursor`, JSON text frames, an empty binary keepalive every 25 s). The guest bearer is the
`Authorization` header only. The bidder routes by NSID prefix and never opens a sealed body.
- Guest memory. The guest image swaps to zram by default (zstd, half the guest's RAM: the generator
`/etc/systemd/system-generators/socialweb-zram-swap` and `/usr/local/sbin/socialweb-zram-swap`; kernel PSI on).
Every VM whose bidder serves socialweb-guest-agent gets `/etc/socialweb-guest-agent/memory.json`
(`guestseed.ProvisionOptions.Memory`), so the agent runs its memory watch (`internal/guestagent/memory.go`,
state in `/run/socialweb-guest-agent/memory-state.json`): edge-triggered guest.memoryPressure (PSI some avg10
>= 10% or swap >= 10% in use; re-armed after 5 min under half the threshold) and guest.oomKill per oom_kill
rise; one per kind per 10 min and ten per contract, enforced again by the bidder from its StateSyncs
(`memoryEventCapped`; a capped event is answered 200 and dropped). The requester logs each market event
of its contracts (those, vm.memoryPressure, vm.oomKilled, vm.terminated with its reason) as one
`requester_market_event` line; swc-test-host forwards it to the composition like Held, and the test modules
decide (`steps::market_event`, first in every module: an OOM kill fails the run naming the VM, pressure goes to
readings.memory, which gate.sh lists). No memory policy in the host.
- Requester images. `run request --image <https base>@<image.json sha256>` (ComputeContract `spec.image {url, sha256}`,
compute.vm `image`) rents a VM booted from the requester's full image (guestimage.Manifest shape plus `marker`; kernel,
rootfs and bootArgs its own, on the bidder's VMM). Only offerings with `images` are asked or accepted (`Bidder.Lacks`).
The bidder checks image.json before it bids (`imageRefused`); the FirecrackerVM fetches the files into the image
store (`Store.EnsureRequested`, content-addressed) and a refusal fails it as lostReason ImageRefused (vm.terminated
`image_refused`). The guest agent posts one guest.image {marker} from `/etc/swc-image`; the requester logs
`requester_guest_image` {marker, expected, ok}. A refusal is re-placed (`MaxAcceptReplacements`, the refusing bidder
denied): a bidder declines an accept whose image it refuses at provision (`imageRefused`, before its reservation), and a
wrong or missing marker or a vm.terminated `image_refused` releases the VM and re-places (`replaceWrongImage`; after a
requester restart `resumeReceipt` only fails the contract). An image contract waits `RequestedFetchTimeout` more than
`DefaultVMReadyTimeout` for the guest (the first fetch of a 4.8 GB rootfs). Verified image.json fetches are cached per
(url, sha256); `Store.Verify` trusts the `.verified` stamp (sha, size, mtime per file) of an image dir, so a reused rootfs is
not hashed again. Requested image dirs carry `.requested` and `.used` (touched on every Prepare): `Store.Collect` keeps them
by last use within `requested-image-budget` (`run bidder`/`install bidder`/`dev up`, `vmruntime.Config.RequestedImageBudget`,
default 16G), evicts the least recently used only over the budget or with under `DefaultMinFree` (8G) free, never one
a FirecrackerVM or work dir names or one used in the last `UsedGrace` (10 min); `FirecrackerVM.status.image.fetched` says a
boot fetched its image (composition 0016 asserts the second rent does not). Fetch guard and its limits: docs/FEATURES.md Known limits. The CI toolchain image
(`deploy/vm-images/ci-toolchain.json`, `scripts/build-ci-image.sh`: the pinned base kernel and rootfs with
`knotless-ci-toolchain.sh --bake` run in it, published content-addressed under `vm-images/ci-toolchain/<image.json sha256>/`)
is the image of the knotless build binding; composition 0016 (`run`, `badsha`) is its testcase.
- Signaling is sealed. Each side publishes a `guest.publicKey` (kind `age-x25519`) per tag; an offer
is age ciphertext sealed to the guest's latest key, with plaintext `{sdp, replyTo}`; the answer is
sealed to `replyTo`. A signal names no recipient: a reader trial-opens every signal for its tag.
No private key, SDP, ICE candidate or IP address is ever in a record, a ContractEvent, a StateSync,
kcp or a log. The requester side of each event is a cluster-scoped `ContractEvent`
(`socialweb.computer/v1alpha1`, outbound ones made by the CLI or the browser, inbound ones by the
requester), owned by its ComputeContract.
- Secrets. `spec.secretsRef` names a bundle in the requester's OpenBao (`[{path, value, mode?}]`). One rule
(owner, decision 66), checked by the requester (`CheckSecretEntry`: `ParseSecretsBundle`, the CLI's `--secrets`)
and by both guest writers (setup-secrets, the agent's sealed writer): a clean absolute path (equal to its cleaned
form, not `/`, no NUL), a mode that is an octal string of at most 0777 (default 0600), a non-string value written as
its compact JSON text; one bad entry refuses the whole bundle, nothing written; only missing directories are made
(root 0700); each file root-owned, written to a temp file in its directory and renamed (a symlink at the path is
replaced, never followed). The channel, in order (owner, decision 64): a peer transport, then https, then the
sealed record only in comms mode Firehose. getSecrets is one XRPC service with one check (the workload identity
token's audience and subject, `secretsState.fetch`); the transports only carry its requests. With an inbound
endpoint the requester writes `/etc/socialweb-guest-agent/requester.json` `{iroh, webrtc}`: `iroh` the ticket of
its Service `svc-requester-<name>-iroh` (handler `iroh`) behind IngressProxy `requester-iroh`, `webrtc` the id of
its guest WebRTC ingress, which offers to the guest over the sealed guest signalling (tag `requester`); the file
makes the bidder install the guest agent, which serves `/run/socialweb-guest-agent/requester.sock` (root 0600) and
forwards over iroh, then WebRTC; setup-secrets tries the socket, then GETs
`/xrpc/com.publicdomainrelay.temp.compute.secrets.getSecrets/<contract>` from `status.secretsUrl`. The requester
records the channel (`status.secretsFetchedVia`, reason `GetSecretsIroh|WebRTC|HTTPS`, log
`requester_guest_secrets_fetched` `channel`, every fetch; status keeps the first) and fails the contract when no fetch
came within `DefaultVMReadyTimeout` (2 min) after `waitGuest` returned (not for a resumed contract with
`status.contract.secretsApplied`); `run request` then releases the VM and exits non-zero, `--hold` or not, as it
does when a Firehose contract's sealed secrets never arrive. The socket forwards every method and path, follows no redirect and drops hop-by-hop headers (decision 72); setup-secrets waits for it only when
`bidder.json` says `"requester": true`; the iroh Service, IngressProxy `requester-iroh` and Exposure exist only while
a contract with a `secretsRef` is live; a path with a control character is refused. With an inbound endpoint the requester records a
`guest.secretsRequested` and never acts on it (`requester_guest_secrets_not_sealed`), fails an unpublished outbound
`guest.secrets` ContractEvent and refuses a `guest.secretsApplied`. Without one (comms mode Firehose; the
embedded and browser requesters, sealed until after the two rolls) the sealed record runs when
`spec.secretsFallback` is `Sealed` (`run request --comms-mode Firehose --secrets-fallback Sealed`,
`Config.SealedSecrets`): user_data writes `/etc/socialweb-guest-agent/secrets.json` `{"fallback": "sealed"}`; the
agent asks only on exactly that object (an older requester's `"getSecrets": false` too; `"getSecrets": true` or
anything else turns it off, `guest_secrets_sealed_refused`), publishes `guest.publicKey` tag `secrets`, posts
`guest.secretsRequested`; the requester then, and only then, seals the bundle to that key as the outbound
ContractEvent `secrets-<contract>` (`guest.secrets`, resealed for a newer key while unanswered); the agent writes
the files and posts `guest.secretsApplied {via: sealed}`. Either path ends in
`/run/socialweb-guest-agent/secrets.ready`; a unit that consumes secrets orders itself
`After=setup-secrets.service socialweb-guest-secrets.service` with `Wants=socialweb-guest-secrets.service`
(`cloudinit.SecretsReadyUnits`). Never send a sealed bundle unasked: published ciphertext stays on the protocol
and can be broken later. A Firehose requester with `None` and a secretsRef fails the contract
("secrets need the requester ingress (getSecrets) or secretsFallback: Sealed").
- The guest fence. socialweb-vmexec loads the nf_tables bridge table `swcguest` in the guest's
namespace before the tap exists, and the VM does not start without it (`internal/guestnet`,
`fence_linux.go`). On frames from the guest's tap it keeps, in order: TCP to the host's loopback at
`192.0.2.1` only on the ports of the ComputeVM's `spec.hostPorts` (every other TCP connection to it
is reset, the rest dropped); DNS (udp and tcp 53) to the pasta gateway 172.30.0.1; ICMPv6 router and
neighbor discovery. It then drops everything to the bidder host's own addresses (`vmruntime`
lists them in the host namespace at Prepare and writes them into the vmexec params) and to the
guest link 172.30.0.0/30, and to 0/8, 10/8, 100.64/10, 127/8, 169.254/16, 172.16/12, 192.0.0/24,
192.168/16, 198.18/15, 224/4, 255.255.255.255, ::1, fc00::/7, fe80::/10 and ff00::/8, and every
ethertype but IPv4, IPv6 and ARP. TCP to the LAN is always closed. One allowance, on by default:
UDP to 10/8, 172.16/12, 192.168/16 and fc00::/7 on destination ports 1024 and up, except 1900,
5353 and 5355, so same-LAN WebRTC can use a LAN candidate; it never reaches the host's own
addresses, 100.64/10, 169.254/16 or 127/8. It is the ComputeVM's and FirecrackerVM's
`spec.guestNetwork.lanUDP` (unset: true), which the bidder writes from its own
`spec.guestNetwork.lanUDP` (MarketBidder, Bidder, NetworkApp), the bidder operator's choice and
never the requester's: `run bidder`, `install bidder` and `dev up` take `--no-guest-lan-udp`, the
hono-bidder passthrough takes `--guest-lan-udp=false`. A second allowance, `hostUDP`, comes before
the drop of the host's own addresses: UDP to those addresses (never loopback or link-local, never
TCP) on the same ports, so WebRTC works when the requester runs on the bidder's own host (the
router does not hairpin). It is the ComputeVM's and FirecrackerVM's `spec.guestNetwork.hostUDP`
(unset: false), which the bidder always writes as its effective value: its own
`spec.guestNetwork.hostUDP` (MarketBidder, Bidder, NetworkApp) when set, else on when its
`spec.policy` is `bidder-only-me` and off otherwise; the Bidder's `status.guestNetwork` reports
`{lanUDP, hostUDP, hostUDPReason}` (`spec`, `policy only-me`, `policy not only-me`). `run bidder`,
`install bidder` and `dev up` take `--guest-host-udp true|false|auto`, the hono-bidder passthrough
`--guest-host-udp`. The one inbound path is the forwarded webrtc UDP port: for a guest with a
webrtc tunnel the bidder leases one UDP port per ComputeVM from its `spec.guestNetwork.webrtcUDPPorts
{min, max}` (MarketBidder, Bidder, NetworkApp; default 31000-31999; a udp port registry under its
state dir, `webrtc-udp-ports`), writes it as the ComputeVM's and FirecrackerVM's
`spec.guestNetwork.webrtcUDPPort`, and writes `{"webrtc": {"udpPort", "hostAddresses"}}` (the
host's RFC1918 and ULA addresses, `guestnet.HostLANAddresses`) into
`/etc/socialweb-guest-agent/bidder.json`. pasta forwards `-u <lan addr>/<port>` to the same guest
port, where socialweb-guest-agent listens with one pion UDPMux and rewrites its host candidates to
those addresses, so a same-LAN peer (mDNS-only host candidates, no hairpin) reaches the guest. The
fence accepts UDP from that source port to the host's own non-loopback addresses and to the LAN
prefixes on ports 1024 and up except 1900, 5353 and 5355, with `lanUDP` and `hostUDP` off too; no
other UDP port is forwarded, and the addresses never go into kcp. Owner-approved (2026-11-19): one more accept rule on
every guest, before the drop of the host's own addresses: UDP from the guest's tap to the host's own non-loopback,
non-link-local addresses with a destination port in the bidder's `webrtcUDPPorts` range (except 1900, 5353, 5355, which
are never leased: `internal/market/bidder/provision.go` webrtcPort releases and refuses a port that fails
`guestnet.ValidWebRTCUDPPort`, so the rule equals the leasable set;
written as the ComputeVM's and FirecrackerVM's `spec.guestNetwork.webrtcUDPPorts`), so a guest reaches another guest's
forwarded webrtc port on the same host; it reaches ANY host UDP socket bound in the range (other guests' forwards and
pasta's outbound sockets for them, other bidders' leases, the bidder's own `bidder-mesh` endpoint, which binds the
wildcard address); it holds with `lanUDP` false too, and `EffectiveFor` always fills the default 31000-31999, so every
guest of a new bidder gets it; every other port on the host's own addresses stays dropped. One owner-approved exception (2026-11-15, plan 0050):
a market-network bidder's guests may open TCP to that network's own address (its `/32`, any port), so they reach the
network's services by name as its VMs do; a bidder outside a network has no such rule. A guest-facing local listener the bidder hands to a guest goes into
`spec.hostPorts` (`GuestHostPorts`, `guestseed.HostPortsOf`); never widen the fence to all ports,
and never open TCP to the LAN or to the host's own addresses.
- A guest's rules follow the running bidder (owner decision 75, 2026-11-20). vmexec loads `swcguest` once, at boot, so a
guest booted by an older core kept that core's rules after a self-update (promote 93fd8c11: decision 54's rule was not
on the promote's own VM). Now the first Running reconcile of each FirecrackerVM in a core process (`firecrackervm`
controller, `firecracker.Provider.Rewire`, `vmruntime` `Rewire`) replaces that guest's `swcguest` in ONE nf_tables
transaction (add table, delete table, add table, chains, rules: never a moment with no rules or with wider ones) by
exactly the rules a guest this core boots now gets: boot (`guestnet.Fence`) and reload (`guestnet.Refence`) build them
with one function (`fenceMessages`) from one wiring (`vmruntime.planFor`): the VM's own `hostPorts`, `webrtcUDPPort` and
`networkTCP` from its FirecrackerVM spec, the bidder-level `lanUDP`, `hostUDP` and `webrtcUDPPorts` from the running
bidder's `EffectiveFor` (`computevm.Config.GuestNetwork`: `run bidder` from the MarketBidder spec it applies, a market
network's bidder app from its NetworkApp; a VM runtime with no bidder, the `vm` commands, keeps the spec's), and the
host's addresses read at that moment. A VM this core booted itself is not reloaded; an adopted one is, once per VM
invocation per core process (a restart with no version change reloads byte-identical rules). Logged
`guest_rules_reloaded` (firecrackervm, unit, vmmPid, invocation, the wiring); a failure is `guest_rules_reload_failed`
(firecrackervm, unit, err) and the reconcile returns the error, so the controller's backoff retries it; the guest is
never stopped for it. The rules match no conntrack state, so a flow in flight passes the new table as it passed the
old one (a guest's iroh ssh session stays up across a reload: 0007:fence). How the bidder reaches the namespace with
no new privilege: `swc-userns-stage` creates the VM's user namespace as the bidder's own uid, so the bidder owns it and
(the kernel's owner rule) holds every capability in it and in pasta's child user namespace, which owns the guest's
network namespace. Go cannot `setns(CLONE_NEWUSER)` (it is multithreaded), so the bidder runs its own release's stub
(the path its assets resolve, never the old install's) as `swc-userns-stage join`: exactly that argv, fd 3 a pidfd of
the VM's recorded VMM pid (the unit's identity checked by the manager, then the pid pinned and its start ticks compared
to the record: `hostsvc.PinProcess`; a recycled pid fails and is retried, never applied to a guess), fd 4 one end of a
unix socketpair. The stub calls `setns(3, CLONE_NEWUSER|CLONE_NEWNET)`, opens one `NETLINK_NETFILTER` socket, sends it
over fd 4 (SCM_RIGHTS) and exits: it runs no command and enters no other namespace. The bidder sends the batch over
that socket from its own process. Rollback: a binary with this code re-applies its own (older) rule set on adopt; a
binary from before it (688c2105 and older) has no `join` and reloads nothing, so after a rollback to one its adopted
guests keep the newer rules until they end. Testcase: composition 0007 entry `fence` (staging only: it upgrades a
bidder under a running guest from its own pinned from binary, `release:v0.0.0-rc186` = 688c2105, never the previous rc).
- After a change to `cmd/socialweb-vmexec` or `internal/guestnet`, rebuild with `--dev --publish` (nothing to
re-pin: socialweb-vmexec is built from the commit and pinned nowhere). A `dev restart ... core` reloads every running
guest's `swcguest` from the new core (above); a change to the rest of `Wire` (bridge, tap, routes) still needs `dev
down` and `dev up` the network.
## Tests: Rust system integration tests
- Tests are Rust system integration tests (`tests/system`, harness `tests/testkit`), plus the VMM wrapper and meshd
integration tests. Never unit tests: no mocks, fakes, golden files, snapshots, ratchets, inventories or source greps.
- The runbooks (`docs/acceptance`) are the spec. Each test in `tests/system/tests/acc_NNNN_*.rs` names the runbook
section it implements; a runbook change and its test change land together.
- New behavior gets a test in `tests/system`.
- Tests are atlibs compositions: step-loop modules (`tests/modules`, `tests/compositions/<runbook>`), host handlers
in Rust (atlibs-kube, atlibs-ssh, `tests/host`); `tests/testkit/README.md`. A runbook's Rust test stays until its
composition passes on staging and prod.
- A failed composition run freezes (the default outside CI; knotless CI's `test` freezes too): attach with
`swc-test-host attach <run dir>` (MCP), read `failure`, `state`, `resources`, inspect with `kube_get` / `ssh_exec`, then
`cleanup`; `swc-test-host replay <run dir>` reruns the module's decisions offline. Never leave a frozen run past its TTL by hand.
On a held staging network: rerun one entry with `--on-fail freeze`, swap in a release build of a fix (never a `--dev`
build: its requester never comes Ready), and get into that entry's guests through its control socket's `ssh_exec`; the
copy-paste commands are docs/ci-debug.md, "Iterate on a held staging network", steps 5 to 5c.
Before queueing a pull read docs/ci-debug.md "Mistakes that cost a cycle": its checklist is the bar for "run out of band".
- atlibs pins: `tests/Cargo.toml`'s atlibs-host, atlibs-iroh, atlibs-kube, atlibs-ssh (one atlibs repo rev: they are its workspace's crates) and identlink follow one tested
set, atlibs `pins.json`, as the knotless runner does (fork branch `runner-atlibs-composition`, deployed by
knotless-fork-infra `versions.env`; each of atlibs and knotless-fork-infra checks its own pins on push in its
`pins` workflow). `pins.sh check` here fails when tests/ is behind; catch up (rewrites the revs,
`cargo update --workspace`, `cargo check`), then commit:
`curl -sSf https://knotless.dev/johnandersen777.bsky.social/atlibs/raw/main/pins.sh | sh -s -- sync`.
A host older than its modules refuses them before they run: `this host lacks <nsid>; update atlibs-host (pins
sync)` (2026-11-13: a runner without `identity.resolveDid` made the requester's vouch policy deny every bid).
- A change is done when the system tests covering it pass on staging.
- No Go tests, except the policy engine specification tests (owner, 2026-10-30): `internal/policyengine/spec`, the
policy decision tests in `internal/market/bidder` and `internal/market/requester`, their harness
`internal/policyengine/policytest`. They must pass; no skip.
- Tests drive a real Go requester (`run requester` / `run request`) as an account with vouches; requests use discovery
(no bidders list); cleanups release VMs and delete records even on failure; one driver per account.
Run (from `tests/`; the target is a target.json, `tests/targets/prod.json` for prod):
```sh
cd tests
SWC_TARGET=$PWD/targets/prod.json SWC_DRY_RUN=1 cargo test # preflight; touches no account
SWC_TARGET=<staging target.json> cargo test # the full suite on staging
SWC_TARGET=$PWD/targets/prod.json cargo test --test acc_0004_vm_atirc_agent # prod: only in a lead-given window
cargo run -p testkit --bin last-pass -- readings/*.json # passing runs onto docs/acceptance/README.md
cargo test -p swc-vmm-tests -- --nocapture # VMM wrapper, KVM host: tests/vmm/README.md
```
Before pushing a `tests/` change (commit it first): `tests/publish-test-host.sh`. It builds the static swc-test-host and
the module wasm of HEAD's `tests/` tree on hat-8 and publishes them to the bucket at `tests/<git rev-parse HEAD:tests>/`
(SHA256SUMS last); knotless CI's `test` workflow downloads and checks them, and builds on its VM only when they are not there
(its log says `fallback: building test host`).
Debug a failed CI run (a composition run dir holds `ticks.ndjson.zst`, `state.json`, `failure.json`,
`resources.json`, `run.json`; no secret in any of them). On a failure, before the freeze or the cleanup, `logs/` too: per
invocation `<name>/local/` (the request log and its requester's dev logs) and, for a VM that was Held, `<name>/vm/` (over
request.ssh: the VM's journal and every `*.log` under /home and /root: core.log, dev logs, step logs; a failed staging
run's `dev down` keeps its state for this). Each its last 512 KiB, 6 MiB in all, smallest first, the rest named in
`logs/skipped.txt` (an artifact reaches the runner through the guest agent's 16 MiB job output ring). Every run also
writes `logs/<name>/release.log`: its cleanup's `run request release` of each invocation, with its time. Logs are
redacted: secret-named JSON keys as in the dump, other text cut after a secret word's `=` or `:`, JWT-like words and PEM
blocks dropped. A failed staging suite also brings its failed gate entries' run dirs from the VM that ran the gate
(host call `logs.fetchRuns`) into `runs/staging-composition-<ms>/logs/gate-runs/<run>/` (failure.json, state, resources,
logs/, ticks; 4 MiB in all): replay or read them as any run dir.
```sh
# 1. the failed check's artifacts (a workflow declares `artifacts: [runs/]` and runs with SWC_RUNS=runs; private to
# the owner and collaborators, kept 14 days); a knotless CLI built from the fork (runner-atlibs-composition),
# logged in again once it asks for dev.knotless.getArtifact (`knotless login <you>`). The check's log shows the
# run as it goes: one swc-test-host line per tick (event, calls, timers, `done pass=`)
export KNOTLESS_APPVIEW=https://knotless.socialweb.computer
knotless run artifacts knotless://socialweb.computer/socialweb-computer-kcp --workflow test --commit <sha> # list
knotless run artifacts knotless://socialweb.computer/socialweb-computer-kcp --workflow test --commit <sha> --out ./artifacts
R=./artifacts/runs/<runbook>-composition-<ms> # locally: ~/.tmp/swc-system-tests/runs/<...>
jq . "$R/failure.json"
# 1b. the market as the run saw it (the `test` workflow captures the relays and Jetstreams into runs/watch.ndjson): load
# ./artifacts/runs/watch.ndjson in https://watch.socialweb.computer/ with its file picker
# 2. replay it offline (from tests/, after `cargo wasm` in tests/modules; a CI run's composition is this checkout's of
# the same name): the first diverging tick, then the states there
cd tests && SWC_TARGET=$PWD/targets/prod.json cargo run -q -p swc-test-host -- replay "$R"
cargo run -q -p swc-test-host -- replay "$R" --to <seq> | jq .statesAt
# 3. a frozen run: attach (MCP on stdio), then the tools state, ticks, state_at, failure, resources, kube_get, ssh_exec,
# run_assert, and continue or cleanup
cargo run -q -p swc-test-host -- attach "$R"
# a frozen CI run (`test` freezes a failure for its --freeze-ttl, then cleans up; its `attach` is what a given key
# runs on the CI VM): the owner or a collaborator gets a handle from the runner (dev.knotless.getAttach, service auth;
# the CI VM's iroh ticket and host keys, never in the check record; the key goes with the VM), then attach or end it
ssh-keygen -q -t ed25519 -N '' -f ~/.tmp/ci-attach/attach_key
knotless run attach knotless://<owner>/<repo> --workflow <wf> --commit <sha> --key ~/.tmp/ci-attach/attach_key.pub > attach.json
cargo run -q -p swc-test-host -- attach --ci attach.json --key ~/.tmp/ci-attach/attach_key # needs dumbpipe on PATH
cargo run -q -p swc-test-host -- cleanup --ci attach.json --key ~/.tmp/ci-attach/attach_key # cleanup: the step ends, VMs released
# 4. a local repro: freeze at the step before the failure
SWC_TARGET=$PWD/targets/prod.json cargo run -q -p swc-test-host -- compositions/<runbook> --on-fail freeze --freeze-at <step>
# 5. leave nothing behind
SWC_TARGET=$PWD/targets/prod.json cargo run -q -p swc-test-host -- cleanup "$R"
```
A later workflow job (`after: [build]`) reads an artifact with
`curl -sfo <file> -H "Authorization: Bearer $KNOTLESS_ARTIFACTS_TOKEN" "$KNOTLESS_ARTIFACTS_URL?workflow=build&name=<path>"`.
Never paste creds anywhere. A run's dump dir (`$SWC_RUNS/<run>`) holds none: secret-named keys are redacted,
kubeconfigs, keys, tokens and the requester state stay in its work dir (`~/.tmp/swc-system-tests/work/<run>`), never
an artifact.
Staging knotless (`dev up --with-knotless <dir>`, with `--with-gateway`): the gateway runs knotless-serve and
knotless-run (the fork's image, binaries copied out) on the network's PLC, PDSes and relay; the runner is the gateway's
owner account, trusts only its repo's compositions and rents from the network's bidders. Guests reach both by the
network's `*.localhost` hosts. Tests: `tests/compositions/knotless/README.md`.
```sh
K=/var/tmp/<work>-knbin; mkdir -p $K; c=$(docker create knotless-fork:<commit12>) # knotless-fork-infra scripts/build.sh
for b in knotless knotless-serve knotless-run git-remote-knotless dumbpipe; do docker cp -q $c:/usr/local/bin/$b $K/; done; docker rm $c
socialweb-computer dev up --instance <i> --profile market-network --with-gateway --with-knotless $K ... # as test-0001
socialweb-computer dev status --instance <i> | jq '.network.status.apps[]|select(.name=="gateway").listeners[]|select(.component=="knotless")'
G=<state>/dev/<i>/apps/dev/<i>-gateway/knotless # client.env (no secret), owner.env (0600), serve.log, run.log
set -a; . $G/client.env; . $G/owner.env; set +a; PATH=$KNOTLESS_BIN:$PATH knotless repo list $KNOTLESS_IDENTIFIER
grep '"run ended"' $G/run.log # a failed run: knotless run list <url> --commit <sha> --log
```
A new repo's refs run only once the runner watches it (it acts on nothing already there): wait for the appview to
list it (`$KNOTLESS_APPVIEW/xrpc/dev.knotless.listRepos?runner=$KNOTLESS_IDENTIFIER`) and a `--sweep` (10 s) before
pushing.
Release pipeline. kcp CI is the knotless workflows only (`.knotless/workflows/`, the fork's runner; GitHub CI is gone, its
`SPACES_*` secrets unused; hat-8's GitHub self-hosted runner, account socialweb-ci from `scripts/gh-runner-install.sh`, is
stopped and disabled with its files kept: `sudo systemctl --user -M socialweb-ci@ status
gh-actions-runner-socialweb-computer-kcp.service`, `--remove` deletes it). The knotless `release` workflow runs the checks, builds once and stages `staging/<sha>/` (plus the content-addressed
`downloads/` and `guest-images/`); on main its last steps (the former `publish` workflow) copy it inside the bucket
(`scripts/copy-spaces.ts`) to `main/<sha>/` and `main/latest/` and check it. No v* tag is built: `releases/<version>/` is
promote's copy of `main/<sha>/` for the stable version it cuts (`scripts/promote.sh`, plan 0040 step 15). `staging/` is
scratch for a build whose gate failed. A release build restores the caches of `scripts/asset-cache-upload.sh`
and `scripts/ci-build-cache.sh` (`build-cache/<kind>/<key>`: cargo target by Cargo.lock, meshd by the rust/ tree; the Go module
and build caches are opt-in: restoring them costs more than they save) and main builds upload what is new; a cached build gives the
bytes of a cold one (`-trimpath`). Until it has passed on main, builds are hat-8
release builds (never `--dev`), published by hand to `local/<sha>/binaries/` (testkit's `local:<sha>`), and the test host
and module wasm to `tests/<tree>/` (`tests/publish-test-host.sh`):
```sh
C=$(git rev-parse HEAD) # on hat-8, a clean checkout at C
scripts/build-socialweb-computer.sh --platform linux/amd64 --platform linux/arm64 --commit $C
(. ~/.spaces_swc_kcp_source && deno run -A --no-lock scripts/publish-spaces.ts \
--dir ~/.tmp/socialweb-computer-build/release/$C --prefix local/$C/binaries)
```
Pipeline, in order; cut the rc only when both suites are green:
1. Build the release of the commit (`scripts/build-socialweb-computer.sh`), `deploy/check.sh --fast`.
2. VMM wrapper tests (QEMU default, Firecracker).
3. Stand up staging on VMs rented from the market: own PLC, PDS with throwaway accounts, relay, Jetstream, gateway, ssh
gateway, bidders; it writes the staging target.json. `tests/compositions/staging`: one VM, or `--input '{"layout":
"multi"}'` (four small VMs on one meshd overlay, the bidders joined by `--market-network-file`; README there).
4. Full suite against staging.
5. Tear staging down.
6. Redeploy prod.
7. Full suite against prod (from rc166 the release gate; no hand runs), on hat-8, as bob, in one command:
```sh
cd tests && SWC_COMPAT_REQUESTER=sha:<previous rc commit> GATE_VMS=6 ./gate.sh $PWD/targets/prod.json --account bob,alice
```
It runs the compositions as many at once as there are accounts (one driver per account: the test host's account lock;
each slot its own iroh key, `SWC_IROH_KEY`), each holding its entry's VM weight of `GATE_VMS` (4) so the bidders are
not oversubscribed; an exclusive entry (0001: k8s nodes one per host; 0007:restart; 0007:upgrade; 0014: bidders filled or booked) runs alone
(`GATE_EXCLUSIVE`); `GATE_JOBS=1` is the serial gate; `GATE_SHARD=i/n` runs share i of n (`./gate.sh --list <staging|prod>` prints it; by hand only: docs/ci-debug.md, "Shares by hand"). `--on-fail freeze`, `GATE_FREEZE_TTL` 900 s; one
`RESULT <runbook>:<entry> PASS|FAIL <seconds> <note>` line each as it ends, a summary table in gate order with the
wall time and `gate: <kind>: N of M PASS`, exit 1 on a FAIL. There is no SKIP: the entries are `tests/entries.txt`'s
(`both` or `staging`), chosen by the target's kind (`network` present: staging), never named on the command line; a
module that cannot do its check ends `unmet: <why>`, a FAIL. The serial
gate2 of 2026-11-13 (entries summing to 3528 s) scheduled this way takes about 2200 s with bob and alice, about 1970 s
with a third account; 0001 (803 s) runs alone, which is the floor. Logs, the
readings and each run's dump path: `~/.tmp/swc-system-tests/gate/<time>/`.
## Prod deployments
A prod deployment guarantees nothing; the routine path preserves most things.
- The routine deploy (`scripts/deploy.sh` on an installed gateway) upgrades in place: state dir, kcp
objects, OpenBao contents, the requester's and bidder's DIDs, association records, web sessions and
running HostProcess units, which the new core adopts by unit name and InvocationID, all survive
(a bidder's adopted guests get the new core's nftables rules: "A guest's rules follow the running bidder")
(read on prod: web session across rc8 to rc9, DIDs and association
across rc11 to rc21). A fresh install, or a lost or purged state dir, preserves nothing and needs
one association per identity again.
- Losing sessions, VMs, clusters, claims, receipts or any other prod state during a deploy is
acceptable. Do not plan migration, carry-over or rollback for a change of shape. Deploy fresh when
the shape changed.
- The atproto credentials of the two prod accounts are on the deploy host, owner-only (mode 0600):
`~/.aliceoa_bsky_social_source` (requester account alice; exports `ALICE_ATPROTO_HANDLE`,
`ALICE_ATPROTO_PASSWORD`) and `~/.bobvmbuilder_bsky_social_source` (bidder account bobvmbuilder;
exports `BOB_ATPROTO_HANDLE`, `BOB_ATPROTO_PASSWORD`). When a deploy loses a session, sign in again
from these files. Source them into the one command that needs them; never print, log, commit or
copy their values, and never put them in a kcp object, a HostProcess argument or a plan. The one exception (plan 0040
decision 7 and step 11): CI `promote` gets alice's and bob's main passwords (the prod gate's only drivers, owner 2026-11-19, docs/ci-debug.md "The hat-8 gate"), the ops read secret (the bearer of the bidders' and
the ssh gateway's read-only `com.fedfork.exec.ops.*` routes, docs/SYSTEM.md; no kubeconfig and no bidder host access), the
droplet deploy key (the deploys only) and the agent config as secrets sealed to `refs/heads/main` (their names: the head of
`.knotless/workflows/promote.yml`); `scripts/ci-prod-target.sh` writes each to a 0600 file of r2 on the release VM and
`scripts/promote.sh` drops them from the environment before the gate; never printed, never an artifact.
- OAuth sign-ins are scripted; never hand a sign-in link to the user. `scripts/oauth-signin.ts` drives
a headless Chrome (`/usr/bin/google-chrome`, puppeteer-core) through the atproto OAuth flow with the
handle and password from a credential file: the QR session-transfer page (qr.fedfork.com, a bidder's
`prod-qr` MarketIdentity `pendingAction.url`) or the web app's sign-in page, the PDS's authorize
page, then "Authorize". Example, for the hat-8 bidder:
`deno run -A --no-lock scripts/oauth-signin.ts --creds ~/.bobvmbuilder_bsky_social_source --account BOB --url "<pendingAction.url>"`.
It reads the file itself (values never on the command line or in logs) and writes step screenshots
to `~/.tmp/oauth-signin/<ts>/` (mode 0700) for debugging. A QR link expires after 5 minutes; read the
current one from the MarketIdentity status right before running it.
- After a fresh gateway deploy, the requester alice has no web app session: the cluster provider logs
"no oauth session stored for did:plc:..." on every claim. Sign alice in with the same script and
`--url https://k8s.socialweb.computer/ --start-api` (the gateway is API-only: `--start-api` POSTs
`/oauth/start` with the handle and drives the returned authorization URL).
- A bidder's guest image comes from `--guest-image-source` (a `file://` or `https://` base that
`install bidder` writes to config.json), or, when that is unset, from the published artifacts:
`run bidder` reads `guest-images/<target>/<pinned image.json sha256>`, then `releases/<version>` and
`main/<commit>` under `--guest-image-release-base`, the public Spaces bucket by default. CI uploads a
guest image once, to that content-addressed key, and only when the key does not hold it yet. On hat-8 the pinned image is in
`/var/lib/socialweb-computer/guest-image` (root-owned; the bidder verifies every file by the sha256
pinned in internal/versions/versions.json).
- The did:key ingress relay at xrpc.fedproxy.com is the Go relay `socialweb-computer internal
ingress-relay` (`internal/atproto/ingress`), behind caddy on a unix socket as the
`30-go.conf` drop-in of `xrpc-relay-server.service`. Deploy it only with
`scripts/deploy-xrpc-relay.sh` (build, ship, restart, health check, rollback on a failed check;
once healthy it points Caddy's on_demand ask at the relay's 127.0.0.1:5556, live and in
/etc/caddy/Caddyfile; `infra/ingress-relay/deploy.sh` wraps it); `--rollback` goes back to the deno
relay, which has no ask listener: it starts `caddy-check-dns-from-config` (127.0.0.1:5555, installed
but disabled otherwise), points the ask (live and Caddyfile, backup kept) at 5555, removes the
drop-in (which stops the Go relay), then runs `xrpc-custom-domains.sh remove`; `--status` only
reads. What runs on fedproxy.com and what depends on it: `docs/infra/FEDPROXY.md`. Never run
`atproto-reverse-proxy/scripts/deploy.sh`: it prints `CF_API_TOKEN`. A change to the ingress
protocol (`internal/atproto/ingress`) ships to both ends: the core and this relay.
Custom domains: a host whose `_xrpc.<host>` TXT holds a live subscriber's did:key (or its
did-key host) is served as that subscriber; a subscriber may also send
`com.fedproxy.temp.xrpc.alias {hosts}` (answered `#result {accepted, rejected}`, 16 per subscriber,
gone on disconnect). Caddy reaches it through the relay itself: with
`--ingress-relay-caddy-custom-domains` (the deploy's `CUSTOM_DOMAINS=1`) it adds, on start and every
30 s over the admin socket, what is missing of the route of every host outside fedproxy.com and of
the HTTP-01 fallback issuer on atprp-ssh-relay's policy (a Caddy restart drops both; admin API only:
a hostless Caddyfile site block would shadow atprp's appended tunnel routes), plus its ask listener
127.0.0.1:5556. `infra/fedproxy-legacy/xrpc-custom-domains.sh status|remove` reads or drops the
route and the issuer; the ask stays on 5556 (it answers every fedproxy.com host as the old checker
did). A response
frame's optional `headers` (name -> string, or -> [string] for Set-Cookie) reach the caller; the relay
drops hop-by-hop, Content-Length, Strict-Transport-Security, Alt-Svc, Proxy-*, Set-Cookie with a
Domain, invalid names or values (CR, LF, NUL), and past 64 values or 32 KiB; `contentType` wins over
a Content-Type header.
- The atproto relay at relay.mini-cloud-0002.chadig.com (bidders' requestCrawl, a firehose source) is
the Go relay `socialweb-computer internal atproto-relay` (`internal/atprotorelaymain` on
`internal/relay/local`), on 127.0.0.1:2584 behind caddy as the `30-go.conf` drop-in of
`atproto-relay.service`, state in `/var/lib/atproto-relay-go`. Deploy it only with
`scripts/deploy-atproto-relay.sh` (`--dry-run` prints the plan, preflight on 127.0.0.1:2585, rollback
on a failed check); `--rollback` removes the drop-in and goes back to the deno relay, `--status` only
reads. Caddy is not touched.
- Every prod bidder runs the policy `bidder-tangled-vouch`, the default when MarketBidder
`spec.policy` is unset (`install bidder` writes `bidder-policy` to config.json); `open` is insecure
and runs only when set explicitly, never on prod. The acceptance requester alice must be vouched by
the bidder operators bob and john (`sh.tangled.graph.vouch`, docs/acceptance/0001-auth-modes.md).
- A market policy always runs, on both sides and in both lanes. No policy named means the default
(`requester-tangled-vouch` on a requester, every contract and RFP carries it, owner-session mode
included; `bidder-tangled-vouch` on a bidder). The bidder's full lane runs its own workflow and then
every policy the RFP attaches, and allows only if all allow; no verdict is cached across decisions,
and a stored bid verdict never skips it. The requester gate (scope and full, contract and session)
runs the body of the attached policy record, which must be in the requester's own repo and, for a
bundled name, equal the bundled workflow; it refuses a contract or session whose policy names
another. On the bidder the requester is the subject. A run whose workflow declares `cache-key` gets
`runs.CacheKey` (workflow, mode, perspective, every DID, the records, the args), so a memoized
verdict never crosses parties. Free winners get the full run too. A missing evaluator, run client
or engine, or an `allow` output that is not a strict boolean, denies. `open` runs the open workflow
only when named; it never skips. A run whose PolicyEngine is not bound yet (a core start) waits
for the binding (`policyengine_run_waits_for_engine`, then `policyengine_run_engine_bound`) within
the caller's run timeout instead of failing at once. A requester restart never fails a contract
for a transient start-up failure: when a resumed contract's recorded-winner policy run does not
complete, or its attached policy record, its secrets bundle or its ingress url cannot be read, it
keeps its phase and tries again (`requester_contract_waits_transient`, a bind or the backoff 2 s
doubling to 30 s) and fails only on a deny or after `TransientResumeGiveUp` (2 min).
- The PolicyEngine run cache (`internal/policyengine/cache`) hands a run only its own entries (its cache-key verdict, the policy/ keys of its rfp, bid and accept, their rec/ entries and the strongRefs in them, its trust snapshot), drops expired verdicts, and its `spec.limits.maxCacheBytes` unset means `sandbox.OutputLimit`/2 (32 KiB; the scope verdict store keeps the set value, unset is no byte cap). A policy action whose stdout passes `sandbox.OutputLimit` (64 KiB) fails with `policy action output exceeded 65536 bytes`.
- Bidder capacity (`internal/market/bidder/vmsize.go`): a guest counts as its vCPUs and its memory plus the VM overhead
(`contract/host.VMCommitMiB`, unchanged; the VM's cgroup itself may use up to 2 x the guest memory, `VMLimits`: MemoryMax = MemoryHigh = 2 x guest, MemorySwapMax = guest, so the host pages to swap rather than throttling or killing a VM; a host needs swap of about its RAM). The one count is `committed`: each live ComputeVM, plus the capacity hold of
each open provision StateSync whose accept has no live ComputeVM (one accept counts once: a VM, or its hold), and the
guest quota (`maxConcurrentGuests`) counts the same guests. That count is decided twice per rental: once per RFP, when
its bid is created (`enqueueBidLocked`, under the bid's lock, by whichever intake sees the RFP first: submitRfp or the
market stream's inbox), and once per accept at admission (below). At the RFP's intake `declineFor` runs (size, kind,
room, image, guest tools, net) before the bid has any status; the bid's first status is the decision (Deciding, or
Failed `DeclinedCapacity` / `DeclinedUnservable`), one write, and a decline is answered only when that write succeeded,
so a bidder never declines and bids on one RFP. An intake that cannot read the RFP or its payload decides nothing and
fails (the stream's inbox retries it; a submitRfp answers 500, which the requester retries), and a bid with no status
is never decided by `decideBid`, so the next intake decides it, with the room; the RFP's bid window bounds the retries (an intake that finds the bid
created but not listed yet fails too, so it is retried), and a bid still with no status when that window ends finishes
`Complete` `Undecided`, so its TTL removes it. What a requester hears: by
submitRfp, the decline in its answer (`declined: {kind, reason}`); by the stream, nothing (that path has no answer),
and no bid; a submitRfp that finds the bid already created answers from its recorded state (a Failed
`Declined*`, `PayloadUnservable`, `PayloadUnavailable`, `GuestToolsUnserved` or `NetRefused` as its decline, anything
else none), so a requester whose RFP the stream brought first still hears the decline. The bid's own decision
(`decideBid`) makes no capacity decision (a `market.bid` promises no capacity), so a bidder that had room at intake bids
and refuses at admission if the room went to another rental's admission meanwhile; its payload refusals fail the bid
`PayloadUnservable` or `PayloadUnavailable` (`bidder_payload_refused`, with the kind). An accept is admitted once
(`admitAccept`, for a submitAccept and for an accept from the firehose inbox alike): after the bidder's objects are
listed (a submitAccept right after a restart waits for that), it resolves the accept's compute.vm and, under one lock,
decides once for that accept: it refuses it (the refusal is remembered under the lock until the bid is over, so every
later send of that accept, a copy or the firehose's, is refused again and none is admitted; a submitAccept answers 409
`CapacityExceeded`; the bid fails `AcceptRefusedCapacity` after the lock; an inbox accept ends there, so a requester that
never calls submitAccept sees no receipt and fails at its `ReceiptTimeout`) or creates the accept's provision StateSync
(a kcp write bounded to 15 s, not cancelled with the request) with the hold `{cpus, memMiB, bid}` in its payload. That
StateSync, durable in kcp, is the reservation: it counts from the 202 until it ends Rejected or Obsolete, settles (its
VM placed, then counted as the VM, or released), or its bid is over (released, failed, expired: the bid TTL, 24 h). A
re-send of an admitted accept is answered 202 with no check. An accept whose records (the accept, its RFP or the RFP's
payload) the bidder cannot read yet, not found or unreachable alike, is answered 202 with a StateSync with no hold: a 202 for an accept the bidder could not read yet promises nothing until it is admitted; its attempts read it again and admit it
as below once it reads; an accept record that stays unreadable is abandoned (`acceptgiveup.go`: one `bidder_accept_abandoned`, the StateSync Obsolete), while an unreadable RFP or payload is an attempt error that retries, as on main. An accept the bidder cannot serve gets a StateSync with no
hold, which holds nothing: with no bid of its own it ends Obsolete; with the bid over or from another requester its
attempts fail with a plain error and the StateSync stalls and retries for ever (as on main). The memory rule (owner
decision 73; admission is the one queue, the starts run in parallel): a guest is admitted, and bid for (at the RFP's
intake, either path), only when its
committed memory fits in the host's MemAvailable less the memory every admitted guest has not taken yet, that is each
unplaced hold's whole committed memory and, for each live VM, its committed memory less what its cgroup's
`memory.stat` counts as anonymous memory (`anon`: the guest RAM the VMM has touched; page cache is not counted, since MemAvailable already counts it as free; the whole committed memory while its cgroup is not found), and the sum of all committed
memory stays within MemTotal. No capacity check runs after admission: not on a retry, not in `placeAccepted`, not at a
re-send, so a capacity shortfall never fails an admitted accept, before or after its receipt. A provision StateSync with
no hold whose accept has no live VM (one an older bidder wrote, or one for an accept that was not readable at intake) is adopted at its next attempt: an accept with a receipt
(or that its bid names) gets its hold written with no check (it is owed), one without a receipt is admitted as above
(and may be refused, its bid failed, before any receipt). After a 409 the requester keeps its accept (never deletes one a bidder may still act on: a bidder that finds it gone retries until `bidder_accept_abandoned`), excludes that bidder and accepts the next bid of
the same RFP (up to 3 times a round); when the bids run out after a refusal it publishes a new RFP at once without the refusers (it keeps excluding
them until an accept works). A 409 past that round's 3 is "full" too: the requester waits for capacity as for a round with no
bid (each waiting round starts the count again, with its refusers cleared), never a contract failure. A round that gets no bid at all, that exclude-the-refusers RFP included, is not a failure: the requester waits
for capacity (`requester_waiting_for_capacity`, status message `waiting for a N-vCPU slot: round k`), deletes the RFP, and after a backoff
of 30 s, 1 min, then 2 min (`CapacityBackoff`) publishes a fresh RFP that asks every bidder again (capacity changed), until the
contract's own deadline (its `vm-expires-at` label, `run request --expires-in`; none: `DefaultCapacityWait`, 30 min), then fails with the
no-bids error and the round count (the compute.vm of every round carries that deadline as its `expiresAt`; a round whose deadline has passed is never
sent: `deadlineErr` fails the contract instead, since every bidder declines a payload whose `expiresAt` is past). A bidderDids restriction that names no bidder, and bids all denied by policy, still fail at once. Only a genuine capacity shortage waits: the
bidder answers submitRfp with `declined: {kind, reason}` (kind `capacity`: full now; `unservable`: its offering lacks the compute kind,
nestedVirt or images, the VM is above its maxCpus or maxMem, the requester image cannot be fetched or verified (every image check refusal, and before a capacity decline: a full bidder still names a refused image), guest tools or net it does not serve),
and when every bidder the RFP went to declined `unservable` (or the discovery dropped every candidate for a capability or size
its offering lacks) the contract fails at once with "no bidder can serve this request" and the reasons (`requester_bidder_declined` logs each);
a bidder that is full, silent, or too old to answer `declined` keeps the round a waiting one. The atlibs requester module needs the same
reading of `declined` for parity.
`run request` waits `--contract-timeout` plus that deadline for the contract to settle.
The bidder polls each live VM's cgroup `memory.events` (15 s) and publishes edge-triggered
`compute.events.vm.memoryPressure` / `vm.oomKilled` reports (one per crossing, at most 10 per contract).
- Only the lead runs a prod deployment (plan 0006 M5-FRESH-DEPLOY). Task agents never touch prod.
## How to verify
- While iterating, build only what you touched: `go build ./internal/<pkg>/... ./cmd/socialweb-computer`.
Do not run the full gate per change.
- The CI gate is THE gate: the knotless `release` workflow's checks step runs `deploy/check.sh --fast` (go vet on every target; it builds nothing)
on every push to `main` and `knotless-ci`; read its check
(`knotless run list knotless://socialweb.computer/socialweb-computer-kcp --commit <sha>`). Do not run it on
hat-8 as well. A local `deploy/check.sh` on hat-8 (under `scripts/swc-heavy.sh`) is only for what CI does not run:
a VMM change (its tests are local only), a pins change (`make check-pins`; it covers only what is built from
pinned third-party source: kcp from the owner's fork, atirc), a guest image change.
- `deploy/check.sh --fast`, in order: `gofmt -l` of `api`, `internal`, `cmd`, `hack` and `lexicons`;
`go build ./...`; apigen into a scratch directory and a diff against every file
`go run ./cmd/socialweb-apigen -root . -list` names (generated code must match its source);
`go generate ./...` must leave `api`, `internal/atproto/market` and `deploy` unchanged;
`go vet ./...`; `scripts/check-architecture.sh`; `scripts/check-entries.sh` (`tests/entries.txt` against plan 0040's Entries table); `scripts/check-shards.sh` (the gate's shares, `GATE_SHARD=i/n`, are disjoint, none empty, and their union is the list, for the staging and the prod list); release.yml, test.yml and promote.yml have the same push `paths:` and promote is `after: [release, test]`;
`scripts/check-oauth-scope.sh` (the Go oauth scope lists against the QR page's); `scripts/check-docs.sh` (the docs checker, plan 0040 step 17:
every fenced block tagged `run=gate` in `README.md`, `docs/quickstart*.md`, `docs/deploy.md` and `docs/bidder.md` runs, a page's blocks in one
`bash -euo pipefail` shell from the repository root, and must pass; `ATLIBS_REPO=<checkout>` adds atlibs `docs/`; `--list` names every block;
docs that act on prod and the debug docs are not in its list); `scripts/check-cli-args.sh` (plan 0043: a host build of the CLI runs `--help` of every verb
that takes an application's options or a command, which fails when `options.json` and `deploy/apps/option-map.yaml` define one name twice, parses
every unit template's ExecStart with `internal parse-args`, and finds no option after `--` in a unit, a drop-in or the cloud-init texts); `scripts/check-pins-records.sh` (plan 0040 step 20: `com.fedfork.exec.pins/atlibs`
of johnandersen777.bsky.social, aliceoa.bsky.social and socialweb.computer, read without a credential, equal apart from `createdAt`; run it
after a `publish-chain.sh` too); the identlink vectors (`go run ./hack/identlinkvectors internal/identlink/vectors.json`); the policy engine specification tests
(`go test ./internal/policyengine/... ./internal/market/bidder/ ./internal/market/requester/`; they
find deno through `SOCIALWEB_POLICYTEST_DENO`, the embedded deno, the build-staged pinned zip in
`~/.tmp/socialweb-computer-build/cache/deno-embed`, or PATH, and fail without one); `go build` and `go vet` of each Go asset module
in `internal/versions/versions.json`; `go build` and `go vet` for darwin/arm64, darwin/amd64,
windows/amd64, windows/arm64 and linux/arm64, each with a `-tags release` build of
`cmd/socialweb-computer`; and the `-tags release` build of every release platform.
- `deploy/check.sh --local` adds `cargo fmt --check` and `cargo test` of `rust/`, and
`socialweb-computer dev verify-assets` on a release build when the host's blobs are staged; it
lists what it skipped. `deploy/check.sh --full` makes a skip a failure.
- The binary for live runs: `scripts/build-socialweb-computer.sh --dev --publish /var/tmp/<work>`
(`make dev` is the same with `/var/tmp/swc`). It builds one socialweb-computer of the working tree
(uncommitted changes included) for the host platform, or for each `--platform`. It embeds every
pinned asset, staged once per `versions.json` entry; the first `--dev` build on a host stages every
asset (about 3.5 minutes), later builds only an asset whose entry changed. `socialweb-computer
version` says "dev build, not reproducible" with the commit and the dirty state. `--publish`
copies the host binary to `<publish>/bin/socialweb-computer` and mirrors the working tree to
`<publish>/src`, both readable by test-0001. Output: `~/.tmp/socialweb-computer-build/dev/<worktree>/`.
- A Go change in a running dev instance or market network: rebuild with `--dev --publish`, then
`socialweb-computer dev restart <instance> core --binary /var/tmp/<work>/bin/socialweb-computer`
(as the account that runs the instance). It stops the core unit, places it again with the new
binary, waits until it is ready, prints a JSON report and fails unless every unit was adopted
unchanged. A child application of a market network takes `dev restart <network> <app>/core` (or
`dev restart <network>-<app> core`); the network's supervisor places that core through its
HostProcess object. Then rerun `run request`. Do not `dev down`/`dev up` for a code change.
- One component: `socialweb-computer dev restart <instance> [<app>/][<workspace>:][<kind>/]<component>`.
It writes `spec.restartedAt` on that component's kcp object and waits until the component reports
the restart. The PolicyEngine restarts only with `core`. A kcp roll is `dev roll kcp`; on an installed host,
`units roll kcp`, `units roll cluster` and `units roll link`.
- The release binary (prod only; the lead builds it): `scripts/build-socialweb-computer.sh` without
`--dev` builds one commit once per release platform of `versions.json` from a clean checkout:
every pinned asset staged by `scripts/stage-assets.sh` and checked against its pin (a signed upstream is
verified with its key vendored at `internal/versions/keys/<fingerprint>.asc`, never fetched at build time;
only `make pins` fetches a key that is not vendored, from its pinned URL, keyserver.ubuntu.com or
keys.openpgp.org, and keeps the first that imports with the pinned primary fingerprint), the embedded
Deno of the policy engine, then `go build -tags "release embed_deno"` with `CGO_ENABLED=0`.
Each step is keyed by its inputs and does not run again when they did not change: asset blobs by
their pinned inputs (assets built from this commit by their source inputs), socialweb-meshd by rust/ and the
toolchain, plus the go build cache and the cargo target dir. Tens of seconds warm; minutes with
empty caches. Every phase prints a `timing` line. `make` builds with `--update-pins`: assets built
from pinned third-party source (kcp, atirc) are rebuilt and their new pins written to
`internal/versions/versions.json`; downloaded assets still verify their pins and never move. kcp's source is the
owner's fork johnandersen777/kcp at d71f020c (tag `swc-apibinding-requeue-1`): kcp-dev/kcp main 74e9fb9c plus one
commit (the APIBinding reconciler reads a missing APIResourceSchema live and requeues on NotFound; kine can drop a
watch event and a kcp first start then stuck for ever). A kcp patch is a commit on a branch of that fork, never upstream.
`make pins` re-stages and re-pins only those. The `pins` CI job runs `make check-pins`, which runs
`make pins` and fails when `versions.json` differs from HEAD.
- **What `versions.json` pins, and what it never does.** It pins THIRD-PARTY inputs only: source archives, toolchains,
upstream binaries, images, debs, and the expected tree of what is built from them (kcp, atirc). The outputs of
what we build from this repository's own commit (every asset whose fetch kind is `go` or `script`:
socialweb-vmexec, the two OpenBao plugins, swc-userns-stage, socialweb-computer-k8s-downloads,
socialweb-guest-downloads) are NEVER pinned there: they change with every Go change under `internal/`
(cmd/socialweb-node and the guest agent import it), so pinning them cost one re-pin commit per change and a
broken `make check-pins` in between (53 pin commits in 477 on main by 2026-11-14). `deploy/check.sh` fails when
`versions.json` carries a `platforms` entry or a fetch `sha256` for one. Instead:
- `scripts/stage-assets.sh` writes what the build produced beside each blob,
`internal/assets/blobs/<goos>_<goarch>/<asset>.built.json` (tree digest, sizes, and for a Go asset the binary
sha256, which is the sha256 the OpenBao plugin catalog registers). It is ignored by git and embedded in the
release binary with the blob (`internal/assets/built.go`); `assets.PlatformPins` gives every consumer the
third-party pin from `versions.json` or the build record. The binary extracts the blob under its tree digest and
re-hashes the tree against the record before every use (`Store.Ensure`), `Ready()` refuses to start without
the record of an embedded built asset, and the plugin placement (`internal/openbaoctl/plugins.go`) checks the
placed binary against the record. A dev build (`--dev`) stages the working tree's assets and their records the
same way (a script-built asset's shared blob is keyed by `stage-assets.sh --print-src-key`).
- The release publishes the produced hashes: `release/<commit>/release.json` (`builds[].builtAssets`) and
`built-assets-<goos>-<goarch>.json`, beside `SHA256SUMS`; the binary's SHA256SUMS entry covers the embedded
blobs and records. The downloads trees carry their own sidecars (`<name>.sha256`): the web app and the
GuestControlPlane refuse a sidecar that does not hash its file, and a node or guest checks the sha256 in the
`/downloads/<sha256>/<name>` path against the bytes it fetched.
- The record is the proof, not a committed hash: the binary re-hashes the unpacked tree against it before every use. CI does not
rebuild the assets to compare them (owner rule: no reproducible-build verification in CI, in any form).
- Tampering is a testcase: composition 0018 (`tests/compositions/0018`, `0018:run` in `tests/gate.sh`, staging only) flips a byte
of a download in the extracted tree (with and without its sidecar) and of the embedded record in a copy of the binary, and
asserts the binary heals the tree or refuses (`docs/acceptance/0018-downloads-integrity.md`).
- Adding such an asset: give it fetch kind `go` or `script` and NO `platforms` / `sha256`; add its inputs to the
script's `# stage-assets inputs:` line so its cache key sees them. Checkouts, caches and outputs stay under
`~/.tmp/socialweb-computer-build`; `release/<commit>/` holds the binaries and `SHA256SUMS` that
`install` and `upgrade --binary --sha256` take. Never build it in the inner loop.
## Where and how to work
- Work in a plain git worktree or branch of this repository. Do not put a worktree inside another
worktree.
- Commit small logical commits and push them to a branch of your own (`git push swc HEAD:refs/heads/<item>`; remote
`swc` = `knotless://socialweb.computer/socialweb-computer-kcp`): a push to a branch runs no workflow. Land through the
merge queue (Landing changes); the queue's batches run no CI (main's push runs it).
- CI's repository is `knotless://socialweb.computer/socialweb-computer-kcp` (plan 0051): its bindings, its secrets, its runner sign-in; changes land there
(johnandersen777.bsky.social has write), never on `knotless://johnandersen777.bsky.social/socialweb-computer-kcp` (a push to its main would run CI a second time).
Push triggers are back (2026-11-16, plan 0044 step 11, after the staging suite and the prod gate passed by hand and CI release + test passed):
CI runs on main only (owner, 2026-11-15; v* tags dropped by plan 0040 step 15): `release` (the gate's checks, the one build, and on main the publish), `test` (the staging gate, after release) and `promote` (the release, after test) run on a
push to `main`; no branch push and no merge-queue batch triggers CI. To test a branch, push it to `swc` (`git push swc HEAD:refs/heads/<branch>`) and ask:
`knotless run rerun knotless://<owner>/socialweb-computer-kcp --ref refs/heads/<branch> --workflow release`, then `--workflow test` (an asked run is one workflow: none follows another), with no bucket key (a branch
restores the caches read-only and publishes nothing; the publish steps and every bucket write stay main), and a newer ask supersedes its running one. Read it with
`knotless run list knotless://<owner>/socialweb-computer-kcp --ref refs/heads/<branch> --wait <secs>`, and `--log` for a failure. Local hat-8 runs
(`deploy/check.sh`, `tests/gate.sh`, a hand-run staging network) are for interactive debugging only, under `swc-heavy`.
- Heavy jobs on hat-8 (VMM tests, a local gate or `make check-pins`, release or guest image builds, staging stand-up
and its suites) run under `scripts/swc-heavy.sh <command...>`: two at a time (`SWC_HEAVY_SLOTS`, default 2: an
`flock -n` on `/var/tmp/swc-heavy.lock.d/slot-<n>` plus a shared `flock` on `/var/tmp/swc-heavy.lock`, which a copy of the
script from before the slots still holds exclusively), or alone with `SWC_HEAVY_EXCLUSIVE=1` (a job that needs the whole
box, VMM tests: running jobs drain, new ones hold off); `SWC_HEAVY_WAIT=<secs>` gives up after that long. `~/.local/bin/swc-heavy`
is a copy of the script for any checkout's use. `tests/gate.sh` takes it itself (for the whole run on a target with a local `network`; on prod, whose VMs run on the prod bidders, only for its build). A dev
network for an item still takes a live slot (`live-slot.ts`) for its whole life, not this lock.
- Never tag or roll a release by hand: rc's and stable come only from CI `promote` (`docs/plans/0040-release-pipeline.md`
Flow); babysit each promote to its end (stable, or the rollback).
- Never use `/tmp`; it is tmpfs and spills to swap. Use `~/.tmp` (set `TMPDIR` under it).
- A one-off script is a `.ts` file under `$TMPDIR`, run with `deno run -A`, using `Deno.Command` for git
and go. Keep each shell call to one plain command.
- Plans live in `docs/plans`. A claim carries its proof: a CI result its knotless workflow log link, a hat-8 gate its
summary and gate dir (in `.plans.yaml` progress); a finding goes into `docs/plans/0040-findings.md` (seen, cause, fix, proof), not into the plan.
- Rebase on `origin/main` often (`git fetch && git rebase origin/main`). When a rebase stops on
conflicts, run `scripts/regen-after-rebase.sh` (no arguments). It stops first when a generator
input is conflicted (`api/*.go`, `internal/apigen`, `cmd/socialweb-apigen`, `hack/codegen.sh`):
resolve those by hand, `git add` them and run it again. Otherwise it drops the conflicted
`api/**/zz_generated.*.go` files, reruns `hack/codegen.sh`, takes one side of every conflicted file
that `go run ./cmd/socialweb-apigen -root . -list` names (the schemas and exports under `deploy/`,
the zz_generated register and conditions files), reruns `go generate ./api/...`, stages the
outputs and prints the hand-written conflicts left, or the command to continue. Never hand-merge
generated output. `deploy/apiresourceschema-revisions.txt` merges itself (`merge=union`); when it
names one schema twice, the script keeps the HEAD line, and the generator may ask for a Revision
bump in `api/v1alpha1/resources.go`.
- Live runs: your own market network only, run as the VM account (test-0001) from the binary and the
mirror that `--dev --publish /var/tmp/<work>` wrote, with an instance name and bidder handle unique to
your work. Keep the network up for the whole item; clean up only your own units and VMs.
## A fix: check the first guess
Owner, 2026-11-15: always double or triple check the first guess of a fix to find the real correct one. Before writing it:
why is the failing value what it is (trace it to where it is made, not where it is used); does the fix remove the cause or
add a condition around it; does the owner's design (the plan of record) already say how this should work; would the
same failure come back one layer up or down (a guest's guest, another client, another port). A fix that adds a special
case, a second field or a fallback is usually the patch, not the fix: find the design it works around and build that.
Example (plan 0050): R2's guests got no network PLC; the first guess (fall back to R2's https PLC when the guest field is
unset) was a condition around the network's PLC still being plain HTTP; the fix was the PLC's https name as the network's
one PLC.
## The fast loop for one item
- Compile: `go build ./cmd/socialweb-computer` (about 4 s in a new worktree, the Go caches are
shared).
- Binary and mirror: `scripts/build-socialweb-computer.sh --dev --publish /var/tmp/<work>` (about 21 s the
first time in a worktree, about 6.5 s after). Later changes: build again, then
`dev restart <instance> core --binary /var/tmp/<work>/bin/socialweb-computer`.
- One network for the whole item: `dev up` once (80 to 95 s, mostly waiting, about one core), then every
proof is a `run request --network <instance>` (80 to 90 s each, 35 s of it the requester's own kcp).
`dev down` costs 70 to 95 s (one-kcp network with a gateway). Do not `dev up` per proof.
- A held guest burns one host core for as long as it lives (an idle firecracker vCPU does not halt), so
release a `--hold` invocation as soon as you have read it.
- Take a live slot before `dev up` and release it after `dev down`:
`deno run -A ~/.tmp/swc-lead/live-slot.ts acquire|beat|release <name>`. The default is 5 slots
(`SWC_LIVE_SLOTS`), and `acquire` also waits while MemAvailable is under 12000 MB
(`SWC_LIVE_MIN_AVAIL_MB`) or cpu pressure avg10 is over 90 (`SWC_LIVE_MAX_CPU_PSI`).
## The loop
- There is no container. Read the state of an instance with `socialweb-computer units list` and its
units' journals (`journalctl --user -u <unit>`); the processes already log a status line.
- NEVER write a sleep-poll loop such as `for i in $(seq ...); do sleep ...; journalctl ...; done`.
It blocks a turn for minutes and cannot fail. Watch the log with the Monitor tool instead (below).
## Turns and background work
- Never end a turn with a build, test or live run in flight. Background a long command and
keep working, or wait for it, but do not hand back a turn that says "still running".
- Never block the foreground on a long run. Start a live run (`dev up`, `run bidder`, `run request`)
or a build with run_in_background and wait for its completion notification, or watch its log with
the Monitor tool (filter for the success line AND the failure lines). Foreground `until` loops
only for waits under about a minute; no `sleep N; tail` chains. Do other work (docs, the next
commit) while it runs.
- Parallelize reads, greps and edits. Live runs go in parallel too: every listener binds port 0, and
the VM account test-0001 has 16 VM slots that vmruntime hands out under a per-account lock (no
whole-run lock; test-0001 linger stays on). Keep heavy builds few: 16 cores.
- Launch subagents for parallel work, then verify the artefact (the file, the exit code, the log
line) rather than the summary.
- Nested subagents hit a depth limit. Do not spawn a subagent from inside a subagent; do that
reading inline.
## Edit discipline
- Read the exact bytes before editing. The edit tool matches text, and a guessed string is a
failed edit.
- A block that repeats: use `replaceAll`, not one edit per copy.
- One failed match is fixed by re-reading and retrying. Do not guess the same string again.
- Do not revert another agent's work. Run `git status` first; the tree is shared.
## Architecture rules (plan 0026)
- Layout: the TypeScript ABC layers in Go. Imports point down this list only;
`scripts/check-architecture.sh` enforces it:
- `api/<group>/v1alpha1` (common): types, contract status, refs.
- `internal/contract/<concept>` (abc): small Go interfaces and pure types; api and stdlib only.
- `internal/kit/<name>` (common helpers): no concept; api, contract, kit and third-party only.
- `internal/controllers/runtime/<pkg>` (transport): the kcp transport (`kube` typed client and
awaits, `kcpapply`, `manifests`); the only home of the dynamic client and unstructured objects.
- `internal/<concept>/<impl>` (impl): one sibling package per backend, the real work, no kcp.
- Adapters: `internal/controllers/<kind>` turns kcp objects into contract calls (one reconciler per
kind on `internal/controllers/runtime`); `internal/services/<svc>` turns XRPC/HTTP routes into
contract calls and holds no role logic.
- `internal/app/<root>` (factory): one composition controller per app root; wires impls into the
bus and picks a backend by the `backendRef` kind, never by a flag. Composition code
(`internal/stack/**`, `internal/devcore/**`, `internal/<x>main`) is this layer too.
- `cmd/` (CLI); tools (`hack/`, `internal/apigen`, `internal/lexgen`) rank with it.
- Gate. `scripts/check-architecture.sh` (`go run ./hack/archcheck`) fails on any violation of: the
layer order; backend deps; a ticker or periodic requeue site missing from `deploy/obligations.txt`;
a System caller missing from `deploy/bus-bootstrap-callers.txt`; a dynamic client or unstructured
import outside the transport and apigen; a GroupVersionResource literal outside `api/` (foreign
groups are vars in `api/kcpgvr`); an annotation read as a signal (kcp and kubelet metadata
conventions are listed in the checker); an endpoint field in spec without `apigen:"userEndpoint"`;
an impl package without a compile-time assertion against an `internal/contract` interface.
- Go idiom. No factory types: `New(cfg, deps)` takes explicit dependencies and returns a concrete
struct; callers accept the contract interface. Every impl states its contracts at compile time
(`var _ compute.Provider = (*Provider)(nil)`). One package, one concept; no util package outside
`internal/kit`. kcp is a transport like XRPC or the CLI: an adapter translates it.
- No polling. Controllers are driven by watches on their own kind and every kind they ref. Deadlines
use delayed requeue; errors use the runtime's rate-limited backoff. A ticker is allowed only for an
outbound obligation and must be listed in `deploy/obligations.txt` with its owner and period.
systemd units are followed through D-Bus signals, processes through wait.
- Bus. Components and controllers get typed contract handles with `bus.Resolve[T]`. The caller is the
reconciled object or the component instance, never a controller and never System; System is allowed
only for the callers in `deploy/bus-bootstrap-callers.txt`. The callee holds credentials; a handle
never exposes a token. A call with no accepted ComponentLink fails with `ErrLinkDenied`. A call waits for its
workspace's links to be loaded once per core start (the componentlink controller loads each engaged workspace's
ComponentLinks from its synced cache: `LinkPolicy.Load`, `bus.LinksLoaded`), bounded by the cache sync timeout
(`bus.CacheSyncTimeout`, counted from the workspace's first wait; one WARN when it runs out); then the settle
(`bus.LinkSettle`) and the deny as before. Plan 0040 Findings "BUS LINKS LOADED BEFORE A CALL IS JUDGED".
- Testable backends. A backend package (`internal/<concept>/<impl>` and the host libraries it drives)
runs without kcp, the bus or a controller runtime: it takes and returns `internal/contract` and api
value types, and host effects (systemd, processes, filesystem root, clock) come in through the small
interfaces of `internal/contract/host`. The controller translates objects to contract values.
`scripts/check-architecture.sh` reports a backend whose deps reach kcp.
- Writers. Every write is server-side apply with field manager `socialweb.computer/<controller>`. A
spec is written only by its composer; a status only by the controller of its kind; a field a second
controller reports is its own status subfield with its own field manager. Every child and intent
object has an ownerReference to its composer or creator. No annotation carries a control signal.
## Standing constraints
- No explanatory comments in code. The code explains itself through its structure: names, layers,
contracts and types. Two kinds of comment are allowed: the `ponytail:` form naming a ceiling and its
upgrade path, and functional comments that a tool reads, such as `//go:` directives, generated-code
headers and code generator markers (`// +k8s:deepcopy-gen=package`, `// +groupName=`,
`// +k8s:deepcopy-gen:interfaces=...`). A functional comment carries no prose.
- ASCII only, everywhere, including this file and `docs/ARCHITECTURE.md`.
- Decisions, measurements and invariants live in `docs/ARCHITECTURE.md`, never in code comments.
An item opens in section 14 and moves to section 15, with its before/after readings, when it
closes.
- The invariants in section 13 are load-bearing; a change must not break one.
- Runtime runs as USER 1000:1000. Nothing on the L0 host needs root.
## Names: use the real name
Write the real name of a thing: the repository, directory, package, binary, command or plan item id.
Do not make up a short name. Write plain, short sentences, with one meaning for each word. The plans
and older notes use the short names below; read them as the real names.
| Short name in the plans | Real name |
| --- | --- |
| the door | the gateway's web app and tenant API: the GatewayAPI component (`internal/services/gatewayapi`) and `internal/gatewayhttp`, served by `socialweb-computer serve gateway xrpc` |
| the ssh door | the SshServer component (`internal/services/sshserver`) of the SshGateway root (`internal/app/sshgateway`), run by `socialweb-computer run gateway ssh` |
| the core | the `socialweb-computer` process of an instance that runs the controllers (`internal/stack/componentcore`) and supervises the components and HostProcesses; `internal dev-core` in a dev instance |
| a fold, folded | one pass that merged a set of decisions into the plans |
| the spine | the RFP flow: requester, RFP, bid, accept, cloud-init `user_data` |
| the front | the stable entry point in front of a service: the kcp front before the kcp slots (0006 M2-KCP-SLOTS), the TLS front of the market network |
| the edge | caddy and the other stable public entry points that keep a known port |
| a stand-in | a small test executable used in place of k3s, kine or postgres |
| the soak | the last run against the real accounts (the bobvmbuilder bidder and alice), after the market network passes |
| a gate | a check that must pass before an item or a merge goes on (`deploy/check.sh`, a plan's done_when) |
| the seed | the first example that a model grows from |
## Landing changes
Agents never push to the base branch (`main`; `runner-atlibs-composition` in the knotless fork): it is protected with
`--queue`, and the merge queue lands a change once its checks pass. Push a branch, open a pull and queue it:
```sh
U=knotless://socialweb.computer/socialweb-computer-kcp
git push swc HEAD:refs/heads/<branch> # remote swc = $U
knotless pull create $U <base> $U <branch> "<title>" -m "<why>"
knotless pull merge $U <pull> --queue
knotless queue list $U
```
`knotless pull show $U <pull>` says where it is; a pull whose checks fail is ejected from the queue (fix the branch, queue it
again).
Out of band first (owner, 2026-11-19; plan 0040 decision 51): a change under the workflows' `paths:` is run by hand,
off CI, before its pull is queued: a local market (`docs/quickstart-local.md`), a held staging network or a hand prod
entry (`docs/ci-debug.md`), a runner module's own test. `deploy/check.sh --fast` and a review are not that run. The
pull's message says what was run and what was seen. A push to main starts `release`, `test`, `promote`, and a green
promote moves the `stable` channel users install from: one code change per cycle.
## Security controls: propose, never implement
Never add a security control unless the owner explicitly approved that exact control. That includes:
- caps and size limits;
- refusals and validation that rejects input;
- allowlists and denylists;
- scope or permission checks;
- sandbox rules;
- timeouts that cut off or refuse a party "for safety" (a time limit that lets a hung request be retried is not a control:
owner, 2026-11-16).
If you think one is needed, add it to `docs/security/proposed-controls.md` and tell the owner. Do not implement it.
```md
## <short name>
- What: <the check, one line>
- Where: <file:line or component>
- Threat: <who does what, and what this stops>
```
Found an existing control the owner never approved? List it there under "Existing, not owner-approved". Do not remove or extend it on your own.
## Plan document before work
Every change executes a plan in `docs/plans/NNNN-<slug>.md`. No plan covers the issue? Write one first, land it through the merge queue, then execute it.
```md
# NNNN - <title>
Status: plan | approved by the owner <date> | done
## Why
<one or two lines>
## Steps
1. …
## Order
1. …
```
- Plans hold the facts: numbers, names, decisions. Keep them current in the plan, not in chat.
- Brief subagents with the plan's path.
- `.plans.yaml` lists the plans in flight: plan path, agent ID, status, next step. Add your entry when you start a plan, keep
it current as you go, remove it when the plan is done. Read it before picking up work: a plan another agent holds is theirs.