~ knotless Search/
↑↓ move↵ open⇧↵ all results

socialweb.computer/socialweb-computer-kcp

socialweb.computer: a compute market on atproto (kcp)

1 collaborator
0
0
208 lines · 14.4 KB Raw History
# protocol: the atproto protocol of socialweb-computer

- every record, XRPC method and stream that socialweb-computer applications speak
- applications talk ONLY over this protocol. a shared kcp is composition and isolation, never a data channel
- lexicon files: `lexicons/` (embedded by `lexicons/embed.go`, Go types generated by `internal/lexgen`)
- companion of [../SYSTEM.md](../SYSTEM.md) (the whole system) and [../KCP.md](../KCP.md) (kcp objects)
- no history here. what changed when: git log, `docs/plans/*.yaml`

## Files In This Tree

| doc | what |
|---|---|
| [README.md](README.md) | this index: NSID families, where each is documented, the end-to-end sequence |
| [flow.md](flow.md) | one contract end to end, step by step, with real records: the story, S0 to S9, the failure branches, a real run |
| [compute-contract-flow-map.md](compute-contract-flow-map.md) | the owner's end-to-end map of the layers and the flow, from the lexicons to the guest transport |
| [glossary.md](glossary.md) | every NSID on the wire, one line each, by namespace |
| [release.md](release.md) | `software.release.*`: channels, versions, manifests; how install.sh and a bidder follow them |
| [market.md](market.md) | the RFP flow: rfp, bid, accept, receipt, event, offering, heartbeat, relays, submit* XRPC, streams, selection |
| [guest-events.md](guest-events.md) | `compute.events.vm.*`, `compute.events.guest.*`, `compute.guest.subscribeEvents`, secrets, agent start, guest requester (R2) |
| [lexicons-market.md](lexicons-market.md) | reference rows for every lexicon under `com/publicdomainrelay/temp/` except `cluster.*` and `badgeBlueKeys` |
| [infra.md](infra.md) | `computer.socialweb.infra`, `ephemeralInfra`, `k8s.*`, `cluster.*`: the deployment records and the gateway cluster API |
| [identity.md](identity.md) | `network.attested.*`, `badgeBlueKeys`, associations (plan 0036), did:key ingress, sign-in |
| [policy.md](policy.md) | policy records (gha-lite workflows), the strongRefs an RFP attaches, tangled-vouch |
| [outside.md](outside.md) | lexicons we use but do not own: `sh.tangled.graph.vouch`, `chat.atirc.*`, `com.atproto.*`, `app.bsky.*`, `com.fedproxy.*`, OAuth scopes per role |

## The Lexicon Reference

The lexicon files under `lexicons/` are the one source of every record's and method's fields. The reference is generated
from them: https://socialweb.computer/docs/lexicons/ (one page per NSID: type, key, fields, refs, errors, the raw schema
and the docs pages that name it) and https://socialweb.computer/docs/xrpc/ (the XRPC methods). A docs page that shows a
lexicon's fields includes the generated table (a `[lexicon: <nsid>](https://socialweb.computer/docs/lexicons/<nsid>.html)`
line, rendered as the table on the site) and adds only what the code does with it:

| row | meaning |
|---|---|
| rkey | `self`, a deterministic key (named), `tid`, or `any` with the scheme the writer uses |
| written by / read by | the application role and the Go package |
| step | the step of the [end-to-end sequence](#end-to-end-sequence) where it appears |

## NSID Families

| family | owner | lexicon files | kind | doc |
|---|---|---|---|---|
| `com.publicdomainrelay.temp.market.*` | us | `lexicons/com/publicdomainrelay/temp/market/` | records + XRPC | [market.md](market.md), [lexicons-market.md](lexicons-market.md) |
| `com.publicdomainrelay.temp.market.association`, `associationRequest` | us | same | records | flow: [identity.md](identity.md#associations-identlink); rows: [lexicons-market.md](lexicons-market.md#marketassociation) |
| `com.publicdomainrelay.temp.compute.vm` | us | `lexicons/com/publicdomainrelay/temp/compute/vm.json` | record (RFP payload) | [market.md](market.md), [lexicons-market.md](lexicons-market.md) |
| `com.publicdomainrelay.temp.compute.events.*`, `compute.guest.subscribeEvents` | us | `lexicons/com/publicdomainrelay/temp/compute/` | event payload records, subscription | [guest-events.md](guest-events.md) |
| `com.publicdomainrelay.temp.ingress.webRTC` | us | `lexicons/com/publicdomainrelay/temp/ingress/` | record | [lexicons-market.md](lexicons-market.md) |
| `com.publicdomainrelay.temp.cluster.*` | us | `lexicons/com/publicdomainrelay/temp/cluster/` | XRPC (gateway clusters) | [infra.md](infra.md#compublicdomainrelaytempcluster) |
| `com.publicdomainrelay.temp.badgeBlueKeys` | us | `lexicons/com/publicdomainrelay/temp/badgeBlueKeys.json` | record (key claim) | [identity.md](identity.md#badgebluekeys) |
| `computer.socialweb.infra`, `computer.socialweb.ephemeralInfra` | us | `lexicons/computer/socialweb/` | records (deployments) | [infra.md](infra.md) |
| `computer.socialweb.k8s.*` | us | `lexicons/computer/socialweb/k8s/` | XRPC (gateway k8s API) | [infra.md](infra.md#computersocialwebk8s) |
| `network.attested.*` | not ours (a copy here) | `lexicons/network/attested/` | object, record, query; `proof` and `verify` are not written, served or called here | [identity.md](identity.md#networkattested) |
| `com.atproto.repo.strongRef` | atproto (vendored copy) | `lexicons/com/atproto/repo/strongRef.json` | object | [outside.md](outside.md#comatprotorepostrongref) |
| `computer.socialweb.temp.policy.*`, `com.publicdomainrelay.temp.market.policies.denoWorker` | us | `lexicons/computer/socialweb/temp/policy/`, `lexicons/com/publicdomainrelay/temp/market/policies/` | records (policy) | [policy.md](policy.md) |
| `com.publicdomainrelay.temp.compute.config.wif.simple`, `compute.secrets.getSecrets` | us | `lexicons/com/publicdomainrelay/temp/compute/` | record (bid config), query (the guest's secrets) | [guest-events.md](guest-events.md#secrets-getsecrets-or-sealed-fallback) |
| `com.publicdomainrelay.temp.auth.allowlist.rbacDid`, `com.publicdomainrelay.temp.keepalive.ping` | us | `lexicons/com/publicdomainrelay/temp/` | record (scope only), the ingress relay's keepalive | [outside.md](outside.md#scope-only-collections) |
| `computer.socialweb.cicd.process.*` | us | `lexicons/computer/socialweb/cicd/process/` | XRPC (the guest agent's shell socket) | - |
| `sh.tangled.graph.vouch` | tangled | none here | record (trust edge) | [outside.md](outside.md#shtangledgraphvouch), [policy.md](policy.md#tangled-vouch) |
| `chat.atirc.mint`, `chat.atirc.revoke` | atirc | atirc repo | XRPC | [outside.md](outside.md#chatatirc) |
| `com.atproto.*`, `app.bsky.*` | atproto, Bluesky | upstream | XRPC, records | [outside.md](outside.md) |
| `com.fedproxy.rbac`, `com.fedproxy.sshPublicKey`, `com.fedproxy.temp.xrpc.*` | us | `lexicons/com/fedproxy/` | records (workload roles, tunnel keys), XRPC (the did:key ingress relay) | [outside.md](outside.md#comfedproxy) |
| `software.release.*` | not ours (a copy here) | `lexicons/software/release/` | records (releases) | [release.md](release.md) |
| `computer.socialweb.ingress.*`, `computer.socialweb.peer` | us | `lexicons/computer/socialweb/` | records (webrtc ingress, peers) | [market.md](market.md#xrpc-procedures), [lexicons-market.md](lexicons-market.md#computersocialwebpeer) |
| `link.ident.endorsement` | ident.link (vendored) | `lexicons/link/ident/endorsement.json` | record | [identity.md](identity.md#associations-identlink) |
| `com.fedfork.exec.ops.*` | us | `lexicons/com/fedfork/exec/ops/` | XRPC (read-only ops routes) | [../SYSTEM.md](../SYSTEM.md#xrpc) |

One line per NSID: [glossary.md](glossary.md).

### NSIDs we own that have no lexicon file

| NSID | where | doc |
|---|---|---|
| `com.fedfork.exec.binding`, `com.fedfork.exec.pins` | their lexicons are atlibs' `lexicons/com/fedfork/exec/`; `hack/pinsrecords` reads pins | [../ci-debug.md](../ci-debug.md) |
| `com.fedfork.atprotoOauthQR` | `internal/atproto/oauthclient/qr.go` `QRNSID` (client), `did-key-associator/hono-backend/mod.ts` (server); one NSID served as GET and POST, which no single lexicon method can describe | [glossary.md](glossary.md) |

## Go Packages

| package | holds |
|---|---|
| `lexicons` | the lexicon JSON, `embed.FS` of `com`, `computer`, `network` |
| `internal/lexgen` | the generator: lexicon JSON -> Go record types, `Validate`, `decodeRecord` |
| `internal/atproto/market` | generated market, compute, ingress, badgeBlueKeys, `network.attested.signature`, strongRef types; NSID constants (`nsid.go`), stable rkeys |
| `internal/atproto/infra` | generated `computer.socialweb.infra` / `ephemeralInfra` types (`nsid.go`) |
| `internal/atproto/oauth` | OAuth scope sets per role (`scope.go`) |
| `internal/xrpcserver` | the gateway XRPC server: `cluster.*`, `computer.socialweb.k8s.*`; validates against the embedded catalog |
| `internal/atproto/pdsclient` | the PDS client: getRecord, putRecord, createRecord, listRecords, getServiceAuth, subscribeRepos |
| `internal/eventstream` | the market event streams and the one handler sink per role |
| `internal/market/{requester,bidder,participant}` | the two sides of the RFP flow |
| `internal/policyengine` | the policy engine: executor, trust (vouches, associations), bundled gha-lite workflows |

## End-To-End Sequence

The steps below name every record in the order it is written. Other docs refer to them as S0 to S9. Each step with its real records: [flow.md](flow.md).

```mermaid
sequenceDiagram
  autonumber
  participant U as user / operator
  participant R as Requester (CLI, page, iOS, gateway)
  participant RP as requester repo
  participant B as Bidder
  participant BP as bidder repo
  participant G as guest VM
  participant A as atirc

  U->>R: S0 sign in (OAuth, scopes per role)
  U->>RP: S1 badgeBlueKeys claim, sh.tangled.graph.vouch
  R->>RP: S2 computer.socialweb.infra (or ephemeralInfra) mirror
  R->>RP: S3 policy record, compute.vm, market.rfp (signed)
  R->>BP: discovery: getRecord market.offering + market.heartbeat at self
  R->>B: submitRfp (and RP-->>B: firehose market.rfp)
  B->>RP: getRecord vm payload + attached policies
  B->>BP: S4 compute.config.wif.simple, bids.free, market.bid
  B->>R: submitBid
  R->>RP: S5 market.accept (+ market.association for a named subject)
  R->>B: submitAccept (capacity checked, reserved)
  B->>BP: S6 market.receipt (payload = the compute.vm ref)
  B->>G: boot from cloud-init user_data
  B->>BP: S7 market.event vm.hostKeys, vm.onNetwork
  B->>R: submitEvent
  R->>RP: S8 market.event guest.* (secrets, agentStart, r2Start; for R2: market.association after its nonce)
  R->>A: chat.atirc.mint (service auth)
  R->>A: S9 chat.atirc.revoke
  R->>RP: S9 market.event vm.delete
  B->>BP: S9 market.event vm.terminated
  R->>RP: S9 delete own contract records (rfp, vm, policy, accept, delete, rbac, association)
  R->>RP: S9 infra status released / ended (record kept)
```

<a id="step-s0"></a>
### S0: sign in

- a MarketIdentity signs in; OAuth sessions live in the app's own OpenBao (`auth/atproto session/<did>`)
- the scope set depends on the role: [outside.md](outside.md#oauth-scopes-per-role)
- dev networks use password accounts (`com.atproto.server.createSession`)
- detail: [identity.md](identity.md#sign-in)

<a id="step-s1"></a>
### S1: trust

- an operator ties a bidder or requester DID to itself with a `badgeBlueKeys` claim: [identity.md](identity.md#badgebluekeys)
- operators and users vouch for each other with `sh.tangled.graph.vouch`: [outside.md](outside.md#shtangledgraphvouch)
- the default policy on both sides is tangled-vouch: [policy.md](policy.md#tangled-vouch)
- standing associations and contract-scoped ones: [identity.md](identity.md#associations-identlink)

<a id="step-s2"></a>
### S2: declare the deployment

- an Infra or EphemeralInfra kcp object mirrors into `computer.socialweb.infra` / `ephemeralInfra` in the owner's repo
- the per-account index sits at rkey `self`. records are never deleted
- detail: [infra.md](infra.md)

<a id="step-s3"></a>
### S3: policy, vm, rfp

- the requester writes its policy record (gha-lite), then `compute.vm` and a signed `market.rfp` whose `policies` strongRefs it
- detail: [flow.md](flow.md#s3-policy-vm-rfp), [policy.md](policy.md#rfp-policy-strongrefs), [market.md](market.md)

<a id="step-s4"></a>
### S4: bid

- bidders read the RFP from the firehose or from `submitRfp`, run their policy, write `market.bid`, its payload `bids.free` (rkey `<bid rkey>-payload`) and its config `compute.config.wif.simple` (rkey `<bid rkey>-config`)
- detail: [flow.md](flow.md#s4-bid), [market.md](market.md)

<a id="step-s5"></a>
### S5: accept

- the requester runs its policy, picks a bid, writes `market.accept` (no payload record); the bidder checks capacity and reserves (409 `CapacityExceeded` otherwise)
- a requester whose guest runs a guest requester (R2) writes the contract-scoped `market.association` after R2 answers the nonce (S8): [identity.md](identity.md#guest-requester-r2)
- detail: [flow.md](flow.md#s5-accept), [market.md](market.md)

<a id="step-s6"></a>
### S6: receipt

- the winner writes `market.receipt` (its `payload` is the RFP's `compute.vm` ref, its `cid` the accept's attestation CID) and boots the guest
- detail: [flow.md](flow.md#s6-receipt), [market.md](market.md)

<a id="step-s7"></a>
### S7: boot and vm events

- `vm.hostKeys`, `vm.onNetwork`; the ComputeContract goes Held on the iroh address and host keys
- detail: [flow.md](flow.md#s7-boot-and-vm-events), [guest-events.md](guest-events.md)

<a id="step-s8"></a>
### S8: guest events and atirc

- secrets bundle, agent start, guest requester start; the requester mints an atirc child token
- detail: [flow.md](flow.md#s8-guest-events-secrets-held), [guest-events.md](guest-events.md), [outside.md](outside.md#chatatirc)

<a id="step-s9"></a>
### S9: release

- atirc token revoked, `vm.delete`, the bidder's `vm.terminated`; the requester deletes the records it wrote for the contract (rfp, compute.vm, policy, accept, its delete records, rbac, association) and a guest requester's DIDs are tombstoned
- the Infra record keeps its history: status `released`, then `ended`. user records are never deleted
- detail: [flow.md](flow.md#s9-release-and-cleanup), [market.md](market.md#release-and-termination), [guest-events.md](guest-events.md), [infra.md](infra.md#lifecycle)

## Records Policy (all families)

- never delete user records. no sweep, retention or GC of people's repo records
- release deletes only the records the app wrote for that contract ([market.md](market.md#release-and-termination) step 9)
- one `getRecord` at a deterministic rkey; `listRecords` only as a legacy fallback
- full list of stable rkeys: [../SYSTEM.md](../SYSTEM.md#records-policy)

## See Also

- [../SYSTEM.md](../SYSTEM.md#market-protocol), [../KCP.md](../KCP.md), [../ARCHITECTURE.md](../ARCHITECTURE.md)
- plans: `docs/plans/0035-atirc-agent.yaml`, `docs/plans/0036-record-association.yaml`, `docs/plans/0039-long-lived-requester.yaml`