~ knotless Search/
↑↓ move↵ open⇧↵ all results

socialweb.computer/socialweb-computer-kcp

socialweb.computer: a compute market on atproto (kcp)

1 collaborator
0
0
208 lines · 14.4 KB Raw History

protocol: the atproto protocol of socialweb-computer

  • every record, XRPC method and stream that socialweb-computer applications speak
  • applications talk ONLY over this protocol. a shared kcp is composition and isolation, never a data channel
  • lexicon files: lexicons/ (embedded by lexicons/embed.go, Go types generated by internal/lexgen)
  • companion of ../SYSTEM.md (the whole system) and ../KCP.md (kcp objects)
  • no history here. what changed when: git log, docs/plans/*.yaml

Files In This Tree

docwhat
README.mdthis index: NSID families, where each is documented, the end-to-end sequence
flow.mdone contract end to end, step by step, with real records: the story, S0 to S9, the failure branches, a real run
compute-contract-flow-map.mdthe owner's end-to-end map of the layers and the flow, from the lexicons to the guest transport
glossary.mdevery NSID on the wire, one line each, by namespace
release.mdsoftware.release.*: channels, versions, manifests; how install.sh and a bidder follow them
market.mdthe RFP flow: rfp, bid, accept, receipt, event, offering, heartbeat, relays, submit* XRPC, streams, selection
guest-events.mdcompute.events.vm.*, compute.events.guest.*, compute.guest.subscribeEvents, secrets, agent start, guest requester (R2)
lexicons-market.mdreference rows for every lexicon under com/publicdomainrelay/temp/ except cluster.* and badgeBlueKeys
infra.mdcomputer.socialweb.infra, ephemeralInfra, k8s.*, cluster.*: the deployment records and the gateway cluster API
identity.mdnetwork.attested.*, badgeBlueKeys, associations (plan 0036), did:key ingress, sign-in
policy.mdpolicy records (gha-lite workflows), the strongRefs an RFP attaches, tangled-vouch
outside.mdlexicons we use but do not own: sh.tangled.graph.vouch, chat.atirc.*, com.atproto.*, app.bsky.*, com.fedproxy.*, OAuth scopes per role

The Lexicon Reference

The lexicon files under lexicons/ are the one source of every record's and method's fields. The reference is generated from them: https://socialweb.computer/docs/lexicons/ (one page per NSID: type, key, fields, refs, errors, the raw schema and the docs pages that name it) and https://socialweb.computer/docs/xrpc/ (the XRPC methods). A docs page that shows a lexicon's fields includes the generated table (a [lexicon: <nsid>](https://socialweb.computer/docs/lexicons/<nsid>.html) line, rendered as the table on the site) and adds only what the code does with it:

rowmeaning
rkeyself, a deterministic key (named), tid, or any with the scheme the writer uses
written by / read bythe application role and the Go package
stepthe step of the end-to-end sequence where it appears

NSID Families

familyownerlexicon fileskinddoc
com.publicdomainrelay.temp.market.*uslexicons/com/publicdomainrelay/temp/market/records + XRPCmarket.md, lexicons-market.md
com.publicdomainrelay.temp.market.association, associationRequestussamerecordsflow: identity.md; rows: lexicons-market.md
com.publicdomainrelay.temp.compute.vmuslexicons/com/publicdomainrelay/temp/compute/vm.jsonrecord (RFP payload)market.md, lexicons-market.md
com.publicdomainrelay.temp.compute.events.*, compute.guest.subscribeEventsuslexicons/com/publicdomainrelay/temp/compute/event payload records, subscriptionguest-events.md
com.publicdomainrelay.temp.ingress.webRTCuslexicons/com/publicdomainrelay/temp/ingress/recordlexicons-market.md
com.publicdomainrelay.temp.cluster.*uslexicons/com/publicdomainrelay/temp/cluster/XRPC (gateway clusters)infra.md
com.publicdomainrelay.temp.badgeBlueKeysuslexicons/com/publicdomainrelay/temp/badgeBlueKeys.jsonrecord (key claim)identity.md
computer.socialweb.infra, computer.socialweb.ephemeralInfrauslexicons/computer/socialweb/records (deployments)infra.md
computer.socialweb.k8s.*uslexicons/computer/socialweb/k8s/XRPC (gateway k8s API)infra.md
network.attested.*not ours (a copy here)lexicons/network/attested/object, record, query; proof and verify are not written, served or called hereidentity.md
com.atproto.repo.strongRefatproto (vendored copy)lexicons/com/atproto/repo/strongRef.jsonobjectoutside.md
computer.socialweb.temp.policy.*, com.publicdomainrelay.temp.market.policies.denoWorkeruslexicons/computer/socialweb/temp/policy/, lexicons/com/publicdomainrelay/temp/market/policies/records (policy)policy.md
com.publicdomainrelay.temp.compute.config.wif.simple, compute.secrets.getSecretsuslexicons/com/publicdomainrelay/temp/compute/record (bid config), query (the guest's secrets)guest-events.md
com.publicdomainrelay.temp.auth.allowlist.rbacDid, com.publicdomainrelay.temp.keepalive.pinguslexicons/com/publicdomainrelay/temp/record (scope only), the ingress relay's keepaliveoutside.md
computer.socialweb.cicd.process.*uslexicons/computer/socialweb/cicd/process/XRPC (the guest agent's shell socket)-
sh.tangled.graph.vouchtanglednone hererecord (trust edge)outside.md, policy.md
chat.atirc.mint, chat.atirc.revokeatircatirc repoXRPCoutside.md
com.atproto.*, app.bsky.*atproto, BlueskyupstreamXRPC, recordsoutside.md
com.fedproxy.rbac, com.fedproxy.sshPublicKey, com.fedproxy.temp.xrpc.*uslexicons/com/fedproxy/records (workload roles, tunnel keys), XRPC (the did:key ingress relay)outside.md
software.release.*not ours (a copy here)lexicons/software/release/records (releases)release.md
computer.socialweb.ingress.*, computer.socialweb.peeruslexicons/computer/socialweb/records (webrtc ingress, peers)market.md, lexicons-market.md
link.ident.endorsementident.link (vendored)lexicons/link/ident/endorsement.jsonrecordidentity.md
com.fedfork.exec.ops.*uslexicons/com/fedfork/exec/ops/XRPC (read-only ops routes)../SYSTEM.md

One line per NSID: glossary.md.

NSIDs we own that have no lexicon file

NSIDwheredoc
com.fedfork.exec.binding, com.fedfork.exec.pinstheir lexicons are atlibs' lexicons/com/fedfork/exec/; hack/pinsrecords reads pins../ci-debug.md
com.fedfork.atprotoOauthQRinternal/atproto/oauthclient/qr.go QRNSID (client), did-key-associator/hono-backend/mod.ts (server); one NSID served as GET and POST, which no single lexicon method can describeglossary.md

Go Packages

packageholds
lexiconsthe lexicon JSON, embed.FS of com, computer, network
internal/lexgenthe generator: lexicon JSON -> Go record types, Validate, decodeRecord
internal/atproto/marketgenerated market, compute, ingress, badgeBlueKeys, network.attested.signature, strongRef types; NSID constants (nsid.go), stable rkeys
internal/atproto/infragenerated computer.socialweb.infra / ephemeralInfra types (nsid.go)
internal/atproto/oauthOAuth scope sets per role (scope.go)
internal/xrpcserverthe gateway XRPC server: cluster.*, computer.socialweb.k8s.*; validates against the embedded catalog
internal/atproto/pdsclientthe PDS client: getRecord, putRecord, createRecord, listRecords, getServiceAuth, subscribeRepos
internal/eventstreamthe market event streams and the one handler sink per role
internal/market/{requester,bidder,participant}the two sides of the RFP flow
internal/policyenginethe policy engine: executor, trust (vouches, associations), bundled gha-lite workflows

End-To-End Sequence

The steps below name every record in the order it is written. Other docs refer to them as S0 to S9. Each step with its real records: flow.md.

sequenceDiagram
  autonumber
  participant U as user / operator
  participant R as Requester (CLI, page, iOS, gateway)
  participant RP as requester repo
  participant B as Bidder
  participant BP as bidder repo
  participant G as guest VM
  participant A as atirc

  U->>R: S0 sign in (OAuth, scopes per role)
  U->>RP: S1 badgeBlueKeys claim, sh.tangled.graph.vouch
  R->>RP: S2 computer.socialweb.infra (or ephemeralInfra) mirror
  R->>RP: S3 policy record, compute.vm, market.rfp (signed)
  R->>BP: discovery: getRecord market.offering + market.heartbeat at self
  R->>B: submitRfp (and RP-->>B: firehose market.rfp)
  B->>RP: getRecord vm payload + attached policies
  B->>BP: S4 compute.config.wif.simple, bids.free, market.bid
  B->>R: submitBid
  R->>RP: S5 market.accept (+ market.association for a named subject)
  R->>B: submitAccept (capacity checked, reserved)
  B->>BP: S6 market.receipt (payload = the compute.vm ref)
  B->>G: boot from cloud-init user_data
  B->>BP: S7 market.event vm.hostKeys, vm.onNetwork
  B->>R: submitEvent
  R->>RP: S8 market.event guest.* (secrets, agentStart, r2Start; for R2: market.association after its nonce)
  R->>A: chat.atirc.mint (service auth)
  R->>A: S9 chat.atirc.revoke
  R->>RP: S9 market.event vm.delete
  B->>BP: S9 market.event vm.terminated
  R->>RP: S9 delete own contract records (rfp, vm, policy, accept, delete, rbac, association)
  R->>RP: S9 infra status released / ended (record kept)

<a id="step-s0"></a>

S0: sign in

  • a MarketIdentity signs in; OAuth sessions live in the app's own OpenBao (auth/atproto session/<did>)
  • the scope set depends on the role: outside.md
  • dev networks use password accounts (com.atproto.server.createSession)
  • detail: identity.md

<a id="step-s1"></a>

S1: trust

  • an operator ties a bidder or requester DID to itself with a badgeBlueKeys claim: identity.md
  • operators and users vouch for each other with sh.tangled.graph.vouch: outside.md
  • the default policy on both sides is tangled-vouch: policy.md
  • standing associations and contract-scoped ones: identity.md

<a id="step-s2"></a>

S2: declare the deployment

  • an Infra or EphemeralInfra kcp object mirrors into computer.socialweb.infra / ephemeralInfra in the owner's repo
  • the per-account index sits at rkey self. records are never deleted
  • detail: infra.md

<a id="step-s3"></a>

S3: policy, vm, rfp

  • the requester writes its policy record (gha-lite), then compute.vm and a signed market.rfp whose policies strongRefs it
  • detail: flow.md, policy.md, market.md

<a id="step-s4"></a>

S4: bid

  • bidders read the RFP from the firehose or from submitRfp, run their policy, write market.bid, its payload bids.free (rkey <bid rkey>-payload) and its config compute.config.wif.simple (rkey <bid rkey>-config)
  • detail: flow.md, market.md

<a id="step-s5"></a>

S5: accept

  • the requester runs its policy, picks a bid, writes market.accept (no payload record); the bidder checks capacity and reserves (409 CapacityExceeded otherwise)
  • a requester whose guest runs a guest requester (R2) writes the contract-scoped market.association after R2 answers the nonce (S8): identity.md
  • detail: flow.md, market.md

<a id="step-s6"></a>

S6: receipt

  • the winner writes market.receipt (its payload is the RFP's compute.vm ref, its cid the accept's attestation CID) and boots the guest
  • detail: flow.md, market.md

<a id="step-s7"></a>

S7: boot and vm events

  • vm.hostKeys, vm.onNetwork; the ComputeContract goes Held on the iroh address and host keys
  • detail: flow.md, guest-events.md

<a id="step-s8"></a>

S8: guest events and atirc

<a id="step-s9"></a>

S9: release

  • atirc token revoked, vm.delete, the bidder's vm.terminated; the requester deletes the records it wrote for the contract (rfp, compute.vm, policy, accept, its delete records, rbac, association) and a guest requester's DIDs are tombstoned
  • the Infra record keeps its history: status released, then ended. user records are never deleted
  • detail: flow.md, market.md, guest-events.md, infra.md

Records Policy (all families)

  • never delete user records. no sweep, retention or GC of people's repo records
  • release deletes only the records the app wrote for that contract (market.md step 9)
  • one getRecord at a deterministic rkey; listRecords only as a legacy fallback
  • full list of stable rkeys: ../SYSTEM.md

See Also