~ knotless Search/
↑↓ move↵ open⇧↵ all results

socialweb.computer/socialweb-computer-kcp

socialweb.computer: a compute market on atproto (kcp)

1 collaborator
0
0
262 lines · 17.1 KB Raw History

infra: deployment records and the gateway cluster API

  • lexicons: lexicons/computer/socialweb/ and lexicons/com/publicdomainrelay/temp/cluster/
  • an Infra or EphemeralInfra kcp object is a deployment; its record in the owner's repo is the same deployment, readable from any device
  • the gateway serves tenant Kubernetes clusters over two XRPC families: cluster.* (lifecycle, kubeconfig) and computer.socialweb.k8s.* (apply, read a Job)
  • index: README.md. sequence step: S2 and S9

Lexicons In This File

NSIDtyperkeydoc
computer.socialweb.infrarecordobject name, successor rkey, self = indexinfra
computer.socialweb.ephemeralInfrarecordobject name, successor rkeyephemeralInfra
computer.socialweb.k8s.kubectlApplyServerSideprocedure-k8s
computer.socialweb.k8s.getJobquery-k8s
com.publicdomainrelay.temp.cluster.createClusterprocedure-cluster
com.publicdomainrelay.temp.cluster.getKubeconfigprocedure-cluster
com.publicdomainrelay.temp.cluster.listClustersquery-cluster
com.publicdomainrelay.temp.cluster.controlPlaneUpprocedure-cluster
com.publicdomainrelay.temp.cluster.controlPlaneDownprocedure-cluster
com.publicdomainrelay.temp.cluster.deleteClusterprocedure-cluster
com.publicdomainrelay.temp.cluster.revokeIdentityprocedure-cluster
com.publicdomainrelay.temp.cluster.listKubeconfigsquery-cluster
com.publicdomainrelay.temp.cluster.revokeKubeconfigprocedure-cluster
com.publicdomainrelay.temp.cluster.subscribeClusterssubscription-cluster

Objects And Records

flowchart LR
  subgraph KCP["owner's kcp (CLI core, gateway, page, iOS)"]
    INF["Infra / EphemeralInfra"]
    AR["AtprotoRecord (child, Retain)"]
    CC["ComputeContract (one per contract in requested state)"]
    IC["InfraCollection / EphemeralInfraCollection"]
  end
  subgraph REPO["owner's repo"]
    REC["computer.socialweb.infra/RKEY"]
    IDX["computer.socialweb.infra/self (index)"]
  end
  INF -->|apply| AR
  AR -->|putRecord| REC
  INF -->|putRecord swap cid| IDX
  REC -->|pull (sync Both)| INF
  IDX -->|getRecord| IC
  IC -->|create child bound to uri| INF
  INF -->|state requested| CC
  CC -->|RFP flow| MKT["market.md"]
  • the infra controller never writes the record itself: it applies a child AtprotoRecord with DeletionPolicy: Retain; the atprotorecord controller does the putRecord (internal/controllers/infra/record.go, internal/controllers/atprotorecord/apply.go)
  • the index at self is written by the infra controller directly, compare-and-swap on the cid (internal/controllers/infra/index.go mutateIndex)
  • the reverse direction is real: a changed record is pulled into the object spec; an InfraCollection reads self and creates one child object per live entry (internal/controllers/infracollection/controller.go)
  • one ComputeContract per contract in state requested runs the RFP flow: market.md
kcp kindapicontrollerrecord
Infraapi/v1alpha1/types_infra.gointernal/controllers/infracomputer.socialweb.infra
EphemeralInfraapi/v1alpha1/types_infra.go (EphemeralInfraSpec)internal/controllers/infracomputer.socialweb.ephemeralInfra
InfraCollection, EphemeralInfraCollectionapi/v1alpha1/types_infracollection.gointernal/controllers/infracollectionreads the index at self
AtprotoRecordapi/v1alpha1internal/controllers/atprotorecordany collection (here: the two above)
  • lifecycle and declaration logic (pure, no I/O): internal/infra (declaration.go, lifecycle.go, sync.go, index.go)
  • generated record types: internal/atproto/infra (nsid.go, zz_generated.lexicons.go)
  • registered in the CLI and gateway core with writer label core (internal/stack/componentcore/controllers.go) and in the iOS library with embedded (pkg/embedded/embedded.go)

computer.socialweb.infra

fieldvalue
NSIDcomputer.socialweb.infra
typerecord, key any
key fieldscontracts[] (#contract), status (#status), index (only at self), description, createdAt
rkeythe object name; a successor rkey when that one is taken by an ended record (rkeys); self is the index
written bythe infra controller of the owner's kcp, via a child AtprotoRecord
read bythe same controller (pull, adopt), InfraCollection (index), any device of the owner
Gointernal/atproto/infra, internal/controllers/infra, internal/infra
stepS2 declare, S9 released / ended

#contract

fieldmeaning
nameDNS-1123 label, unique in the record
statestaged (declared, no RFP), requested (run the RFP flow, hold the guest), released (release the guest)
releaseRequestedAtwith released: a newer time asks a failed release again
vmName, computeKindvm (firecracker, the default) or deno (a deno sandbox that runs userData as its program)
userDatacloud-init user_data. public here: never a secret
bidderDids, denyBidderDidsbidder allow / deny lists
sshAuthorizedKeyskeys for the guest
policy, policyArgsthe policy workflow name and its args: policy.md
vmReadyTimeoutSecreadiness bound
tunnels[]tag, port (guest 127.0.0.1), transport opentunnel (default, https URL) or webrtc, optional server, shell.envFile
agentan atirc agent on the guest, envFile (default /etc/atirc/atirc.env); token arrives in the secrets bundle: guest-events.md
associationa guest requester (R2): subject or fromGuest, refFile (default /etc/socialweb-computer/association-ref): identity.md
guestToolstools the bidder installs from its own downloads

Contract state to ComputeContract

record statewhat the controller does (internal/infra/lifecycle.go, internal/controllers/infra/contracts.go)
stagedno ComputeContract; a finished one is deleted
requestedcreates the ComputeContract when none exists; it runs the RFP flow
releasedsets the ComputeContract's release; a releaseRequestedAt newer than the contract's own sends the release again; a Failed contract is released again
(object ends)every contract is forced to released

#status

fieldvalues
phasepending, staged, deploying, deployed, releasing, released, degraded, ending, deleting, ended
contracts[] (#contractStatus)name, phase (staged, requesting, deployed, releasing, released, failed), winner (bidder DID), receipt (at-uri), message
endedAt, endReasonset once, when the deployment ends. endReason deleted on Infra
  • status comes from the observed object status; progress writes are rate limited (records policy: at most once a minute)

#index (rkey self)

fieldmeaning
live[] (#indexEntry)collection (computer.socialweb.infra or ephemeralInfra), rkey, createdAt, expiresAt (EphemeralInfra), addedAt, writer
recentEnded[] (#indexEnded)collection, rkey, endedAt, endReason; newest first
updatedAtlast write
  • one index per account, at computer.socialweb.infra/self, for both collections
  • limits (internal/infra/index.go): at most 256 live entries (IndexMaxLive), 32 recent ended (IndexMaxEnded); an entry whose record is gone counts as gone after 10 minutes (IndexGoneAfter)
  • indexPass runs on every reconcile: adds or refreshes the entry while the record is live, removes it once the record ended. indexEnd runs at deletion or unbind
  • writes: putRecord with swap cid, createRecord when no index exists yet; retried on a swap mismatch
  • a record with index is not a deployment; nothing else is ever written at self
  • writer: the lexicon names page:<origin host>, ios, gateway:<host>, cli. this repo passes core (componentcore: CLI, gateway, bidder cores) and embedded (pkg/embedded, the iOS app); the label is trimmed to 64 characters

computer.socialweb.ephemeralInfra

fieldvalue
NSIDcomputer.socialweb.ephemeralInfra
typerecord, key any
key fieldscontracts[] (computer.socialweb.infra#contract), ttlSeconds (required), endWhen, status, createdAt
rkeythe object name or a successor, as for Infra
written by / read byas for Infra; listed in the same index at computer.socialweb.infra/self
Gosame packages
stepS2, S9
  • ttlSeconds: kcp default 3600, minimum 60. expiresAt = createdAt + ttlSeconds
  • endWhen: complete (default: ends when every contract is released) or deadline (ends only at expiresAt)
  • status.endReason: complete, deadline or deleted
  • an ended or expired record is never live again: re-creating the object takes a successor rkey

Rkeys

  • default rkey = the kcp object name. deterministic, never a TID (internal/controllers/infra/record.go)
  • spec.record.uri set: its rkey is followed instead (bind to an existing record)
  • successor rkeys (internal/infra/declaration.go Successor, InSuccession):
nrkey
0<name>
1<name>-<first 8 hex of the object uid>
2..64<name>-<uid8>-<n> (MaxSuccessors = 64)
  • a fresh bind moves to the next successor when the record at the current rkey has ended, is past its TTL, or is the reserved self
  • that covers an object deleted and re-created under the same name: the old record keeps its history, the new object gets a new rkey
  • a stored URI stays bound while its rkey is still in the succession of the same name and uid

Sync Modes

spec.record.sync on the object (InfraRecordBinding, api/v1alpha1/types_infra.go):

modedirection
Both (default)kcp -> record and record -> kcp
Publishkcp -> record only

internal/infra/sync.go Decide picks one action per reconcile:

actionwhen
Createno record yet
Pushthe kcp spec changed (or Publish and the record differs)
Pullonly the record changed (Both); server-side applied onto the spec
InSyncthe record hash equals the spec hash
Conflictboth changed since the last stored declaration hash
DeleteBoth, the bound record is gone: the kcp object is deleted
Endedthe record has ended: never written again
Reservedthe rkey is self

Lifecycle

sequenceDiagram
  autonumber
  participant U as owner (kubectl, page, iOS)
  participant I as infra controller
  participant AR as AtprotoRecord
  participant P as owner PDS
  participant C as ComputeContract

  U->>I: apply Infra (contracts staged)
  I->>AR: apply child, Retain
  AR->>P: putRecord computer.socialweb.infra/NAME
  I->>P: putRecord self (index, swap cid)
  U->>I: contract state requested
  I->>C: create ComputeContract
  C->>C: RFP flow (market.md)
  I->>P: status deployed, winner, receipt
  U->>I: contract state released
  I->>C: release
  U->>I: delete Infra
  I->>P: final write: phase ended, endedAt, endReason deleted (swap cid)
  I->>P: index: move entry to recentEnded
  I->>AR: force Retain, delete object (record stays)

Never Deleted

  • deleting the object runs finalize -> finalWrite: phase ended, endedAt (the deletion time), endReason (default deleted), written with swap cid (internal/controllers/infra/record.go)
  • unbindRecord forces the child AtprotoRecord to Retain before it deletes the child object
  • the atprotorecord finalizer never calls deleteRecord for computer.socialweb.infra or ephemeralInfra (keptCollections, internal/controllers/atprotorecord/controller.go)
  • a deletion that gives up, or an orphaned repo, leaves the record with its last status
  • one exception, the cluster mirror: an Infra with spec.clusterRef (the gateway's nodes-<hex>, internal/controllers/nodeclaim/infra.go) mirrors a tenant Cluster's nodes and is not a deployment anyone declared. Its end (delete, or unbind when the Cluster goes) pins the child AtprotoRecord to the record's cid, deletes the child, then deletes the record with swap cid (dropMirror); indexEnd drops its entry without a recentEnded line (IndexChange.Forget), and deletes self when no live or ended entry is left
  • see README.md

computer.socialweb.k8s

NSIDtypeinputoutput
computer.socialweb.k8s.kubectlApplyServerSideprocedurecluster, manifest (one object), namespace, name, fieldManager (default socialweb-computer), force, dryRunapiVersion, kind, name, uid, resourceVersion, generation, object
computer.socialweb.k8s.getJobquerycluster, namespace, name, tailLines (default 100, 64 KiB cap per log)phase (Pending, Running, Succeeded, Failed), counts, conditions, pods[].containers[].log
  • served by the GatewayAPI: internal/xrpcserver (K8sNSIDPrefix), wired in internal/gatewaymain/wiring.go and internal/services/gatewayapi/service.go
  • auth: bearer service-auth JWT, lxm = the NSID, aud one of the gateway audiences (did:web:k8s.socialweb.computer); internal/atprotoauth/atprotoauth.go Verify
  • the gateway checks the revocation permit and that the cluster is Ready (internal/gateway/tenantapi.go), then applies as the account's own ServiceAccount (internal/tenantapply/tenantapply.go)
  • no kubeconfig needed: apply a Job, then poll getJob until Succeeded
  • callers: any atproto client with service auth (README curl and goat account service-auth --lxm examples)

com.publicdomainrelay.temp.cluster

NSIDtypeinputoutputdoes
cluster.createClusterprocedurenamephasecreate on read, then desired state running
cluster.getKubeconfigprocedurename, ttlSeconds (default 28800, 60..86400, 0 is noExpiry, refused outside), noExpirykubeconfig (client cert, no exec, no token), instancecreate on read (spec.running=false)
cluster.listClustersquery-clusters[] {name, phase, running, instance, teardownBlocked, server}the caller's own clusters only (account from the token iss)
cluster.controlPlaneUpprocedurenamephasedesired running; idempotent
cluster.controlPlaneDownprocedurenamephasedesired not running; store retained
cluster.deleteClusterprocedurenamephase Drainingteardown behind a finalizer gate; never blocks
cluster.revokeIdentityproceduredid, namerevoked[] {name, state}deny record first, then delete the ServiceAccount; grant kept as Revoked
cluster.listKubeconfigsquerynamekubeconfigs[] {serial, createdAt, notAfter, noExpiry, revoked, revokedAt}the ClusterClientCertificate records; no key
cluster.revokeKubeconfigprocedurename, serialserial, revokedsets spec.revoked; the front refuses the cert
cluster.subscribeClusterssubscriptionnameframes #cluster, #member, #nodeClaim, #info (SnapshotComplete)connect snapshot, then changes
  • phases: Unavailable (cold, store kept), Initializing, Ready, Draining
  • served by internal/xrpcserver (NSIDPrefix = com.publicdomainrelay.temp.cluster.); subscribeClusters verifies on the WebSocket upgrade (internal/xrpcserver/subscribe.go)
  • served command: socialweb-computer serve gateway xrpc
  • auth: the same service-auth check (lxm = the NSID)
  • callers: internal/gatewayhttp/cluster.go XRPCClusterGateway (service auth per audience and NSID); the web /api/clusters/* routes call the gateway directly, not over XRPC
  • OAuth scopes: rpc:com.publicdomainrelay.temp.cluster.<method>?aud=* in the gateway scope (internal/app/compose/atprotoscope.go clusterScope) and K8sOAuthScope (internal/atproto/oauth/scope.go). revokeIdentity, subscribeClusters, listKubeconfigs and revokeKubeconfig are not in those sets
  • node scaling: a NodeClaim rents a market VM through the same RFP flow: market.md; the system view: ../SYSTEM.md

See Also