johnandersen777.bsky.social/socialweb-computer-kcp

Switch repository

Loading…

plan: "0025"
title: "Defect register from the rc81 live proof and the architecture audit"
status: done
source: >
  Defects observed during the rc81 live proof of plans 0023 and 0024 and
  found by the 2026-10-22 architecture audit. Each item records the defect
  and where it shows. Fixes are not decided here; they follow the refactor
  plan that comes out of the audit.
items:
  - id: D01-RECORD-WRITE-AUTH
    title: "AtprotoRecord writes to a local PDS have no token"
    status: done
    defect: >
      The gateway AtprotoRecord named association stays Failed with
      "pdsclient: authentication required: no access token is available".
      The record controller RepoResolver is built with no Tokens and no DPoP
      (internal/stack/componentcore/controllers.go:350-351). bindingAuth
      returns nil without an AuthRef (internal/atproto/pdsclient/repo.go:311-318).
      The RemotePDS branch and the legacy spec.repo branch never set AuthRef
      (internal/atproto/recordcontroller/controller.go:351-363, :375,
      :381-416). PutRecord is called with WriteOptions{} (:180). The
      requester writes the same record through its own LocalRepo handle, so
      the CR status does not reflect what is on the PDS.
  - id: D02-AUTHREF-SHAPE
    title: "AtprotoRepo authRef carries a vault path in RepoRef.Name"
    status: done
    defect: >
      appctl writes AuthRef as RepoRef{Kind: "Vault", Name: "openbao:"+path}
      (internal/appctl/statusobjects.go:30). The record controller reads only
      Name and drops workspace and kind
      (internal/atproto/recordcontroller/controller.go:347-348, :410-411).
  - id: D03-FIREHOSE-URLS-LOST
    title: "Bidder restart clears firehoseUrls"
    status: done
    defect: >
      After each bidder restart the root loses its firehoseUrls and goes
      Failed with "comms mode Firehose has no relay or firehose URL". The
      operator has to patch the root again. Seen on hat-7, hat-9 and local
      during the rc81 proof.
  - id: D04-SUSPEND-STOPS-RELEASE
    title: "Bidder suspend also stops release consumption"
    status: not-an-issue
    defect: >
      A paused Bidder stops taking new RFPs and also stops consuming release
      events for VMs it already runs. During the rc81 proof the paused
      winners had to be resumed before the cluster could scale to zero.
  - id: D05-SUSPEND-ROOT-PHASE
    title: "A suspended root reports Pending and still runs ensureDesired"
    status: not-an-issue
    defect: >
      ApplicationPhase has no Suspended value, so a suspended root reports
      Phase Pending with message "suspended" (internal/appctl/bidder.go:782),
      while ComponentPhase has Suspended (api/v1alpha1/types_component.go:34).
      A suspended root still runs ensureDesired (internal/appctl/bidder.go:761-770).
  - id: D06-SUSPEND-OVERRIDE
    title: "An override cannot un-suspend a component under a suspended root"
    status: not-an-issue
    defect: >
      The effective suspend is rootSuspend || *ov.Suspend
      (internal/appctl/overrides.go:406-407). NetworkApp uses stopped, not
      suspend (api/v1alpha1/types_application.go:689). No evidence that
      ComputeVM, Bid, MarketPresence or StateSync honor root suspend
      (inferred from the audit).
  - id: D07-SUSPEND-WRITERS
    title: "Suspend fields have competing writers"
    status: not-an-issue
    defect: >
      SuspendComponentIn merge-patches component spec.suspend
      (internal/appctl/components.go:762-771) and the next whole-spec Update
      overwrites it; it has no callers outside the package. Child root
      spec.suspend is written by SSA in applyRoot
      (internal/appctl/child.go:282) and by a merge patch
      (internal/appctl/network.go:926).
  - id: D08-COMPUTEVM-MARKET-WRITERS
    title: "ComputeVM status.market has four writers"
    status: not-an-issue
    defect: >
      The bidder merge patch (internal/services/bidder/kcp.go:246),
      guestcontrolplane SSA (internal/services/guestcontrolplane/store.go:125),
      the computevm controller (internal/computevm/computevm.go:695) and
      internal/computevm/client.go:252 all write status.market.
  - id: D09-BID-STATUS-WRITERS
    title: "Bid status has two writers, one with no field manager"
    status: not-an-issue
    defect: >
      The bidder writes Bid status with PatchStatus
      (internal/services/bidder/kcp.go:119-138); computevm writes it with
      UpdateStatus and no field manager (internal/computevm/bids.go:134).
  - id: D10-ROOT-STATUS-RMW
    title: "Root status is written by read-modify-write next to a component-owned field"
    status: not-an-issue
    defect: >
      appctl overwrites root status except status.market with
      read-modify-write UpdateStatus (internal/appctl/appctl.go:358-427),
      while a component owns status.market.
  - id: D11-DEAD-REPO-CONTROLLER
    title: "Duplicate AtprotoRepo status controller is never registered"
    status: not-an-issue
    defect: >
      internal/statusstate/repo.go:20 reconciles AtprotoRepo status but is
      never registered (internal/statusstate/statusstate.go:199-210). The
      live writer is internal/appctl/atproto_repo.go:49.
  - id: D12-MISSING-OWNERREFS
    title: "StateSync, MarketPresence, Bid and ComputeContract have no ownerRefs"
    status: done
    defect: >
      None of these objects carry ownerReferences. Garbage collection relies
      on labels, ttlctl and finalizers. computevm termination creates a
      StateSync with labels only (internal/computevm/termination.go:103).
      Stale termination StateSyncs and ComputeVMs with dead processes had to
      be removed by hand during the rc81 proof.
  - id: D13-CA-POOL-STARTUP-ONLY
    title: "Some CA pools are read only at startup"
    status: done
    defect: >
      The dispatchertls upstream CA pool and the exposure and PDS client
      verification CA pools are loaded once at startup. Leaf certificates
      reload on file change (internal/tlsreload); the CA pools do not.
  - id: D14-VAULT-TOKEN-STAMPEDE
    title: "Global OpenBao token cache has no request dedupe"
    status: not-an-issue
    defect: >
      The package-level vaultTokens cache
      (internal/clusterprovider/openbaoserver.go:137) has no single-flight,
      so concurrent callers can each log in when the token is missing or
      expired (inferred from the code).
  - id: D15-UNJOINED-GOROUTINES
    title: "Goroutines started without a join"
    status: done
    defect: >
      28 files start goroutines with no WaitGroup, done channel or errgroup,
      for example internal/appctl/appctl.go:229-230,
      internal/computevm/computevm.go:914,1102,1130,
      internal/services/gatewayapi/vault.go:138,
      internal/cloudprovider/controller.go:131. Shutdown does not wait for
      them.
  - id: D16-NIL-CONTEXT
    title: "Library code accepts a nil context and uses Background for shutdown"
    status: done
    defect: >
      67 "if ctx == nil { ctx = context.Background() }" guards, for example
      internal/component/runtime.go:168,185,225,288,412,765. Shutdown paths
      use context.Background() (internal/services/requester/service.go:203,213,237).
  - id: D17-MUTABLE-GLOBALS
    title: "Package globals mutated at runtime"
    status: done
    defect: >
      appPdsPublicURLs is mutated at runtime (internal/appctl/pds.go:155-168),
      a package-level sync.Map sits at internal/appctl/pds.go:47, and
      internal/services/guestcontrolplane/jwtcompat.go:37 replaces the jwt
      library's global ES256 signing method in init.
  - id: D18-ANNOTATION-SIGNALS
    title: "Annotations carry control signals between controllers"
    status: not-an-issue
    defect: >
      release-requested/release-request (written in
      internal/services/bidder/statesync.go:344-345,1274,1418, read in
      internal/computevm/provider.go:236-250), accept
      (internal/policyaction/computevm.go:136, read in
      internal/computevm/bids.go:113), bidder-dids
      (internal/appctl/network.go:989, read in internal/appctl/bidder.go:704)
      and adopt-joined-node (internal/nodemarket, read in
      internal/gatewayapi/nodeclaim.go:90) are signals with no schema,
      status or conditions.
  - id: D19-CABUNDLEREF-FILE
    title: "caBundleRef is a host file path"
    status: done
    defect: >
      caBundleRef is read with os.ReadFile
      (internal/appctl/atprotoscope.go:113); it is not a reference to an
      object.
  - id: D20-ERROR-RETRY-RESYNC
    title: "Watch-based loops retry errors only on the 30s resync"
    status: done
    defect: >
      The componentcore runtime reconciles inline with no queue or backoff
      (internal/stack/componentcore/controllers.go:104-215); a failed
      reconcile waits for the 30s resync. The workspace list is read once at
      Run (:113), so workspaces created later are not watched.
  - id: D21-REQUESTER-ASSOCIATION-LOST
    title: "Requester MarketIdentity loses its operator association after a gateway restart"
    status: done
    defect: >
      After a gateway restart, requester bids were denied with "not vouched"
      and "scope denied" until scripts/associate-bidder.ts was rerun and
      spec.approvedAssociations was patched again on the requester
      MarketIdentity.
  - id: D22-UNREGISTERED-KINDS
    title: "Eleven kinds have no DeepCopyObject and are not in the scheme"
    status: not-an-issue
    defect: >
      Bid, ComputeContract, Gateway, MarketBidder, MarketNetwork,
      MarketRequester, SshGateway, AtprotoRecord, AtprotoRecordWatch,
      ClusterClientCertificate and GuestIdentityGrant have no DeepCopyObject
      in api/v1alpha1/zz_generated.deepcopy.go and are missing from
      AddKnownTypes (api/v1alpha1/groupversion_info.go:19). 46 kinds are
      registered against 57 schemas. The deepcopy file is kept by hand; no
      generator writes it (scripts/regen-after-rebase.sh:21).
  - id: D23-APP-WORKSPACE-NO-COMPUTEVMS
    title: "socialweb-application workspaces never initialize in a network without bidders"
    status: done
    defect: >
      In a market network brought up with --bidders 0, a workspace of type
      socialweb-application never leaves initializing, because the
      computevms APIExport that the type binds by default does not exist.
      Seen in the plan 0026 W0-RUNTIME live run (instance w0rt).
  - id: D24-WORKSPACE-NO-CREATEDBY
    title: "A workspace created without the owner annotation stalls"
    status: done
    defect: >
      A workspace created without the experimental.tenancy.kcp.io/owner
      annotation stalls with "no createdBy recorded". Seen in the plan 0026
      W0-RUNTIME live run (instance w0rt).
  - id: D25-MOUNT-BEARER-KCP-MAIN
    title: "Tenant bearer requests through the kcp mount fail on kcp main"
    status: done
    defect: >
      kcp main (#4427, pkg/server/mountproxy.go) verifies the mount target's
      TLS certificate against the system roots plus
      --mount-proxy-server-ca-file, authenticates the caller itself, strips
      Authorization and forwards X-Remote-* headers. The tenant mount target
      https://127.0.0.1:<advertisePort> (internal/materialiser/materialiser.go:635)
      presents the cluster's own k3s CA, so every mount request gets 502;
      a tenant service-account bearer token gets 401 at kcp. Registry.MountStatus
      (internal/clusterprovider/registry.go:234) then always reads 502 and
      materialiser.shut() (internal/materialiser/materialiser.go:595-602) counts
      502 as shut while the tenant apiserver is up. The revoker's mount calls
      (internal/kcpclient/kcpclient.go:794, 814) fail.
  - id: D26-LEASE-MIRROR-ROOT-APP
    title: "The lease mirror looks for a root:app workspace that does not exist"
    status: done
    defect: >
      In a market network on kcp main the lease mirror keeps failing with
      'workspaces.tenancy.kcp.io "root:app" not found'. Seen in the plan
      0026 W0-RUNTIME rerun (instance w0rt2).