plan: "0025" title: "Defect register from the rc81 live proof and the architecture audit" status: done source: > Defects observed during the rc81 live proof of plans 0023 and 0024 and found by the 2026-10-22 architecture audit. Each item records the defect and where it shows. Fixes are not decided here; they follow the refactor plan that comes out of the audit. items: - id: D01-RECORD-WRITE-AUTH title: "AtprotoRecord writes to a local PDS have no token" status: done defect: > The gateway AtprotoRecord named association stays Failed with "pdsclient: authentication required: no access token is available". The record controller RepoResolver is built with no Tokens and no DPoP (internal/stack/componentcore/controllers.go:350-351). bindingAuth returns nil without an AuthRef (internal/atproto/pdsclient/repo.go:311-318). The RemotePDS branch and the legacy spec.repo branch never set AuthRef (internal/atproto/recordcontroller/controller.go:351-363, :375, :381-416). PutRecord is called with WriteOptions{} (:180). The requester writes the same record through its own LocalRepo handle, so the CR status does not reflect what is on the PDS. - id: D02-AUTHREF-SHAPE title: "AtprotoRepo authRef carries a vault path in RepoRef.Name" status: done defect: > appctl writes AuthRef as RepoRef{Kind: "Vault", Name: "openbao:"+path} (internal/appctl/statusobjects.go:30). The record controller reads only Name and drops workspace and kind (internal/atproto/recordcontroller/controller.go:347-348, :410-411). - id: D03-FIREHOSE-URLS-LOST title: "Bidder restart clears firehoseUrls" status: done defect: > After each bidder restart the root loses its firehoseUrls and goes Failed with "comms mode Firehose has no relay or firehose URL". The operator has to patch the root again. Seen on hat-7, hat-9 and local during the rc81 proof. - id: D04-SUSPEND-STOPS-RELEASE title: "Bidder suspend also stops release consumption" status: not-an-issue defect: > A paused Bidder stops taking new RFPs and also stops consuming release events for VMs it already runs. During the rc81 proof the paused winners had to be resumed before the cluster could scale to zero. - id: D05-SUSPEND-ROOT-PHASE title: "A suspended root reports Pending and still runs ensureDesired" status: not-an-issue defect: > ApplicationPhase has no Suspended value, so a suspended root reports Phase Pending with message "suspended" (internal/appctl/bidder.go:782), while ComponentPhase has Suspended (api/v1alpha1/types_component.go:34). A suspended root still runs ensureDesired (internal/appctl/bidder.go:761-770). - id: D06-SUSPEND-OVERRIDE title: "An override cannot un-suspend a component under a suspended root" status: not-an-issue defect: > The effective suspend is rootSuspend || *ov.Suspend (internal/appctl/overrides.go:406-407). NetworkApp uses stopped, not suspend (api/v1alpha1/types_application.go:689). No evidence that ComputeVM, Bid, MarketPresence or StateSync honor root suspend (inferred from the audit). - id: D07-SUSPEND-WRITERS title: "Suspend fields have competing writers" status: not-an-issue defect: > SuspendComponentIn merge-patches component spec.suspend (internal/appctl/components.go:762-771) and the next whole-spec Update overwrites it; it has no callers outside the package. Child root spec.suspend is written by SSA in applyRoot (internal/appctl/child.go:282) and by a merge patch (internal/appctl/network.go:926). - id: D08-COMPUTEVM-MARKET-WRITERS title: "ComputeVM status.market has four writers" status: not-an-issue defect: > The bidder merge patch (internal/services/bidder/kcp.go:246), guestcontrolplane SSA (internal/services/guestcontrolplane/store.go:125), the computevm controller (internal/computevm/computevm.go:695) and internal/computevm/client.go:252 all write status.market. - id: D09-BID-STATUS-WRITERS title: "Bid status has two writers, one with no field manager" status: not-an-issue defect: > The bidder writes Bid status with PatchStatus (internal/services/bidder/kcp.go:119-138); computevm writes it with UpdateStatus and no field manager (internal/computevm/bids.go:134). - id: D10-ROOT-STATUS-RMW title: "Root status is written by read-modify-write next to a component-owned field" status: not-an-issue defect: > appctl overwrites root status except status.market with read-modify-write UpdateStatus (internal/appctl/appctl.go:358-427), while a component owns status.market. - id: D11-DEAD-REPO-CONTROLLER title: "Duplicate AtprotoRepo status controller is never registered" status: not-an-issue defect: > internal/statusstate/repo.go:20 reconciles AtprotoRepo status but is never registered (internal/statusstate/statusstate.go:199-210). The live writer is internal/appctl/atproto_repo.go:49. - id: D12-MISSING-OWNERREFS title: "StateSync, MarketPresence, Bid and ComputeContract have no ownerRefs" status: done defect: > None of these objects carry ownerReferences. Garbage collection relies on labels, ttlctl and finalizers. computevm termination creates a StateSync with labels only (internal/computevm/termination.go:103). Stale termination StateSyncs and ComputeVMs with dead processes had to be removed by hand during the rc81 proof. - id: D13-CA-POOL-STARTUP-ONLY title: "Some CA pools are read only at startup" status: done defect: > The dispatchertls upstream CA pool and the exposure and PDS client verification CA pools are loaded once at startup. Leaf certificates reload on file change (internal/tlsreload); the CA pools do not. - id: D14-VAULT-TOKEN-STAMPEDE title: "Global OpenBao token cache has no request dedupe" status: not-an-issue defect: > The package-level vaultTokens cache (internal/clusterprovider/openbaoserver.go:137) has no single-flight, so concurrent callers can each log in when the token is missing or expired (inferred from the code). - id: D15-UNJOINED-GOROUTINES title: "Goroutines started without a join" status: done defect: > 28 files start goroutines with no WaitGroup, done channel or errgroup, for example internal/appctl/appctl.go:229-230, internal/computevm/computevm.go:914,1102,1130, internal/services/gatewayapi/vault.go:138, internal/cloudprovider/controller.go:131. Shutdown does not wait for them. - id: D16-NIL-CONTEXT title: "Library code accepts a nil context and uses Background for shutdown" status: done defect: > 67 "if ctx == nil { ctx = context.Background() }" guards, for example internal/component/runtime.go:168,185,225,288,412,765. Shutdown paths use context.Background() (internal/services/requester/service.go:203,213,237). - id: D17-MUTABLE-GLOBALS title: "Package globals mutated at runtime" status: done defect: > appPdsPublicURLs is mutated at runtime (internal/appctl/pds.go:155-168), a package-level sync.Map sits at internal/appctl/pds.go:47, and internal/services/guestcontrolplane/jwtcompat.go:37 replaces the jwt library's global ES256 signing method in init. - id: D18-ANNOTATION-SIGNALS title: "Annotations carry control signals between controllers" status: not-an-issue defect: > release-requested/release-request (written in internal/services/bidder/statesync.go:344-345,1274,1418, read in internal/computevm/provider.go:236-250), accept (internal/policyaction/computevm.go:136, read in internal/computevm/bids.go:113), bidder-dids (internal/appctl/network.go:989, read in internal/appctl/bidder.go:704) and adopt-joined-node (internal/nodemarket, read in internal/gatewayapi/nodeclaim.go:90) are signals with no schema, status or conditions. - id: D19-CABUNDLEREF-FILE title: "caBundleRef is a host file path" status: done defect: > caBundleRef is read with os.ReadFile (internal/appctl/atprotoscope.go:113); it is not a reference to an object. - id: D20-ERROR-RETRY-RESYNC title: "Watch-based loops retry errors only on the 30s resync" status: done defect: > The componentcore runtime reconciles inline with no queue or backoff (internal/stack/componentcore/controllers.go:104-215); a failed reconcile waits for the 30s resync. The workspace list is read once at Run (:113), so workspaces created later are not watched. - id: D21-REQUESTER-ASSOCIATION-LOST title: "Requester MarketIdentity loses its operator association after a gateway restart" status: done defect: > After a gateway restart, requester bids were denied with "not vouched" and "scope denied" until scripts/associate-bidder.ts was rerun and spec.approvedAssociations was patched again on the requester MarketIdentity. - id: D22-UNREGISTERED-KINDS title: "Eleven kinds have no DeepCopyObject and are not in the scheme" status: not-an-issue defect: > Bid, ComputeContract, Gateway, MarketBidder, MarketNetwork, MarketRequester, SshGateway, AtprotoRecord, AtprotoRecordWatch, ClusterClientCertificate and GuestIdentityGrant have no DeepCopyObject in api/v1alpha1/zz_generated.deepcopy.go and are missing from AddKnownTypes (api/v1alpha1/groupversion_info.go:19). 46 kinds are registered against 57 schemas. The deepcopy file is kept by hand; no generator writes it (scripts/regen-after-rebase.sh:21). - id: D23-APP-WORKSPACE-NO-COMPUTEVMS title: "socialweb-application workspaces never initialize in a network without bidders" status: done defect: > In a market network brought up with --bidders 0, a workspace of type socialweb-application never leaves initializing, because the computevms APIExport that the type binds by default does not exist. Seen in the plan 0026 W0-RUNTIME live run (instance w0rt). - id: D24-WORKSPACE-NO-CREATEDBY title: "A workspace created without the owner annotation stalls" status: done defect: > A workspace created without the experimental.tenancy.kcp.io/owner annotation stalls with "no createdBy recorded". Seen in the plan 0026 W0-RUNTIME live run (instance w0rt). - id: D25-MOUNT-BEARER-KCP-MAIN title: "Tenant bearer requests through the kcp mount fail on kcp main" status: done defect: > kcp main (#4427, pkg/server/mountproxy.go) verifies the mount target's TLS certificate against the system roots plus --mount-proxy-server-ca-file, authenticates the caller itself, strips Authorization and forwards X-Remote-* headers. The tenant mount target https://127.0.0.1: (internal/materialiser/materialiser.go:635) presents the cluster's own k3s CA, so every mount request gets 502; a tenant service-account bearer token gets 401 at kcp. Registry.MountStatus (internal/clusterprovider/registry.go:234) then always reads 502 and materialiser.shut() (internal/materialiser/materialiser.go:595-602) counts 502 as shut while the tenant apiserver is up. The revoker's mount calls (internal/kcpclient/kcpclient.go:794, 814) fail. - id: D26-LEASE-MIRROR-ROOT-APP title: "The lease mirror looks for a root:app workspace that does not exist" status: done defect: > In a market network on kcp main the lease mirror keeps failing with 'workspaces.tenancy.kcp.io "root:app" not found'. Seen in the plan 0026 W0-RUNTIME rerun (instance w0rt2).