plan: "0024" title: "Per-component suspend and typed configuration overrides for app roots" status: done source: > Every app root has one spec.suspend and one set of root-wide fields. appctl passes that one suspend to every child it creates, including the shared IngressProxy and every Service/Exposure. Suspending a Bidder therefore also suspends GuestControlPlane, Pds, IngressProxy and all exposures. Operators cannot pause one component, cannot tune one component, and cannot use the per-component fields that already exist on some child CRDs because ensureComponentIn replaces the whole child spec on every pass. The concrete case is hat-7: it runs an oauthQr bidder as the bobvmbuilder account and has no local Pds. An operator wants to stop only the Bidder market loop so Bob's PDS stops receiving heartbeat writes, while GuestControlPlane and IngressProxy stay Ready and their public URLs keep answering. Today the only supported lever, spec.suspend, stops all of them. principle: > The app root is the only writer of its children. A per-component override is desired state on the root, resolved by appctl into one child spec. The child CR stays appctl-owned; direct edits are reverted. Suspend is per component and never cascades unless the root itself is suspended. Configuration is typed and validated; no arbitrary pod specs, no arbitrary commands, no secrets, and no bypass of root ownership. old_behavior: | Old bidder discovery: offering refresh was a per-bidder option, offering-refresh-sec, default 300, and 0 disabled it (old tree: atproto-market/hono-bidder/cli-args-env.ts:328, atproto-market/hono-bidder/mod.ts:704; accepted by the root option map at deploy/apps/option-map.yaml:79). The old requester exposed max-concurrent and per-account limits through options (deploy/apps/option-map.yaml:188). The old SSH service exposed max-sessions, sessions-per-account, max-connections, auth-timeout-sec and session-max-sec with explicit defaults (old tree: socialweb-computer-ssh/hono-socialweb-computer-ssh/cli-args-env.json:75-108). The old ingress proxy exposed relay-timeout-ms, keepalive-ms, keepalive-timeout-ms, reconnect-grace-ms and nonce-ttl-ms (old tree: did-key-ingress-proxy/hono-did-key-ingress-proxy/cli-args-env.json:1-60). The old requester bid collection polled every 250 ms (old tree: atproto-market/lib/compute-request-xrpc/mod.ts:1589). The rewrite kept the API names but moved most of these values to constants in the services, and appctl never feeds them from a root CR. current_gaps: | Ranked by operator impact. Every entry was verified in the current tree. 1. App-wide suspend. bidder.go:546 and :751, requester.go:234 and :370, gateway.go:264 and :443, sshgateway.go:270 and :358, network.go:312 and :643 all use one root bool. NewComponentSpec (components.go:161-163) gives it to every child; serviceDesiredNamed and exposureDesiredForService (components.go:262-291) give it to every Service/Exposure; policyEngineDesired (bidder.go:221-224) gives it to PolicyEngine. IngressProxyDesired (components.go:379-386) gives it to the shared proxy. One pause stops the app. 2. No override path. ensureComponentIn replaces child spec wholesale on every pass (components.go:572-579). Child fields such as ComponentSpec.Limits and ComponentSpec.ServiceAuth (types_component.go:105-118) are unreachable from a root and any direct child edit is lost. 3. Bidder market timing is hardcoded. heartbeat is market.HeartbeatInterval = 60s (atproto/market/nsid.go:56) and offering refresh is DefaultOfferingRefresh = 5m (bidder/config.go:52), assigned at bidder/service.go:366-367. BidderSpec has no timing fields (types_component.go:166-203). The offering loop only reads these fields (bidder/offering.go:105-114), so exposure is small. 4. Requester concurrency and timing are hardcoded. MaxConcurrent 64, PerAccount 16, ContractPoll 2s, BidWindow 30s, ReceiptTimeout 5m, ReceiptPoll 500ms, SessionCap 1h, SessionIdle 5m, DirectoryTTL 60s, DirectoryKeys 10000 (requester/config.go:44-90, defaults at :553-599). RequesterSpec (types_component.go:327-365) has no such fields. 5. Comms is root-wide. The same MarketComms is copied into BidderSpec (appctl/bidder.go:609), RequesterSpec (requester.go:257), Gateway requester (gateway.go:336) and SshGateway requester (sshgateway.go:299). NetworkApp has no Comms field (types_application.go:620-643), and EventStreams are created per role with no tuning (marketstreams.go:80-110). A gateway cannot use XRPC while its bidder uses firehose. 6. Ingress is root-wide. One IngressConfig feeds every Pds and IngressProxy. NetworkIngressConfig forces ingressRef to the network's dispatcher (ingressselect.go:40-56, network.go:330-365); app IngressProxy is always named ingress with an empty state dir (components.go:379-386). Per-PDS public host/provider is impossible without a second root. 7. PolicyEngine is hardcoded. policyEngineDesired sets only Suspend and BundledActionsEnabled (bidder.go:221-224). PolicyEngineSpec supports Limits, Trust and Cache (api/policy/v1alpha1/types_policyengine.go:22-50), but no root feeds them. 8. Component limits and service auth are declared but dead. ComponentLimits and ComponentServiceAuth exist (types_component.go:63-87, 105-118), but appctl sets neither (no Limits: or ServiceAuth: in internal/appctl). ComponentLimits is not enforced by componentcore; HostProcessLimits (types_hostprocess.go:100-108, hostprocess/reconcile.go:106-115) is the only enforced path. 9. Service identity is fixed by root kind. bidder_associate (association.go:21), requester_associate (gateway.go:44, :697), SSH always validates requester_associate (sshtransport/authorizer.go:24), and trust accepts only those two IDs (policyengine/trust/trust.go:18-19). Gateway audience is one root option (gateway.go:291, :360). ComponentServiceAuth is unused. 10. GuestControlPlane has no toggles. Its spec exposes only identity, refs, hostnames, storage, serve and HostKeyRef (types_component.go:221-237); appctl never sets HostKeyRef (bidder.go:566-574). Login mount, role, grant sync, issuer retry, host-key attempts and relay are constants (guestcontrolplane/config.go:21-33, :122-178), and ConfigFrom never populates Relay, so relay is always off (guestcontrolplane/relay.go:19). 11. PDS settings are hardcoded. appPdsSpec forces Mode Full and AllowLocalhostClients true (appctl/pds.go:201-221); network PDS does the same (network.go:295-306). PdsOAuth TTLs and localhost policy exist (types_component.go:300-310) but no root sets them. Blob and request limits are not in PdsSpec at all (pds/config.go:77-81, atproto/pds/server.go:24-25, :107-113). OAuth scope is only atproto (atproto/oauth/server.go:194). The authoritative role registry is typescript-helpers/lib/oauth-scope/mod.ts (the source of did-key-associator's generated/oauth-scope.js); port it to internal/atproto/oauth/scope.go and use the Requester, Bidder and DIDKeyAssociator sets instead of a blanket transition:generic. 12. SshServer limits and timeouts are hardcoded. MaxConnections, MaxSessions, SessionsPerAccount, MaxAuthAttempts, AuthTimeout, SessionMax and association cache settings are constants (sshserver/config.go:43-67); SshServerSpec has no fields (types_component.go:449-466). 13. EventStream timing is hardcoded. Retry, status and reconnect constants (eventstream/controller.go:38-41); EventStreamSpec has no tuning fields. 14. Relay and IngressProxy timing is hardcoded. Relay crawl interval 30s (relay/service.go:33, :140); IngressProxy keepalive/nonce/timeouts are provider defaults with no CR fields (ingressproxy/service.go:55-59). 15. Gateway child fields are partly reachable only through root options. RequiredDownloads is built from a fixed list (gateway.go:53, :301); gateway-audience, node-poll-interval-ms and post-sign-in-url are root options (gateway.go:254-257, :290-295), not child CR fields users can set through spec.components. old_gaps_addressed: | This plan restores the old per-service knobs without restoring the old untyped CLI/env surface: offering refresh (including 0 to disable), requester concurrency, SSH limits, ingress timeouts, PDS/OAuth settings and policy-engine cache/limits all become typed child settings resolved from the root. target_model: spec_components: | Add one field to MarketBidderSpec, MarketRequesterSpec, GatewaySpec, SshGatewaySpec and MarketNetworkSpec: components: map[string]ComponentOverride The key is the component kind for a singleton (Bidder, Pds, ...) or Kind/name for an explicit instance (Pds/alice, Pds/bob). The existing ComponentKey(kind, name) helper is kind+"/"+name (components.go:157), so the key is already the internal identity. Matching is case-sensitive against CRD kinds. An ambiguous singleton (two Pds children) requires the explicit form. Names are the constants in appctl: bidder, guest-kcp, pds, policy-engine, requester, gateway-api, ssh-server, ingress, plc, relay, dispatcher. NetworkApp also gains components: ComponentOverrides, applied to the child application root it creates; NetworkApp.Stopped remains the coarse per-app stop. crd_shape: | spec: suspend: false # unchanged master kill switch components: Bidder: # singleton; or Bidder/bidder suspend: true # *bool: absent = inherit, false = resume restartPolicy: Always # Never | OnFailure | Always limits: # ComponentLimits, typed cpuMillicores: 500 memoryBytes: 1073741824 diskBytes: 0 serviceAuth: # no issuer/key material here audience: [did:web:example] hostname: example comms: # per-component transport mode: Firehose relayUrls: [wss://relay.example] firehoseUrls: [] ingress: # per-component ingress selection provider: xrpc-subscription publicHost: pds.example ingressRef: {kind: IngressProxy, name: ingress} settings: # exactly one block, matching this entry's kind; all are shown bidder: heartbeatIntervalSeconds: 5 offeringRefreshSeconds: 0 bidTtlSeconds: 86400 maxConcurrentGuests: 2 policyPollMilliseconds: 250 policyTimeoutSeconds: 120 requester: maxConcurrent: 32 perAccount: 8 contractPollMilliseconds: 2000 bidWindowSeconds: 30 receiptTimeoutSeconds: 300 receiptPollMilliseconds: 500 sessionCapSeconds: 3600 sessionIdleSeconds: 300 directoryTTLSeconds: 60 directoryKeys: 10000 policyPollMilliseconds: 250 policyTimeoutSeconds: 120 oauthScopes: [] # defaults to the RequesterOAuthScope role set guestControlPlane: guestLoginMount: sshsig role: kcp grantSyncSeconds: 30 issuerRetrySeconds: 30 hostKeyAttempts: 10 relayEnabled: false pds: maxBlobSize: 5242880 maxRequestBytes: 4194304 contactEmail: ops@example oauth: allowLocalhostClients: true accessTokenTTLSeconds: 0 refreshTokenTTLSeconds: 0 disableLocalhostClients: false scopes: [atproto] sshServer: maxConnections: 10000 maxSessions: 1000 sessionsPerAccount: 1000 maxAuthAttempts: 3 authTimeoutSeconds: 120 sessionMaxSeconds: 3600 cacheEntries: 10000 gatewayAPI: nodePollIntervalSeconds: 30 requiredDownloads: [socialweb-meshd-x86_64] ingressProxy: relayTimeoutMilliseconds: 30000 keepaliveMilliseconds: 5000 keepaliveTimeoutMilliseconds: 5000 reconnectGraceMilliseconds: 10000 nonceTTLMilliseconds: 60000 relay: crawlIntervalSeconds: 30 policyEngine: limits: maxConcurrentRuns: 4 queueDepth: 64 activeDeadlineSeconds: 120 maxConsoleBytes: 1048576 maxOutputBytes: 1048576 maxCacheEntries: 512 maxCacheBytes: 67108864 trust: ttlSeconds: 300 cache: verdictTTLSeconds: 300 scopeTTLSeconds: 300 recordTTLSeconds: 300 Zero means "use the service default" except offeringRefreshSeconds, where 0 disables refresh (matching old behavior). Unset pointers inherit. go_types: | type ComponentOverrides map[string]ComponentOverride type ComponentOverride struct { Suspend *bool `json:"suspend,omitempty"` RestartPolicy ComponentRestartPolicy `json:"restartPolicy,omitempty"` Limits *ComponentLimits `json:"limits,omitempty"` ServiceAuth *ComponentServiceAuthOverride `json:"serviceAuth,omitempty"` Comms *MarketComms `json:"comms,omitempty"` Ingress *IngressConfig `json:"ingress,omitempty"` Settings *ComponentSettings `json:"settings,omitempty"` } type ComponentServiceAuthOverride struct { Audience []string `json:"audience,omitempty"` Hostname string `json:"hostname,omitempty"` } type ComponentSettings struct { Bidder *BidderSettings `json:"bidder,omitempty"` Requester *RequesterSettings `json:"requester,omitempty"` GuestControlPlane *GuestControlPlaneSettings `json:"guestControlPlane,omitempty"` Pds *PdsSettings `json:"pds,omitempty"` SshServer *SshServerSettings `json:"sshServer,omitempty"` GatewayAPI *GatewayAPISettings `json:"gatewayAPI,omitempty"` IngressProxy *IngressProxySettings `json:"ingressProxy,omitempty"` Relay *RelaySettings `json:"relay,omitempty"` PolicyEngine *PolicyEngineSettings `json:"policyEngine,omitempty"` } NetworkApp gains Components ComponentOverrides `json:"components,omitempty"`. PolicyEngineSettings is a local typed mirror of PolicyEngineLimits, TrustConfig and CacheConfig; appctl translates it into policyv1alpha1.PolicyEngineSpec so api/v1alpha1 does not import the policy API. A ComponentSettings block is valid only when it matches the entry's kind; appctl rejects a mismatch. Interval fields are *int32 seconds or milliseconds; unset means default. The per-kind structs use the field names in the YAML above, with Go names HeartbeatIntervalSeconds etc. typed_vs_options: | Typed fields win. Generic options (ApplicationOption{Name,Value} at types_application.go:61-65, option-map at deploy/apps/option-map.yaml) are stringly typed, have no component target, cannot be validated by the CRD, cannot express units or pointers, make GitOps diffs and SSA ownership poor, and invite secrets into a field that status/docs will echo. They are retained only for existing CLI compatibility; this plan does not extend them. Typed fields give schema validation, deterministic defaults, per-field precedence, readable status and safe concatenation into child CRs. The only map in the design is the component-keyed override map, which is a namespacing device, not a bag of arbitrary settings. precedence: | For every field: component override > root default > service default. effectiveSuspend = root.Spec.Suspend || (override.Suspend != nil && *override.Suspend). Root suspend is a master kill switch: an override of suspend:false cannot resume anything while root.Spec.Suspend is true. Removing a key restores the root/service default on the next pass; it does not delete the child because the desired component set is unchanged. A key must resolve to a component in the desired set for the current profile; a conditional component that is absent (for example Pds on an oauthQr bidder with no local PDS) is invalid. A malformed key, an unresolvable component or an inapplicable settings block fails the whole override set closed: appctl makes no child spec changes for that pass, sets ComponentOverridesValid=False and Phase=Failed, and names the key. A typo therefore cannot silently resume a paused component. ownership: | spec.components is the only operator interface. appctl computes the child spec from root fields plus the override and writes it with ensureComponentIn, which replaces the whole spec (components.go:572-579). Direct child spec edits are unsupported and revert on the next pass; child annotations are preserved, so the existing restart-at annotation remains an emergency restart path (components.go:706, contract.go:20). The root spec is never written by appctl; only status is. Root objects are applied by stack components with server-side apply, FieldManager socialweb-computer and Force=true (stack/bidder/bidder.go:197-199, stack/request/request.go:252-254). New components is omitempty and absent from those applies, so a GitOps manager can own spec.components without the stack taking it; verify this in W0. If two managers set the same components key, Kubernetes reports the SSA conflict instead of appctl choosing silently. gitops_and_ssa: | Use a map, not a list: SSA owns and merges each map key independently, so two GitOps changes to different components do not conflict and do not churn the whole list. A list would be atomic in this codebase because neither the Go types nor generated schemas use x-kubernetes-list-type markers today. The trade-off is that the schema cannot constrain map keys; appctl validates every key against the desired component set and fails closed on unknown keys. GitOps owns roots only. It must not apply child Bidder/Pds/... specs; if it does, appctl's Update wins next pass and sets ComponentDrift=True with the child key until the next successful apply. Do not add Force to that drift path; make the conflict visible. restart_and_immutability: | Any applied change to restartPolicy, limits, serviceAuth, comms, ingress or settings changes the child generation; component/runtime.go:250-278 stops and starts the process, and the restarts counter increments. suspend stops the process without deleting its state; resume starts it from the same state dir. Immutable and rejected: the map key (kind/name), identity DID, PdsRef/MarketIdentityRef, association serviceID, and a PDS publicHost after StableHost has been published (pdsingress.go:164-169). Computed and not overrideable: stateDir, network, hostnames, serve bind/port, owner refs, labels, and the Service/Exposure set. Public host changes before publication are allowed. status: | Extend ApplicationComponent with Suspended bool, ObservedGeneration int64, Restarts int32 and Message string. Add ComponentSummary {Desired, Ready, Suspended, NotReady} and ComponentOverrideStatus {Component, Generation, Suspend, RestartPolicy, Applied, Message}. Add conditions ComponentsReady, ComponentSuspended, ComponentOverridesValid and ComponentDrift. Root Phase stays Ready when every desired component is either Ready or Suspended by intent; Provisioning when a non-suspended component is not Ready; Pending when root.Spec.Suspend is true; Failed when an override is invalid. This replaces snap.complete(allDesired) (bidder.go:100-110), which would never report Ready with a suspended child. Do not echo setting values or any secret in status. suspend_propagation: | Root suspend keeps today's behavior: every child, Service, Exposure and the shared IngressProxy is suspended. Per-component suspend sets only the named child's ComponentSpec.Suspend. It must not suspend that child's Service/Exposure or a shared IngressProxy, because that would stop other components. Consequence: a suspended Bidder's own market URL will fail while stopped, but GuestControlPlane, Pds and IngressProxy URLs remain Ready and answer. This is the hat-7 contract. state_rules: - Read AGENTS.md first. status is todo, doing, done or blocked; the lead flips statuses. One writer per directory; tracks touching the same file are serialized even in a parallel wave. - Every item ends with gofmt, go build ./..., go vet ./..., and the relevant live proof. No unit tests and no test-first work. Integration tests are written only in W4, after the live proof, per AGENTS.md. No push. - Keep defaults identical when components is absent. Existing roots must reconcile without changing any child generation. - Regenerate with go generate ./api/...; deploy/check.sh --fast must pass. Bump Revision in api/v1alpha1/resources.go only if the generator requires it; record schema revisions in deploy/apiresourceschema-revisions.txt. - Never put a secret, token, password, private key or credential value in a root spec, child spec or status. Key refs only. - The old tree is reference only. Do not reintroduce Deno components or a second configuration path. waves: - id: W0 title: "Contracts, schema and validation" status: done serial: true tracks: [W0-CONTRACTS] item: W0-CONTRACTS gate: "types and schemas compile; invalid keys and setting blocks fail closed; no child behavior changes." - id: W1 title: "Per-component suspend and truthful status" status: done serial: true tracks: [W1-SUSPEND] item: W1-SUSPEND depends_on: [W0] gate: "local pause proof passes; hat-7/A1 reproduces it in W4; root Ready with one suspended child; resume advances heartbeat." - id: W2 title: "Bidder and Requester typed tuning" status: done parallel: true tracks: [W2-TUNING] item: W2-TUNING depends_on: [W1] gate: "heartbeat/offering and requester limits take effect from settings and are visible in status." - id: W3 title: "Remaining components, overrides and immutability" status: done parallel: true tracks: [W3-COMPONENTS] item: W3-COMPONENTS depends_on: [W2] gate: "every child kind has a verified override path or an explicit not-supported reason; immutable changes are rejected." - id: W4 title: "Integration and live proof" status: done serial: true tracks: [W4-PROOF] item: W4-PROOF depends_on: [W3] gate: "acceptance matrix passes on this host, hat-9, hat-7 and prod gateway; no secrets; no app-wide collateral stop." items: - id: W0-CONTRACTS title: "Add spec.components and typed override contracts" wave: W0 track: contracts parallel_with: [] depends_on: [] status: done decision: > Add ComponentOverrides map[string]ComponentOverride to all five app roots, with Suspend *bool, RestartPolicy, Limits, ServiceAuth, Comms, Ingress and per-kind Settings. Add the per-kind settings structs listed in target_model. Extend ApplicationStatus with ComponentSummary, ComponentOverrideStatus and the four conditions. Implement one resolver in appctl that resolves Kind or Kind/name against the desired set, rejects unknown keys, ambiguous singletons, mismatched settings blocks and immutable fields, and returns the effective override per component. Add Components ComponentOverrides to NetworkApp so a network's child application roots receive the same map. Do not change child specs yet. files: - api/v1alpha1/types_application.go - api/v1alpha1/types_component.go - api/v1alpha1/resources.go - api/v1alpha1/zz_generated.deepcopy.go - deploy/marketbidder-apiresourceschema.yaml - deploy/marketrequester-apiresourceschema.yaml - deploy/gateway-apiresourceschema.yaml - deploy/sshgateway-apiresourceschema.yaml - deploy/marketnetwork-apiresourceschema.yaml - deploy/workspacetype-application.yaml - deploy/apiresourceschema-revisions.txt - internal/appctl/components.go - docs/ARCHITECTURE.md done_when: > go generate ./api/... and deploy/check.sh --fast are clean; each root accepts a components map; a live apply with an unknown key, an ambiguous name, a mismatched settings block or an immutable field fails closed with a named condition; an absent components map produces byte-identical child specs and does not restart any component. - id: W1-SUSPEND title: "Resolve per-component suspend without collateral stops" wave: W1 track: suspend parallel_with: [] depends_on: [W0-CONTRACTS] status: done decision: > Replace each root's single suspend := o.Spec.Suspend with suspendFor(kind, name) = root.Spec.Suspend || override.Suspend, and pass only the named child's effective suspend to that child. Keep root suspend propagating to exposures and the shared IngressProxy; a per-component suspend must not. Replace snap.complete(allDesired) with an evaluator that is satisfied when a desired component is Ready or Suspended by intent. Add ComponentSummary, per-component Suspended, ObservedGeneration, Restarts and Message, plus ComponentsReady, ComponentSuspended and ComponentOverridesValid. A suspended child must report Phase Suspended, Ready false and Ready condition reason Suspended from the existing runtime (component/runtime.go:262-270). Persist no new state; resume starts from the same state dir. files: - internal/appctl/components.go - internal/appctl/bidder.go - internal/appctl/requester.go - internal/appctl/gateway.go - internal/appctl/sshgateway.go - internal/appctl/network.go - internal/appctl/statusobjects.go - api/v1alpha1/types_application.go - docs/ARCHITECTURE.md done_when: > On a disposable local root with a Bidder-only override, Bidder is Suspended/not Ready, its heartbeat record createdAt stops advancing, GuestControlPlane and IngressProxy stay Ready and their listener URLs answer, no other component changes phase, and the root stays Ready with ComponentSuspended=True and componentSummary.suspended=1. Clearing the override returns Bidder to Ready and the heartbeat advances again. Root spec.suspend=true still suspends everything and cannot be overridden by a component false. W4/A1 reproduces this on hat-7. - id: W2-TUNING title: "Wire Bidder and Requester settings to services" wave: W2 track: tuning parallel_with: [] depends_on: [W1-SUSPEND] status: done decision: > Add the Bidder and Requester settings to their child specs and read them in the service configs. Bidder: heartbeatIntervalSeconds, offeringRefreshSeconds (0 disables), bidTtlSeconds, maxConcurrentGuests, policyPollMilliseconds, policyTimeoutSeconds. Requester: maxConcurrent, perAccount, contractPollMilliseconds, bidWindowSeconds, receiptTimeoutSeconds, receiptPollMilliseconds, sessionCapSeconds, sessionIdleSeconds, directoryTTLSeconds, directoryKeys, policyPollMilliseconds, policyTimeoutSeconds. appctl resolves and writes them; services read them instead of constants. ComponentOverrideStatus records the child generation and Applied=true, and ComponentOverridesValid=True reports that appctl accepted the override, so an operator can see that the root value reached the child. A settings change restarts only that child. files: - api/v1alpha1/types_component.go - internal/services/bidder/config.go - internal/services/bidder/service.go - internal/services/bidder/offering.go - internal/services/bidder/lifecycle.go - internal/services/bidder/report.go - internal/services/requester/config.go - internal/appctl/bidder.go - internal/appctl/requester.go - internal/appctl/components.go - deploy/bidder-apiresourceschema.yaml - deploy/requester-apiresourceschema.yaml done_when: > settings.bidder.heartbeatIntervalSeconds=5 makes heartbeat createdAt advance about every 5s instead of 60s; offeringRefreshSeconds=0 stops offering re-commits; requester maxConcurrent and contract poll change the observed service behavior. The root reports the override generation and Applied=True; removing the override restores defaults without a root or component recreation. - id: W3-COMPONENTS title: "Complete per-component settings, overrides and immutability" wave: W3 track: components parallel_with: [] depends_on: [W2-TUNING] status: done decision: > Add typed settings and override translation for GuestControlPlane, Pds, SshServer, GatewayAPI and PolicyEngine. GuestControlPlane: guestLoginMount, role, grantSyncSeconds, issuerRetrySeconds, hostKeyAttempts, relayEnabled and HostKeyRef. Pds: maxBlobSize, maxRequestBytes, contactEmail, PdsOAuth TTLs/localhost policy, and an explicit OAuth scope list consistent with atproto. SshServer: the limits/timeouts/cache fields from target_model. GatewayAPI: nodePollIntervalSeconds and requiredDownloads. IngressProxy: provider relay/keepalive/nonce timeouts. Relay: crawlIntervalSeconds. PolicyEngine: limits/trust/cache. Requester: oauthScopes. Wire per-component Comms, Ingress, ServiceAuth and Limits where the child supports them. Keep identity DID, PDS ref, serviceID and published StableHost immutable. ComponentLimits is resolved and reported; wire enforcement only where the backend can enforce it (HostProcessLimits) and otherwise set a named LimitsEnforced=False condition rather than pretending. Add drift detection: a child spec that differs from appctl's prior desired spec sets ComponentDrift=True with the child key. NetworkApp.Components carries the same override map into each child application root; NetworkApp.Stopped stays the coarse app stop. files: - api/v1alpha1/types_component.go - api/v1alpha1/types_application.go - internal/services/guestcontrolplane/config.go - internal/services/guestcontrolplane/relay.go - internal/services/pds/config.go - internal/services/sshserver/config.go - internal/services/gatewayapi/config.go - internal/services/ingressproxy/service.go - internal/services/relay/service.go - internal/policyengine/controller/controller.go - internal/appctl/components.go - internal/appctl/pds.go - internal/appctl/ingressselect.go - internal/appctl/pdsingress.go - internal/appctl/gateway.go - internal/appctl/sshgateway.go - internal/appctl/network.go - deploy/guestcontrolplane-apiresourceschema.yaml - deploy/pds-apiresourceschema.yaml - deploy/sshserver-apiresourceschema.yaml - deploy/gatewayapi-apiresourceschema.yaml - deploy/policyengine-apiresourceschema.yaml done_when: > Every child kind in target_model has a verified override path or a documented not-supported reason; settings reach the service config; per-component comms and ingress select distinct providers/hosts; immutable changes are rejected with a named condition; direct child edits are reverted and set ComponentDrift=True; limits report whether they are enforced; network children and NetworkApp apps can be tuned independently. - id: W4-PROOF title: "Prove the matrix locally and on the hosts" wave: W4 track: proof parallel_with: [] depends_on: [W3-COMPONENTS] status: done decision: > Run the live matrix on this host, hat-9, hat-7 and the prod gateway first. Only after that proof works, write the integration suite in test/integration, per AGENTS.md. The hat-7 proof is the canonical pause: components.Bidder.suspend=true, heartbeat createdAt unchanged across at least three default heartbeat intervals, GuestControlPlane and IngressProxy Ready and their URLs answering, all other components unchanged, root Ready. Then resume and prove the heartbeat advances. Then set a visible typed override (heartbeatIntervalSeconds=5) and prove the cadence and status. The lead runs the prod gateway row; task agents do not. Do not push. Update operator docs with the key grammar, default names, precedence and the pause/resume recipe. files: - docs/plans/0024-crd-configurability.yaml - test/integration/ - test/soak/ - BIDDER_ARCH.md - GATEWAY_ARCH.md - docs/ARCHITECTURE.md done_when: > Every acceptance row has live evidence or a named blocked reason; no secret appears in any root or child status; root remains Ready with a suspended child; resume and per-component tuning work on all four hosts; gofmt, go build ./..., go vet ./..., deploy/check.sh --fast and make check-pins pass; no push. test_matrix: roots: - MarketBidder - MarketRequester - Gateway - SshGateway - MarketNetwork components: - Bidder - GuestControlPlane - Pds - PolicyEngine - Requester - GatewayAPI - SshServer - IngressProxy - Plc - Relay - DispatcherTLS scenarios: - per-component suspend and resume - root suspend precedence over a component false - per-component settings take effect - per-component comms and ingress selection - component limits and service auth - unknown key, ambiguous singleton, mismatched settings block - immutable identity, serviceID and published public host - direct child edit reverted with drift condition - GitOps SSA ownership of spec.components - root Ready with one suspended child, no collateral stop - no secret in spec or status live: - this host - hat-9 - hat-7 (oauthQr, bobvmbuilder account, no local Pds) - socialweb.computer prod gateway acceptance_matrix: rule: > W0 freezes these rows. Every row names a live host or a fixture. A row with neither is blocked, never silently skipped. proof_sources: | this host: local bidder, associated hat-9: associated bidder host hat-7: oauthQr bidder as bobvmbuilder, no local Pds prod gateway: socialweb.computer, Alice sign-in test/integration: final regression suite, written in W4 only after the live proof, per AGENTS.md fixtures: paused-bidder: components.Bidder.suspend=true on a disposable root resumed-bidder: the same root with the override removed fast-heartbeat: settings.bidder.heartbeatIntervalSeconds=5 drift-child: a direct patch to a child spec.suspend typo-override: an unknown components key rows: - id: A1 check: "pause Bidder only; Bidder not Ready/Suspended; heartbeat createdAt stops advancing; GuestControlPlane and IngressProxy stay Ready; public URLs still answer" proof: > On hat-7 set components.Bidder.suspend=true. kubectl get the root: status.components[Bidder].phase=Suspended and ready=false; GuestControlPlane and IngressProxy ready=true. Fetch com.publicdomainrelay.temp.market.heartbeat/self through Bob's PDS (com.atproto.repo.getRecord with the bidder DID, collection com.publicdomainrelay.temp.market.heartbeat, rkey self) at T0 and T0+180s; value.createdAt is identical. curl the GuestControlPlane /healthz listener URL from status and, where the root has a Pds, its /xrpc/_health listener URL; both answer below 400. On hat-7 there is no local Pds, so GuestControlPlane and IngressProxy are the check. The bidder market URL may fail because the bidder process is stopped; that is expected and is not a collateral stop. hosts: [hat-7, this host] - id: A2 check: "resume -> Bidder Ready; heartbeat advances again" proof: > Remove the components.Bidder entry or set suspend=false. Bidder phase returns Ready, ready=true, componentSummary.suspended returns 0. Fetch the same heartbeat record at T1 and T1+120s; createdAt changes and the delta is within two heartbeat intervals. No root or other component recreation is observed. hosts: [hat-7, this host] - id: A3 check: "per-component options override takes effect and is reflected in status/conditions" proof: > Set settings.bidder.heartbeatIntervalSeconds=5 on the running Bidder. The root reports ComponentOverridesValid=True, and ComponentOverrideStatus has the child component key, its observedGeneration and Applied=true; the child reports the same generation. The heartbeat record createdAt advances about every 5s, not 60s. Set offeringRefreshSeconds=0 and confirm no offering refresh write in two default refresh periods. Remove the settings and confirm the 60s cadence returns. hosts: [this host, hat-9] - id: A4 check: "no app-wide collateral stop" proof: > During A1, root phase is Ready; status.componentSummary shows desired=N, ready=N-1, suspended=1, notReady=0; every non-Bidder component has the same phase and ready value as before the pause; the shared IngressProxy is Ready; listener URLs for GuestControlPlane and Pds are unchanged. No root-wide condition is Failed. hosts: [hat-7, this host] - id: A5 check: "root suspend remains the master kill switch" proof: > With components.Bidder.suspend=false, set spec.suspend=true. Every child including the shared IngressProxy and all Exposures is Suspended. Clearing spec.suspend resumes only components whose own override is not suspending them. hosts: [this host] - id: A6 check: "direct child edit is reverted and observed" proof: > Patch a child Bidder spec.suspend=true while the root says false. On the next appctl pass the child spec.suspend returns to false and the root sets ComponentDrift=True naming Bidder/bidder. No manual child edit is required to keep the root authoritative. hosts: [this host] - id: A7 check: "invalid override fails closed without partial apply" proof: > Add components.Bldier.suspend=true (typo). The root sets ComponentOverridesValid=False with reason InvalidKey and Phase Failed. No child generation changes in that pass. Removing the typo restores Ready without touching the children. hosts: [this host] - id: A8 check: "immutable fields are rejected" proof: > Attempt to change identity DID, PdsRef, association serviceID or a published PDS publicHost through components. The root sets ComponentOverridesValid=False with reason ImmutableField or StableHostMismatch and does not change the child. The existing stable-host path still reports StableHostMismatch. hosts: [this host, prod gateway] - id: A9 check: "no secret in spec or status" proof: > Dump every root and child CR spec and status. No token, password, private key, session, audience key material or credential value appears. Status contains only refs, counts, phases, generations, condition reasons and messages. hosts: [this host, hat-7, hat-9, prod gateway] - id: A10 check: "GitOps SSA owns spec.components without conflict" proof: > Apply a second SSA field manager that sets only spec.components.Bidder.suspend. The stack's forced root apply leaves the field intact; appctl resolves it. Two edits to different component keys merge; two edits to the same key produce a visible conflict rather than a silent winner. hosts: [this host] risks: | A per-component suspend leaves that component's Service/Exposure and the shared IngressProxy Ready, so the paused component's own URL fails while the rest of the app answers. This is deliberate: propagating suspend to the shared proxy would recreate the app-wide collateral stop. Document it in the pause recipe. Per-component suspend can create dependency breakage (for example a suspended Pds with a running Bidder). The plan does not block combinations; the status shows every component and the operator owns the combination. ComponentLimits is declared but not enforced by the in-process component runtime. Wiring it to HostProcessLimits is only possible where the backend has cgroups and a per-component unit. Report LimitsEnforced=False rather than claiming enforcement. Public host, DID and association serviceID changes are not transparent once accounts exist; DID documents and association records must move. This plan rejects them instead of attempting a migration. The components map cannot be key-validated by the generated CRD (no CEL in this tree), so appctl validation is the only guard. A typo fails the root closed; it cannot silently resume a paused component. appctl uses Update, not server-side apply, on children. Direct child edits and child-level GitOps are overwritten; drift is visible only through the ComponentDrift condition/event. Operators must move child configuration to the root. Adding typed settings grows the CRD surface and schema revisions. Keep every new field optional and every default identical to today, or existing roots will restart components on upgrade. Status must not echo settings values. Only names, generations, booleans and counts are safe; a future settings field must be reviewed for secret risk. decisions: key_shape: "spec.components is a map keyed by Kind (singleton) or Kind/name; it is not a list" key_reason: "map keys give SSA per-component merge and mirror ComponentKey(kind,name); list is atomic here because no list-type markers exist" typed_over_options: "typed fields and per-kind settings; generic ApplicationOption remains legacy CLI compatibility only" master_suspend: "effective suspend is root.Spec.Suspend OR component override; root cannot be overridden" no_cascade: "per-component suspend does not suspend its Service/Exposure or a shared IngressProxy; root suspend still cascades" root_ready_with_suspended_child: "root Phase is Ready when every component is Ready or Suspended by intent" invalid_overrides: "fail closed: no child spec changes, Phase Failed, ComponentOverridesValid=False naming the key" immutable: "kind/name key, identity DID/PdsRef, association serviceID and published StableHost are immutable" child_ownership: "appctl owns child spec; direct child edits revert; annotations including restart-at survive" limits: "resolve and report component limits; enforce only through HostProcessLimits, otherwise LimitsEnforced=False" secrets: "no secret or setting value in status; refs, counts, enums and generations only" service_id: "serviceID stays root-kind-derived in this plan; only audience and hostname are per-component overridable" do_not_do: - "No secrets, tokens, passwords, private keys or credential values in root spec, child spec or status." - "No arbitrary pod specs, container images, commands, argv, env, volumes, hostPath, service accounts or raw Kubernetes objects in overrides." - "No overriding computed fields: stateDir, network, hostnames, serve bind/port, owner references, labels, Service/Exposure set." - "No bypass of root ownership: do not patch child specs, do not let GitOps own child components, do not let appctl write root spec." - "No override that creates or deletes a component; unknown keys are errors, not deletions." - "No per-component suspend cascading to a shared IngressProxy or another component's Exposure." - "No per-component override of root spec.suspend; it is the master kill switch." - "No untyped options bag for new settings; do not extend ApplicationOption except for legacy CLI compatibility." - "No mutable DID, PdsRef, association serviceID or published public host; reject with a named condition." - "No silent ignore of invalid overrides and no partial apply of a bad override set." - "No heartbeat or app data in status; do not poll a PDS from appctl to report application health." - "No reintroduction of Deno application components or a second configuration path."