plan: 0013-bounded-for-years written: 2026-10-12 revised: > 2026-10-12, first version, from the lead's review of what grows when a bidder or the gateway runs for months or years (user question of 2026-10-12: use the kcp garbage collector and our kinds so the hosts they run on never run out of memory). Split out of the draft milestone M5 of plan 0009 at the user's request. queued: > M1-BACKUP-RETENTION and M1-PDS-POINT-READS have no dependency and may start now: both are growth that already exists on main and reaches prod first. Every other item waits for 0006 M5-FRESH-DEPLOY and for the plan 0009 kinds it names. source: > A bidder handle and the gateway are meant to run for months or years. Memory in kcp, in the Go core's informers and in each DenoPod is roughly the number of objects times their size, plus caches; the store grows with every write until it is compacted; vaults, disks and PDS repositories grow with every contract. On main (checked 2026-10-12): nothing configures kine compaction or sqlite VACUUM; OpenBao has no pki tidy, no no_store and no kv max_versions; every kcp roll writes a pg_dump backup that is never removed; a DenoPod has no memory limit; ownerReferences are almost unused; kcp has no event TTL and no object-count quota; eleven listRecordsAll calls in the market libraries read every record of a DID on paths that run per accept or per event. principle: stated: > Every long-running application instance is bounded: object count, object size, store history, process memory, vault storage, disk and own PDS records each have a limit or a lifecycle that ends. A limit that is reached shows in kcp status and degrades service (a declined RFP, a restarted pod), never the host. rules: - "Every child object has an owner that ends (ownerReferences, cascading inside one workspace) or a TTL after it finishes. An object with neither is a leak and is counted." - "Every list in a status has a cap." - "A per-workspace object-count quota is the hard ceiling; reaching it refuses the create and the caller declines or waits, with the reason in status." - "Every process has a memory limit; a breach restarts that process, never the host." - "Proof is a soak reading whose series plateau, not a test." state_rules: - "No tests until prod works (user, 2026-10-12). done_when is a live reading." - "Real names only; commit and push only socialweb-computer-kcp; the lead flips status." ownership: repository: socialweb-computer-kcp go: "internal/appctl, internal/computevm, internal/clusterprovider (owners), a TTL controller in the core, internal/bootstrap and internal/pgunit (store), internal/denokcp (DenoPod limits, runs dirs), internal/openbao and internal/openbaoctl (vault)" typescript: "ts/atproto-market (point reads, own-record retention, cache caps), ts/typescript-helpers/lib (kcp client watch discipline)" research: - "kcp runs the Kubernetes garbage collector per logical cluster: ownerReferences cascade inside one workspace, never across workspaces." - "The Kubernetes TTL-after-finished controller covers Jobs only; the vendored deno-kcp kinds carry their own run TTL (PolicyWorkflowRun runTTLSecondsAfterFinished, DenoJob ttlSecondsAfterFinished)." - "kine keeps a row per revision and compacts old revisions on an interval; an sqlite file does not shrink without VACUUM." - "OpenBao pki stores every issued leaf unless the role sets no_store; pki tidy removes expired and revoked ones." items: - id: M1-BACKUP-RETENTION milestone: M1 title: Keep a bounded number of kcp store backups depends_on: [] decision: > The pg_dump written before each kcp roll (/kcp/backups/kine-.sql) is kept only for the last 5 backups and those newer than 14 days; older ones are removed after a new backup is written and checked (exit 0, size > 0). Both numbers are options with those defaults. Decided by Claude, user may veto. files: ["internal/pgunit", "internal/kcpstore"] done_when: "after 8 rolls on a dev gateway instance, 5 backups remain, the newest is the last roll's." risks: [] status: done evidence: - id: M1-PDS-POINT-READS milestone: M1 title: Market paths read one record by key, never every record of a DID depends_on: [] decision: > Each listRecordsAll(pds, did, collection) call in the market libraries that looks for one record whose uri is already known becomes a getRecord(repo, collection, rkey) on that uri (accept watch, event watch, receipts, contract lookups). Calls that truly need a scan page with a cursor and stop at the first match. Decided by Claude, user may veto. files: ["ts/atproto-market/lib/market-bidder", "ts/atproto-market/lib/compute-request-xrpc", "ts/atproto-market/lib/requester-xrpc", "the other libraries that call listRecordsAll"] done_when: "no listRecordsAll call remains on a per-accept or per-event path; a market network run with --secrets passes." risks: [] status: done evidence: - id: M2-OWNERSHIP-TREE milestone: M2 title: Every child object hangs from an owner that ends depends_on: ["0006:M5-FRESH-DEPLOY"] decision: > ownerReferences on every child kind, inside its own workspace: a ComputeVM owned by its won Bid (by its MarketBidder until plan 0009 M2-BID-KIND), a VM HostProcess mirror by its ComputeVM, a NodeClaim by its Cluster, a ComputeContract by its NodeClaim (gateway) or its MarketRequester (CLI), a PolicyWorkflowRun by the object it evaluates, each profile DenoPod, PolicyEngine and OpenBaoServer by its root. The root status gains `gc`: objects with no owner and no TTL, by kind, and objects terminating for more than 10 minutes, by kind and finalizer. Every finalizer names the one controller that removes it. Decided by Claude, user may veto. files: ["internal/appctl", "internal/computevm", "internal/clusterprovider", "ts/atproto-market (creators of child objects)", "api/types_application.go"] done_when: "deleting a MarketBidder, a Cluster or a NodeClaim leaves no child object; the root status of a busy market network shows 0 orphans and 0 stuck terminations." risks: ["A cascade that deletes a ComputeVM whose guest still runs: a ComputeVM's own finalizer stops its unit first, as today."] status: done evidence: - id: M2-TTL-CONTROLLER milestone: M2 title: One TTL-after-finished controller in the core for every finished object depends_on: [M2-OWNERSHIP-TREE] decision: > One controller in the Go core deletes an object whose annotation socialweb.computer/ttl-after-finished (a duration) has passed since the lastTransitionTime of its Complete or Failed condition. Defaults set by the creators: a Lost or Expired Bid 24 h, a Won Bid 24 h after its ComputeVM is gone, a finished CLI ComputeContract 7 d, a DenoRun and DenoJob 1 h. It runs from watches, not a poll, and reports deletions in the root status. Decided by Claude, user may veto. files: ["internal/ttlctl (new)", "internal/denokcp"] done_when: "on a market network with short TTLs set, finished objects of every kind disappear at their TTL and the counts stay flat under load." risks: [] status: done evidence: - id: M2-QUOTAS milestone: M2 title: Object-count quotas per workspace; a quota hit declines, never OOMs depends_on: [M2-TTL-CONTROLLER] decision: > Each application workspace gets an object-count quota per kind (Bids, ComputeVMs, ComputeContracts, NodeClaims, PolicyWorkflowRuns, DenoRuns; defaults sized from M4-SOAK-READING, options on the root). A refused create makes the bidder decline with reason "capacity" and the requester keep the contract Pending with an AtCapacity condition. First check whether kcp enforces ResourceQuota for resources bound through an APIBinding; if it does not, the same count check runs in the creating controller against its watch. Decided by Claude, user may veto. files: ["deploy/apps/*/ (quota objects)", "internal/appctl", "ts/atproto-market/lib/market-bidder"] done_when: "with a quota of 10 Bids, the 11th RFP is declined with reason capacity and kcp memory does not move." risks: [] status: done evidence: - id: M3-STORE-COMPACTION milestone: M3 title: The kcp store keeps a bounded history depends_on: ["0006:M5-FRESH-DEPLOY"] decision: > kine's compaction stays the apiserver's, not kine's own: kine's --compact-interval defaults to 0 and its Compact is a documented no-op once that interval is positive ("manual compact is a no-op unless automatic compaction is disabled"), so setting it would hand history-bounding to kine's ticker, the reverse of the design. The sqlite DSN does carry _journal_mode=WAL and the rest of kine's own five parameters (see the kine bullet in docs/ARCHITECTURE.md section 15). Still to do, and what this item is now about: a bidder's sqlite store uses auto_vacuum=INCREMENTAL and the core runs incremental_vacuum on an interval; the gateway's postgres keeps autovacuum on for the kine table and the root status reports its table size and dead tuples; kcp runs with a short --event-ttl (1 h) and our controllers write status, not Events. Decided by Claude, user may veto. files: ["internal/bootstrap", "internal/kcpserver", "internal/pgunit", "internal/postgresctl"] done_when: "under M4-SOAK-READING load the sqlite file size and the kine row count plateau on a bidder, and the kine table size plateaus on the gateway." risks: ["An sqlite store created before auto_vacuum was set needs one full VACUUM; the core runs it once at the next start, during the slot roll."] status: done evidence: - id: M3-DENOPOD-LIMITS milestone: M3 title: Every process has a memory limit; a breach restarts that process depends_on: ["0006:M5-FRESH-DEPLOY"] decision: > DenoPod spec gains resources.memory. The deno-kcp provider passes --v8-flags=--max-old-space-size at about 80% of it and runs the pod in its own delegated cgroup scope under the core's unit with MemoryMax and MemorySwapMax=0; a breach restarts the pod and is counted with its reason in the pod status. Every HostProcess (kcp, kine, OpenBao, postgres, VM units) gets a MemoryMax (hostsvc supports it today). The TypeScript caches get size caps: the scope verdict cache an LRU of 10000 entries on top of its TTLs, and any remaining in-memory dedup sets bounded. The profiles set every limit. Decided by Claude, user may veto. files: ["internal/denokcp/api/v1alpha1/types_denopod.go", "internal/denokcp/provider", "internal/hostsvc", "deploy/apps/*/", "ts/policy-engine/lib/policy-engine-scope-cache"] done_when: "a DenoPod made to allocate past its limit is restarted alone (core and other pods keep their pids); every HostProcess unit shows a MemoryMax." risks: ["Delegated cgroups under a user manager need Delegate=yes on the core unit; 0006 M1-INSTALL writes the delegation drop-in already."] status: done evidence: - id: M3-VAULT-TIDY milestone: M3 title: Vault storage stays bounded depends_on: ["0006:M5-FRESH-DEPLOY"] decision: > Every pki role that issues DenoPod and service leaves sets no_store=true and a short TTL (renewed by the provider); each OpenBaoServer runs pki tidy (tidy_cert_store, tidy_revoked_certs, safety_buffer 72 h) on an interval; kv mounts set max_versions=3; token and lease tidy runs on an interval. OpenBaoServer spec declares these, and the OpenBaoServer controller keeps them in sync like its mounts. Decided by Claude, user may veto. files: ["internal/openbao", "internal/openbaoctl", "internal/baosync", "api (OpenBaoServer spec)"] done_when: "after 500 DenoPod restarts the OpenBao storage size of the instance is within 5% of its start." risks: [] status: done evidence: - id: M3-DISK-RETENTION milestone: M3 title: Disk use stays bounded depends_on: [M1-BACKUP-RETENTION] decision: > The deno-kcp provider removes a pod's runs/ directory when the pod or run is deleted and at start for directories no object names. User units log to the journal with SystemMaxUse-style limits set for the account; any file log is rotated. Guest images that no ComputeVM, version entry or the current and previous binary names are removed. The existing DENO_DIR, asset and VM work dir rules stay. The root status reports the state dir size. Decided by Claude, user may veto. files: ["internal/denokcp/provider", "internal/guestimage", "internal/install"] done_when: "under M4-SOAK-READING load the state dir size plateaus." risks: [] status: doing evidence: - id: M3-PDS-RECORD-RETENTION milestone: M3 title: Own market records are removed when they are no longer needed depends_on: [M1-PDS-POINT-READS, "0009:M2-BID-KIND"] decision: > The bidder deletes its own market.bid record when its Bid is Lost or Expired and its TTL passes, every due record each pass in applyWrites batches with a time bound per pass, not a count cap (changed 2026-10-18, since plan 0018 the records are in the bidder's own repo PDS); the requester deletes its own RFP and accept records after the contract is Released and a retention time (default 7 d) passes, keeping what a later vm.delete answer or receipt check needs. No application ever deletes a record of another DID. Decided by Claude, user may veto. files: ["ts/atproto-market/lib/market-bidder", "ts/atproto-market/lib/requester-xrpc"] done_when: "on a market network after 1000 contracts and their retention, each PDS repository holds only live and retained records." risks: ["A requester that checks a receipt after the bidder removed its bid record: the receipt and the ComputeVM tombstone answer, not the bid record."] status: dropped evidence: - id: M4-SOAK-READING milestone: M4 title: Soak reading - every series plateaus depends_on: [M2-QUOTAS, M3-STORE-COMPACTION, M3-DENOPOD-LIMITS, M3-VAULT-TIDY, M3-DISK-RETENTION, M3-PDS-RECORD-RETENTION] decision: > Changed by the user 2026-10-17: 3 h of accelerated load is enough, with every TTL and sweep interval shortened so each fires several times inside the window (e.g. Bid TTL 15 min, contract TTL 30 min); the slope is fitted after warm-up. The text below is the earlier decision; its counts scale to 3 h. A market network with a gateway under accelerated load for about 24 h: at least 50000 RFPs, 2000 contracts, 200 restarts and upgrades of bidder and gateway components. Sampled every minute: RSS of kcp, kine, the core, each deno, OpenBao and postgres; object counts per kind; kine rows and store size; OpenBao storage size; state dir size. Recorded in docs/ARCHITECTURE.md section 15. Pass: every series is flat after warm-up (a fitted slope within noise); any series that keeps growing is named and fixed. The readings also size the M2-QUOTAS defaults. Decided by Claude, user may veto. files: ["docs/ARCHITECTURE.md"] done_when: "the readings are recorded and every series plateaus." risks: ["24 h on hat-8 holds VM slots; it runs in its own stand-in account with its own subuid range."] status: doing evidence: acceptance: where: hat-8 checks: - id: ACC13-BOUNDED text: "Under M4-SOAK-READING load, memory of every process, object counts per kind, the kcp store, the vault store, the state dir and own PDS records all plateau." status: todo evidence: [] - id: ACC13-NO-HOST-OOM text: "A quota hit declines or waits with a reason in status, and a memory breach restarts only the process that breached; the host never runs out of memory." status: done evidence: [] - id: ACC13-NO-ORPHANS text: "Every object has an owner that ends or a TTL; the root status of each application reports 0 orphans and 0 stuck terminations after the soak." status: done evidence: [] questions_for_user: []