# hatk privateCollections Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Let a hatk app declare collections that are indexed and queryable in-process but never served by the built-in `dev.hatk.*` record endpoints. **Architecture:** hatk currently conflates "is this collection indexed" with "should it be served publicly" — both are answered by `getSchema(collection)` returning a schema, so adding a lexicon publishes it. This adds a `privateCollections` config field held in a small module-level registry (mirroring how the schema registry already works), and a guard applied as the *first* check in each generic record endpoint, before any schema or database lookup. **Tech Stack:** TypeScript (ESM, `.ts` extension imports), Node, vitest (introduced by this plan), oxlint + oxfmt. ## Global Constraints - Package is ESM: `"type": "module"`. Relative imports carry the `.ts` extension (`from './server.ts'`). - Formatting is oxfmt: 2-space indent, single quotes, no semicolons. Run `npm run format` before committing. - `npm run check` (`tsc --noEmit && oxlint . && oxfmt --check .`) must pass at every commit. - The guard must **fail closed**: run before `getSchema`, before any DB access, and return 404 (never 403 — a 403 confirms the collection exists). - No breaking changes. `privateCollections` defaults to `[]`, so existing apps behave exactly as before. - Work happens in `~/code/hatk`, in the `packages/hatk` workspace. --- ### Task 1: Test infrastructure hatk has no test runner. This adds one so the rest of the plan can be test-driven. **Files:** - Modify: `packages/hatk/package.json` - Modify: `package.json` (repo root) - Create: `packages/hatk/vitest.config.ts` - Create: `packages/hatk/test/smoke.test.ts` **Interfaces:** - Consumes: nothing - Produces: `npm test` at the repo root runs vitest against `packages/hatk/test/**/*.test.ts` - [ ] **Step 1: Confirm vitest is already available** `vitest` is already a **production** dependency of `@hatk/hatk` at `^4` — the scaffolding templates in `src/templates/*.tpl` import from it, so hatk supplies it to consuming apps. Do **not** install, move, or re-pin it: changing it alters the published dependency surface and breaks scaffolded tests downstream. ```bash cd ~/code/hatk && node -e "console.log(require('./packages/hatk/package.json').dependencies.vitest)" ``` Expected: `^4`. If it is absent from `dependencies`, stop and report BLOCKED. - [ ] **Step 2: Create the vitest config** Create `packages/hatk/vitest.config.ts`: ```ts import { defineConfig } from 'vitest/config' export default defineConfig({ test: { include: ['test/**/*.test.ts'], }, }) ``` - [ ] **Step 3: Add test scripts** In `packages/hatk/package.json`, add to `"scripts"`: ```json "test": "vitest run" ``` In the repo root `package.json`, add to `"scripts"`: ```json "test": "npm run test -w @hatk/hatk" ``` - [ ] **Step 4: Write a smoke test** Create `packages/hatk/test/smoke.test.ts`: ```ts import { expect, test } from 'vitest' test('vitest runs', () => { expect(1 + 1).toBe(2) }) ``` - [ ] **Step 5: Run the test suite** Run: `cd ~/code/hatk && npm test` Expected: PASS, 1 test. - [ ] **Step 6: Verify checks still pass** Run: `cd ~/code/hatk && npm run check` Expected: no errors. - [ ] **Step 7: Commit** ```bash cd ~/code/hatk git add package.json packages/hatk/package.json packages/hatk/vitest.config.ts packages/hatk/test/smoke.test.ts git commit -m "chore: add vitest test infrastructure" ``` --- ### Task 2: The guard module A pure module holding the private-collection set and the AT-URI collection parser. Pure so it can be tested without a database or server. **Files:** - Create: `packages/hatk/src/private-collections.ts` - Create: `packages/hatk/test/private-collections.test.ts` **Interfaces:** - Consumes: nothing - Produces: - `setPrivateCollections(list: string[]): void` - `isPrivateCollection(nsid: string | null | undefined): boolean` - `collectionFromUri(uri: string): string | undefined` - [ ] **Step 1: Write the failing test** Create `packages/hatk/test/private-collections.test.ts`: ```ts import { beforeEach, expect, test } from 'vitest' import { collectionFromUri, isPrivateCollection, setPrivateCollections } from '../src/private-collections.ts' beforeEach(() => { setPrivateCollections([]) }) test('no collections are private by default', () => { expect(isPrivateCollection('social.switchback.activity')).toBe(false) }) test('listed collections are private', () => { setPrivateCollections(['social.switchback.activity']) expect(isPrivateCollection('social.switchback.activity')).toBe(true) expect(isPrivateCollection('app.bsky.actor.profile')).toBe(false) }) test('setPrivateCollections replaces rather than accumulates', () => { setPrivateCollections(['a.b.c']) setPrivateCollections(['d.e.f']) expect(isPrivateCollection('a.b.c')).toBe(false) expect(isPrivateCollection('d.e.f')).toBe(true) }) test('a missing collection is not private', () => { setPrivateCollections(['a.b.c']) expect(isPrivateCollection(null)).toBe(false) expect(isPrivateCollection(undefined)).toBe(false) }) test('collectionFromUri extracts the collection segment', () => { expect(collectionFromUri('at://did:plc:abc/social.switchback.activity/3kx')).toBe( 'social.switchback.activity', ) }) test('collectionFromUri returns undefined for a malformed uri', () => { expect(collectionFromUri('not-a-uri')).toBeUndefined() expect(collectionFromUri('at://did:plc:abc')).toBeUndefined() }) ``` - [ ] **Step 2: Run the test to verify it fails** Run: `cd ~/code/hatk && npm test` Expected: FAIL — cannot resolve `../src/private-collections.ts`. - [ ] **Step 3: Write the implementation** Create `packages/hatk/src/private-collections.ts`: ```ts /** * Collections that are indexed and queryable in-process but must never be * served by the built-in dev.hatk.* record endpoints. * * Held module-level, mirroring the schema registry, so the guard can be applied * inside handlers without threading config through every signature. */ let privateCollections = new Set() export function setPrivateCollections(list: string[]): void { privateCollections = new Set(list) } export function isPrivateCollection(nsid: string | null | undefined): boolean { return nsid != null && privateCollections.has(nsid) } /** * The collection segment of an AT-URI: at://{did}/{collection}/{rkey}. * Splitting yields ['at:', '', did, collection, rkey]. */ export function collectionFromUri(uri: string): string | undefined { const parts = uri.split('/') return parts.length > 3 ? parts[3] : undefined } ``` - [ ] **Step 4: Run the test to verify it passes** Run: `cd ~/code/hatk && npm test` Expected: PASS, 7 tests. - [ ] **Step 5: Format and check** Run: `cd ~/code/hatk && npm run format && npm run check` Expected: no errors. - [ ] **Step 6: Commit** ```bash cd ~/code/hatk git add packages/hatk/src/private-collections.ts packages/hatk/test/private-collections.test.ts git commit -m "feat: add private collection registry and at-uri collection parser" ``` --- ### Task 3: The config field **Files:** - Modify: `packages/hatk/src/config.ts:59-74` (the `HatkConfig` interface) and `:117-140` (the config literal) **Interfaces:** - Consumes: nothing - Produces: `HatkConfig.privateCollections: string[]`, defaulting to `[]` There is no unit test for this task. The field's existence is a type-level fact, and vitest does not type-check, so a runtime test would pass whether or not the field exists. `npm run typecheck` is the gate, and Task 4's tests fail without the field. Do not add a test that cannot fail. - [ ] **Step 1: Establish the current state** Run: `cd ~/code/hatk && npm run typecheck` Expected: passes — this is the baseline. - [ ] **Step 2: Add the field to the interface** In `packages/hatk/src/config.ts`, in the `HatkConfig` interface, immediately after the `collections` line: ```ts collections: string[] // optional — auto-derived from lexicons if empty privateCollections: string[] // never served by the built-in dev.hatk.* record endpoints ``` - [ ] **Step 3: Add the default to the config literal** In the same file, in the `const config: HatkConfig = {` literal, immediately after the `collections:` line: ```ts collections: parsed.collections || [], privateCollections: parsed.privateCollections || [], ``` - [ ] **Step 4: Run typecheck** Run: `cd ~/code/hatk && npm run typecheck` Expected: no type errors. - [ ] **Step 5: Format, check, commit** ```bash cd ~/code/hatk npm run format && npm run check git add packages/hatk/src/config.ts git commit -m "feat: add privateCollections config field" ``` --- ### Task 4: Guard the HTTP record endpoints The three generic record branches in `createHandler`. The guard goes **first** in each branch, before `getSchema` and before any DB call, so a private collection is rejected without touching storage. **Files:** - Modify: `packages/hatk/src/server.ts` — `getRecords` branch (~line 347), `getRecord` branch (~line 378), `searchRecords` branch (~line 410) - Create: `packages/hatk/test/private-endpoints.test.ts` **Interfaces:** - Consumes: `isPrivateCollection`, `collectionFromUri` from Task 2; `createHandler(config: HandlerConfig)` from `packages/hatk/src/server.ts:298` - Produces: 404 responses for private collections on all three endpoints - [ ] **Step 1: Write the failing test** `createHandler` needs no database for these assertions, because the guard returns before any lookup. `oauth: null` skips viewer resolution. Create `packages/hatk/test/private-endpoints.test.ts`: ```ts import { beforeEach, expect, test } from 'vitest' import { createHandler } from '../src/server.ts' import { setPrivateCollections } from '../src/private-collections.ts' const PRIVATE = 'social.switchback.activity' function handler() { return createHandler({ collections: [PRIVATE, 'app.bsky.actor.profile'], publicDir: null, oauth: null, admins: [], }) } beforeEach(() => { setPrivateCollections([PRIVATE]) }) test('getRecords 404s for a private collection', async () => { const res = await handler()( new Request(`http://localhost/xrpc/dev.hatk.getRecords?collection=${PRIVATE}`), ) expect(res.status).toBe(404) }) test('searchRecords 404s for a private collection', async () => { const res = await handler()( new Request(`http://localhost/xrpc/dev.hatk.searchRecords?collection=${PRIVATE}&q=run`), ) expect(res.status).toBe(404) }) test('getRecord 404s for a uri in a private collection', async () => { const res = await handler()( new Request( `http://localhost/xrpc/dev.hatk.getRecord?uri=at://did:plc:abc/${PRIVATE}/3kx`, ), ) expect(res.status).toBe(404) }) test('the guard does not 403, which would confirm the collection exists', async () => { const res = await handler()( new Request(`http://localhost/xrpc/dev.hatk.getRecords?collection=${PRIVATE}`), ) expect(res.status).not.toBe(403) }) test('a collection that is not private is not blocked by the guard', async () => { const res = await handler()( new Request('http://localhost/xrpc/dev.hatk.getRecords?collection=app.bsky.actor.profile'), ) // No schema is registered in this test, so the endpoint 404s for a different // reason. What matters is that the guard is not what rejected it: the body // carries the unknown-collection message rather than a bare not-found. expect(await res.text()).toContain('Unknown collection') }) ``` - [ ] **Step 2: Run the test to verify it fails** Run: `cd ~/code/hatk && npm test` Expected: FAIL — the private-collection requests return an "Unknown collection" 404 or reach the DB, and the last test's assertion ordering is not yet meaningful. - [ ] **Step 3: Import the guard in server.ts** At the top of `packages/hatk/src/server.ts`, alongside the existing imports: ```ts import { collectionFromUri, isPrivateCollection } from './private-collections.ts' ``` - [ ] **Step 4: Guard the getRecords branch** In the `if (url.pathname === coreXrpc('getRecords')) {` branch, insert immediately after the `collection` is read and its missing-parameter check, before the `getSchema` check: ```ts const collection = url.searchParams.get('collection') if (!collection) return withCors(jsonError(400, 'Missing collection parameter', acceptEncoding)) if (isPrivateCollection(collection)) return withCors(jsonError(404, `Unknown collection: ${collection}`, acceptEncoding)) if (!getSchema(collection)) return withCors(jsonError(404, `Unknown collection: ${collection}`, acceptEncoding)) ``` - [ ] **Step 5: Guard the searchRecords branch** In the `if (url.pathname === coreXrpc('searchRecords')) {` branch, apply the same insertion — after the missing-parameter checks for `collection` and `q`, before `getSchema`: ```ts if (isPrivateCollection(collection)) return withCors(jsonError(404, `Unknown collection: ${collection}`, acceptEncoding)) ``` - [ ] **Step 6: Guard the getRecord branch** This branch has no `collection` parameter — it is parsed from the URI. Insert immediately after the missing-`uri` check, before `getRecordByUri`: ```ts if (url.pathname === coreXrpc('getRecord')) { const uri = url.searchParams.get('uri') if (!uri) return withCors(jsonError(400, 'Missing uri parameter', acceptEncoding)) if (isPrivateCollection(collectionFromUri(uri))) return withCors(jsonError(404, 'Record not found', acceptEncoding)) const record = await getRecordByUri(uri) ``` - [ ] **Step 7: Run the tests to verify they pass** Run: `cd ~/code/hatk && npm test` Expected: PASS, all 5 tests in this file. - [ ] **Step 8: Format, check, commit** ```bash cd ~/code/hatk npm run format && npm run check git add packages/hatk/src/server.ts packages/hatk/test/private-endpoints.test.ts git commit -m "feat: block private collections from generic HTTP record endpoints" ``` --- ### Task 5: Guard the in-process handler registry `registerCoreHandlers` registers the same three endpoints for `callXrpc()` in SSR and server code. These are a separate code path from Task 4 and need the same guard, or SSR bypasses it. **Files:** - Modify: `packages/hatk/src/server.ts` — `registerCoreHandlers` at `:90`, the `getRecords` handler at `:91`, `getRecord` at `:116`, `searchRecords` at `:135` - Modify: `packages/hatk/test/private-endpoints.test.ts` **Interfaces:** - Consumes: `isPrivateCollection`, `collectionFromUri` from Task 2 (already imported by Task 4) - Produces: `NotFoundError` thrown from the three registry handlers for private collections - [ ] **Step 1: Write the failing test** Append to `packages/hatk/test/private-endpoints.test.ts`: ```ts import { registerCoreHandlers } from '../src/server.ts' import { callXrpc } from '../src/xrpc.ts' test('the in-process getRecords handler rejects a private collection', async () => { registerCoreHandlers([PRIVATE], null) await expect(callXrpc('dev.hatk.getRecords', { collection: PRIVATE })).rejects.toThrow( /Unknown collection/, ) }) test('the in-process getRecord handler rejects a private uri', async () => { registerCoreHandlers([PRIVATE], null) await expect( callXrpc('dev.hatk.getRecord', { uri: `at://did:plc:abc/${PRIVATE}/3kx` }), ).rejects.toThrow(/not found/i) }) ``` `callXrpc(nsid, params, input)` is exported from `packages/hatk/src/xrpc.ts:354` and dispatches to handlers registered via `registerCoreXrpcHandler`. - [ ] **Step 2: Run the test to verify it fails** Run: `cd ~/code/hatk && npm test` Expected: FAIL — the handlers resolve or throw a different error. - [ ] **Step 3: Guard the registry getRecords handler** In `registerCoreHandlers`, in the `dev.hatk.getRecords` handler, after the missing-collection check and before `getSchema`: ```ts if (!collection) throw new InvalidRequestError('Missing collection parameter') if (isPrivateCollection(collection)) throw new NotFoundError(`Unknown collection: ${collection}`) if (!getSchema(collection)) throw new NotFoundError(`Unknown collection: ${collection}`) ``` - [ ] **Step 4: Guard the registry searchRecords handler** In the `dev.hatk.searchRecords` handler, after the missing-parameter checks and before `getSchema`: ```ts if (isPrivateCollection(collection)) throw new NotFoundError(`Unknown collection: ${collection}`) ``` - [ ] **Step 5: Guard the registry getRecord handler** In the `dev.hatk.getRecord` handler, after the missing-`uri` check and before `getRecordByUri`: ```ts if (!uri) throw new InvalidRequestError('Missing uri parameter') if (isPrivateCollection(collectionFromUri(uri))) throw new NotFoundError('Record not found') ``` - [ ] **Step 6: Run the tests to verify they pass** Run: `cd ~/code/hatk && npm test` Expected: PASS. - [ ] **Step 7: Format, check, commit** ```bash cd ~/code/hatk npm run format && npm run check git add packages/hatk/src/server.ts packages/hatk/test/private-endpoints.test.ts git commit -m "feat: block private collections from in-process record handlers" ``` --- ### Task 6: Omit private collections from describeCollections `describeCollections` publishes each collection's schema, including column names. Listing a private collection advertises its shape even if its rows are unreachable. **Files:** - Modify: `packages/hatk/src/server.ts` — registry handler at `:153`, HTTP branch at `:434` - Modify: `packages/hatk/test/private-endpoints.test.ts` **Interfaces:** - Consumes: `isPrivateCollection` from Task 2 - Produces: `describeCollections` output excluding private collections - [ ] **Step 1: Write the failing test** Append to `packages/hatk/test/private-endpoints.test.ts`: ```ts test('describeCollections omits private collections', async () => { const res = await handler()(new Request('http://localhost/xrpc/dev.hatk.describeCollections')) const body = (await res.json()) as { collections: { collection: string }[] } const names = body.collections.map((c) => c.collection) expect(names).not.toContain(PRIVATE) expect(names).toContain('app.bsky.actor.profile') }) ``` - [ ] **Step 2: Run the test to verify it fails** Run: `cd ~/code/hatk && npm test` Expected: FAIL — `names` contains the private collection. - [ ] **Step 3: Filter the HTTP branch** In the `if (url.pathname === coreXrpc('describeCollections')) {` branch, filter before mapping: ```ts const collectionInfo = collections .filter((c) => !isPrivateCollection(c)) .map((c) => { const schema = getSchema(c) return { collection: c, columns: schema?.columns.map((col) => ({ name: col.name, originalName: col.originalName, type: col.sqlType, required: col.notNull, })), } }) ``` - [ ] **Step 4: Filter the registry handler** Apply the same `.filter((c) => !isPrivateCollection(c))` before `.map(` in the `dev.hatk.describeCollections` handler registered in `registerCoreHandlers`. - [ ] **Step 5: Run the tests to verify they pass** Run: `cd ~/code/hatk && npm test` Expected: PASS. - [ ] **Step 6: Format, check, commit** ```bash cd ~/code/hatk npm run format && npm run check git add packages/hatk/src/server.ts packages/hatk/test/private-endpoints.test.ts git commit -m "feat: omit private collections from describeCollections" ``` --- ### Task 7: Wire config into the registry at startup Nothing has called `setPrivateCollections` yet, so the guard is inert in a real app. This connects config to the registry in all three entry points, including the test harness — without which apps cannot test their own private collections. **Files:** - Modify: `packages/hatk/src/main.ts:119` (near `registerCoreHandlers(collections, config.oauth)`) - Modify: `packages/hatk/src/dev-entry.ts:77` (same call) - Modify: `packages/hatk/src/test.ts` — inside `startTestServer`, before `startServer(...)` at `:285` - Modify: `packages/hatk/test/private-endpoints.test.ts` **Interfaces:** - Consumes: `setPrivateCollections` from Task 2; `HatkConfig.privateCollections` from Task 3 - Produces: the guard active in `hatk start`, `vp dev`, and `startTestServer()` - [ ] **Step 1: Write the failing test** Append to `packages/hatk/test/private-endpoints.test.ts`: ```ts test('setPrivateCollections drives the guard, so an empty list serves normally', async () => { setPrivateCollections([]) const res = await handler()( new Request(`http://localhost/xrpc/dev.hatk.getRecords?collection=${PRIVATE}`), ) // Not blocked by the guard: rejected as an unregistered schema instead. expect(await res.text()).toContain('Unknown collection') }) ``` - [ ] **Step 2: Run the test to verify it passes already** Run: `cd ~/code/hatk && npm test` Expected: PASS. This one passes before the change by design — it is not a TDD test. It pins the registry as the only source of the private set, so the wiring in the steps below cannot satisfy the other tests by hardcoding a collection name. Keep it. - [ ] **Step 3: Wire main.ts** In `packages/hatk/src/main.ts`, add the import alongside the existing `./server.ts` import: ```ts import { setPrivateCollections } from './private-collections.ts' ``` and immediately before `registerCoreHandlers(collections, config.oauth)`: ```ts setPrivateCollections(config.privateCollections) ``` - [ ] **Step 4: Wire dev-entry.ts** In `packages/hatk/src/dev-entry.ts`, add the same import, and immediately before `registerCoreHandlers(collections, config.oauth)`: ```ts setPrivateCollections(config.privateCollections) ``` - [ ] **Step 5: Wire the test harness** In `packages/hatk/src/test.ts`, add the same import, and inside `startTestServer`, immediately before the `const httpServer = startServer(` call: ```ts setPrivateCollections(ctx._config.privateCollections) ``` - [ ] **Step 6: Run tests and checks** Run: `cd ~/code/hatk && npm test && npm run check` Expected: PASS, no errors. - [ ] **Step 7: Commit** ```bash cd ~/code/hatk git add packages/hatk/src/main.ts packages/hatk/src/dev-entry.ts packages/hatk/src/test.ts packages/hatk/test/private-endpoints.test.ts git commit -m "feat: apply privateCollections config at startup and in the test harness" ``` --- ### Task 8: Document and release **Files:** - Modify: `docs/` — the configuration reference page (locate with the command in Step 1) - Modify: `packages/hatk/package.json` (version) **Interfaces:** - Consumes: everything above - Produces: a published `@hatk/hatk` alpha that switchback can depend on - [ ] **Step 1: Find the config documentation page** Run: `cd ~/code/hatk && grep -rln "collections" docs/ | head` Open the configuration reference page from the results. - [ ] **Step 2: Document the field** Add an entry for `privateCollections` next to `collections`: ```markdown ### `privateCollections` `string[]` — defaults to `[]`. Collections listed here are indexed, typed, and queryable from your own feeds and XRPC handlers, but are never served by the built-in `dev.hatk.*` record endpoints. `getRecords`, `getRecord`, and `searchRecords` return 404 for them, and `describeCollections` omits them. Use this for any collection holding data that is not public. Authorization for your own handlers remains yours to enforce — this only ensures hatk does not serve the collection on your behalf. ```ts export default defineConfig({ privateCollections: ['social.switchback.activity'], }) ``` ``` - [ ] **Step 3: Bump the version** In `packages/hatk/package.json`, increment the alpha: `0.0.1-alpha.63` → `0.0.1-alpha.64`. - [ ] **Step 4: Verify the build** Run: `cd ~/code/hatk && npm run build && npm test && npm run check` Expected: build succeeds, tests pass, no check errors. - [ ] **Step 5: Commit** ```bash cd ~/code/hatk git add docs packages/hatk/package.json git commit -m "docs: document privateCollections and bump to alpha.64" ``` - [ ] **Step 6: Publish — NOT performed by the implementer** Stop here and hand the branch over. The repository owner runs the publish themselves, so that the dependency surface of a released package gets a human look before it reaches consumers: ```bash cd ~/code/hatk && npm run release ``` Expected: `@hatk/hatk@0.0.1-alpha.64` published with the `alpha` tag. --- ## Verification After all tasks, confirm the whole feature from a consumer's perspective: - [ ] `cd ~/code/hatk && npm test` — all tests pass - [ ] `cd ~/code/hatk && npm run check` — clean - [ ] An app with `privateCollections: ['x.y.z']` gets 404 from `/xrpc/dev.hatk.getRecords?collection=x.y.z` and no `x.y.z` entry in `/xrpc/dev.hatk.describeCollections` - [ ] An app with `privateCollections: []` behaves exactly as before this change