chadtmiller.com/pds.js
An atproto PDS in JavaScript, for Cloudflare Workers and Node
0
0
# Lexicon Resolver Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
**Goal:** Add optional lexicon validation to record writes with dynamic schema resolution.
**Architecture:** Core defines a `LexiconResolverPort` interface and wires it into write handlers (createRecord, putRecord, applyWrites). A separate `@pdsjs/lexicon-resolver` package wraps `@atproto/lex-resolver` for schema resolution and `@atproto/lexicon` for validation.
**Tech Stack:** `@atproto/lexicon` (validation), `@atproto/lex-resolver` (DNS + PDS schema fetching)
---
## Task 1: Add LexiconResolverPort to ports.js
**Files:**
- Modify: `packages/core/src/ports.js:127-129`
**Step 1: Add the port typedef**
Add before the final `export {}`:
```javascript
/**
* @typedef {'valid' | 'unknown'} ValidationStatus
*/
/**
* Lexicon resolver port for validating records against schemas.
* Implementations may resolve schemas dynamically (DNS + fetch) or use static schemas.
*
* @typedef {Object} LexiconResolverPort
* @property {(collection: string, record: object) => Promise<ValidationStatus>} assertValid
* Resolves schema if needed, validates record.
* Returns 'valid' on success, 'unknown' if schema unresolvable.
* Throws Error with descriptive message if record is invalid against known schema.
*/
```
**Step 2: Commit**
```bash
git add packages/core/src/ports.js
git commit -m "feat(core): add LexiconResolverPort typedef
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>"
```
---
## Task 2: Add lexiconResolver to PDS constructor
**Files:**
- Modify: `packages/core/src/pds.js:300-340`
**Step 1: Add JSDoc for the new parameter**
Add to the constructor JSDoc (after line 311):
```javascript
* @param {import('./ports.js').LexiconResolverPort} [config.lexiconResolver] - Optional lexicon resolver for record validation
```
**Step 2: Add to constructor destructuring**
Add `lexiconResolver` to the destructuring (after `webSocket,`):
```javascript
webSocket,
lexiconResolver,
```
**Step 3: Store on instance**
Add after `this.webSocket = webSocket;` (line 334):
```javascript
this.lexiconResolver = lexiconResolver;
```
**Step 4: Commit**
```bash
git add packages/core/src/pds.js
git commit -m "feat(core): add lexiconResolver to PDS constructor
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>"
```
---
## Task 3: Add validation to handleCreateRecord
**Files:**
- Modify: `packages/core/src/pds.js:742-819`
- Test: `packages/core/test/lexicon.test.js` (create)
**Step 1: Write failing test**
Create `packages/core/test/lexicon.test.js`:
```javascript
import { describe, expect, test } from 'vitest';
// Mock resolver that tracks calls
function createMockResolver(behavior = 'valid') {
const calls = [];
return {
calls,
async assertValid(collection, record) {
calls.push({ collection, record });
if (behavior === 'valid') return 'valid';
if (behavior === 'unknown') return 'unknown';
if (behavior === 'invalid') throw new Error('Invalid record: missing required field');
return 'valid';
},
};
}
describe('Lexicon Validation', () => {
describe('LexiconResolverPort mock', () => {
test('calls resolver with collection and record', async () => {
const resolver = createMockResolver('valid');
expect(resolver.calls).toHaveLength(0);
const status = await resolver.assertValid('app.bsky.feed.post', { text: 'hello' });
expect(resolver.calls).toHaveLength(1);
expect(resolver.calls[0].collection).toBe('app.bsky.feed.post');
expect(resolver.calls[0].record).toEqual({ text: 'hello' });
expect(status).toBe('valid');
});
test('returns unknown when schema not found', async () => {
const resolver = createMockResolver('unknown');
const status = await resolver.assertValid('com.example.unknown', {});
expect(status).toBe('unknown');
});
test('throws when record is invalid', async () => {
const resolver = createMockResolver('invalid');
await expect(
resolver.assertValid('app.bsky.feed.post', {})
).rejects.toThrow('Invalid record');
});
});
});
```
**Step 2: Run test to verify it passes**
```bash
cd /Users/chadmiller/code/pds-experiment && pnpm test packages/core/test/lexicon.test.js
```
Expected: PASS (tests the mock behavior)
**Step 3: Update handleCreateRecord to use resolver**
In `handleCreateRecord`, replace the current record handling (around lines 785-818) with:
```javascript
const rkey = providedRkey || createTid();
const uri = `at://${did}/${collection}/${rkey}`;
// Ensure $type is set
const recordWithType = { $type: collection, ...record };
// Validate record against schema if resolver provided
const shouldValidate = _validate !== false;
let validationStatus;
if (shouldValidate && this.lexiconResolver) {
try {
validationStatus = await this.lexiconResolver.assertValid(
collection,
recordWithType,
);
} catch (err) {
return Response.json(
{
error: 'InvalidRecord',
message: `Invalid ${collection} record: ${err.message}`,
},
{ status: 400 },
);
}
} else if (shouldValidate && !this.lexiconResolver) {
validationStatus = 'unknown';
}
// If _validate === false, validationStatus remains undefined
// Encode record and create CID
const encoded = cborEncodeDagCbor(recordWithType);
const cid = cidToString(await createCid(encoded));
// Store block
await this.actorStorage.putBlock(cid, encoded);
// Store record
await this.actorStorage.putRecord(uri, cid, collection, rkey, encoded);
// Link blobs to record
const blobRefs = findBlobRefs(record);
for (const blobCid of blobRefs) {
await this.actorStorage.linkBlobToRecord(blobCid, uri);
}
// Create commit
await this.createCommit(did, [{ action: 'create', uri, cid }]);
// Get commit info
const latestCommit = await this.actorStorage.getLatestCommit();
return Response.json({
uri,
cid,
commit: latestCommit
? { cid: latestCommit.cid, rev: latestCommit.rev }
: null,
validationStatus,
});
```
**Step 4: Run tests**
```bash
cd /Users/chadmiller/code/pds-experiment && pnpm test
```
**Step 5: Commit**
```bash
git add packages/core/src/pds.js packages/core/test/lexicon.test.js
git commit -m "feat(core): add lexicon validation to createRecord
- Calls lexiconResolver.assertValid when resolver provided
- Returns InvalidRecord error when validation fails
- Returns validationStatus in response (valid/unknown/undefined)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>"
```
---
## Task 4: Add validation to handlePutRecord
**Files:**
- Modify: `packages/core/src/pds.js:915-1018`
**Step 1: Add validation logic to handlePutRecord**
Find the section after scope permissions check (around line 950) and add the same validation pattern. Update to include `_validate` in destructuring and add validation logic before encoding:
```javascript
const {
repo,
collection,
rkey,
record,
validate: _validate,
} = await request.json();
```
Then after scope check, before encoding:
```javascript
// Ensure $type is set
const recordWithType = { $type: collection, ...record };
// Validate record against schema if resolver provided
const shouldValidate = _validate !== false;
let validationStatus;
if (shouldValidate && this.lexiconResolver) {
try {
validationStatus = await this.lexiconResolver.assertValid(
collection,
recordWithType,
);
} catch (err) {
return Response.json(
{
error: 'InvalidRecord',
message: `Invalid ${collection} record: ${err.message}`,
},
{ status: 400 },
);
}
} else if (shouldValidate && !this.lexiconResolver) {
validationStatus = 'unknown';
}
```
**Step 2: Update response to include validationStatus**
Add `validationStatus` to the response object.
**Step 3: Run tests**
```bash
cd /Users/chadmiller/code/pds-experiment && pnpm test
```
**Step 4: Commit**
```bash
git add packages/core/src/pds.js
git commit -m "feat(core): add lexicon validation to putRecord
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>"
```
---
## Task 5: Add validation to handleApplyWrites
**Files:**
- Modify: `packages/core/src/pds.js:1021-1170`
**Step 1: Extract validate param from request body**
Add `validate` to the destructuring:
```javascript
const { repo, writes, validate, swapCommit: _swapCommit } = await request.json();
```
**Step 2: Add validation inside create case**
For create case (around line 1070), add validation before storing:
```javascript
if ($type === 'com.atproto.repo.applyWrites#create') {
const { collection, rkey: providedRkey, value } = write;
// Validate if resolver provided and validate !== false
const shouldValidate = validate !== false;
let validationStatus;
const recordWithType = { $type: collection, ...value };
if (shouldValidate && this.lexiconResolver) {
try {
validationStatus = await this.lexiconResolver.assertValid(
collection,
recordWithType,
);
} catch (err) {
return Response.json(
{
error: 'InvalidRecord',
message: `Invalid ${collection} record: ${err.message}`,
},
{ status: 400 },
);
}
} else if (shouldValidate && !this.lexiconResolver) {
validationStatus = 'unknown';
}
// ... rest of create logic, using recordWithType for encoding
```
**Step 3: Add validation to update case**
Same pattern for the update case.
**Step 4: Add validationStatus to create/update results**
```javascript
results.push({
$type: 'com.atproto.repo.applyWrites#createResult',
uri,
cid,
validationStatus,
});
```
**Step 5: Run tests**
```bash
cd /Users/chadmiller/code/pds-experiment && pnpm test
```
**Step 6: Commit**
```bash
git add packages/core/src/pds.js
git commit -m "feat(core): add lexicon validation to applyWrites
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>"
```
---
## Task 6: Create lexicon-resolver package
**Files:**
- Create: `packages/lexicon-resolver/package.json`
- Create: `packages/lexicon-resolver/src/index.js`
- Create: `packages/lexicon-resolver/src/resolver.js`
**Step 1: Create package.json**
```json
{
"name": "@pdsjs/lexicon-resolver",
"version": "0.1.0",
"type": "module",
"main": "src/index.js",
"exports": {
".": "./src/index.js",
"./resolver": "./src/resolver.js"
},
"dependencies": {
"@atproto/lexicon": "^0.4.0",
"@atproto/lex-resolver": "^0.1.0"
},
"devDependencies": {
"vitest": "^2.1.8"
}
}
```
**Step 2: Create src/index.js**
```javascript
export { LexiconResolver } from './resolver.js';
```
**Step 3: Create src/resolver.js**
```javascript
import { Lexicons } from '@atproto/lexicon';
import { LexResolver } from '@atproto/lex-resolver';
/**
* @typedef {import('@pdsjs/core/ports').ValidationStatus} ValidationStatus
*/
/**
* Lexicon resolver using @atproto/lex-resolver for schema resolution
* and @atproto/lexicon for validation.
* Implements LexiconResolverPort.
*/
export class LexiconResolver {
/**
* @param {Object} [opts]
* @param {number} [opts.ttl=3600000] - Cache TTL in ms (default 1 hour)
* @param {typeof fetch} [opts.fetch] - Fetch implementation
* @param {string} [opts.plcDirectoryUrl] - PLC directory URL
* @param {Array<object>} [opts.schemas] - Pre-loaded static schemas (LexiconDoc[])
*/
constructor(opts = {}) {
this.lexicons = new Lexicons();
this.resolver = new LexResolver({
fetch: opts.fetch,
plcDirectoryUrl: opts.plcDirectoryUrl,
});
this.ttl = opts.ttl ?? 3600000;
this.cache = new Map(); // nsid -> { fetchedAt }
// Pre-load static schemas
for (const schema of opts.schemas ?? []) {
try {
this.lexicons.add(schema);
this.cache.set(schema.id, { fetchedAt: Date.now() });
} catch {
// Schema already added or invalid, skip
}
}
}
/**
* Validate a record against its schema.
* @param {string} collection - The collection/NSID (e.g., 'app.bsky.feed.post')
* @param {object} record - The record to validate
* @returns {Promise<ValidationStatus>}
*/
async assertValid(collection, record) {
// Ensure schema is loaded
const hasSchema = await this.ensureSchema(collection);
if (!hasSchema) {
return 'unknown';
}
// Validate using @atproto/lexicon
this.lexicons.assertValidRecord(collection, record);
return 'valid';
}
/**
* Ensure a schema is loaded, fetching if necessary.
* @param {string} nsid
* @returns {Promise<boolean>} - true if schema is available
*/
async ensureSchema(nsid) {
// Check if already loaded and not stale
const cached = this.cache.get(nsid);
if (cached && Date.now() - cached.fetchedAt < this.ttl) {
return this.lexicons.get(nsid) !== undefined;
}
// Already have it but checking staleness
if (this.lexicons.get(nsid)) {
return true;
}
// Try to resolve
try {
const result = await this.resolver.get(nsid);
if (result?.lexicon) {
this.lexicons.add(result.lexicon);
this.cache.set(nsid, { fetchedAt: Date.now() });
return true;
}
} catch {
// Resolution failed
}
return false;
}
}
```
**Step 4: Commit**
```bash
git add packages/lexicon-resolver/
git commit -m "feat(lexicon-resolver): create package with @atproto/lexicon + lex-resolver
- LexiconResolver class implementing LexiconResolverPort
- Uses @atproto/lex-resolver for DNS + PDS schema fetching
- Uses @atproto/lexicon for validation
- In-memory caching with TTL
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>"
```
---
## Task 7: Add tests for LexiconResolver
**Files:**
- Create: `packages/lexicon-resolver/test/resolver.test.js`
**Step 1: Create test file**
```javascript
import { describe, expect, test, vi } from 'vitest';
import { LexiconResolver } from '../src/resolver.js';
// Sample lexicon schema for testing
const postSchema = {
lexicon: 1,
id: 'app.bsky.feed.post',
defs: {
main: {
type: 'record',
key: 'tid',
record: {
type: 'object',
required: ['text', 'createdAt'],
properties: {
text: { type: 'string', maxLength: 3000 },
createdAt: { type: 'string', format: 'datetime' },
},
},
},
},
};
describe('LexiconResolver', () => {
describe('with static schemas', () => {
test('validates record against pre-loaded schema', async () => {
const resolver = new LexiconResolver({
schemas: [postSchema],
});
const status = await resolver.assertValid('app.bsky.feed.post', {
$type: 'app.bsky.feed.post',
text: 'Hello world',
createdAt: new Date().toISOString(),
});
expect(status).toBe('valid');
});
test('throws on invalid record', async () => {
const resolver = new LexiconResolver({
schemas: [postSchema],
});
await expect(
resolver.assertValid('app.bsky.feed.post', {
$type: 'app.bsky.feed.post',
// missing required fields
})
).rejects.toThrow();
});
test('returns unknown for unloaded schema', async () => {
const resolver = new LexiconResolver({
schemas: [postSchema],
});
const status = await resolver.assertValid('com.example.unknown', {
$type: 'com.example.unknown',
});
expect(status).toBe('unknown');
});
});
describe('ensureSchema', () => {
test('returns true for pre-loaded schema', async () => {
const resolver = new LexiconResolver({
schemas: [postSchema],
});
const result = await resolver.ensureSchema('app.bsky.feed.post');
expect(result).toBe(true);
});
test('returns false for unknown schema when resolution fails', async () => {
const resolver = new LexiconResolver();
// Mock the resolver to fail
resolver.resolver.get = vi.fn().mockRejectedValue(new Error('Not found'));
const result = await resolver.ensureSchema('com.example.unknown');
expect(result).toBe(false);
});
});
});
```
**Step 2: Run tests**
```bash
cd /Users/chadmiller/code/pds-experiment && pnpm test packages/lexicon-resolver/test/resolver.test.js
```
**Step 3: Commit**
```bash
git add packages/lexicon-resolver/test/
git commit -m "test(lexicon-resolver): add unit tests
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>"
```
---
## Task 8: Install dependencies and update workspace
**Step 1: Install dependencies**
```bash
cd /Users/chadmiller/code/pds-experiment && pnpm install
```
**Step 2: Run full test suite**
```bash
cd /Users/chadmiller/code/pds-experiment && pnpm test
```
**Step 3: Run linter**
```bash
cd /Users/chadmiller/code/pds-experiment && pnpm lint
```
**Step 4: Fix any issues and commit**
```bash
git add pnpm-lock.yaml
git commit -m "chore: install lexicon-resolver dependencies
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>"
```
---
## Task 9: Update endpoint comparison docs
**Files:**
- Modify: `docs/endpoint-comparison.md`
**Step 1: Update the missing parameters table**
Change from:
```markdown
| repo.createRecord | collection, record, rkey | **repo**, **validate**, swapCommit |
| repo.putRecord | collection, rkey, record | **repo**, **validate**, swapCommit, swapRecord |
| repo.applyWrites | writes | **repo**, validate, swapCommit |
```
To:
```markdown
| repo.createRecord | collection, record, rkey, **validate** | **repo**, swapCommit |
| repo.putRecord | collection, rkey, record, **validate** | **repo**, swapCommit, swapRecord |
| repo.applyWrites | writes, **validate** | **repo**, swapCommit |
```
**Step 2: Commit**
```bash
git add docs/endpoint-comparison.md
git commit -m "docs: update endpoint comparison with validate param support
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>"
```
---
## Summary
After completing all tasks:
1. **@pdsjs/core** has:
- `LexiconResolverPort` typedef in `ports.js`
- Optional `lexiconResolver` in PDS constructor
- `validate` param on createRecord, putRecord, applyWrites
- `validationStatus` in responses
2. **@pdsjs/lexicon-resolver** provides:
- `LexiconResolver` class implementing the port
- Uses `@atproto/lex-resolver` for DNS + PDS schema fetching
- Uses `@atproto/lexicon` for validation
- In-memory caching with TTL
- Pre-loadable static schemas
3. **Usage:**
```javascript
import { PersonalDataServer } from '@pdsjs/core';
import { LexiconResolver } from '@pdsjs/lexicon-resolver';
const pds = new PersonalDataServer({
// ... other config
lexiconResolver: new LexiconResolver({
schemas: [/* optional static schemas */],
}),
});
```