chadtmiller.com/pds.js
An atproto PDS in JavaScript, for Cloudflare Workers and Node
0
0
# MinIO S3 E2E Testing Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
**Goal:** Add MinIO to docker-compose and enable E2E testing of the `@pdsjs/blobs-s3` package via `BLOB_STORAGE=s3` environment variable.
**Architecture:** MinIO runs alongside existing docker services. The `@pdsjs/node` package accepts an optional `blobs` adapter parameter. Test helper configures S3 when `BLOB_STORAGE=s3` is set, otherwise uses filesystem.
**Tech Stack:** MinIO (S3-compatible storage), @aws-sdk/client-s3, @pdsjs/blobs-s3
---
### Task 1: Add MinIO to Docker Compose
**Files:**
- Modify: `docker-compose.yml`
**Step 1: Add MinIO service**
Add after the `postgres` service:
```yaml
minio:
image: minio/minio:latest
ports:
- "9000:9000"
- "9001:9001"
environment:
- MINIO_ROOT_USER=minioadmin
- MINIO_ROOT_PASSWORD=minioadmin
command: server /data --console-address ":9001"
volumes:
- minio_data:/data
healthcheck:
test: ["CMD", "mc", "ready", "local"]
interval: 2s
timeout: 5s
retries: 10
```
**Step 2: Add minio_data volume**
Add to `volumes:` section:
```yaml
minio_data:
```
**Step 3: Verify MinIO starts**
Run: `docker compose up minio -d && docker compose logs -f minio`
Expected: "API: http://0.0.0.0:9000" and "Console: http://0.0.0.0:9001"
**Step 4: Commit**
```bash
git add docker-compose.yml
git commit -m "feat: add MinIO to docker-compose for S3 testing"
```
---
### Task 2: Add AWS SDK Dev Dependency
**Files:**
- Modify: `package.json`
**Step 1: Install @aws-sdk/client-s3**
Run: `pnpm add -D @aws-sdk/client-s3`
Expected: Package added to devDependencies
**Step 2: Commit**
```bash
git add package.json pnpm-lock.yaml
git commit -m "chore: add @aws-sdk/client-s3 dev dependency"
```
---
### Task 3: Update @pdsjs/node to Accept Blobs Adapter
**Files:**
- Modify: `packages/node/index.js`
- Modify: `packages/node/index.d.ts`
**Step 1: Update createServer to accept blobs parameter**
In `packages/node/index.js`, change the function signature and body:
```javascript
export async function createServer({
dbPath,
blobsDir,
blobs: blobsArg,
jwtSecret,
port = 3000,
hostname,
appviewUrl,
appviewDid,
relayUrl,
password,
}) {
```
Then change line 106 from:
```javascript
const blobs = createFsBlobs(blobsDir);
```
To:
```javascript
const blobs = blobsArg ?? createFsBlobs(blobsDir);
```
**Step 2: Update type definitions**
In `packages/node/index.d.ts`, update ServerOptions interface to add blobs parameter:
```typescript
export function createServer(options: {
dbPath: string;
blobsDir?: string;
blobs?: import('@pdsjs/core/ports').BlobPort;
jwtSecret: string;
port?: number;
hostname?: string;
appviewUrl?: string;
appviewDid?: string;
relayUrl?: string;
password?: string;
}): Promise<{
pds: import('@pdsjs/core').PersonalDataServer;
server: import('node:http').Server;
db: unknown;
listen: () => Promise<import('node:http').Server>;
close: () => Promise<void>;
}>;
```
**Step 3: Verify existing tests still pass**
Run: `PLATFORM=node npm test -- --run`
Expected: All tests pass (no S3 configured, uses blobsDir fallback)
**Step 4: Commit**
```bash
git add packages/node/index.js packages/node/index.d.ts
git commit -m "feat(node): accept optional blobs adapter parameter"
```
---
### Task 4: Update Docker Services Helper for MinIO
**Files:**
- Modify: `test/helpers/docker-services.js`
**Step 1: Add MinIO constants and readiness check**
Add after line 4 (`const PLC_URL = ...`):
```javascript
const MINIO_URL = 'http://localhost:9000';
const USE_S3 = process.env.BLOB_STORAGE === 's3';
```
**Step 2: Add MinIO wait in ensureDockerServices**
Add after `await waitForService(RELAY_URL, 'Relay');` (around line 42):
```javascript
if (USE_S3) {
await waitForService(`${MINIO_URL}/minio/health/ready`, 'MinIO');
}
```
**Step 3: Export USE_S3**
Update the export at the end of the file:
```javascript
export { RELAY_URL, PLC_URL, MINIO_URL, USE_S3 };
```
**Step 4: Commit**
```bash
git add test/helpers/docker-services.js
git commit -m "feat(test): add MinIO readiness check when BLOB_STORAGE=s3"
```
---
### Task 5: Update Node Server Helper for S3 Blobs
**Files:**
- Modify: `test/helpers/node-server.js`
**Step 1: Add USE_S3 import and S3 configuration**
Replace the entire file with:
```javascript
// test/helpers/node-server.js
import { mkdirSync, rmSync } from 'node:fs';
import { createServer } from '@pdsjs/node';
const TEST_DATA_DIR = './test-data';
const TEST_PORT = 3000;
const USE_LOCAL_INFRA = process.env.USE_LOCAL_INFRA !== 'false';
const USE_S3 = process.env.BLOB_STORAGE === 's3';
/**
* Create S3 blobs adapter for MinIO
* @returns {Promise<import('@pdsjs/core/ports').BlobPort>}
*/
async function createMinioBlobs() {
const {
S3Client,
GetObjectCommand,
PutObjectCommand,
DeleteObjectCommand,
CreateBucketCommand,
} = await import('@aws-sdk/client-s3');
const { createS3Blobs } = await import('@pdsjs/blobs-s3');
const client = new S3Client({
endpoint: 'http://localhost:9000',
region: 'us-east-1',
credentials: {
accessKeyId: 'minioadmin',
secretAccessKey: 'minioadmin',
},
forcePathStyle: true,
});
// Ensure bucket exists
try {
await client.send(new CreateBucketCommand({ Bucket: 'pds-blobs' }));
console.log('Created S3 bucket: pds-blobs');
} catch (e) {
if (e.name !== 'BucketAlreadyOwnedByYou' && e.name !== 'BucketAlreadyExists') {
throw e;
}
}
return createS3Blobs({
client,
bucket: 'pds-blobs',
GetObjectCommand,
PutObjectCommand,
DeleteObjectCommand,
});
}
/**
* Start Node.js PDS server for e2e tests
* Cleans test data directory for fresh state each run
* @returns {Promise<{close: () => Promise<void>}>} Server instance with close() method
*/
export async function startNodeServer() {
// Fresh data each run
rmSync(TEST_DATA_DIR, { recursive: true, force: true });
mkdirSync(TEST_DATA_DIR, { recursive: true });
// Configure blobs adapter
const blobs = USE_S3 ? await createMinioBlobs() : undefined;
if (USE_S3) {
console.log('Using S3 blob storage (MinIO)');
}
const server = await createServer({
port: TEST_PORT,
dbPath: `${TEST_DATA_DIR}/pds.db`,
blobsDir: `${TEST_DATA_DIR}/blobs`,
blobs,
jwtSecret: 'test-secret-for-e2e',
// Use HTTPS hostname (via Caddy proxy) when docker infra is enabled
hostname: USE_LOCAL_INFRA
? 'host.docker.internal:3443'
: `localhost:${TEST_PORT}`,
password: 'test-password',
// Use local relay when docker infrastructure is available
relayUrl: USE_LOCAL_INFRA ? 'http://localhost:2470' : undefined,
// Keep appview pointing to production (for proxy tests)
appviewUrl: 'https://api.bsky.app',
appviewDid: 'did:web:api.bsky.app',
});
await server.listen();
return server;
}
/**
* Stop Node.js PDS server
* @param {{close: () => Promise<void>}} server - Server instance from startNodeServer
*/
export async function stopNodeServer(server) {
await server.close();
}
export { USE_LOCAL_INFRA, TEST_PORT, USE_S3 };
```
**Step 2: Verify tests pass with filesystem**
Run: `docker compose up -d && PLATFORM=node USE_LOCAL_INFRA=true npm test -- --run`
Expected: All tests pass using filesystem blobs
**Step 3: Commit**
```bash
git add test/helpers/node-server.js
git commit -m "feat(test): configure S3 blobs when BLOB_STORAGE=s3"
```
---
### Task 6: End-to-End Verification
**Step 1: Start all docker services**
Run: `docker compose down -v && docker compose up -d`
Expected: All services start including MinIO
**Step 2: Run tests with S3 backend**
Run: `PLATFORM=node USE_LOCAL_INFRA=true BLOB_STORAGE=s3 npm test -- --run`
Expected: All tests pass, console shows "Using S3 blob storage (MinIO)"
**Step 3: Verify blobs in MinIO console (optional)**
Open: `http://localhost:9001` (login: minioadmin/minioadmin)
Expected: `pds-blobs` bucket exists with blob files after test run
**Step 4: Final commit (if any cleanup needed)**
```bash
git status
# If clean, no commit needed
```
---
## Summary
After completing all tasks:
- `docker compose up -d` starts MinIO alongside other services
- `PLATFORM=node npm test` runs with filesystem blobs (default)
- `PLATFORM=node BLOB_STORAGE=s3 npm test` runs with S3 blobs via MinIO
- Existing tests cover blob operations, validating the S3 adapter works E2E