MinIO S3 E2E Testing Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Add MinIO to docker-compose and enable E2E testing of the @pdsjs/blobs-s3 package via BLOB_STORAGE=s3 environment variable.

Architecture: MinIO runs alongside existing docker services. The @pdsjs/node package accepts an optional blobs adapter parameter. Test helper configures S3 when BLOB_STORAGE=s3 is set, otherwise uses filesystem.

Tech Stack: MinIO (S3-compatible storage), @aws-sdk/client-s3, @pdsjs/blobs-s3

---

Task 1: Add MinIO to Docker Compose

Files:

• Modify: docker-compose.yml

Step 1: Add MinIO service

Add after the postgres service:

  minio:
    image: minio/minio:latest
    ports:
      - "9000:9000"
      - "9001:9001"
    environment:
      - MINIO_ROOT_USER=minioadmin
      - MINIO_ROOT_PASSWORD=minioadmin
    command: server /data --console-address ":9001"
    volumes:
      - minio_data:/data
    healthcheck:
      test: ["CMD", "mc", "ready", "local"]
      interval: 2s
      timeout: 5s
      retries: 10

Step 2: Add minio_data volume

Add to volumes: section:

  minio_data:

Step 3: Verify MinIO starts

Run: docker compose up minio -d && docker compose logs -f minio Expected: "API: http://0.0.0.0:9000" and "Console: http://0.0.0.0:9001"

Step 4: Commit

git add docker-compose.yml
git commit -m "feat: add MinIO to docker-compose for S3 testing"

---

Task 2: Add AWS SDK Dev Dependency

Files:

• Modify: package.json

Step 1: Install @aws-sdk/client-s3

Run: pnpm add -D @aws-sdk/client-s3 Expected: Package added to devDependencies

Step 2: Commit

git add package.json pnpm-lock.yaml
git commit -m "chore: add @aws-sdk/client-s3 dev dependency"

---

Task 3: Update @pdsjs/node to Accept Blobs Adapter

Files:

• Modify: packages/node/index.js
• Modify: packages/node/index.d.ts

Step 1: Update createServer to accept blobs parameter

In packages/node/index.js, change the function signature and body:

export async function createServer({
  dbPath,
  blobsDir,
  blobs: blobsArg,
  jwtSecret,
  port = 3000,
  hostname,
  appviewUrl,
  appviewDid,
  relayUrl,
  password,
}) {

Then change line 106 from:

  const blobs = createFsBlobs(blobsDir);

To:

  const blobs = blobsArg ?? createFsBlobs(blobsDir);

Step 2: Update type definitions

In packages/node/index.d.ts, update ServerOptions interface to add blobs parameter:

export function createServer(options: {
  dbPath: string;
  blobsDir?: string;
  blobs?: import('@pdsjs/core/ports').BlobPort;
  jwtSecret: string;
  port?: number;
  hostname?: string;
  appviewUrl?: string;
  appviewDid?: string;
  relayUrl?: string;
  password?: string;
}): Promise<{
  pds: import('@pdsjs/core').PersonalDataServer;
  server: import('node:http').Server;
  db: unknown;
  listen: () => Promise<import('node:http').Server>;
  close: () => Promise<void>;
}>;

Step 3: Verify existing tests still pass

Run: PLATFORM=node npm test -- --run Expected: All tests pass (no S3 configured, uses blobsDir fallback)

Step 4: Commit

git add packages/node/index.js packages/node/index.d.ts
git commit -m "feat(node): accept optional blobs adapter parameter"

---

Task 4: Update Docker Services Helper for MinIO

Files:

• Modify: test/helpers/docker-services.js

Step 1: Add MinIO constants and readiness check

Add after line 4 (const PLC_URL = ...):

const MINIO_URL = 'http://localhost:9000';
const USE_S3 = process.env.BLOB_STORAGE === 's3';

Step 2: Add MinIO wait in ensureDockerServices

Add after await waitForService(RELAY_URL, 'Relay'); (around line 42):

  if (USE_S3) {
    await waitForService(`${MINIO_URL}/minio/health/ready`, 'MinIO');
  }

Step 3: Export USE_S3

Update the export at the end of the file:

export { RELAY_URL, PLC_URL, MINIO_URL, USE_S3 };

Step 4: Commit

git add test/helpers/docker-services.js
git commit -m "feat(test): add MinIO readiness check when BLOB_STORAGE=s3"

---

Task 5: Update Node Server Helper for S3 Blobs

Files:

• Modify: test/helpers/node-server.js

Step 1: Add USE_S3 import and S3 configuration

Replace the entire file with:

// test/helpers/node-server.js
import { mkdirSync, rmSync } from 'node:fs';
import { createServer } from '@pdsjs/node';

const TEST_DATA_DIR = './test-data';
const TEST_PORT = 3000;
const USE_LOCAL_INFRA = process.env.USE_LOCAL_INFRA !== 'false';
const USE_S3 = process.env.BLOB_STORAGE === 's3';

/**
 * Create S3 blobs adapter for MinIO
 * @returns {Promise<import('@pdsjs/core/ports').BlobPort>}
 */
async function createMinioBlobs() {
  const {
    S3Client,
    GetObjectCommand,
    PutObjectCommand,
    DeleteObjectCommand,
    CreateBucketCommand,
  } = await import('@aws-sdk/client-s3');
  const { createS3Blobs } = await import('@pdsjs/blobs-s3');

  const client = new S3Client({
    endpoint: 'http://localhost:9000',
    region: 'us-east-1',
    credentials: {
      accessKeyId: 'minioadmin',
      secretAccessKey: 'minioadmin',
    },
    forcePathStyle: true,
  });

  // Ensure bucket exists
  try {
    await client.send(new CreateBucketCommand({ Bucket: 'pds-blobs' }));
    console.log('Created S3 bucket: pds-blobs');
  } catch (e) {
    if (e.name !== 'BucketAlreadyOwnedByYou' && e.name !== 'BucketAlreadyExists') {
      throw e;
    }
  }

  return createS3Blobs({
    client,
    bucket: 'pds-blobs',
    GetObjectCommand,
    PutObjectCommand,
    DeleteObjectCommand,
  });
}

/**
 * Start Node.js PDS server for e2e tests
 * Cleans test data directory for fresh state each run
 * @returns {Promise<{close: () => Promise<void>}>} Server instance with close() method
 */
export async function startNodeServer() {
  // Fresh data each run
  rmSync(TEST_DATA_DIR, { recursive: true, force: true });
  mkdirSync(TEST_DATA_DIR, { recursive: true });

  // Configure blobs adapter
  const blobs = USE_S3 ? await createMinioBlobs() : undefined;
  if (USE_S3) {
    console.log('Using S3 blob storage (MinIO)');
  }

  const server = await createServer({
    port: TEST_PORT,
    dbPath: `${TEST_DATA_DIR}/pds.db`,
    blobsDir: `${TEST_DATA_DIR}/blobs`,
    blobs,
    jwtSecret: 'test-secret-for-e2e',
    // Use HTTPS hostname (via Caddy proxy) when docker infra is enabled
    hostname: USE_LOCAL_INFRA
      ? 'host.docker.internal:3443'
      : `localhost:${TEST_PORT}`,
    password: 'test-password',
    // Use local relay when docker infrastructure is available
    relayUrl: USE_LOCAL_INFRA ? 'http://localhost:2470' : undefined,
    // Keep appview pointing to production (for proxy tests)
    appviewUrl: 'https://api.bsky.app',
    appviewDid: 'did:web:api.bsky.app',
  });

  await server.listen();
  return server;
}

/**
 * Stop Node.js PDS server
 * @param {{close: () => Promise<void>}} server - Server instance from startNodeServer
 */
export async function stopNodeServer(server) {
  await server.close();
}

export { USE_LOCAL_INFRA, TEST_PORT, USE_S3 };

Step 2: Verify tests pass with filesystem

Run: docker compose up -d && PLATFORM=node USE_LOCAL_INFRA=true npm test -- --run Expected: All tests pass using filesystem blobs

Step 3: Commit

git add test/helpers/node-server.js
git commit -m "feat(test): configure S3 blobs when BLOB_STORAGE=s3"

---

Task 6: End-to-End Verification

Step 1: Start all docker services

Run: docker compose down -v && docker compose up -d Expected: All services start including MinIO

Step 2: Run tests with S3 backend

Run: PLATFORM=node USE_LOCAL_INFRA=true BLOB_STORAGE=s3 npm test -- --run Expected: All tests pass, console shows "Using S3 blob storage (MinIO)"

Step 3: Verify blobs in MinIO console (optional)

Open: http://localhost:9001 (login: minioadmin/minioadmin) Expected: pds-blobs bucket exists with blob files after test run

Step 4: Final commit (if any cleanup needed)

git status
# If clean, no commit needed

---

Summary

After completing all tasks:

• docker compose up -d starts MinIO alongside other services
• PLATFORM=node npm test runs with filesystem blobs (default)
• PLATFORM=node BLOB_STORAGE=s3 npm test runs with S3 blobs via MinIO
• Existing tests cover blob operations, validating the S3 adapter works E2E