chadtmiller.com/pds.js
An atproto PDS in JavaScript, for Cloudflare Workers and Node
0
0
# @pdsjs/git-ci
Continuous integration for git repositories hosted in an atproto account by
[`@pdsjs/git`](../git). A daemon watches the PDS firehose, runs a workflow for
every push, and publishes the result as a record in its own repo.
Nothing about the trigger, the checkout, or the result needs a forge. The
repository is a record, so the firehose already announces every push. The PDS
serves the repository over read-only smart HTTP, so the checkout is a plain
`git clone`. The result is a record signed by the runner's DID, so a reader
verifies it the same way it verifies any other atproto record.
## How it works
- One `com.atproto.sync.subscribeRepos` connection per PDS. A commit carries
its own blocks, so a `dev.pdsjs.git.repo` write arrives whole and no read
back to the PDS is needed to see what changed.
- The same connection carries `dev.pdsjs.git.checkRequest`, which is how a
repository's owner asks for a run.
- The record lists every ref with its object id. The daemon compares that list
against the one it last saw and starts a run for each ref that moved.
- The run clones from `https://<pds-host>/git/<did>/<repo>`, detaches at the
commit, and reads the workflows under `.pdsjs/workflows/` from the tree.
- Steps run through an `Executor`. The shipped one runs them in a shell.
Each workflow that names the ref runs, and each publishes its own check.
- The run publishes a `dev.pdsjs.git.check` record as `running`, then swaps it
for the outcome with the logs attached as a blob.
- Each write also points a `dev.pdsjs.git.latestCheck` record at the check, so
a reader finds the current state of one ref without a listing.
## Reading a check
Checks live in the runner's repo, not the repository owner's, and atproto has
no reverse index: a check names its subject, and nothing lets the subject ask
what names it. Two records close that gap.
Name the runner in the repository's `dev.pdsjs.git.config` record, beside the
branch protection rules under the same rkey, which is the repository's own:
```json
{ "runner": "did:plc:therunner" }
```
A reader resolves that DID to its PDS and asks for one record. The key is
derived from the repository's own record key and the ref, so nothing has to be
listed or filtered, and a rename leaves every check where it is:
```js
import { latestCheckKey, GIT_LATEST_CHECK_COLLECTION } from '@pdsjs/git-ci';
const rkey = latestCheckKey(ownerDid, repoRkey, 'refs/heads/main');
// -> did:plc:owner:3mup7vnsuqc3f:refs_heads_main
const res = await fetch(
`${runnerPds}/xrpc/com.atproto.repo.getRecord` +
`?repo=${runnerDid}&collection=${GIT_LATEST_CHECK_COLLECTION}&rkey=${rkey}`,
);
```
The record carries one entry per workflow, each with the status, the commit,
and a strongref to the full check. Checks are public and pds.js serves XRPC
reads with
`Access-Control-Allow-Origin: *`, so a browser reads this directly with no
proxy and no credentials.
The TID-keyed `dev.pdsjs.git.check` collection holds the history. Page it when
someone opens a repository's check list; the pointer answers everything else.
pds.js does this for you on the account page. Set the runner under a
repository's settings, and `GET /xrpc/dev.pdsjs.git.listChecks?repo=<repo>` returns
the latest check on each branch, with the server resolving the runner's DID.
Trust rests on the DID. A check is signed by the runner's repo, so a reader
that honors only the runner named in the config ignores a record any other
account publishes about the same commit.
## Asking for a run
The daemon listens on no port. It reads the firehose, so the channel that
already reaches it is a record, and a request is one: the repository's owner
writes `dev.pdsjs.git.checkRequest` into their own repo.
```js
import { checkRequestKey, GIT_CHECK_REQUEST_COLLECTION } from '@pdsjs/git-ci';
await agent.com.atproto.repo.putRecord({
repo: ownerDid,
collection: GIT_CHECK_REQUEST_COLLECTION,
rkey: checkRequestKey(repoRkey, 'refs/heads/main'),
record: {
$type: GIT_CHECK_REQUEST_COLLECTION,
repo: repoRkey,
ref: 'refs/heads/main',
requestedAt: new Date().toISOString(),
},
});
```
The record's authority is the authorisation. A request is in the owner's own
repo, so the PDS has already authenticated the write and the firehose event is
signed; the daemon runs a request only for a repository it watches whose owner
is the account that wrote it. Nothing else can name that repository here.
A request names a repository by its record key, and the daemon watches by
name, so it reads the record to learn which repository the key belongs to. A
request from an account it watches nothing of is answered before that read.
`sha` is optional. Without it the run takes whatever the ref points at when
the daemon reads the repository record, which is one `getRecord` at the moment
the request arrives. With it the run takes that commit, which is how a flaky
result gets a second opinion.
`requestedAt` is what makes a second request a second run. The record key is
derived from the repository's own record key and the ref, so asking again
replaces the record
rather than leaving one per click, and the daemon tells a fresh request from a
replayed one by the record's version. A request whose version it has already
acted on starts no run.
pds.js does this for you. `POST /xrpc/dev.pdsjs.git.requestCheck` with `{name, ref}`
writes the record, and the repository page has a "Run again" button beside the
check.
## What a check names
A check record carries the full ref name it ran against, and its subject names
the repository record in the copy it read. The account in that subject and the
ref are what identify a pull request, so a rebase or a force-push leaves the
run attached to the work rather than to a sha nothing points at.
## Usage
```sh
npm install -g @pdsjs/git-ci
export PDSJS_CI_IDENTIFIER=ci.example.com
export PDSJS_CI_PASSWORD=<app password>
pdsjs-git-ci watch alice.example.com/my-project alice.example.com/other
```
Node 22 or later, for the global `WebSocket` the firehose connection uses. The
daemon also calls `git`, so the host needs it on the PATH.
The PDS hosting the repositories needs the smart HTTP endpoint enabled:
`experimental: { git: { http: true } }` in the Node adapter, or
`PDS_EXPERIMENTAL_GIT_HTTP = "true"` on Cloudflare.
Give the runner its own account. Its DID is the check records' authority, and
an app password is enough: it writes only to its own repo and reads the
repositories under test anonymously.
### Workflow definitions
One file per workflow under `.pdsjs/workflows/`, at the commit under test,
named `*.yml`, `*.yaml` or `*.json`:
```yaml
name: ci
refs: [refs/heads/main]
env:
NODE_ENV: test
steps:
- name: install
run: pnpm install --frozen-lockfile
- name: check
# A block scalar carries a script as written. The same step in JSON is
# one line of escaped newlines and escaped quotes.
run: |
set -e
npm run ci
```
The parser is YAML 1.2, which subsumes JSON, so a `.json` workflow reads the
same as it always did. Quote a value that YAML would otherwise read as
something else: `on`, `no` and `2.4` are a boolean, a boolean and a number.
`name` defaults to the filename without its extension, and two workflows may
not share one: the name distinguishes their entries in the latest-check
record. `refs` defaults to `refs/heads/*`. A pattern matches exactly, or by a
single trailing `*`; `refs/tags/v*` runs a workflow on version tags. Steps
run in order and stop at the first non-zero exit. Each step sees `PDSJS_CI`,
`PDSJS_CI_REPO`, `PDSJS_CI_REF` and `PDSJS_CI_SHA` on top of the daemon's own
environment, minus the runner's credentials.
Every workflow that names the ref runs, in filename order, and each publishes
its own check record. A repository without the directory falls back to a
single workflow at `.pdsjs/ci.json`, so a commit from before the directory
convention still runs.
A repository with no workflows, or none naming a matching ref, produces no
record. Watching a repository before it opts in is harmless.
### Secrets
A step sees a secret only when both sides name it. The workflow declares the
names it needs, in the tree and in public:
```json
{ "secrets": ["NPM_TOKEN"] }
```
The runner's host grants values per repository, in the file `PDSJS_CI_SECRETS`
points at:
```json
{
"alice.example.com/my-project": { "NPM_TOKEN": "npm_..." }
}
```
The intersection reaches the step environment. A declared name with no grant
is absent, so a workflow guards with `${NPM_TOKEN:-}` and reports what is
missing in its own logs. A grant one repository holds never reaches another
repository's steps, and the runner's own `PDSJS_CI_PASSWORD` reaches no step
at all.
The names are public in the workflow file; the values exist only on the
runner host. Grant a secret only to a repository whose collaborators you
trust with it: every step of that repository's workflow can read it.
## Environment variables
| Variable | Purpose | Default |
| --- | --- | --- |
| `PDSJS_CI_IDENTIFIER` | Runner account handle or DID | required |
| `PDSJS_CI_PASSWORD` | App password for that account | required |
| `PDSJS_CI_SECRETS` | Per-repository secrets file path | none |
| `PDSJS_CI_SERVICE` | Runner's PDS base URL, skips identity resolution | resolved from the identifier |
| `PDSJS_CI_STATE` | State file path | `$XDG_STATE_HOME/pdsjs-git-ci/state.json` |
| `PDSJS_CI_STEP_TIMEOUT` | Per-step timeout in milliseconds | 30 minutes |
`ATPROTO_GIT_SERVICE` and `ATPROTO_GIT_PLC_URL` resolve the watched
repositories, the same as they do for the remote helper.
## Executors
The daemon holds an `Executor` and knows nothing else about it:
```js
/** @typedef {(request: ExecutionRequest) => Promise<StepResult[]>} Executor */
```
A request names a working tree, the steps, the environment, and a log sink. The
shipped `createShellExecutor` runs each step through the host's shell, which is
what makes any workflow work today: the step is a command, and the machine has
whatever the machine has.
That is also the interface's point. A container or isolate backend replaces it
without touching the trigger, the checkout, or the publishing path. An isolate
backend is the interesting one and the more limited one, since it can run only
JavaScript and Wasm, but it can run a check per push for a fraction of the
cost.
## Restarts
The state file holds three things: the firehose sequence reached on each PDS,
the object id last seen on every ref, and the version of every check request
acted on. All advance after a run finishes, so a daemon killed mid-run replays
the event and runs it again. A ref that has not moved starts no run however
many times its event replays, and neither does a request already acted on.
Rewinding the cursor by hand replays every event after it. Refs that have not
moved since start no run, but a ref that moved twice runs again at the commit
it passed through, and a request made in that window is acted on again.
## Limits
- Public repositories only. A repository in a space needs the credential chain
the remote helper implements, and the runner is a third party to the space.
- **Steps are not sandboxed.** `createShellExecutor` runs each step directly on
the host, as the daemon's user. The runner's credentials and the secrets
file stay out of the step environment, but a step still shares the host with
the daemon: it can read any file the daemon's user can, including the state
file and the secrets file itself. Run the daemon on a host you are willing
to lose, and watch only repositories whose collaborators you trust. A step
that needs a boundary can ask for one itself, since a step is a shell
command: `docker run --rm -v "$PWD:/w" -w /w node:22 npm test`. A container
executor is the standing fix, and the `Executor` interface is where it goes.
- No merge gate. Branch protection enforces at the moment of the push and a
check finishes after it, so a check can report a failure but not prevent one.
This is the same limit that rules out required reviews in `@pdsjs/git`.
- One run at a time per daemon. Events are handled in order so a run cannot
publish out of order with the push that caused it.
## Stability
An experiment, like `@pdsjs/git`. The package versions at 0.x and the
`dev.pdsjs.git.check` and `dev.pdsjs.git.checkRequest` lexicons are
unpublished, so consumers see `validationStatus: 'unknown'` and know there is
no contract yet.