# Operations What the server tells whoever runs it, and how fast it must be: its log, its health, its metrics and its traces, what its admin reads of them over HTTP, and the performance it is held to beside the reference. What the server serves, and how it is checked, is in [`PDS.md`](PDS.md). Of this, the reference shows only `/xrpc/_health`, and the harness holds the server to it. The reference's log is its own (pino's), and it serves no metrics or traces, so the rest is this server's design, following the ordinary conventions: a log of JSON lines, Prometheus' text format for metrics, and OpenTelemetry for traces (OTLP, W3C Trace Context, its HTTP semantic conventions and its standard `OTEL_*` variables). The harness holds the server to this page. ## Environment Besides those the other pages list: | Variable | Is | Default | | --- | --- | --- | | `PDS_LOG_LEVEL` | the least level of log line written: `debug`, `info`, `warn` or `error`; or `off`, for none | `info` | | `PDS_LOG_DIR` | a directory the log is kept in too, for the admin to read back ("Kept", below); made if missing; unset, the log is kept nowhere but standard output | unset | | `PDS_LOG_KEEP_BYTES` | the most bytes the files in `PDS_LOG_DIR` take together; at least `65536` | `67108864` (64 MiB) | | `PDS_METRICS_PORT` | the port `/metrics` is served on, on 127.0.0.1; unset, metrics are not served | unset | | `OTEL_EXPORTER_OTLP_ENDPOINT` | the base URL of the OTLP receiver traces go to: they are posted to it with `v1/traces` appended, after a `/` if it does not end in one | unset | | `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | the URL traces are posted to, as it is; before `OTEL_EXPORTER_OTLP_ENDPOINT` | unset | | `OTEL_EXPORTER_OTLP_PROTOCOL`, `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` | `http/protobuf`, the only protocol the server speaks | `http/protobuf` | | `OTEL_EXPORTER_OTLP_HEADERS`, `OTEL_EXPORTER_OTLP_TRACES_HEADERS` | headers sent with each export, `key=value` pairs joined by `,`, each value percent-decoded; the second's pairs win | none | | `OTEL_EXPORTER_OTLP_COMPRESSION`, `OTEL_EXPORTER_OTLP_TRACES_COMPRESSION` | `gzip` or `none` | `none` | | `OTEL_SERVICE_NAME` | the `service.name` of the server's spans | `unknown_service:pds` | | `OTEL_SDK_DISABLED` | `true` turns traces off, whatever else is set | `false` | | `OTEL_BSP_SCHEDULE_DELAY` | milliseconds between exports | `5000` | | `OTEL_BSP_MAX_QUEUE_SIZE` | spans held for export at most; past it, new spans are dropped | `2048` | | `OTEL_BSP_MAX_EXPORT_BATCH_SIZE` | spans in one export at most | `512` | | `OTEL_BSP_EXPORT_TIMEOUT` | milliseconds an export may take, retries and all, before its spans are dropped | `30000` | A value that does not parse, a level not listed, a protocol other than `http/protobuf` (as `grpc` or `http/json`), a `PDS_METRICS_PORT` equal to `PDS_PORT`, a `PDS_LOG_KEEP_BYTES` below `65536`, or a `PDS_LOG_DIR` that cannot be made or written to, makes `pds serve` exit 1 before it writes anything to standard output, with why on standard error, as `LIMITS.md` says of its variables. An empty variable is as if unset. ## The log The server writes its log to **standard output**: one JSON object per line, UTF-8, each ended by `\n`; and with `PDS_LOG_DIR` set, keeps each line in that directory too ("Kept", below). Standard error is for one thing only, why `pds serve` exits 1 (`COMMANDS.md`, `DURABILITY.md`, `LIMITS.md`): a server that runs writes nothing there. *Note:* standard output is the stream a container runtime or a supervisor collects; standard error is kept for the human reading why the server would not start. Each line is written whole, by one `write`, so lines of requests answered at once never mix. A line is not held in the process: once written, it survives the server being killed with SIGKILL. ### Every line | Field | Is | | --- | --- | | `time` | when the line was written, RFC 3339 in UTC with milliseconds: `2026-11-09T17:23:00.123Z` | | `level` | `debug`, `info`, `warn` or `error` | | `msg` | what happened, a fixed text: one of the events below, or a short text of the server's own that holds no value from a request | and the event's own fields. A field with no value is left out, never `null`. Lines below `PDS_LOG_LEVEL` are not written; the levels go `debug`, `info`, `warn`, `error`, and `off` writes none. ### Events | `msg` | `level` | Fields | When | | --- | --- | --- | --- | | `start` | `info` | `version`, `port`, `sync`, and `metrics_port` when it is set | the first line of each start, once the environment is read and before the data is | | `recovered` | `warn` | `dropped_bytes` | a start on a data directory the last server did not shut down: killed, crashed, or the machine lost power | | `request` | `info`, or `error` for a status of 500 or more | `method`, `status`, `ms`, `trace_id`, and `did` when known; of a `getBlob`, `repo`, `cid` and `bytes` | each request answered | | `plc failure` | `warn` | `error`, `status` when the directory answered, and `did` when the call concerns one | each call to the PLC directory that fails: no answer, a timeout, or a status other than 2xx | | `blobstore failure` | `warn` | `op`: `put`, `get` or `delete`; `error`, `status` when the store answered, and `did` | each request to the bucket that fails (`BLOBS.md`): no answer, a timeout, or a status other than 2xx | | `trace export failed` | `warn` | `error` | an export whose spans are dropped (Export, below), at most one line a minute while exports keep failing | | `store failure` | `error` | `error` | what the server does to its data in the background and could not (`STORE.md`, `DURABILITY.md`): a compaction; a checkpoint of the journal; or a sync of the journal, after which it answers no write until it is started again | | `shutdown` | `info` | `signal`: `SIGTERM` or `SIGINT` | the last line of a server shut down | - **`start`**: `version` is the server's version, as `/xrpc/_health` answers it; `port` and `metrics_port` are numbers; `sync` is `together` or `each`, how waiting writes are made durable (`DURABILITY.md`, `PDS_SYNC`). - **`recovered`**: `dropped_bytes` is a number, the bytes of writes cut short that recovery dropped from the end of the logs the server keeps its state in (`DURABILITY.md`, `STORE.md`): a torn tail, and the logs of generations after one cut short or not sealed; 0 when it dropped none. A start on an empty directory, or on one the last server shut down, writes no `recovered`. The server may choose how it knows, as by a mark it makes on shutdown and takes away as it starts. - **`request`**: one line for each request the server answers, written once its answer is whole and before the last byte of it is sent, so that a client that has the whole answer knows the line is written. A response streamed, such as `getRepo`'s, has its line once its body is made. An upgrade to a WebSocket (`subscribeRepos`) is answered by its `101`: the line is written then, not when the subscription ends. A request whose client leaves before the answer is sent whole still has its line. - `method`: the request's route, as the metrics label it (below); - `status`: the HTTP status, a number; - `ms`: a number, the milliseconds from reading the request's head to writing the line, with any fraction; - `trace_id`: the request's trace, 32 lowercase hex digits (Traces, below), whether or not traces are exported; - `did`: the DID of the account whose credentials the request carried and the server verified, as a bearer token it gave out; left out otherwise; - `repo` and `cid`: of `com.atproto.sync.getBlob` alone, the DID of the account and the CID of the blob asked for, when the account holds that blob and the server answers with it, `200`, or failed to read it, `500` (`BLOBS.md`, "getBlob", "The objects"); left out of a `getBlob` answered otherwise, and of every other line. `did` stays the caller's, so a `getBlob` with no token has `repo` and no `did`; - `bytes`: with `repo` and `cid`, when the answer is `200`: a number, the bytes of the blob the server had handed to the connection by the time it wrote the line, which it writes as it hands on the last of them. That is the blob's size when it was sent whole, and fewer when the client left first or the store's read failed after its first byte; `0` for `HEAD`. It counts what the server handed on, not what reached the client: a client that left may have had fewer. - **`plc failure`**: `error` says why, as `timeout` or the status; `status` is a number. - **`shutdown`**: on SIGTERM or SIGINT the server stops taking connections, lets the requests it has finish, for up to 10 seconds, closes the firehose's subscriptions, syncs what it has written, exports the spans it holds, for up to 5 seconds, writes this line, and exits 0. Besides these the server may write lines of its own, as when a crawler does not answer or a write fails, at `warn` or `error`, with fields of their own, and at `debug` whatever helps. ### What is never logged No line holds, in any field, whole or in part: - a password, an account's or an app password; the admin password; - a token: an access or refresh JWT, a delegation token or space credential (`SPACES-CREDENTIALS.md`), an OAuth code or token, a DPoP proof, or any `Authorization` header; - an invite code, or a code sent by email, as an OAuth sign-in's `emailOtp`; - a cookie's value, as a device's or its session's id (`OAUTH.md`); - a key: the PLC rotation key, an account's signing or KEM key, the key the server signs its JWTs with; - the values of `OTEL_EXPORTER_OTLP_HEADERS`; - an account's email; - a request's body, query string or headers, or an answer's body; but for `getBlob`'s `repo` and `cid` (Events, above): the DID of an account here and the CID of a blob it holds, as public as the blob. `method` is a route the server serves, never the path asked, so nothing else of a URL is logged. ### Kept With `PDS_LOG_DIR` set, each line written to standard output is written to a file in `PDS_LOG_DIR` too, the same bytes, by one `write` of its own, just after its `write` to standard output and before the next line is written to either. So a line, once on standard output, is kept as well, and survives the server being killed with SIGKILL as it does, but for the one line a server killed between the two writes was writing: it is on standard output alone. Lines below `PDS_LOG_LEVEL` are written to neither. A write to the directory that fails is not tried again, and the server serves on: standard output is the log, and the files a copy of it to read back. The files in `PDS_LOG_DIR` take at most `PDS_LOG_KEEP_BYTES` bytes together, at every moment. Past that the oldest lines are let go first, as many at once as the server likes, but never one of the newest lines whose bytes, with their `\n`, add up to no more than half of `PDS_LOG_KEEP_BYTES`: those are always kept. A line whose bytes, with its `\n`, are more than half of it is not kept at all. A server started on a `PDS_LOG_DIR` that holds lines carries on after them, so the kept lines run across starts, oldest first. How the files are named and laid out is the server's own; only their total size, and the lines `_logs` reads back, are seen. One server writes to a `PDS_LOG_DIR` at a time. *Note:* `PDS_LOG_DIR` is apart from `PDS_DATA_DIR`: the lines are no state of the server's, no backup needs them, and the data directory's size is held to its bounds without them (`STORE.md`, "Bounds"). A container's standard output is lost with the container, as when a deploy makes it again; a volume of their own keeps the lines across it. ## Health `GET /xrpc/_health` answers as the reference does: `200` with `Content-Type: application/json; charset=utf-8` and ```json {"version": "0.0.0"} ``` `version` being the server's version, that of the `pds` package. Its query, if any, is ignored. `HEAD` answers the same status and headers with no body; `POST` is `400 InvalidRequest`. It is answered while the server serves, and is in no rate limit: past `global` (`LIMITS.md`) it is still `200`, with no `RateLimit-*` headers. Its route is `_health`. ## The root `GET /` says what the server is, as the reference's does, and how to connect an AI app to it (`COMMANDS.md`, `OAUTH-MCP.md`). It answers `200` with `Content-Type: text/plain; charset=utf-8` and this text, where `` is the origin of `PDS_PUBLIC_URL`, every line ending in `\n`, the first line empty, and none with spaces at its end: ```text __ __ /\ \__ /\ \__ __ \ \ ,_\ _____ _ __ ___\ \ ,_\ ___ /'__'\ \ \ \/ /\ '__'\/\''__\/ __'\ \ \/ / __'\ /\ \L\.\_\ \ \_\ \ \L\ \ \ \//\ \L\ \ \ \_/\ \L\ \ \ \__/.\_\\ \__\\ \ ,__/\ \_\\ \____/\ \__\ \____/ \/__/\/_/ \/__/ \ \ \/ \/_/ \/___/ \/__/\/___/ \ \_\ \/_/ This is an AT Protocol Personal Data Server (aka, an atproto PDS) Most API routes are under /xrpc/ Connect an AI app 1. Add this address to your AI app as a connector: /mcp In Claude: Settings > Connectors > Add custom connector, and choose "Sign in when needed". 2. When it asks, sign in here and approve it. 3. Ask it things like: What have I posted this week? Post "hello from Claude" to Bluesky Who have I followed lately? Set my Bluesky bio to "Software engineer in Seattle" Code: https://knotless.dev/chadtmiller.com/monolith Protocol: https://atproto.com ``` The banner is the reference's; the rest is this server's own, and only the status and `Content-Type` are compared. Its query, if any, is ignored. `HEAD` answers the same status and headers with no body; any other method is `404`, as the reference's. It is in no rate limit and carries no `RateLimit-*` headers, as the reference's does not. Its route is `/`. ## Metrics `GET /metrics` on `127.0.0.1:PDS_METRICS_PORT` answers the server's metrics, `200` with `Content-Type: text/plain; version=0.0.4; charset=utf-8`, in Prometheus' text exposition format, version 0.0.4. Any other request on that port is `404`. With `PDS_METRICS_PORT` unset nothing listens, and `/metrics` on the server's own port is `404` as any route it does not serve. It is a port of its own, with no auth. Being apart, a scrape is no request of the server's: it has no log line or span, is in no metric and in no rate limit. *Note:* the server's own port is the one put before the internet; a separate one, on the loopback, is never in front of it, so nothing there needs guarding, and a Prometheus scrape, which carries no credentials, needs none. Each family has a `# HELP` line and a `# TYPE` line, then its samples; every family below is there from the start, its series once they have a value. Counters and histograms count from the start of the process, so they start again from 0 when it does, as Prometheus expects. A histogram has a `_bucket` sample for each bound below, cumulative, then `le="+Inf"`, then `_sum` and `_count`. Durations are in seconds. | Metric | Type | Labels | Is | | --- | --- | --- | --- | | `pds_requests_total` | counter | `method`, `status` | requests answered: one for each `request` line | | `pds_request_duration_seconds` | histogram | `method` | each request's `ms`, in seconds | | `pds_rate_limited_total` | counter | `method` | requests refused by a rate limit (`429`) | | `pds_firehose_seq` | gauge | | the `seq` of the firehose's last event; 0 before any | | `pds_firehose_subscribers` | gauge | | `subscribeRepos` connections open | | `pds_accounts` | gauge | | the accounts the server holds, those `listRepos` lists | | `pds_log_bytes` | gauge | | the bytes of the logs the server keeps its state in, in `PDS_DATA_DIR`, all of them: every generation's not yet folded away, and the checkpoints (`STORE.md`) | | `pds_fsync_duration_seconds` | histogram | | each sync of written data to the disk, every one the server makes: a log's as a commit is answered and as it is opened, a sealed log's, a directory's, a table's, a blob's file, the firehose's index, its keys and marks, and the whole data directory's after a start that followed a kill | | `pds_outbound_requests_total` | counter | `target`, `outcome` | each request the server makes | | `pds_outbound_request_duration_seconds` | histogram | `target` | each request the server makes, until its answer or failure | `pds_log_bytes` grows by what each write puts in the logs, and falls only as the server lets go of what they held: a log's generations folded into its checkpoint (an account's, its table: `STORE.md`, "Bounds"), which a write may start and which the server may run in the background at any time; and the journal's records, once what they hold is durable where it is kept (`DURABILITY.md`). It lets go of nothing but as it makes, deletes, replaces or cuts short a file in `PDS_DATA_DIR`, and what it lets go of leaves the gauge as that file changes, neither before nor after: a read of the gauge begun once the change can be seen in the directory counts none of it, and one ended before the change counts all of it, however long the server takes over what it does next (a journal's segment deleted leaves the gauge as it goes, not once the directory is synced or the other segments are deleted). So across a write during which no file there was made, deleted, replaced or cut short, each only grown or left as it was, the gauge grows, by however little; across one during which one was, it may be less after than before. The buckets: - requests and outbound requests: `0.005`, `0.01`, `0.025`, `0.05`, `0.1`, `0.25`, `0.5`, `1`, `2.5`, `5`, `10` (Prometheus' own default); - syncs: `0.0001`, `0.00025`, `0.0005`, `0.001`, `0.0025`, `0.005`, `0.01`, `0.025`, `0.05`, `0.1`, `0.25`, `1`. The labels: - `method` is the request's route, the same as its log line's: - a method the server serves, by its NSID, for `/xrpc/`, whatever its HTTP method; - `_health`, `_logs` and `_stats`, for `/xrpc/_health` and the admin's reads (below), `_oauthSessions`, `_oauthRevokeSession`, `_oauthDevices` and `_oauthRevokeDevice`, for the account's OAuth routes (`OAUTH-ACCOUNT.md`), and `_passkeyList`, `_passkeyAdd` and `_passkeyRemove`, for its passkey routes (`PASSKEYS.md`); - a route outside `/xrpc/` the server serves, as its path: `/` ("The root"), `/mcp` (`COMMANDS.md`), `/.well-known/atproto-did` (`IDENTITY.md`), and OAuth's (`OAUTH.md`): `/.well-known/oauth-protected-resource`, `/.well-known/oauth-authorization-server`, `/oauth/jwks`, `/oauth/par`, `/oauth/token`, `/oauth/revoke`, `/oauth/authorize`, `/oauth/authorize/redirect`, and the authorization page's `/@atproto/oauth-provider/~api/sign-in`, `~api/consent`, `~api/reject`, `~api/sign-out`, `~api/passkey-options` and `~api/passkey-sign-in`, the passkey page `/oauth/passkey` and its `~api/passkey-register` (`PASSKEYS.md`), and `/oauth/assets/{name}` for every path under `/oauth/assets/`, whatever its name (`OAUTH-PAGES.md`); the MCP issuer's (`OAUTH-MCP.md`): `/.well-known/oauth-protected-resource/mcp`, `/.well-known/oauth-authorization-server/oauth/mcp`, `/oauth/mcp/authorize`, `/oauth/mcp/token` and `/oauth/mcp/revoke`; a page that adds such a route adds it to this list; - `_unknown` for anything else: an NSID the server does not serve, a path under `/xrpc/` that is no NSID, any other path. *Note:* so no label holds what a client chose, and the series are bounded by the routes served. - `status` is the HTTP status, in decimal: `"200"`. - `target` is where the request goes: `plc` for the PLC directory, `crawler` for a crawler (`FEDERATION.md`), `service` for the AppView and the services `atproto-proxy` names (`PROXY.md`), `did-web` for a `did:web` document, `client` for a client's metadata document or key set (`OAUTH-CLIENTS.md`, `SPACES-CREDENTIALS.md`), `space` for a space's notification to another service (`SPACES-SYNC.md`), `lexicon` for a permission set's record (`OAUTH-PERMISSION-SETS.md`), `handle` for a handle's `/.well-known/atproto-did` (`IDENTITY.md`), and `blobstore` for the bucket blobs are kept in (`BLOBS.md`); `outcome` is `ok` for a 2xx answer and `error` for anything else, no answer included. ## The admin's reads The admin, whoever has `PDS_ADMIN_PASSWORD`, reads what the server logged and counted over the server's own port, without reaching its machine: `GET /xrpc/_logs` and `GET /xrpc/_stats`. They are the server's own, as `/xrpc/_health` is, of no lexicon and no NSID: so they are listed under no page of `PDS.md`, and their MCP tools and CLI commands are given in `COMMANDS.md`, "The admin's reads". Each takes the admin's `Authorization: Basic` as `LIMITS.md`'s "The admin" says, and checks it first: anything else is `401 AuthenticationRequired`. Then a method other than `GET` is `400 InvalidRequest`; then a parameter that does not parse, or is out of its range, is `400 InvalidRequest`. A parameter neither takes is ignored. Each answers `200` with `Content-Type: application/json; charset=utf-8`, costs a point of `global` (`LIMITS.md`), and has its route, `_logs` or `_stats`, in its `request` line, its metrics and its span, as any request. ### _logs The lines kept in `PDS_LOG_DIR` ("Kept", above), oldest first: ```json {"lines": [{"time": "2026-11-09T17:23:00.123Z", "level": "info", "msg": "request", "method": "com.atproto.repo.createRecord", "status": 200, "ms": 3.2, "trace_id": "…", "did": "did:plc:…"}]} ``` `lines` holds the last `limit` of the lines kept that match every parameter given, each the JSON object the line is, as written. They are the lines kept when the read began: its own `request` line is written once its answer is made, so it is never among them, and is among those of the next read. A line in the files that is not a JSON object, as one cut short by a lost power, is skipped. | Parameter | Matches a line | | --- | --- | | `trace` | whose `trace_id` is this | | `msg` | whose `msg` is this | | `method` | whose `method` is this: a route, as `com.atproto.repo.createRecord` or `/oauth/token` | | `level` | at this level or above: `debug`, `info`, `warn` or `error`; any other value is `400` | | `minMs` | with an `ms` of this many milliseconds or more, an integer of 0 or more; a line with no `ms` does not match | | `limit` | how many, the last of those matched: an integer from 1 to 500, 100 if not given | With `PDS_LOG_DIR` unset, `_logs` is `404 LogNotKept`, once the auth and the parameters are checked. ### _stats What the server counted since it started, as its metrics count it (above), whether or not `PDS_METRICS_PORT` is set: ```json { "version": "0.1.0", "started": "2026-11-09T17:20:01.004Z", "accounts": 3, "logBytes": 1048576, "firehose": {"seq": 120, "subscribers": 1}, "routes": [ {"method": "com.atproto.repo.createRecord", "count": 40, "statuses": {"200": 38, "400": 2}, "rateLimited": 0, "meanMs": 3.1} ], "outbound": [{"target": "plc", "ok": 3, "error": 0, "meanMs": 41.7}], "fsync": {"count": 212, "meanMs": 0.4} } ``` - `version` is the server's, as `/xrpc/_health` answers it; `started` is the `time` of this start's `start` line. - `accounts`, `logBytes`, and `firehose`'s `seq` and `subscribers` are `pds_accounts`, `pds_log_bytes`, `pds_firehose_seq` and `pds_firehose_subscribers`. - `routes` has one entry for each route with a request answered, the most `count` first, then by `method`: `count` is its `pds_request_duration_seconds_count`, `statuses` its `pds_requests_total` by status, each status as a string, `rateLimited` its `pds_rate_limited_total`, 0 if none, and `meanMs` its `pds_request_duration_seconds_sum` over its `count`, in milliseconds. - `outbound` has one entry for each `target` with a request made, by `target`: `ok` and `error` are its `pds_outbound_requests_total` of each `outcome`, 0 if none, and `meanMs` its `pds_outbound_request_duration_seconds` sum over its count, in milliseconds. - `fsync` is `pds_fsync_duration_seconds`' count, and its sum over its count in milliseconds, or 0 with no sync. Every number is as the metrics are when the answer is made: the `_stats` request is counted once answered, so not in its own answer. `meanMs` may have any fraction. ## Traces With `OTEL_EXPORTER_OTLP_ENDPOINT` or `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` set, and `OTEL_SDK_DISABLED` not `true`, the server exports a span for each request it answers, and spans within it, by OTLP over HTTP with protobuf bodies: a `POST` of an `ExportTraceServiceRequest` with `Content-Type: application/x-protobuf`, gzipped with `Content-Encoding: gzip` if the compression is `gzip`. With neither set, no span is exported, but each request still has its trace id. *Note:* protobuf because it is OTLP/HTTP's default: an operator who sets only the endpoint, as the OpenTelemetry specification lets them, gets what every collector takes. ### The request's trace A request's `traceparent` header (W3C Trace Context) names its trace: a request with a valid one is in that trace, its span the child of the span it names. A header that is not valid is ignored, as if absent: valid is `00---`, neither id all zeros; a version other than `00` and not `ff` is read for those fields alike, and what follows them ignored. A request with no valid one starts a trace of its own, with a new random trace id. `tracestate` is not kept. Spans are sampled as OpenTelemetry's `parentbased_always_on`: a request whose `traceparent` has its sampled flag (`01`) clear exports no span, nor any span within it; every other request's spans are exported. ### Spans Each span has the resource attributes `service.name`, as `OTEL_SERVICE_NAME`, and `service.version`, the server's version, and the instrumentation scope `pds`. - **The request**: one span, of kind `SERVER`, from reading the request's head to writing its log line, named ` `, or the HTTP method alone for a route of `_unknown`, as `GET /xrpc/com.atproto.repo.getRecord`. Its attributes, of OpenTelemetry's semantic conventions for an HTTP server: | Attribute | Is | | --- | --- | | `http.request.method` | the HTTP method, as `GET`; one the server does not know, `_OTHER` | | `url.path` | the request's path as it came, percent-encoding and all, without its query | | `url.scheme` | `http` | | `http.route` | `/xrpc/` for a method served, `/xrpc/_health`, `/xrpc/_logs`, `/xrpc/_stats`, `/xrpc/_oauthSessions` and the account's other OAuth and passkey routes, or the route's path, as `/mcp`; none for `_unknown` | | `http.response.status_code` | the status, an integer | | `error.type` | the status, as a string, when it is 500 or more | Its status is `ERROR` when the status is 500 or more, and unset otherwise. - **A sync**: a span of kind `INTERNAL` named `fsync`, the child of the request's, for each time it waits on its writes reaching the disk. Every request whose answer says a commit was made (`commit` in its body) has at least one. - **A request the server makes**, to any target above: a span of kind `CLIENT` named with its HTTP method, as `POST`, with `http.request.method`, `server.address`, `server.port`, `url.full` (without its query, fragment or credentials), and `http.response.status_code` when it was answered, `error.type` (the status as a string, or `timeout`) when it failed, and status `ERROR` then. One made for a request is the child of the request's span; one made on its own, as a crawl or settling a handle change after a start, is the root of a trace of its own. Each such request carries a `traceparent` naming this span, with the trace's sampled flag. No attribute holds a query, a header, a body or anything the log may not. ### Export Spans are exported in the background, in batches, as the `OTEL_BSP_*` variables say. No request waits on an export: a receiver that refuses connections, or takes them and never answers, changes no answer and no request's time. A full queue drops new spans; an export that fails may be tried again until `OTEL_BSP_EXPORT_TIMEOUT` has passed since its first try, and its spans are then dropped. Once the receiver answers again, the next export goes to it, within `OTEL_BSP_EXPORT_TIMEOUT`. ## Performance `harness/perf.mjs` runs one workload against the reference and against the server, each in a process of its own (the reference in a Node process of its own, with its PLC directory, which the server uses too), with the rate limits off. The workload is in parts, run in order: | Part | What | Requests | At once | | --- | --- | --- | --- | | `createAccount` | new accounts, each with a password | 40 | 4 | | `createRecord` | records of about 100 bytes, spread over the accounts | 2000 | 16 | | `getRecord` | records written, chosen at random | 4000 | 16 | | `getRepo` | an account's whole repository, of 50 records | 200 | 8 | | `firehose` | 20 subscribers to `subscribeRepos` from now, each from an address of its own, then writes one at a time; each time is from sending the write to a subscriber receiving its `#commit` | 200 writes, 4000 deliveries | 1 write | | `uploadBlob` | 64 KiB of random bytes, `application/octet-stream` | 200 | 8 | Each part is run in three rounds of a third of its requests, the sides taking turns: the reference's first round, the server's, the reference's second, and so on. A round's throughput is its requests, or for `firehose` its deliveries, a second over the round's whole time. Each side is judged by its best round: the lowest 50th and 99th percentile of a round's times, and the highest throughput. *Note:* other work on the machine only ever makes a round slower, so the best of three is the nearest to what the side can do, and the turns spread a disturbance over both sides. A part's **ratios** are the server's best over the reference's, in the same run: its p99 ratio, of the best 99th percentiles, and its throughput ratio, of the best throughputs. *Note:* the two sides take turns on the same machine, so a slower or busier machine slows both; the ratios keep what is the server's own and drop most of what is the machine's, which times in milliseconds would not. `harness/perf-baseline.json` is the **baseline**: each part's ratios as runs on the runner measured them, the nightly's or the `perf` workflow's (the same run, asked for alone), each part's the weakest of at least five runs (the highest p99 ratio and the lowest throughput ratio), with the runs named in it. The run prints each part's ratios beside the baseline's, and all of them on one line as the baseline holds them. A pull that makes the server faster raises the baseline, from what those runs print; one that lowers it says why. A part the baseline does not name is held to the reference alone, as if its ratios were 1. *Note:* the weakest of several runs, rather than one, so that one lucky run does not make the baseline more than the server does on most nights; on a laptop with other work on it, three of nine runs fell short of the weakest of the others by a little, so five is the least. The server must, for every part: - have a p99 ratio no more than **1.5 times** the baseline's, or than 1 where the baseline's is above 1: never a best 99th percentile more than 1.5 times the reference's, and no more than 1.5 times the baseline's where the server is faster. *Note:* on a shared machine, the 99th percentiles of one side's three rounds of a part spread by 1.05 to 1.4 times where the part is bound by the server's own work (`uploadBlob`, `firehose`, `createAccount`), and further where other work lands on a round; the best of three takes out most of that, the weakest of several runs in the baseline more, and 1.5 the rest, so a server no slower than it was passes while one twice as slow fails. - and a throughput ratio at least **0.8 times** the baseline's, or than 1 where the baseline's is below 1. *Note:* where a part is bound by the same work on both sides (a password hash, the directory, a write's flush, the firehose's fan-out of writes made one at a time), the two sides' best throughputs differ from run to run by a few percent either way; a floor of 1.0 turns that noise into failures of a server that is level, while 0.8 still fails one that is slower. Where the server is many times faster (`getRecord`, `getRepo`), its ratio moves further from run to run, which the weakest of several runs takes in. Then, against the server alone, **subscribers come and go**: cohorts of 20 subscribers to `subscribeRepos` from now, each from an address of its own, the same 20 addresses each time (`LIMITS.md`: 16 at most from one). In a cohort, all 20 connect, one `putRecord` is made, each subscriber receives its `#commit` within 10 s, and all 20 close and see their connections closed within 10 s. One cohort warms the server, and 20 more follow. The server process's open file descriptors, threads and memory are read 15 s after the first cohort, and after the last: on Linux from `/proc//fd` and `/proc//status` (its memory resident and swapped out, `VmRSS` and `VmSwap`), on macOS from `lsof`, `ps` and `footprint` (its footprint, resident and compressed). After the last, they are read again every quarter second, for up to 20 s, taking the least of each reading, until all three are within these bounds. The server must, after the last cohort, hold: - no more open file descriptors than after the first; - no more threads than after the first; *Note:* the server's idle threads end 10 s after their last work, and their memory goes back with them: read at once after the first cohort, the threads the parts before used are still there, and as many threads held for good would be hidden. A cohort starts some 20 again, and the up to 20 s after the last lets them end. A thread or two the server starts and ends on its own may come some seconds later; what is held for good is in every reading, and such a thread in only some, so the least of each leaves it out. Read so, the descriptors and threads after the last cohort come back to those after the first, or fewer, so neither has any allowance. - and no more memory than after the first and **8 MiB**. *Note:* the allocator keeps some of what was freed for a while. On macOS, once the idle threads had ended, the server's resident memory after the last cohort was 0 to 6.1 MiB above the first's in three runs (up to 13 MiB before they had), and its footprint 0.2 and 2.3 MiB above in two. 400 subscribers each holding 256 KiB for good would be 100 MiB more. Resident memory alone is not enough on macOS, which compresses pages no one touches, as memory held for good is not: it missed such a leak, which the footprint showed as 107 MiB more. The reference is not held to this. The run exits 1 if any of these fails, naming the part, or the descriptors, threads or memory. ## How it is checked `harness/checks/operations.mjs` and `harness/checks/logging.mjs`, at the end of every `compare.mjs` run, of any kind. The server is started with `PDS_METRICS_PORT` set, `PDS_LOG_DIR` a directory of its own apart from the data directory, `PDS_LOG_KEEP_BYTES` of `262144`, so that the kept lines are let go of in a run, its traces going to a stub receiver in the harness (`harness/lib/otlp.mjs`) with a header in `OTEL_EXPORTER_OTLP_HEADERS`, `OTEL_SERVICE_NAME`, and `OTEL_BSP_SCHEDULE_DELAY` of 200 ms and `OTEL_BSP_EXPORT_TIMEOUT` of 3 s. Every request the harness sends the server carries a `traceparent` of its own, one in ten unsampled, and is noted with its answer and how long it took; and every password, token, code, cookie, email and key the run handles is kept. In a run of random steps, the receiver stops answering a third of the way through, is gone at half way, once an export could have timed out, and is back at two thirds; the steps go on. Then, at the end: - **Health**: `GET`, `HEAD` and `POST /xrpc/_health`, and `GET` with a query, against the reference: the status, `Content-Type`, and the body's form; `/metrics` on the server's own port, `404` on both; with `--limits`, `_health` past `global` on both. - **The root**: `GET`, `HEAD` and `POST /`, and `GET` with a query, against the reference: the status and `Content-Type`; the server's text held to "The root" above, read from this page, with its own origin; and no `RateLimit-*` headers on it. - **Routes**: paths under `/xrpc/` of no method served, one that is no NSID, and another path, each of route `_unknown`. - **Trace context**: `_health` with no `traceparent`, ones not valid (no parse, a trace id of 0, version `ff`, upper case), an unsampled one, and one of version `01` with more after it: each line's `trace_id` is new, or the one sent; each span the root of a new trace, or the child of the one sent; none for the unsampled. - **Metrics**: an account and records of the harness's own, then: one more subscriber while it subscribes and `pds_firehose_seq` the last event's; `pds_accounts` as `listRepos` lists; `pds_log_bytes` no more than the data directory holds, and grown across a record written while no file in the data directory was made, deleted, replaced or cut short, the directory read before the metrics and after them, around the write (a record written again, up to 20 in all, while one was); every family's type, labels, label values and buckets; `pds_requests_total` equal to the `request` lines since the server started, by route and status, and no less than the harness's requests; each route's `_count` equal to its requests; `pds_rate_limited_total` equal to the 429s; the syncs grown by the records; a PLC request for every account made; `404` for another path. - **The admin's reads**: `_logs` and `_stats` with no `Authorization`, a bearer token, the user `admin` with another password, and another user: each `401 AuthenticationRequired`; `POST`, `400 InvalidRequest`; `_logs` with a `level` not listed, a `minMs` of `-1`, a `limit` of `0`, of `501` and of `x`: each `400 InvalidRequest`. Then `_logs` with no parameter, and with a `limit`, its lines the last that many written to standard output by the starts that kept them, before the read; by the `trace` of each of the last requests the harness sent, that request's `request` line alone; by `msg`, `method`, `level` and `minMs`, each and together, the last lines of standard output that match, and none that does not. `_stats` between two scrapes of `/metrics`, each number between the two scrapes', `started` the `start` line's `time`, `routes` and `outbound` in their order, and the second scrape's `_stats` count one more than the first's. At every check, the files in `PDS_LOG_DIR` take no more than `PDS_LOG_KEEP_BYTES`, and `_logs` reads back the newest lines of half that, across a kill and a start. Started with `PDS_LOG_DIR` unset, `_logs` is `404 LogNotKept` and `_stats` answers. - **Spans**: once the server is shut down with SIGTERM, which it must exit 0 on within 15 s: for every request the harness sent and had answered, one `SERVER` span in its trace, the child of its `traceparent`, with the name, attributes, status, resource and times the spec gives, or none if unsampled; an `fsync` span under each that made a commit; a `CLIENT` span under each `createAccount`, of its call for the DID (the `POST` of its genesis operation to the directory, or, for an account moving in with its `did`, the `GET` of the DID's document, from the directory or, for a `did:web`, from its host, `MIGRATION.md`); a root span for every request line of a client that sent no `traceparent`; no `SERVER` span without a line; every export a protobuf `POST /v1/traces` with the header. Spans of requests near the outage or just before a kill may be lost, and those of an export the server says it dropped (Export, above): a span missing, with a `trace export failed` line from the request's start until `OTEL_BSP_SCHEDULE_DELAY`, twice `OTEL_BSP_EXPORT_TIMEOUT` and 1.5 s after its answer; or with the last such line before its start within a minute and the timeout, and no export taken since, while exports kept failing unlogged. While the receiver hung, the requests' median time is under a second, and a `trace export failed` line is written. - **Starts**: at `warn`, with a PLC directory that answers 500 and `OTEL_SERVICE_NAME` unset: `createAccount` is refused, the directory gets a `traceparent` in the request's trace, a `plc failure` line names the status and the DID, its `CLIENT` span is an error of `unknown_service:pds`, and an outbound error is counted. Killed, then started with no OTLP endpoint: no span. Where the machine has an IPv4 address on an interface other than the loopback, the server started with `PDS_BIND_ADDRESS` unset does not answer `_health` on it, and started with `0.0.0.0` answers on it and on `127.0.0.1`; where the machine has IPv6's loopback, started with `::1` it answers on `[::1]` and not on `127.0.0.1` (`PDS.md`). Then `PDS_LOG_LEVEL=loud`, the protocols `grpc` and `http/json`, a `PDS_METRICS_PORT` that is no number and one equal to `PDS_PORT`, a `PDS_LOG_KEEP_BYTES` of `65535` and of `many`, a `PDS_LOG_DIR` under a file, a `PDS_BIND_ADDRESS` of `localhost` and of `[::1]`, and one of `203.0.113.1`, an address no interface of the machine has: each exits 1 with why on standard error. - **The log**, whole: every line one JSON object with the fields its event has, at no level below the start's `PDS_LOG_LEVEL`; each start's lines starting with `start`, with `recovered` after a start the last server did not shut down and not otherwise, and ending with `shutdown` after a SIGTERM; one `request` line for each request the harness had answered, but one whose answer was cut short once its server was signalled to stop, found by its trace id, with its route and status, an `ms` no longer than the harness waited, and the `did` of its token for a method that acts as one; of each `getBlob` answered `200` or `500`, `repo` and `cid` the `did` and `cid` it asked for, and of one answered `200`, `bytes` the bytes the harness read of it, or, of one whose answer ended short (`BLOBS.md`, "How it is checked"), no fewer than it read and fewer than its `Content-Length`; no `repo`, `cid` or `bytes` on any other `request` line; nothing on standard error but from a start that exited 1; and none of the run's secrets, nor the text of a record the harness wrote, anywhere in standard output, standard error or the files in `PDS_LOG_DIR`. `harness/perf.mjs` is a gate of its own, run alone after every other.