~ knotless Search/
↑↓ move↵ open⇧↵ all results
0
0
766 lines · 42.0 KB Raw History

Operations

What the server tells whoever runs it, and how fast it must be: its log, its health, its metrics and its traces, what its admin reads of them over HTTP, and the performance it is held to beside the reference. What the server serves, and how it is checked, is in PDS.md.

Of this, the reference shows only /xrpc/_health, and the harness holds the server to it. The reference's log is its own (pino's), and it serves no metrics or traces, so the rest is this server's design, following the ordinary conventions: a log of JSON lines, Prometheus' text format for metrics, and OpenTelemetry for traces (OTLP, W3C Trace Context, its HTTP semantic conventions and its standard OTEL_* variables). The harness holds the server to this page.

Environment

Besides those the other pages list:

VariableIsDefault
PDS_LOG_LEVELthe least level of log line written: debug, info, warn or error; or off, for noneinfo
PDS_LOG_DIRa directory the log is kept in too, for the admin to read back ("Kept", below); made if missing; unset, the log is kept nowhere but standard outputunset
PDS_LOG_KEEP_BYTESthe most bytes the files in PDS_LOG_DIR take together; at least 6553667108864 (64 MiB)
PDS_METRICS_PORTthe port /metrics is served on, on 127.0.0.1; unset, metrics are not servedunset
OTEL_EXPORTER_OTLP_ENDPOINTthe base URL of the OTLP receiver traces go to: they are posted to it with v1/traces appended, after a / if it does not end in oneunset
OTEL_EXPORTER_OTLP_TRACES_ENDPOINTthe URL traces are posted to, as it is; before OTEL_EXPORTER_OTLP_ENDPOINTunset
OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_EXPORTER_OTLP_TRACES_PROTOCOLhttp/protobuf, the only protocol the server speakshttp/protobuf
OTEL_EXPORTER_OTLP_HEADERS, OTEL_EXPORTER_OTLP_TRACES_HEADERSheaders sent with each export, key=value pairs joined by ,, each value percent-decoded; the second's pairs winnone
OTEL_EXPORTER_OTLP_COMPRESSION, OTEL_EXPORTER_OTLP_TRACES_COMPRESSIONgzip or nonenone
OTEL_SERVICE_NAMEthe service.name of the server's spansunknown_service:pds
OTEL_SDK_DISABLEDtrue turns traces off, whatever else is setfalse
OTEL_BSP_SCHEDULE_DELAYmilliseconds between exports5000
OTEL_BSP_MAX_QUEUE_SIZEspans held for export at most; past it, new spans are dropped2048
OTEL_BSP_MAX_EXPORT_BATCH_SIZEspans in one export at most512
OTEL_BSP_EXPORT_TIMEOUTmilliseconds an export may take, retries and all, before its spans are dropped30000

A value that does not parse, a level not listed, a protocol other than http/protobuf (as grpc or http/json), a PDS_METRICS_PORT equal to PDS_PORT, a PDS_LOG_KEEP_BYTES below 65536, or a PDS_LOG_DIR that cannot be made or written to, makes pds serve exit 1 before it writes anything to standard output, with why on standard error, as LIMITS.md says of its variables. An empty variable is as if unset.

The log

The server writes its log to standard output: one JSON object per line, UTF-8, each ended by \n; and with PDS_LOG_DIR set, keeps each line in that directory too ("Kept", below). Standard error is for one thing only, why pds serve exits 1 (COMMANDS.md, DURABILITY.md, LIMITS.md): a server that runs writes nothing there.

Note: standard output is the stream a container runtime or a supervisor collects; standard error is kept for the human reading why the server would not start.

Each line is written whole, by one write, so lines of requests answered at once never mix. A line is not held in the process: once written, it survives the server being killed with SIGKILL.

Every line

FieldIs
timewhen the line was written, RFC 3339 in UTC with milliseconds: 2026-11-09T17:23:00.123Z
leveldebug, info, warn or error
msgwhat happened, a fixed text: one of the events below, or a short text of the server's own that holds no value from a request

and the event's own fields. A field with no value is left out, never null. Lines below PDS_LOG_LEVEL are not written; the levels go debug, info, warn, error, and off writes none.

Events

msglevelFieldsWhen
startinfoversion, port, sync, and metrics_port when it is setthe first line of each start, once the environment is read and before the data is
recoveredwarndropped_bytesa start on a data directory the last server did not shut down: killed, crashed, or the machine lost power
requestinfo, or error for a status of 500 or moremethod, status, ms, trace_id, and did when known; of a getBlob, repo, cid and byteseach request answered
plc failurewarnerror, status when the directory answered, and did when the call concerns oneeach call to the PLC directory that fails: no answer, a timeout, or a status other than 2xx
blobstore failurewarnop: put, get or delete; error, status when the store answered, and dideach request to the bucket that fails (BLOBS.md): no answer, a timeout, or a status other than 2xx
trace export failedwarnerroran export whose spans are dropped (Export, below), at most one line a minute while exports keep failing
store failureerrorerrorwhat the server does to its data in the background and could not (STORE.md, DURABILITY.md): a compaction; a checkpoint of the journal; or a sync of the journal, after which it answers no write until it is started again
shutdowninfosignal: SIGTERM or SIGINTthe last line of a server shut down
  • start: version is the server's version, as /xrpc/_health answers it; port and metrics_port are numbers; sync is together or each, how waiting writes are made durable (DURABILITY.md, PDS_SYNC).
  • recovered: dropped_bytes is a number, the bytes of writes cut short that recovery dropped from the end of the logs the server keeps its state in (DURABILITY.md, STORE.md): a torn tail, and the logs of generations after one cut short or not sealed; 0 when it dropped none. A start on an empty directory, or on one the last server shut down, writes no recovered. The server may choose how it knows, as by a mark it makes on shutdown and takes away as it starts.
  • request: one line for each request the server answers, written once its answer is whole and before the last byte of it is sent, so that a client that has the whole answer knows the line is written. A response streamed, such as getRepo's, has its line once its body is made. An upgrade to a WebSocket (subscribeRepos) is answered by its 101: the line is written then, not when the subscription ends. A request whose client leaves before the answer is sent whole still has its line.
    • method: the request's route, as the metrics label it (below);
    • status: the HTTP status, a number;
    • ms: a number, the milliseconds from reading the request's head to writing the line, with any fraction;
    • trace_id: the request's trace, 32 lowercase hex digits (Traces, below), whether or not traces are exported;
    • did: the DID of the account whose credentials the request carried and the server verified, as a bearer token it gave out; left out otherwise;
    • repo and cid: of com.atproto.sync.getBlob alone, the DID of the account and the CID of the blob asked for, when the account holds that blob and the server answers with it, 200, or failed to read it, 500 (BLOBS.md, "getBlob", "The objects"); left out of a getBlob answered otherwise, and of every other line. did stays the caller's, so a getBlob with no token has repo and no did;
    • bytes: with repo and cid, when the answer is 200: a number, the bytes of the blob the server had handed to the connection by the time it wrote the line, which it writes as it hands on the last of them. That is the blob's size when it was sent whole, and fewer when the client left first or the store's read failed after its first byte; 0 for HEAD. It counts what the server handed on, not what reached the client: a client that left may have had fewer.
  • plc failure: error says why, as timeout or the status; status is a number.
  • shutdown: on SIGTERM or SIGINT the server stops taking connections, lets the requests it has finish, for up to 10 seconds, closes the firehose's subscriptions, syncs what it has written, exports the spans it holds, for up to 5 seconds, writes this line, and exits 0.

Besides these the server may write lines of its own, as when a crawler does not answer or a write fails, at warn or error, with fields of their own, and at debug whatever helps.

What is never logged

No line holds, in any field, whole or in part:

  • a password, an account's or an app password; the admin password;
  • a token: an access or refresh JWT, a delegation token or space credential (SPACES-CREDENTIALS.md), an OAuth code or token, a DPoP proof, or any Authorization header;
  • an invite code, or a code sent by email, as an OAuth sign-in's emailOtp;
  • a cookie's value, as a device's or its session's id (OAUTH.md);
  • a key: the PLC rotation key, an account's signing or KEM key, the key the server signs its JWTs with;
  • the values of OTEL_EXPORTER_OTLP_HEADERS;
  • an account's email;
  • a request's body, query string or headers, or an answer's body; but for getBlob's repo and cid (Events, above): the DID of an account here and the CID of a blob it holds, as public as the blob.

method is a route the server serves, never the path asked, so nothing else of a URL is logged.

Kept

With PDS_LOG_DIR set, each line written to standard output is written to a file in PDS_LOG_DIR too, the same bytes, by one write of its own, just after its write to standard output and before the next line is written to either. So a line, once on standard output, is kept as well, and survives the server being killed with SIGKILL as it does, but for the one line a server killed between the two writes was writing: it is on standard output alone. Lines below PDS_LOG_LEVEL are written to neither. A write to the directory that fails is not tried again, and the server serves on: standard output is the log, and the files a copy of it to read back.

The files in PDS_LOG_DIR take at most PDS_LOG_KEEP_BYTES bytes together, at every moment. Past that the oldest lines are let go first, as many at once as the server likes, but never one of the newest lines whose bytes, with their \n, add up to no more than half of PDS_LOG_KEEP_BYTES: those are always kept. A line whose bytes, with its \n, are more than half of it is not kept at all. A server started on a PDS_LOG_DIR that holds lines carries on after them, so the kept lines run across starts, oldest first. How the files are named and laid out is the server's own; only their total size, and the lines _logs reads back, are seen. One server writes to a PDS_LOG_DIR at a time.

Note: PDS_LOG_DIR is apart from PDS_DATA_DIR: the lines are no state of the server's, no backup needs them, and the data directory's size is held to its bounds without them (STORE.md, "Bounds"). A container's standard output is lost with the container, as when a deploy makes it again; a volume of their own keeps the lines across it.

Health

GET /xrpc/_health answers as the reference does: 200 with Content-Type: application/json; charset=utf-8 and

{"version": "0.0.0"}

version being the server's version, that of the pds package. Its query, if any, is ignored. HEAD answers the same status and headers with no body; POST is 400 InvalidRequest. It is answered while the server serves, and is in no rate limit: past global (LIMITS.md) it is still 200, with no RateLimit-* headers. Its route is _health.

The root

GET / says what the server is, as the reference's does, and how to connect an AI app to it (COMMANDS.md, OAUTH-MCP.md). It answers 200 with Content-Type: text/plain; charset=utf-8 and this text, where <origin> is the origin of PDS_PUBLIC_URL, every line ending in \n, the first line empty, and none with spaces at its end:


         __                         __
        /\ \__                     /\ \__
    __  \ \ ,_\  _____   _ __   ___\ \ ,_\   ___
  /'__'\ \ \ \/ /\ '__'\/\''__\/ __'\ \ \/  / __'\
 /\ \L\.\_\ \ \_\ \ \L\ \ \ \//\ \L\ \ \ \_/\ \L\ \
 \ \__/.\_\\ \__\\ \ ,__/\ \_\\ \____/\ \__\ \____/
  \/__/\/_/ \/__/ \ \ \/  \/_/ \/___/  \/__/\/___/
                   \ \_\
                    \/_/


This is an AT Protocol Personal Data Server (aka, an atproto PDS)

Most API routes are under /xrpc/

Connect an AI app

  1. Add this address to your AI app as a connector:

       <origin>/mcp

     In Claude: Settings > Connectors > Add custom connector,
     and choose "Sign in when needed".

  2. When it asks, sign in here and approve it.

  3. Ask it things like:

       What have I posted this week?
       Post "hello from Claude" to Bluesky
       Who have I followed lately?
       Set my Bluesky bio to "Software engineer in Seattle"

      Code: https://knotless.dev/chadtmiller.com/monolith
  Protocol: https://atproto.com

The banner is the reference's; the rest is this server's own, and only the status and Content-Type are compared. Its query, if any, is ignored. HEAD answers the same status and headers with no body; any other method is 404, as the reference's. It is in no rate limit and carries no RateLimit-* headers, as the reference's does not. Its route is /.

Metrics

GET /metrics on 127.0.0.1:PDS_METRICS_PORT answers the server's metrics, 200 with Content-Type: text/plain; version=0.0.4; charset=utf-8, in Prometheus' text exposition format, version 0.0.4. Any other request on that port is 404. With PDS_METRICS_PORT unset nothing listens, and /metrics on the server's own port is 404 as any route it does not serve.

It is a port of its own, with no auth. Being apart, a scrape is no request of the server's: it has no log line or span, is in no metric and in no rate limit.

Note: the server's own port is the one put before the internet; a separate one, on the loopback, is never in front of it, so nothing there needs guarding, and a Prometheus scrape, which carries no credentials, needs none.

Each family has a # HELP line and a # TYPE line, then its samples; every family below is there from the start, its series once they have a value. Counters and histograms count from the start of the process, so they start again from 0 when it does, as Prometheus expects. A histogram has a _bucket sample for each bound below, cumulative, then le="+Inf", then _sum and _count. Durations are in seconds.

MetricTypeLabelsIs
pds_requests_totalcountermethod, statusrequests answered: one for each request line
pds_request_duration_secondshistogrammethodeach request's ms, in seconds
pds_rate_limited_totalcountermethodrequests refused by a rate limit (429)
pds_firehose_seqgaugethe seq of the firehose's last event; 0 before any
pds_firehose_subscribersgaugesubscribeRepos connections open
pds_accountsgaugethe accounts the server holds, those listRepos lists
pds_log_bytesgaugethe bytes of the logs the server keeps its state in, in PDS_DATA_DIR, all of them: every generation's not yet folded away, and the checkpoints (STORE.md)
pds_fsync_duration_secondshistogrameach sync of written data to the disk, every one the server makes: a log's as a commit is answered and as it is opened, a sealed log's, a directory's, a table's, a blob's file, the firehose's index, its keys and marks, and the whole data directory's after a start that followed a kill
pds_outbound_requests_totalcountertarget, outcomeeach request the server makes
pds_outbound_request_duration_secondshistogramtargeteach request the server makes, until its answer or failure

pds_log_bytes grows by what each write puts in the logs, and falls only as the server lets go of what they held: a log's generations folded into its checkpoint (an account's, its table: STORE.md, "Bounds"), which a write may start and which the server may run in the background at any time; and the journal's records, once what they hold is durable where it is kept (DURABILITY.md). It lets go of nothing but as it makes, deletes, replaces or cuts short a file in PDS_DATA_DIR, and what it lets go of leaves the gauge as that file changes, neither before nor after: a read of the gauge begun once the change can be seen in the directory counts none of it, and one ended before the change counts all of it, however long the server takes over what it does next (a journal's segment deleted leaves the gauge as it goes, not once the directory is synced or the other segments are deleted). So across a write during which no file there was made, deleted, replaced or cut short, each only grown or left as it was, the gauge grows, by however little; across one during which one was, it may be less after than before.

The buckets:

  • requests and outbound requests: 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10 (Prometheus' own default);
  • syncs: 0.0001, 0.00025, 0.0005, 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 1.

The labels:

  • method is the request's route, the same as its log line's:

    • a method the server serves, by its NSID, for /xrpc/<nsid>, whatever its HTTP method;
    • _health, _logs and _stats, for /xrpc/_health and the admin's reads (below), _oauthSessions, _oauthRevokeSession, _oauthDevices and _oauthRevokeDevice, for the account's OAuth routes (OAUTH-ACCOUNT.md), and _passkeyList, _passkeyAdd and _passkeyRemove, for its passkey routes (PASSKEYS.md);
    • a route outside /xrpc/ the server serves, as its path: / ("The root"), /mcp (COMMANDS.md), /.well-known/atproto-did (IDENTITY.md), and OAuth's (OAUTH.md): /.well-known/oauth-protected-resource, /.well-known/oauth-authorization-server, /oauth/jwks, /oauth/par, /oauth/token, /oauth/revoke, /oauth/authorize, /oauth/authorize/redirect, and the authorization page's /@atproto/oauth-provider/~api/sign-in, ~api/consent, ~api/reject, ~api/sign-out, ~api/passkey-options and ~api/passkey-sign-in, the passkey page /oauth/passkey and its ~api/passkey-register (PASSKEYS.md), and /oauth/assets/{name} for every path under /oauth/assets/, whatever its name (OAUTH-PAGES.md); the MCP issuer's (OAUTH-MCP.md): /.well-known/oauth-protected-resource/mcp, /.well-known/oauth-authorization-server/oauth/mcp, /oauth/mcp/authorize, /oauth/mcp/token and /oauth/mcp/revoke; a page that adds such a route adds it to this list;
    • _unknown for anything else: an NSID the server does not serve, a path under /xrpc/ that is no NSID, any other path.

    Note: so no label holds what a client chose, and the series are bounded by the routes served.

  • status is the HTTP status, in decimal: "200".

  • target is where the request goes: plc for the PLC directory, crawler for a crawler (FEDERATION.md), service for the AppView and the services atproto-proxy names (PROXY.md), did-web for a did:web document, client for a client's metadata document or key set (OAUTH-CLIENTS.md, SPACES-CREDENTIALS.md), space for a space's notification to another service (SPACES-SYNC.md), lexicon for a permission set's record (OAUTH-PERMISSION-SETS.md), handle for a handle's /.well-known/atproto-did (IDENTITY.md), and blobstore for the bucket blobs are kept in (BLOBS.md); outcome is ok for a 2xx answer and error for anything else, no answer included.

The admin's reads

The admin, whoever has PDS_ADMIN_PASSWORD, reads what the server logged and counted over the server's own port, without reaching its machine: GET /xrpc/_logs and GET /xrpc/_stats. They are the server's own, as /xrpc/_health is, of no lexicon and no NSID: so they are listed under no page of PDS.md, and their MCP tools and CLI commands are given in COMMANDS.md, "The admin's reads".

Each takes the admin's Authorization: Basic as LIMITS.md's "The admin" says, and checks it first: anything else is 401 AuthenticationRequired. Then a method other than GET is 400 InvalidRequest; then a parameter that does not parse, or is out of its range, is 400 InvalidRequest. A parameter neither takes is ignored. Each answers 200 with Content-Type: application/json; charset=utf-8, costs a point of global (LIMITS.md), and has its route, _logs or _stats, in its request line, its metrics and its span, as any request.

_logs

The lines kept in PDS_LOG_DIR ("Kept", above), oldest first:

{"lines": [{"time": "2026-11-09T17:23:00.123Z", "level": "info", "msg": "request", "method": "com.atproto.repo.createRecord", "status": 200, "ms": 3.2, "trace_id": "…", "did": "did:plc:…"}]}

lines holds the last limit of the lines kept that match every parameter given, each the JSON object the line is, as written. They are the lines kept when the read began: its own request line is written once its answer is made, so it is never among them, and is among those of the next read. A line in the files that is not a JSON object, as one cut short by a lost power, is skipped.

ParameterMatches a line
tracewhose trace_id is this
msgwhose msg is this
methodwhose method is this: a route, as com.atproto.repo.createRecord or /oauth/token
levelat this level or above: debug, info, warn or error; any other value is 400
minMswith an ms of this many milliseconds or more, an integer of 0 or more; a line with no ms does not match
limithow many, the last of those matched: an integer from 1 to 500, 100 if not given

With PDS_LOG_DIR unset, _logs is 404 LogNotKept, once the auth and the parameters are checked.

_stats

What the server counted since it started, as its metrics count it (above), whether or not PDS_METRICS_PORT is set:

{
  "version": "0.1.0",
  "started": "2026-11-09T17:20:01.004Z",
  "accounts": 3,
  "logBytes": 1048576,
  "firehose": {"seq": 120, "subscribers": 1},
  "routes": [
    {"method": "com.atproto.repo.createRecord", "count": 40, "statuses": {"200": 38, "400": 2}, "rateLimited": 0, "meanMs": 3.1}
  ],
  "outbound": [{"target": "plc", "ok": 3, "error": 0, "meanMs": 41.7}],
  "fsync": {"count": 212, "meanMs": 0.4}
}
  • version is the server's, as /xrpc/_health answers it; started is the time of this start's start line.
  • accounts, logBytes, and firehose's seq and subscribers are pds_accounts, pds_log_bytes, pds_firehose_seq and pds_firehose_subscribers.
  • routes has one entry for each route with a request answered, the most count first, then by method: count is its pds_request_duration_seconds_count, statuses its pds_requests_total by status, each status as a string, rateLimited its pds_rate_limited_total, 0 if none, and meanMs its pds_request_duration_seconds_sum over its count, in milliseconds.
  • outbound has one entry for each target with a request made, by target: ok and error are its pds_outbound_requests_total of each outcome, 0 if none, and meanMs its pds_outbound_request_duration_seconds sum over its count, in milliseconds.
  • fsync is pds_fsync_duration_seconds' count, and its sum over its count in milliseconds, or 0 with no sync.

Every number is as the metrics are when the answer is made: the _stats request is counted once answered, so not in its own answer. meanMs may have any fraction.

Traces

With OTEL_EXPORTER_OTLP_ENDPOINT or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT set, and OTEL_SDK_DISABLED not true, the server exports a span for each request it answers, and spans within it, by OTLP over HTTP with protobuf bodies: a POST of an ExportTraceServiceRequest with Content-Type: application/x-protobuf, gzipped with Content-Encoding: gzip if the compression is gzip. With neither set, no span is exported, but each request still has its trace id.

Note: protobuf because it is OTLP/HTTP's default: an operator who sets only the endpoint, as the OpenTelemetry specification lets them, gets what every collector takes.

The request's trace

A request's traceparent header (W3C Trace Context) names its trace: a request with a valid one is in that trace, its span the child of the span it names. A header that is not valid is ignored, as if absent: valid is 00-<trace id, 32 lowercase hex>-<parent id, 16 lowercase hex>-<flags, 2 lowercase hex>, neither id all zeros; a version other than 00 and not ff is read for those fields alike, and what follows them ignored. A request with no valid one starts a trace of its own, with a new random trace id. tracestate is not kept.

Spans are sampled as OpenTelemetry's parentbased_always_on: a request whose traceparent has its sampled flag (01) clear exports no span, nor any span within it; every other request's spans are exported.

Spans

Each span has the resource attributes service.name, as OTEL_SERVICE_NAME, and service.version, the server's version, and the instrumentation scope pds.

  • The request: one span, of kind SERVER, from reading the request's head to writing its log line, named <HTTP method> <http.route>, or the HTTP method alone for a route of _unknown, as GET /xrpc/com.atproto.repo.getRecord. Its attributes, of OpenTelemetry's semantic conventions for an HTTP server:

    AttributeIs
    http.request.methodthe HTTP method, as GET; one the server does not know, _OTHER
    url.paththe request's path as it came, percent-encoding and all, without its query
    url.schemehttp
    http.route/xrpc/<nsid> for a method served, /xrpc/_health, /xrpc/_logs, /xrpc/_stats, /xrpc/_oauthSessions and the account's other OAuth and passkey routes, or the route's path, as /mcp; none for _unknown
    http.response.status_codethe status, an integer
    error.typethe status, as a string, when it is 500 or more

    Its status is ERROR when the status is 500 or more, and unset otherwise.

  • A sync: a span of kind INTERNAL named fsync, the child of the request's, for each time it waits on its writes reaching the disk. Every request whose answer says a commit was made (commit in its body) has at least one.

  • A request the server makes, to any target above: a span of kind CLIENT named with its HTTP method, as POST, with http.request.method, server.address, server.port, url.full (without its query, fragment or credentials), and http.response.status_code when it was answered, error.type (the status as a string, or timeout) when it failed, and status ERROR then. One made for a request is the child of the request's span; one made on its own, as a crawl or settling a handle change after a start, is the root of a trace of its own. Each such request carries a traceparent naming this span, with the trace's sampled flag.

No attribute holds a query, a header, a body or anything the log may not.

Export

Spans are exported in the background, in batches, as the OTEL_BSP_* variables say. No request waits on an export: a receiver that refuses connections, or takes them and never answers, changes no answer and no request's time. A full queue drops new spans; an export that fails may be tried again until OTEL_BSP_EXPORT_TIMEOUT has passed since its first try, and its spans are then dropped. Once the receiver answers again, the next export goes to it, within OTEL_BSP_EXPORT_TIMEOUT.

Performance

harness/perf.mjs runs one workload against the reference and against the server, each in a process of its own (the reference in a Node process of its own, with its PLC directory, which the server uses too), with the rate limits off. The workload is in parts, run in order:

PartWhatRequestsAt once
createAccountnew accounts, each with a password404
createRecordrecords of about 100 bytes, spread over the accounts200016
getRecordrecords written, chosen at random400016
getRepoan account's whole repository, of 50 records2008
firehose20 subscribers to subscribeRepos from now, each from an address of its own, then writes one at a time; each time is from sending the write to a subscriber receiving its #commit200 writes, 4000 deliveries1 write
uploadBlob64 KiB of random bytes, application/octet-stream2008

Each part is run in three rounds of a third of its requests, the sides taking turns: the reference's first round, the server's, the reference's second, and so on. A round's throughput is its requests, or for firehose its deliveries, a second over the round's whole time. Each side is judged by its best round: the lowest 50th and 99th percentile of a round's times, and the highest throughput.

Note: other work on the machine only ever makes a round slower, so the best of three is the nearest to what the side can do, and the turns spread a disturbance over both sides.

A part's ratios are the server's best over the reference's, in the same run: its p99 ratio, of the best 99th percentiles, and its throughput ratio, of the best throughputs.

Note: the two sides take turns on the same machine, so a slower or busier machine slows both; the ratios keep what is the server's own and drop most of what is the machine's, which times in milliseconds would not.

harness/perf-baseline.json is the baseline: each part's ratios as runs on the runner measured them, the nightly's or the perf workflow's (the same run, asked for alone), each part's the weakest of at least five runs (the highest p99 ratio and the lowest throughput ratio), with the runs named in it. The run prints each part's ratios beside the baseline's, and all of them on one line as the baseline holds them. A pull that makes the server faster raises the baseline, from what those runs print; one that lowers it says why. A part the baseline does not name is held to the reference alone, as if its ratios were 1.

Note: the weakest of several runs, rather than one, so that one lucky run does not make the baseline more than the server does on most nights; on a laptop with other work on it, three of nine runs fell short of the weakest of the others by a little, so five is the least.

The server must, for every part:

  • have a p99 ratio no more than 1.5 times the baseline's, or than 1 where the baseline's is above 1: never a best 99th percentile more than 1.5 times the reference's, and no more than 1.5 times the baseline's where the server is faster.

    Note: on a shared machine, the 99th percentiles of one side's three rounds of a part spread by 1.05 to 1.4 times where the part is bound by the server's own work (uploadBlob, firehose, createAccount), and further where other work lands on a round; the best of three takes out most of that, the weakest of several runs in the baseline more, and 1.5 the rest, so a server no slower than it was passes while one twice as slow fails.

  • and a throughput ratio at least 0.8 times the baseline's, or than 1 where the baseline's is below 1.

    Note: where a part is bound by the same work on both sides (a password hash, the directory, a write's flush, the firehose's fan-out of writes made one at a time), the two sides' best throughputs differ from run to run by a few percent either way; a floor of 1.0 turns that noise into failures of a server that is level, while 0.8 still fails one that is slower. Where the server is many times faster (getRecord, getRepo), its ratio moves further from run to run, which the weakest of several runs takes in.

Then, against the server alone, subscribers come and go: cohorts of 20 subscribers to subscribeRepos from now, each from an address of its own, the same 20 addresses each time (LIMITS.md: 16 at most from one). In a cohort, all 20 connect, one putRecord is made, each subscriber receives its #commit within 10 s, and all 20 close and see their connections closed within 10 s. One cohort warms the server, and 20 more follow. The server process's open file descriptors, threads and memory are read 15 s after the first cohort, and after the last: on Linux from /proc/<pid>/fd and /proc/<pid>/status (its memory resident and swapped out, VmRSS and VmSwap), on macOS from lsof, ps and footprint (its footprint, resident and compressed). After the last, they are read again every quarter second, for up to 20 s, taking the least of each reading, until all three are within these bounds. The server must, after the last cohort, hold:

  • no more open file descriptors than after the first;

  • no more threads than after the first;

    Note: the server's idle threads end 10 s after their last work, and their memory goes back with them: read at once after the first cohort, the threads the parts before used are still there, and as many threads held for good would be hidden. A cohort starts some 20 again, and the up to 20 s after the last lets them end. A thread or two the server starts and ends on its own may come some seconds later; what is held for good is in every reading, and such a thread in only some, so the least of each leaves it out. Read so, the descriptors and threads after the last cohort come back to those after the first, or fewer, so neither has any allowance.

  • and no more memory than after the first and 8 MiB.

    Note: the allocator keeps some of what was freed for a while. On macOS, once the idle threads had ended, the server's resident memory after the last cohort was 0 to 6.1 MiB above the first's in three runs (up to 13 MiB before they had), and its footprint 0.2 and 2.3 MiB above in two. 400 subscribers each holding 256 KiB for good would be 100 MiB more. Resident memory alone is not enough on macOS, which compresses pages no one touches, as memory held for good is not: it missed such a leak, which the footprint showed as 107 MiB more.

The reference is not held to this.

The run exits 1 if any of these fails, naming the part, or the descriptors, threads or memory.

How it is checked

harness/checks/operations.mjs and harness/checks/logging.mjs, at the end of every compare.mjs run, of any kind. The server is started with PDS_METRICS_PORT set, PDS_LOG_DIR a directory of its own apart from the data directory, PDS_LOG_KEEP_BYTES of 262144, so that the kept lines are let go of in a run, its traces going to a stub receiver in the harness (harness/lib/otlp.mjs) with a header in OTEL_EXPORTER_OTLP_HEADERS, OTEL_SERVICE_NAME, and OTEL_BSP_SCHEDULE_DELAY of 200 ms and OTEL_BSP_EXPORT_TIMEOUT of 3 s. Every request the harness sends the server carries a traceparent of its own, one in ten unsampled, and is noted with its answer and how long it took; and every password, token, code, cookie, email and key the run handles is kept.

In a run of random steps, the receiver stops answering a third of the way through, is gone at half way, once an export could have timed out, and is back at two thirds; the steps go on. Then, at the end:

  • Health: GET, HEAD and POST /xrpc/_health, and GET with a query, against the reference: the status, Content-Type, and the body's form; /metrics on the server's own port, 404 on both; with --limits, _health past global on both.
  • The root: GET, HEAD and POST /, and GET with a query, against the reference: the status and Content-Type; the server's text held to "The root" above, read from this page, with its own origin; and no RateLimit-* headers on it.
  • Routes: paths under /xrpc/ of no method served, one that is no NSID, and another path, each of route _unknown.
  • Trace context: _health with no traceparent, ones not valid (no parse, a trace id of 0, version ff, upper case), an unsampled one, and one of version 01 with more after it: each line's trace_id is new, or the one sent; each span the root of a new trace, or the child of the one sent; none for the unsampled.
  • Metrics: an account and records of the harness's own, then: one more subscriber while it subscribes and pds_firehose_seq the last event's; pds_accounts as listRepos lists; pds_log_bytes no more than the data directory holds, and grown across a record written while no file in the data directory was made, deleted, replaced or cut short, the directory read before the metrics and after them, around the write (a record written again, up to 20 in all, while one was); every family's type, labels, label values and buckets; pds_requests_total equal to the request lines since the server started, by route and status, and no less than the harness's requests; each route's _count equal to its requests; pds_rate_limited_total equal to the 429s; the syncs grown by the records; a PLC request for every account made; 404 for another path.
  • The admin's reads: _logs and _stats with no Authorization, a bearer token, the user admin with another password, and another user: each 401 AuthenticationRequired; POST, 400 InvalidRequest; _logs with a level not listed, a minMs of -1, a limit of 0, of 501 and of x: each 400 InvalidRequest. Then _logs with no parameter, and with a limit, its lines the last that many written to standard output by the starts that kept them, before the read; by the trace of each of the last requests the harness sent, that request's request line alone; by msg, method, level and minMs, each and together, the last lines of standard output that match, and none that does not. _stats between two scrapes of /metrics, each number between the two scrapes', started the start line's time, routes and outbound in their order, and the second scrape's _stats count one more than the first's. At every check, the files in PDS_LOG_DIR take no more than PDS_LOG_KEEP_BYTES, and _logs reads back the newest lines of half that, across a kill and a start. Started with PDS_LOG_DIR unset, _logs is 404 LogNotKept and _stats answers.
  • Spans: once the server is shut down with SIGTERM, which it must exit 0 on within 15 s: for every request the harness sent and had answered, one SERVER span in its trace, the child of its traceparent, with the name, attributes, status, resource and times the spec gives, or none if unsampled; an fsync span under each that made a commit; a CLIENT span under each createAccount, of its call for the DID (the POST of its genesis operation to the directory, or, for an account moving in with its did, the GET of the DID's document, from the directory or, for a did:web, from its host, MIGRATION.md); a root span for every request line of a client that sent no traceparent; no SERVER span without a line; every export a protobuf POST /v1/traces with the header. Spans of requests near the outage or just before a kill may be lost, and those of an export the server says it dropped (Export, above): a span missing, with a trace export failed line from the request's start until OTEL_BSP_SCHEDULE_DELAY, twice OTEL_BSP_EXPORT_TIMEOUT and 1.5 s after its answer; or with the last such line before its start within a minute and the timeout, and no export taken since, while exports kept failing unlogged. While the receiver hung, the requests' median time is under a second, and a trace export failed line is written.
  • Starts: at warn, with a PLC directory that answers 500 and OTEL_SERVICE_NAME unset: createAccount is refused, the directory gets a traceparent in the request's trace, a plc failure line names the status and the DID, its CLIENT span is an error of unknown_service:pds, and an outbound error is counted. Killed, then started with no OTLP endpoint: no span. Where the machine has an IPv4 address on an interface other than the loopback, the server started with PDS_BIND_ADDRESS unset does not answer _health on it, and started with 0.0.0.0 answers on it and on 127.0.0.1; where the machine has IPv6's loopback, started with ::1 it answers on [::1] and not on 127.0.0.1 (PDS.md). Then PDS_LOG_LEVEL=loud, the protocols grpc and http/json, a PDS_METRICS_PORT that is no number and one equal to PDS_PORT, a PDS_LOG_KEEP_BYTES of 65535 and of many, a PDS_LOG_DIR under a file, a PDS_BIND_ADDRESS of localhost and of [::1], and one of 203.0.113.1, an address no interface of the machine has: each exits 1 with why on standard error.
  • The log, whole: every line one JSON object with the fields its event has, at no level below the start's PDS_LOG_LEVEL; each start's lines starting with start, with recovered after a start the last server did not shut down and not otherwise, and ending with shutdown after a SIGTERM; one request line for each request the harness had answered, but one whose answer was cut short once its server was signalled to stop, found by its trace id, with its route and status, an ms no longer than the harness waited, and the did of its token for a method that acts as one; of each getBlob answered 200 or 500, repo and cid the did and cid it asked for, and of one answered 200, bytes the bytes the harness read of it, or, of one whose answer ended short (BLOBS.md, "How it is checked"), no fewer than it read and fewer than its Content-Length; no repo, cid or bytes on any other request line; nothing on standard error but from a start that exited 1; and none of the run's secrets, nor the text of a record the harness wrote, anywhere in standard output, standard error or the files in PDS_LOG_DIR.

harness/perf.mjs is a gate of its own, run alone after every other.