~ knotless Search/
↑↓ move↵ open⇧↵ all results
0
0
163 lines · 7.6 KB Raw History

Operating

Once the server runs, you'll want to know what it tells you, how it stops and starts again, and what it needs from the machine. The rules are in OPERATIONS.md, STORE.md and LIMITS.md.

The log

The server writes its log to standard output as one JSON object per line. There's a line for every request it answers, and lines for events such as a start, a recovery or a shutdown. It never logs passwords, tokens, codes, emails or keys. Standard error is only for why pds serve exits 1, for example when a variable doesn't parse.

PDS_LOG_LEVEL sets the lowest level written: debug, info (the default), warn or error, or off for none (OPERATIONS.md).

Set PDS_LOG_DIR to keep each line in that directory too. Then you can read it back without reaching the machine, and it outlives a rebuilt container. Its files take at most PDS_LOG_KEEP_BYTES (64 MiB by default), and the oldest lines go first. Keep it apart from PDS_DATA_DIR, on its own volume, since no backup needs it (OPERATIONS.md).

Reading the log and stats

When PDS_ADMIN_PASSWORD is set, the admin can read what the server logged and counted over its own port. Use GET /xrpc/_logs and GET /xrpc/_stats with Authorization: Basic and the user admin, the CLI's pds admin logs and pds admin stats, or the MCP tools admin_logs and admin_stats (OPERATIONS.md):

export PDS_URL=https://pds.example.com PDS_ADMIN_PASSWORD=…
pds admin logs                                   # the last 100 lines kept
pds admin logs --level warn --limit 20           # the last warnings and errors
pds admin logs --trace 4bf92f3577b34da6a3ce929d0e0e4736   # one request
pds admin logs --method com.atproto.repo.createRecord --min-ms 500
pds admin stats                                  # requests by route, since the start

_logs answers with the last limit lines (100 by default, at most 500) kept in PDS_LOG_DIR that match every filter given, oldest first. Without PDS_LOG_DIR it answers 404 LogNotKept. _stats answers with what the metrics have counted since the server started, as JSON. That's requests by route and status with their mean times, the requests it made, its syncs, its accounts and its firehose. It doesn't need PDS_METRICS_PORT.

Health

GET /xrpc/_health answers 200 with {"version": "..."} while the server serves, and it isn't rate limited. Point a load balancer or a container's health check at it (OPERATIONS.md).

Metrics

Set PDS_METRICS_PORT to serve Prometheus metrics at /metrics on 127.0.0.1 and that port. It has no auth, and it's never on the server's own port. When it's unset, nothing listens. A scrape doesn't count as a request to the server, so it gets no log line, no span and no rate limit (OPERATIONS.md).

Traces

Set OTEL_EXPORTER_OTLP_ENDPOINT (or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT) and the server exports a span for each request, and spans within it, by OTLP over HTTP with protobuf. The other OTEL_* variables set headers, compression, the service name and batching. A receiver that's down doesn't slow any request (OPERATIONS.md).

Stopping

On SIGTERM or SIGINT the server stops taking connections and gives the requests under way up to 10 seconds to finish. Then it closes the firehose's subscriptions, syncs what it has written, exports its spans for up to 5 seconds, logs shutdown, and exits 0 (OPERATIONS.md).

If the server is killed or the machine loses power, no answered write is lost. The next start recovers and logs recovered (DURABILITY.md). After a power cut, a write that was never answered may turn up whole with nothing kept of its event. That event is never sent. Instead, the next time its account is used, the firehose has the events an activation makes (its #sync among them), once (STORE.md).

Syncing writes

A write is on disk before the server answers it, and writes waiting at the same time share a sync. PDS_SYNC picks how:

  • together (the default): every write goes where its account keeps it, and also to one shared journal in PDS_DATA_DIR/journal/. The write waits for the journal's sync, so one sync answers every write waiting when it began, whatever accounts they're for. In the background, the server makes what the journal holds durable where it's kept, two seconds after the last write or once the journal holds 16 MiB, and then lets it go. A start after a kill or a power cut restores what's missing from the journal before it serves, and a shutdown leaves none of it. Its files take a few bytes when the server is quiet, and up to twice what was written in the last seconds when it's busy.
  • each: each account's writes are made durable by their own sync, which only the writes waiting on that account share.

Any other value makes the server exit 1 as it starts, with pds: PDS_SYNC: and the reason. The log's start line shows which one it is, as sync. Either is safe, and a server started with one carries on from a data directory the other left. together answers more writes a second when many accounts are written at once, such as on a disk whose sync takes a millisecond or more (DURABILITY.md).

Upgrades

Stop the server, put the new binary or image in place, and start it on the same data directory. The server starts on a data directory the last release wrote, whether it was shut down or killed, and carries on from it. Its spaces carry on too when 0.2.0 or a later release wrote it; the spaces 0.1.0 kept are let go on the first start. It may refuse a data directory a later version wrote and exit 1, but it never serves part of one (STORE.md). If a release from before the firehose's index wrote the data directory, the first start reads every account's files once for the earlier events, and takes that much longer to serve. So back up the data directory before an upgrade. Once a later version has written to it, the earlier one may not start on it.

Disk

The data directory stays within three times what its accounts hold, plus a constant for each account, since the server compacts it in the background (STORE.md).

When free space on the filesystem of PDS_DATA_DIR drops below PDS_MIN_FREE_BYTES (256 MiB by default), the server refuses writes with 507 InsufficientStorage but keeps serving reads (LIMITS.md). Watch the free space, and grow the volume before it gets there.

Memory

The server's resident memory stays within its base, plus PDS_CACHE_BYTES (256 MiB by default), plus 4 KiB for each account, no matter what the accounts hold. Set PDS_CACHE_BYTES to what the machine can spare (STORE.md).

Open files

When it starts, the server raises its file descriptor limit to the hard limit it was given. It keeps at most a quarter of them open as accounts' logs and a quarter as their tables, each capped at 4096. If an account's files aren't among them, the server opens them again the next time it's read or written. It serves every account under a hard limit of 256, and a limit of 16384 or more lets it hold the most it will (LIMITS.md).

Backups

Deploying explains how to back up the server, and what a start on a backup does.