# Operating Once the server runs, you'll want to know what it tells you, how it stops and starts again, and what it needs from the machine. The rules are in [OPERATIONS.md](../../spec/OPERATIONS.md), [STORE.md](../../spec/STORE.md) and [LIMITS.md](../../spec/LIMITS.md). ## The log The server writes its log to standard output as one JSON object per line. There's a line for every request it answers, and lines for events such as a start, a recovery or a shutdown. It never logs passwords, tokens, codes, emails or keys. Standard error is only for why `pds serve` exits 1, for example when a variable doesn't parse. `PDS_LOG_LEVEL` sets the lowest level written: `debug`, `info` (the default), `warn` or `error`, or `off` for none ([OPERATIONS.md](../../spec/OPERATIONS.md#the-log)). Set `PDS_LOG_DIR` to keep each line in that directory too. Then you can read it back without reaching the machine, and it outlives a rebuilt container. Its files take at most `PDS_LOG_KEEP_BYTES` (64 MiB by default), and the oldest lines go first. Keep it apart from `PDS_DATA_DIR`, on its own volume, since no backup needs it ([OPERATIONS.md](../../spec/OPERATIONS.md#kept)). ## Reading the log and stats When `PDS_ADMIN_PASSWORD` is set, the admin can read what the server logged and counted over its own port. Use `GET /xrpc/_logs` and `GET /xrpc/_stats` with `Authorization: Basic` and the user `admin`, the CLI's `pds admin logs` and `pds admin stats`, or the MCP tools `admin_logs` and `admin_stats` ([OPERATIONS.md](../../spec/OPERATIONS.md#the-admins-reads)): ```sh export PDS_URL=https://pds.example.com PDS_ADMIN_PASSWORD=… pds admin logs # the last 100 lines kept pds admin logs --level warn --limit 20 # the last warnings and errors pds admin logs --trace 4bf92f3577b34da6a3ce929d0e0e4736 # one request pds admin logs --method com.atproto.repo.createRecord --min-ms 500 pds admin stats # requests by route, since the start ``` `_logs` answers with the last `limit` lines (100 by default, at most 500) kept in `PDS_LOG_DIR` that match every filter given, oldest first. Without `PDS_LOG_DIR` it answers `404 LogNotKept`. `_stats` answers with what the metrics have counted since the server started, as JSON. That's requests by route and status with their mean times, the requests it made, its syncs, its accounts and its firehose. It doesn't need `PDS_METRICS_PORT`. ## Health `GET /xrpc/_health` answers `200` with `{"version": "..."}` while the server serves, and it isn't rate limited. Point a load balancer or a container's health check at it ([OPERATIONS.md](../../spec/OPERATIONS.md#health)). ## Metrics Set `PDS_METRICS_PORT` to serve Prometheus metrics at `/metrics` on `127.0.0.1` and that port. It has no auth, and it's never on the server's own port. When it's unset, nothing listens. A scrape doesn't count as a request to the server, so it gets no log line, no span and no rate limit ([OPERATIONS.md](../../spec/OPERATIONS.md#metrics)). ## Traces Set `OTEL_EXPORTER_OTLP_ENDPOINT` (or `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`) and the server exports a span for each request, and spans within it, by OTLP over HTTP with protobuf. The other `OTEL_*` variables set headers, compression, the service name and batching. A receiver that's down doesn't slow any request ([OPERATIONS.md](../../spec/OPERATIONS.md#traces)). ## Stopping On SIGTERM or SIGINT the server stops taking connections and gives the requests under way up to 10 seconds to finish. Then it closes the firehose's subscriptions, syncs what it has written, exports its spans for up to 5 seconds, logs `shutdown`, and exits 0 ([OPERATIONS.md](../../spec/OPERATIONS.md#events)). If the server is killed or the machine loses power, no answered write is lost. The next start recovers and logs `recovered` ([DURABILITY.md](../../spec/DURABILITY.md)). After a power cut, a write that was never answered may turn up whole with nothing kept of its event. That event is never sent. Instead, the next time its account is used, the firehose has the events an activation makes (its `#sync` among them), once ([STORE.md](../../spec/STORE.md#scale)). ## Syncing writes A write is on disk before the server answers it, and writes waiting at the same time share a sync. `PDS_SYNC` picks how: - `together` (the default): every write goes where its account keeps it, and also to one shared journal in `PDS_DATA_DIR/journal/`. The write waits for the journal's sync, so one sync answers every write waiting when it began, whatever accounts they're for. In the background, the server makes what the journal holds durable where it's kept, two seconds after the last write or once the journal holds 16 MiB, and then lets it go. A start after a kill or a power cut restores what's missing from the journal before it serves, and a shutdown leaves none of it. Its files take a few bytes when the server is quiet, and up to twice what was written in the last seconds when it's busy. - `each`: each account's writes are made durable by their own sync, which only the writes waiting on that account share. Any other value makes the server exit 1 as it starts, with `pds: PDS_SYNC: ` and the reason. The log's `start` line shows which one it is, as `sync`. Either is safe, and a server started with one carries on from a data directory the other left. `together` answers more writes a second when many accounts are written at once, such as on a disk whose sync takes a millisecond or more ([DURABILITY.md](../../spec/DURABILITY.md#writes-waiting-at-once)). ## Upgrades Stop the server, put the new binary or image in place, and start it on the same data directory. The server starts on a data directory the last release wrote, whether it was shut down or killed, and carries on from it. Its spaces carry on too when 0.2.0 or a later release wrote it; the spaces 0.1.0 kept are let go on the first start. It may refuse a data directory a later version wrote and exit 1, but it never serves part of one ([STORE.md](../../spec/STORE.md#upgrades)). If a release from before the firehose's index wrote the data directory, the first start reads every account's files once for the earlier events, and takes that much longer to serve. So back up the data directory before an upgrade. Once a later version has written to it, the earlier one may not start on it. ## Disk The data directory stays within three times what its accounts hold, plus a constant for each account, since the server compacts it in the background ([STORE.md](../../spec/STORE.md#bounds)). When free space on the filesystem of `PDS_DATA_DIR` drops below `PDS_MIN_FREE_BYTES` (256 MiB by default), the server refuses writes with `507 InsufficientStorage` but keeps serving reads ([LIMITS.md](../../spec/LIMITS.md#disk)). Watch the free space, and grow the volume before it gets there. ## Memory The server's resident memory stays within its base, plus `PDS_CACHE_BYTES` (256 MiB by default), plus 4 KiB for each account, no matter what the accounts hold. Set `PDS_CACHE_BYTES` to what the machine can spare ([STORE.md](../../spec/STORE.md#scale)). ## Open files When it starts, the server raises its file descriptor limit to the hard limit it was given. It keeps at most a quarter of them open as accounts' logs and a quarter as their tables, each capped at 4096. If an account's files aren't among them, the server opens them again the next time it's read or written. It serves every account under a hard limit of 256, and a limit of 16384 or more lets it hold the most it will ([LIMITS.md](../../spec/LIMITS.md#open-files-and-threads)). ## Backups [Deploying](deploying.md#backups) explains how to back up the server, and what a start on a backup does.