chadtmiller.com/monolith
An atproto PDS
0
0
# Operating
Once the server runs, you'll want to know what it tells you, how it stops
and starts again, and what it needs from the machine. The rules are in
[OPERATIONS.md](../../spec/OPERATIONS.md), [STORE.md](../../spec/STORE.md)
and [LIMITS.md](../../spec/LIMITS.md).
## The log
The server writes its log to standard output as one JSON object per line.
There's a line for every request it answers, and lines for events such as
a start, a recovery or a shutdown. It never logs passwords, tokens, codes,
emails or keys. Standard error is only for why `pds serve` exits 1, for
example when a variable doesn't parse.
`PDS_LOG_LEVEL` sets the lowest level written: `debug`, `info` (the
default), `warn` or `error`, or `off` for none
([OPERATIONS.md](../../spec/OPERATIONS.md#the-log)).
Set `PDS_LOG_DIR` to keep each line in that directory too. Then you can
read it back without reaching the machine, and it outlives a rebuilt
container. Its files take at most `PDS_LOG_KEEP_BYTES` (64 MiB by
default), and the oldest lines go first. Keep it apart from
`PDS_DATA_DIR`, on its own volume, since no backup needs it
([OPERATIONS.md](../../spec/OPERATIONS.md#kept)).
## Reading the log and stats
When `PDS_ADMIN_PASSWORD` is set, the admin can read what the server
logged and counted over its own port. Use `GET /xrpc/_logs` and
`GET /xrpc/_stats` with `Authorization: Basic` and the user `admin`, the
CLI's `pds admin logs` and `pds admin stats`, or the MCP tools
`admin_logs` and `admin_stats`
([OPERATIONS.md](../../spec/OPERATIONS.md#the-admins-reads)):
```sh
export PDS_URL=https://pds.example.com PDS_ADMIN_PASSWORD=…
pds admin logs # the last 100 lines kept
pds admin logs --level warn --limit 20 # the last warnings and errors
pds admin logs --trace 4bf92f3577b34da6a3ce929d0e0e4736 # one request
pds admin logs --method com.atproto.repo.createRecord --min-ms 500
pds admin stats # requests by route, since the start
```
`_logs` answers with the last `limit` lines (100 by default, at most 500)
kept in `PDS_LOG_DIR` that match every filter given, oldest first. Without
`PDS_LOG_DIR` it answers `404 LogNotKept`. `_stats` answers with what the
metrics have counted since the server started, as JSON. That's requests by
route and status with their mean times, the requests it made, its syncs,
its accounts and its firehose. It doesn't need `PDS_METRICS_PORT`.
## Health
`GET /xrpc/_health` answers `200` with `{"version": "..."}` while the
server serves, and it isn't rate limited. Point a load balancer or a
container's health check at it
([OPERATIONS.md](../../spec/OPERATIONS.md#health)).
## Metrics
Set `PDS_METRICS_PORT` to serve Prometheus metrics at `/metrics` on
`127.0.0.1` and that port. It has no auth, and it's never on the server's
own port. When it's unset, nothing listens. A scrape doesn't count as a
request to the server, so it gets no log line, no span and no rate limit
([OPERATIONS.md](../../spec/OPERATIONS.md#metrics)).
## Traces
Set `OTEL_EXPORTER_OTLP_ENDPOINT` (or `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`)
and the server exports a span for each request, and spans within it, by
OTLP over HTTP with protobuf. The other `OTEL_*` variables set headers,
compression, the service name and batching. A receiver that's down doesn't
slow any request ([OPERATIONS.md](../../spec/OPERATIONS.md#traces)).
## Stopping
On SIGTERM or SIGINT the server stops taking connections and gives the
requests under way up to 10 seconds to finish. Then it closes the
firehose's subscriptions, syncs what it has written, exports its spans for
up to 5 seconds, logs `shutdown`, and exits 0
([OPERATIONS.md](../../spec/OPERATIONS.md#events)).
If the server is killed or the machine loses power, no answered write is
lost. The next start recovers and logs `recovered`
([DURABILITY.md](../../spec/DURABILITY.md)). After a power cut, a write
that was never answered may turn up whole with nothing kept of its event.
That event is never sent. Instead, the next time its account is used, the
firehose has the events an activation makes (its `#sync` among them),
once ([STORE.md](../../spec/STORE.md#scale)).
## Syncing writes
A write is on disk before the server answers it, and writes waiting at the
same time share a sync. `PDS_SYNC` picks how:
- `together` (the default): every write goes where its account keeps it,
and also to one shared journal in `PDS_DATA_DIR/journal/`. The write
waits for the journal's sync, so one sync answers every write waiting
when it began, whatever accounts they're for. In the background, the
server makes what the journal holds durable where it's kept, two seconds
after the last write or once the journal holds 16 MiB, and then lets it
go. A start after a kill or a power cut restores what's missing from the
journal before it serves, and a shutdown leaves none of it. Its files
take a few bytes when the server is quiet, and up to twice what was
written in the last seconds when it's busy.
- `each`: each account's writes are made durable by their own sync, which
only the writes waiting on that account share.
Any other value makes the server exit 1 as it starts, with `pds:
PDS_SYNC: ` and the reason. The log's `start` line shows which one it is,
as `sync`. Either is safe, and a server started with one carries on from a
data directory the other left. `together` answers more writes a second
when many accounts are written at once, such as on a disk whose sync takes
a millisecond or more
([DURABILITY.md](../../spec/DURABILITY.md#writes-waiting-at-once)).
## Upgrades
Stop the server, put the new binary or image in place, and start it on the
same data directory. The server starts on a data directory the last
release wrote, whether it was shut down or killed, and carries on from it.
Its spaces carry on too when 0.2.0 or a later release wrote it; the spaces
0.1.0 kept are let go on the first start. It may refuse a data directory a
later version wrote and exit 1, but it never serves part of one
([STORE.md](../../spec/STORE.md#upgrades)). If a release from before the
firehose's index wrote the data directory, the first start reads every
account's files once for the earlier events, and takes that much longer to
serve. So back up the data directory before an upgrade. Once a later
version has written to it, the earlier one may not start on it.
## Disk
The data directory stays within three times what its accounts hold, plus a
constant for each account, since the server compacts it in the background
([STORE.md](../../spec/STORE.md#bounds)).
When free space on the filesystem of `PDS_DATA_DIR` drops below
`PDS_MIN_FREE_BYTES` (256 MiB by default), the server refuses writes with
`507 InsufficientStorage` but keeps serving reads
([LIMITS.md](../../spec/LIMITS.md#disk)). Watch the free space, and grow
the volume before it gets there.
## Memory
The server's resident memory stays within its base, plus
`PDS_CACHE_BYTES` (256 MiB by default), plus 4 KiB for each account, no
matter what the accounts hold. Set `PDS_CACHE_BYTES` to what the machine
can spare ([STORE.md](../../spec/STORE.md#scale)).
## Open files
When it starts, the server raises its file descriptor limit to the hard
limit it was given. It keeps at most a quarter of them open as accounts'
logs and a quarter as their tables, each capped at 4096. If an account's
files aren't among them, the server opens them again the next time it's
read or written. It serves every account under a hard limit of 256, and a
limit of 16384 or more lets it hold the most it will
([LIMITS.md](../../spec/LIMITS.md#open-files-and-threads)).
## Backups
[Deploying](deploying.md#backups) explains how to back up the server, and
what a start on a backup does.