--- title: Observability section: Operations order: 102 summary: What HUD logs, its traces and metrics, the lines it keeps for its operators, and how to read them. --- `hud serve` logs JSON lines on stderr, one per event, at the level `RUST_LOG` sets (`info` by default). Every request is in a trace, and every line logged while serving it names that trace. So one `trace_id` gives you the request, its tool call and the calls it made. With `HUD_LOGS=on`, HUD also keeps every line in a SQLite file for its operators, who read it back over XRPC or MCP without a shell on the host. ## The lines Each line has `timestamp`, `level` (`ERROR`, `WARN`, `INFO`, `DEBUG`), `message`, `target` and, inside a request, `trace_id`. The other fields follow OpenTelemetry's semantic conventions where there's one, so a collector can take the lines as they are. | `message` | Level | Fields | | --- | --- | --- | | `request` | `info`. `warn` past `HUD_SLOW_MS` or for a 502 or 504, `error` for any other 5xx, `debug` for `/_health` | `http.route`, `http.request.method`, `url.path`, `url.query`, `user_agent.original`, `http.response.status_code`, `http.response.body.size`, `duration_ms`, `trace_id`, `error` | | `mcp tool` | `info`. `warn` past `HUD_SLOW_MS` | `gen_ai.tool.name`, `did`, `kit`, `is_error`, `reason`, `duration_ms` | | `mcp tool failed` | `warn` | `gen_ai.tool.name`, `did`, `kit`, `is_error`, `error`, `duration_ms` | | `mcp error` | `info` | `rpc.method`, `error` | | `outgoing` | `debug`. `warn` when it failed, timed out, answered 5xx or was slow | `server.address`, `http.request.method`, `url.path`, `rpc.method`, `http.response.status_code`, `error.type`, `duration_ms` | | `summary: requests` | `info`, each minute | `count`, `failed`, `slow`, `mean_ms`, `max_ms` | | `summary: outgoing` | `info`, each minute, per host | `server.address`, `count`, `failed`, `slow`, `mean_ms`, `max_ms` | | `summary: cache` | `info`, each minute, per cache | `cache`, `hits`, `misses` | | `logs read` | `info` | `did`, `reason` (what was asked) | | `hud starting` | `info` | `version`, `commit` | | `panicked` | `error` | `error`, with the file and line | `request` is one wide line per request, written when the answer goes out. `http.route` is the route's template (`/blob/{did}/{cid}`), and each of HUD's own XRPC methods by its path (`/xrpc/place.hud.getCanvas`). A request no route takes is `fallback` in the counts, or `web` where HUD hands it to the web app ([the web app's pages](internals.md#the-web-apps-pages)). `http.response.body.size` is the answer's size before HUD compresses it ([how it works](internals.md#compression)), and it's missing for a web app's page, which streams. The workspace's event stream (`getWorkspaceEvents`) is never slow, since it stays open on purpose. `error` names an XRPC method's error, the blob proxy's reason, and for a 5xx why it failed. A 502 or 504 is a service HUD called failing, not HUD (a PDS for the blob proxy, the favicon service for an icon, the web app for a page), so it's a warning, and the call's own `outgoing` line names the service. `url.query` masks the value of any parameter a secret could be in (`code`, `state`, `iss`, `token`, `access_token`, `refresh_token`, `password`, `secret`, `key`) as `_`, and cuts it at 200 characters. A request under `/oauth` logs no query. `mcp tool` is one line per tool call, a `tools/call` on `/mcp` (the iOS app, agents) or a `place.hud.callTool` (the workspace's page), with the caller's DID when signed in. Its `kit` is the renderer its tree was fitted to, as it said itself: `ios 0.3.0`, `android 0.3.0`, `web 0.3.0`, `view` (an MCP host drawing HUD's view), `workspace` or `playground`. A client that said nothing has none, and is drawn for the oldest kit ([old builds and new views](views.md#old-builds-and-new-views)). A tool that answered an error is `mcp tool failed`, with the error the client saw. A call that asked the person to sign in has `reason: sign-in asked`. HUD never logs a tool's arguments or its result, since tokens, records and messages can be in them. `outgoing` is one line per call through hud-core's outbound client (`hud_core::net`), which every request to a PDS, PLC, an AppView, a feed generator, the web app or any other service goes through. `rpc.method` is the XRPC method, for a path under `/xrpc/`. `error.type` is why it failed before an answer (`timeout`, `connect`, `body` or `request`, with the cause), or the status of a 5xx. A call its wait cut off is `timeout`. One dropped before then, because what asked for it stopped waiting (a search typed past, the loser of a race), is `abandoned`: no failure, at debug, and not counted as failed. Calls are at debug, so a busy minute doesn't drown the log. Set `hud::outgoing=debug` to see each one. Other lines (sign-ins, canvases, deletions) are as their code writes them, with the same `timestamp`, `level`, `message` and `trace_id`. HUD never logs a header, a token, a cookie, a body or an outbound call's query. It logs DIDs. It never logs an IP address. ## Traces A request is in the trace its W3C `traceparent` header names, if valid, or a new one. The answer carries it back as `x-trace-id`. Every line logged while serving the request names it as `trace_id`, including the lines of a task the request starts (`hud_core::telemetry::spawn`). Every outbound call sends its own `traceparent` in the same trace. ## Metrics and the level Set `HUD_METRICS` to a loopback address and port (`127.0.0.1:9464`) to serve two things on a listener of their own. HUD refuses any other address, so nothing outside the machine (or the container) reaches it. - `GET /metrics` gives what was counted since the start, in Prometheus's text format: `http_server_request_duration_seconds` by route, method and status, `http_client_request_duration_seconds` by host and outcome (a status or why it failed), `hud_cache_lookups_total` by cache and result (`hit`, `miss`), `hud_build_info` and `process_start_time_seconds`. HUD counts up to 200 hosts and the rest as `other`. - `PUT /log` with a `RUST_LOG` directive as its body changes what is logged until the next restart. The image has no HTTP client, so reach them from a container in the server's network namespace: ```sh docker run --rm --network container:hud-hud-1 curlimages/curl -s http://127.0.0.1:9464/metrics | grep -v '^#' docker run --rm --network container:hud-hud-1 curlimages/curl -s -X PUT http://127.0.0.1:9464/log -d 'info,hud::outgoing=debug' docker run --rm --network container:hud-hud-1 curlimages/curl -s -X PUT http://127.0.0.1:9464/log -d info ``` `PUT` it back to `info` after, or restart, which goes back to `RUST_LOG`. ## Kept lines With `HUD_LOGS=on`, HUD keeps every line it writes to stderr in `HUD_DATA_DIR/logs.db` too, a SQLite file of mode 0600 (its WAL gets the same). A task writes them every second, off the requests' way. If writes can't keep up, HUD drops the oldest lines waiting (past 50,000) and never makes a request wait. It keeps the last 500,000 lines at info and below (`HUD_LOGS_KEEP`), and no line longer than 30 days, pruning each minute. The file holds what stderr has and nothing more, so no token, query secret or body. Lines logged before the store opens (`hud starting`) are on stderr only. Every server writing to the same `HUD_DATA_DIR` writes to the one file, so a deploy that starts a new container on the same volume loses none of the old one's lines. ### Reading the kept lines HUD's operators are the accounts `HUD_OPERATOR_DIDS` names. They read the kept lines with `place.hud.getLogs`, as the workspace's web session or with a HUD access token as `Bearer`, or with the MCP tool `admin_logs`, which takes the same arguments and is listed to them alone. Anyone else gets `Forbidden` (a tool error from `admin_logs`). | Parameter | | | --- | --- | | `trace` | All of one request's lines, by its `x-trace-id` | | `message` | Only lines with this message | | `level` | Only lines at this level or above | | `minMs` | Only lines with a `duration_ms` of this or more | | `limit` | How many, the last of them. At most 500. Unless given, 500 for a trace and 100 otherwise | A read needs one of `trace`, `message`, `level` and `minMs`, and answers the lines oldest first. Each read is itself logged as `logs read`, with who asked and for what. With `HUD_LOGS` off, the method answers `NotFound`. ```sh curl -s -H "Authorization: Bearer $TOKEN" 'https://hud.place/xrpc/place.hud.getLogs?trace=4bf92f3577b34da6a3ce929d0e0e4736' curl -s -H "Authorization: Bearer $TOKEN" 'https://hud.place/xrpc/place.hud.getLogs?level=warn&minMs=2000' ``` ## Reading the log Docker keeps a container's stderr. Read it with `docker logs`, or with `journalctl` where Docker's log driver is `journald`, and filter it with jq: ```sh docker logs hud-hud-1 2>&1 | jq -c 'select(.level == "WARN" or .level == "ERROR")' docker logs hud-hud-1 2>&1 | jq -c 'select(.trace_id == "4bf92f3577b34da6a3ce929d0e0e4736")' docker logs hud-hud-1 2>&1 | jq -c 'select(.message == "mcp tool" or .message == "mcp tool failed") | {t: .timestamp, tool: ."gen_ai.tool.name", did, error, ms: .duration_ms}' journalctl -o cat CONTAINER_NAME=hud-hud-1 --since -1h | jq -Rc 'fromjson? | select(.duration_ms > 1000)' journalctl -o cat CONTAINER_NAME=hud-hud-1 | jq -Rc 'fromjson? | select(.message | startswith("summary"))' ``` The default `json-file` driver loses a container's log when a deploy replaces the container. The `journald` driver keeps it in the host's journal for as long as the journal keeps anything, and `HUD_LOGS` keeps it in `HUD_DATA_DIR`. ## What isn't built - The OpenTelemetry SDK and OTLP export. They pay off with a backend to send to, and HUD has none. The ids and field names are W3C's and OpenTelemetry's, so adding the layer later changes no line's meaning. - Spans for each call inside a request. A trace id on every line, and a line per call, give the part of a trace you read by hand. A waterfall needs a trace store. - Sampling. At HUD's traffic every request's line costs less than deciding which to keep, and the calls, the bulk, are at debug.