chadtmiller.com/hud
A remote MCP server that shows atproto records as interactive UI inside MCP Apps hosts
place.hud.hudkit 0.3.0 Latest 5 packages
title: Observability section: Operations order: 102 summary: What HUD logs, its traces and metrics, the lines it keeps for its operators, and how to read them.
hud serve logs JSON lines on stderr, one per event, at the level RUST_LOG sets (info by default). Every request is in a trace, and every line logged while serving it names that trace. So one trace_id gives you the request, its tool call and the calls it made.
With HUD_LOGS=on, HUD also keeps every line in a SQLite file for its operators, who read it back over XRPC or MCP without a shell on the host.
The lines
Each line has timestamp, level (ERROR, WARN, INFO, DEBUG), message, target and, inside a request, trace_id. The other fields follow OpenTelemetry's semantic conventions where there's one, so a collector can take the lines as they are.
message | Level | Fields |
|---|---|---|
request | info. warn past HUD_SLOW_MS or for a 502 or 504, error for any other 5xx, debug for /_health | http.route, http.request.method, url.path, url.query, user_agent.original, http.response.status_code, http.response.body.size, duration_ms, trace_id, error |
mcp tool | info. warn past HUD_SLOW_MS | gen_ai.tool.name, did, kit, is_error, reason, duration_ms |
mcp tool failed | warn | gen_ai.tool.name, did, kit, is_error, error, duration_ms |
mcp error | info | rpc.method, error |
outgoing | debug. warn when it failed, timed out, answered 5xx or was slow | server.address, http.request.method, url.path, rpc.method, http.response.status_code, error.type, duration_ms |
summary: requests | info, each minute | count, failed, slow, mean_ms, max_ms |
summary: outgoing | info, each minute, per host | server.address, count, failed, slow, mean_ms, max_ms |
summary: cache | info, each minute, per cache | cache, hits, misses |
logs read | info | did, reason (what was asked) |
hud starting | info | version, commit |
panicked | error | error, with the file and line |
request is one wide line per request, written when the answer goes out. http.route is the route's template (/blob/{did}/{cid}), and each of HUD's own XRPC methods by its path (/xrpc/place.hud.getCanvas). A request no route takes is fallback in the counts, or web where HUD hands it to the web app (the web app's pages). http.response.body.size is the answer's size before HUD compresses it (how it works), and it's missing for a web app's page, which streams. The workspace's event stream (getWorkspaceEvents) is never slow, since it stays open on purpose. error names an XRPC method's error, the blob proxy's reason, and for a 5xx why it failed. A 502 or 504 is a service HUD called failing, not HUD (a PDS for the blob proxy, the favicon service for an icon, the web app for a page), so it's a warning, and the call's own outgoing line names the service.
url.query masks the value of any parameter a secret could be in (code, state, iss, token, access_token, refresh_token, password, secret, key) as _, and cuts it at 200 characters. A request under /oauth logs no query.
mcp tool is one line per tool call, a tools/call on /mcp (the iOS app, agents) or a place.hud.callTool (the workspace's page), with the caller's DID when signed in. Its kit is the renderer its tree was fitted to, as it said itself: ios 0.3.0, android 0.3.0, web 0.3.0, view (an MCP host drawing HUD's view), workspace or playground. A client that said nothing has none, and is drawn for the oldest kit (old builds and new views). A tool that answered an error is mcp tool failed, with the error the client saw. A call that asked the person to sign in has reason: sign-in asked. HUD never logs a tool's arguments or its result, since tokens, records and messages can be in them.
outgoing is one line per call through hud-core's outbound client (hud_core::net), which every request to a PDS, PLC, an AppView, a feed generator, the web app or any other service goes through. rpc.method is the XRPC method, for a path under /xrpc/. error.type is why it failed before an answer (timeout, connect, body or request, with the cause), or the status of a 5xx. A call its wait cut off is timeout. One dropped before then, because what asked for it stopped waiting (a search typed past, the loser of a race), is abandoned: no failure, at debug, and not counted as failed. Calls are at debug, so a busy minute doesn't drown the log. Set hud::outgoing=debug to see each one.
Other lines (sign-ins, canvases, deletions) are as their code writes them, with the same timestamp, level, message and trace_id.
HUD never logs a header, a token, a cookie, a body or an outbound call's query. It logs DIDs. It never logs an IP address.
Traces
A request is in the trace its W3C traceparent header names, if valid, or a new one. The answer carries it back as x-trace-id. Every line logged while serving the request names it as trace_id, including the lines of a task the request starts (hud_core::telemetry::spawn). Every outbound call sends its own traceparent in the same trace.
Metrics and the level
Set HUD_METRICS to a loopback address and port (127.0.0.1:9464) to serve two things on a listener of their own. HUD refuses any other address, so nothing outside the machine (or the container) reaches it.
GET /metricsgives what was counted since the start, in Prometheus's text format:http_server_request_duration_secondsby route, method and status,http_client_request_duration_secondsby host and outcome (a status or why it failed),hud_cache_lookups_totalby cache and result (hit,miss),hud_build_infoandprocess_start_time_seconds. HUD counts up to 200 hosts and the rest asother.PUT /logwith aRUST_LOGdirective as its body changes what is logged until the next restart.
The image has no HTTP client, so reach them from a container in the server's network namespace:
docker run --rm --network container:hud-hud-1 curlimages/curl -s http://127.0.0.1:9464/metrics | grep -v '^#'
docker run --rm --network container:hud-hud-1 curlimages/curl -s -X PUT http://127.0.0.1:9464/log -d 'info,hud::outgoing=debug'
docker run --rm --network container:hud-hud-1 curlimages/curl -s -X PUT http://127.0.0.1:9464/log -d info
PUT it back to info after, or restart, which goes back to RUST_LOG.
Kept lines
With HUD_LOGS=on, HUD keeps every line it writes to stderr in HUD_DATA_DIR/logs.db too, a SQLite file of mode 0600 (its WAL gets the same). A task writes them every second, off the requests' way. If writes can't keep up, HUD drops the oldest lines waiting (past 50,000) and never makes a request wait. It keeps the last 500,000 lines at info and below (HUD_LOGS_KEEP), and no line longer than 30 days, pruning each minute. The file holds what stderr has and nothing more, so no token, query secret or body. Lines logged before the store opens (hud starting) are on stderr only.
Every server writing to the same HUD_DATA_DIR writes to the one file, so a deploy that starts a new container on the same volume loses none of the old one's lines.
Reading the kept lines
HUD's operators are the accounts HUD_OPERATOR_DIDS names. They read the kept lines with place.hud.getLogs, as the workspace's web session or with a HUD access token as Bearer, or with the MCP tool admin_logs, which takes the same arguments and is listed to them alone. Anyone else gets Forbidden (a tool error from admin_logs).
| Parameter | |
|---|---|
trace | All of one request's lines, by its x-trace-id |
message | Only lines with this message |
level | Only lines at this level or above |
minMs | Only lines with a duration_ms of this or more |
limit | How many, the last of them. At most 500. Unless given, 500 for a trace and 100 otherwise |
A read needs one of trace, message, level and minMs, and answers the lines oldest first. Each read is itself logged as logs read, with who asked and for what. With HUD_LOGS off, the method answers NotFound.
curl -s -H "Authorization: Bearer $TOKEN" 'https://hud.place/xrpc/place.hud.getLogs?trace=4bf92f3577b34da6a3ce929d0e0e4736'
curl -s -H "Authorization: Bearer $TOKEN" 'https://hud.place/xrpc/place.hud.getLogs?level=warn&minMs=2000'
Reading the log
Docker keeps a container's stderr. Read it with docker logs, or with journalctl where Docker's log driver is journald, and filter it with jq:
docker logs hud-hud-1 2>&1 | jq -c 'select(.level == "WARN" or .level == "ERROR")'
docker logs hud-hud-1 2>&1 | jq -c 'select(.trace_id == "4bf92f3577b34da6a3ce929d0e0e4736")'
docker logs hud-hud-1 2>&1 | jq -c 'select(.message == "mcp tool" or .message == "mcp tool failed") | {t: .timestamp, tool: ."gen_ai.tool.name", did, error, ms: .duration_ms}'
journalctl -o cat CONTAINER_NAME=hud-hud-1 --since -1h | jq -Rc 'fromjson? | select(.duration_ms > 1000)'
journalctl -o cat CONTAINER_NAME=hud-hud-1 | jq -Rc 'fromjson? | select(.message | startswith("summary"))'
The default json-file driver loses a container's log when a deploy replaces the container. The journald driver keeps it in the host's journal for as long as the journal keeps anything, and HUD_LOGS keeps it in HUD_DATA_DIR.
What isn't built
- The OpenTelemetry SDK and OTLP export. They pay off with a backend to send to, and HUD has none. The ids and field names are W3C's and OpenTelemetry's, so adding the layer later changes no line's meaning.
- Spans for each call inside a request. A trace id on every line, and a line per call, give the part of a trace you read by hand. A waterfall needs a trace store.
- Sampling. At HUD's traffic every request's line costs less than deciding which to keep, and the calls, the bulk, are at debug.