chadtmiller.com/knotless
git that lives in your Atmosphere account
1
10
Request lines log their query, with what a secret could be in left out
crates/knotless/src/telemetry.rs+51−0
| @@ -279,6 +279,38 @@ fn hex(bytes: usize) -> String { | ||
}) | ||
} | ||
| ||
/// The names of a query's parameters whose values a log line leaves out: | ||
/// login codes, OAuth state, tokens, passwords and keys. | ||
const SECRETS: [&str; 9] = [ | ||
"code", | ||
"state", | ||
"iss", | ||
"token", | ||
"access_token", | ||
"refresh_token", | ||
"password", | ||
"secret", | ||
"key", | ||
]; | ||
| ||
/// The query of a request to `path`, as a log line gives it: each value a | ||
/// secret could be in ([`SECRETS`]) replaced by `_`, and not too long; none | ||
/// for a login's (`/oauth`), whose query is all codes and state. | ||
pub fn query(path: &str, query: Option<&str>) -> Option<String> { | ||
let query = query.filter(|_| !path.starts_with("/oauth"))?; | ||
let pairs = query.split('&').map(|pair| { | ||
let name = pair.split('=').next().unwrap_or_default(); | ||
let secret = SECRETS.iter().any(|s| name.eq_ignore_ascii_case(s)); | ||
if secret { | ||
format!("{name}=_") | ||
} else { | ||
pair.to_string() | ||
} | ||
}); | ||
let kept = pairs.collect::<Vec<_>>().join("&"); | ||
Some(kept.chars().take(200).collect()) | ||
} | ||
| ||
/// A path as a log line gives it: no query, and not too long. | ||
pub fn short(path: &str) -> String { | ||
path.chars().take(120).collect() | ||
| @@ -514,6 +546,8 @@ pub struct Served<'a> { | ||
pub route: Option<&'a str>, | ||
pub method: &'a str, | ||
pub path: &'a str, | ||
/// Its query, as [`query`] gives it for a log line. | ||
pub query: Option<&'a str>, | ||
/// The client, as its `User-Agent` names itself. | ||
pub agent: Option<&'a str>, | ||
pub status: u16, | ||
| @@ -550,6 +584,7 @@ pub fn served(r: &Served) { | ||
http.route = r.route, | ||
http.request.method = r.method, | ||
url.path = r.path, | ||
url.query = r.query, | ||
user_agent.original = r.agent, | ||
http.response.status_code = r.status, | ||
http.response.body.size = r.bytes, | ||
| @@ -950,6 +985,19 @@ mod tests { | ||
assert_eq!(stats.call("new", "200", 1, false, false), "other"); | ||
} | ||
| ||
#[test] | ||
fn queries_leave_secrets_out() { | ||
let logged = |q: &str| query("/a/b/issues", Some(q)).unwrap(); | ||
assert_eq!( | ||
logged("q=author:a.test&state=open"), | ||
"q=author:a.test&state=_" | ||
); | ||
assert_eq!(logged("code=abc&ISS=https://pds&x"), "code=_&ISS=_&x"); | ||
assert_eq!(logged(&"a".repeat(300)).len(), 200); | ||
assert_eq!(query("/oauth/callback", Some("code=abc")), None); | ||
assert_eq!(query("/a/b/issues", None), None); | ||
} | ||
| ||
#[test] | ||
fn traces_handed_on() { | ||
assert_eq!(Trace::or_new(Some(&Trace("t".into()))), "t"); | ||
| @@ -964,6 +1012,7 @@ mod tests { | ||
route: Some("/xrpc/x.awaitWrites"), | ||
method: "GET", | ||
path: "/xrpc/x.awaitWrites", | ||
query: None, | ||
agent: None, | ||
status: 200, | ||
bytes: None, | ||
| @@ -990,6 +1039,7 @@ mod tests { | ||
route, | ||
method: "GET", | ||
path: "/alice.test/tally", | ||
query: Some("q=author:alice.test"), | ||
agent: Some("Mozilla/5.0"), | ||
status, | ||
bytes: Some(10), | ||
| @@ -1011,6 +1061,7 @@ mod tests { | ||
assert_eq!(lines[2]["http.route"], "/{authority}/{repo}"); | ||
assert_eq!(lines[1]["http.response.status_code"], 200); | ||
assert_eq!(lines[1]["user_agent.original"], "Mozilla/5.0"); | ||
assert_eq!(lines[1]["url.query"], "q=author:alice.test"); | ||
assert_eq!(lines[4]["message"], "cache fill failed"); | ||
let summary = |m: &str| lines.iter().find(|l| l["message"] == m).cloned(); | ||
let requests = summary("summary: requests").unwrap(); | ||
crates/knotless-serve/src/telemetry.rs+2−0
| @@ -67,6 +67,7 @@ async fn observe(mut req: Request, next: Next) -> Response { | ||
req.extensions_mut().insert(route.clone()); | ||
let method = req.method().to_string(); | ||
let path = telemetry::short(req.uri().path()); | ||
let query = telemetry::query(&path, req.uri().query()); | ||
let agent = req.headers().get(header::USER_AGENT); | ||
let agent = agent.map(HeaderValue::to_str).and_then(Result::ok); | ||
let agent = agent.map(str::to_string); | ||
| @@ -83,6 +84,7 @@ async fn observe(mut req: Request, next: Next) -> Response { | ||
route: matched, | ||
method: &method, | ||
path: &path, | ||
query: query.as_deref(), | ||
agent: agent.as_deref(), | ||
status: res.status().as_u16(), | ||
bytes, | ||
deploy/README.md+3−1
| @@ -721,7 +721,9 @@ ended`; `container started` and `stopped`; `batch started`, `landed`, | ||
write retried`; `feed connected` or `down`; `failed` for anything that | ||
went wrong. | ||
| ||
None logs a query string, a header, a token or a body. DIDs are logged; | ||
None logs a header, a token or a body. A request line logs its query | ||
(`url.query`), each value a secret could be in replaced by `_`, and none | ||
for `/oauth`; no call made logs one. DIDs are logged; | ||
handles only as a path or a `/.well-known` host names them; client IPs | ||
never. | ||
| ||
docs/observability.md+8−3
| @@ -12,11 +12,16 @@ to read it is in [deploy/README.md](../deploy/README.md#logs). | ||
after a rollout removes their container, for 30 days, as the privacy | ||
policy says; jq reads them (`just logs`). | ||
- **One wide line a request**: a `request` line when each request ends, | ||
with its route, path, status, size, duration and trace, as Stripe's | ||
with its route, path, query, status, size, duration and trace, as Stripe's | ||
canonical log lines are [1]: a question about traffic is a jq filter | ||
over one kind of line, not a join across many. Slow ones (over | ||
`KNOTLESS_SLOW_MS`, 1000) are at warn, but for long polls, which wait | ||
on purpose; 5xx at error with the cause. | ||
on purpose; 5xx at error with the cause. The query (`url.query`) is | ||
there so a slow page can be told apart by what it asked for, as a filter | ||
or a cursor: each value of a parameter a secret could be in (`code`, | ||
`state`, `iss`, `token`, `access_token`, `refresh_token`, `password`, | ||
`secret`, `key`) is replaced by `_`, the whole cut to 200 characters, | ||
and a request under `/oauth` logs none. | ||
- **A line for each MCP tool call**: `mcp tool`, with the tool's name | ||
(`gen_ai.tool.name`), the caller's DID and `duration_ms`, at info, or | ||
at warn when slow and not asked to wait; `mcp tool failed`, at warn, | ||
| @@ -40,7 +45,7 @@ to read it is in [deploy/README.md](../deploy/README.md#logs). | ||
the pages, and the pages' SvelteKit hook hands it back on their | ||
XRPC calls, so a page and the data it drew share one trace. | ||
- **Field names from OpenTelemetry's semantic conventions** [3]: | ||
`http.route`, `http.request.method`, `url.path`, `user_agent.original`, | ||
`http.route`, `http.request.method`, `url.path`, `url.query`, `user_agent.original`, | ||
`http.response.status_code`, `server.address`, `error.type`, | ||
`http.request.resend_count`, `rpc.method`. A collector can take the | ||
lines as they are, and a backend's HTTP views know them. | ||
web/src/routes/privacy/+page.svelte+1−1
| @@ -25,7 +25,7 @@ | ||
<p>Sessions. When you log in, knotless.dev holds your OAuth session on the server. A session from the web pages can do what knotless needs, including reading private repositories you are a member of. A session for the git gateway lasts up to two years if used at least every three months. An MCP session is dropped after 90 days unused.</p> | ||
<p>Notifications. Your notifications are worked out from the index. knotless.dev keeps when you last looked, and which notifications you chose to hear of.</p> | ||
<p>Usage counts. Requests are counted by the hour, by route and by the kind of client that sent them. Pages drawn in a browser report the route and, if you came from another site, that site. Visitors are counted once a day by a hash of their IP address, with its last byte dropped, and their browser's user agent, salted with a key made for that day. The key and the hashes are deleted once the day is over, so no address is kept and no one can be followed from one day to the next. A browser that sends Global Privacy Control is not counted. Which accounts used which kinds of action each day is counted by DID, and the DIDs are dropped after 30 days, leaving only totals that name no one. Why an action failed, as the server answered, and the user agents of crawlers and scripts, never of browsers, are counted by the hour and dropped after 30 days.</p> | ||
<p>Logs. The server logs each request's route, path, status, size, duration and a trace ID, and for some requests the DID and repository involved. It never logs query strings, headers or request bodies, and its logs hold no IP addresses. Logs are deleted after 30 days.</p> | ||
<p>Logs. The server logs each request's route, path, query string, status, size, duration and a trace ID, and for some requests the DID and repository involved. Before a query string is logged, any value that could be a secret is removed, such as a login code, a token or a password. Login requests have no query string logged at all. The server never logs headers or request bodies, and its logs hold no IP addresses. Logs are deleted after 30 days.</p> | ||
<p>Hosted CI. The minutes a public repository's CI uses are published as records. Private repositories' minutes are not.</p> | ||
<h2>What knotless.dev never does</h2> | ||
<ul> | ||