~ knotless Search/
↑↓ move↵ open⇧↵ all results

chadtmiller.com/knotless

git that lives in your Atmosphere account

knotless.dev
1 collaborator

knotless 0.17.0 Latest 2 packages, 1 image

1
10

A test server's OAuth client ID is never one an earlier server had

Merged
Merged chadtmiller.com asked to merge into main from chadtmiller.com/knotless:gateway/flaky-client-assertion

The gateway binary's tests failed now and then with invalid_client: Validation of "client_assertion" failed: signature verification failed, or a pages login answering 502, a different test each time (on knotless-serve answers Maven's repository layout for a repository's artifacts, orgs::organizations_with_invitations_only, then reviewed::protected_branches_take_reviewed_pulls and subscriptions::watching_and_subscribing_from_the_pages). The PDS took a test server for an earlier one that had had its port. Only the harness and the tests' helpers change; no reader's attention needed beyond a look.

Claude Opus 5.5

What changes

Each TLS front has a hostname of its own. tls_proxy (crates/knotless/tests/common/mod.rs), which every gateway, pages, forge and MCP server the tests start sits behind, now puts its front at <tid>.gateway.knotless.internal:<port> rather than gateway.knotless.internal:<port>, so each server's OAuth client ID is one no other server had.

The harness covers those names. harness/pds.mjs makes its certificate for gateway.knotless.internal and *.gateway.knotless.internal, and resolves both to 127.0.0.1. The bare name still works, for e2e, the scenarios and the video camera, which use it as before. authorize.mjs already resolves every https name to 127.0.0.1, and tls-proxy.mjs serves one certificate whatever the name, so neither changes.

racing_gateways_rebuild_on_the_moved_branch leaves quarantine. Its entry in harness/quarantine.json, past its day since 2026-11-14, was for this failure.

Why

A gateway signs its client_assertion with the key in its own config directory, a new one for each test server, and publishes it in its client metadata, at its client ID, https://gateway.knotless.internal:<front port>/oauth-client-metadata.json (or /oauth/web-client-metadata.json for the pages). The reference PDS's OAuth provider keeps a client's metadata, its JWKS among it, for ten minutes by client ID (clientMetadataCache, ttl: 600e3). The fronts listen on port 0, and every test binary on a harness shares its PDS: once a front comes to a port an earlier server's front had within those ten minutes, the PDS checks the new server's assertion against the old server's key. Both keys have the same kid, so jose finds the key and the signature fails, which is the error the checks showed; the gateway answers the PDS's refusal with a 502.

On the runner it is likelier than it looks. Linux gives a socket with SO_REUSEADDR, as node's and tokio's listeners are, a port from the lower half of ip_local_port_range first, at a random odd offset (inet_csk_find_open_port): about 7,000 ports. The gateway binary alone starts about 50 servers, and the pages, forge and MCP binaries more on the same harnesses, and a binary that fails is run again on them, by then with every earlier server's metadata kept.

A key the tests share would have hidden the signature failures, but not the rest of what the PDS keeps of a client (its redirect URIs and scope, which differ between the gateway's, the pages' and the MCP server's metadata at a reused port), and would no longer make a key for each server. Turning the PDS's cache off would change the PDS under test. A client ID no earlier server had is the fault itself fixed: the label is a TID, made from the time and the process, so it never repeats.

Fixes Test racing_gateways_rebuild_on_the_moved_branch fails a client_assertion signature, where racing_gateways_rebuild_on_the_moved_branch failed the same signature.

How it was checked

A spy loaded into the harness's PDS (node --import, not committed) logged each confidential client it loaded, with its key, and each assertion it refused, with the key it held and the one the client serves at that moment.

The gateway binary, built once from main and once from this branch, run here against harness PDSes with that spy:

mainthis pull
Two gateways one after the other, their fronts made to take the same port (oauth_logins_reach_a_gateway twice, one PDS)the second fails: knotless: UpstreamFailure (502): invalid_client: Validation of "client_assertion" failed: signature verification failed; the spy: REFUSED https://gateway.knotless.internal:47123/oauth-client-metadata.json held x=vHBhKN7L0R serves x=3C37247khdboth pass, as 3n2vjcxwwvqur.gateway.knotless.internal:47123 and 3n2vjd5v77sfs.gateway.knotless.internal:47123
As the test check runs it, with the fronts taking ports as Linux does (a preload into their node: a random odd port in the lower half of 32768–60999, then the next free one up): 30 fresh PDSes, the binary twice on each, as a failed binary is run again, two PDSes at a time; 60 runs4 runs failed on a refused assertion (6.7%), each matched by the spy's refusal: sessions::logging_out_one_login_of_several, private_flow::delegating_writes_from_the_pages and themes_from_the_pages at gateway.rs:1598 (502 for 200), org_index::the_appview_counts_organization_members on account delegate (invalid_client). One more failed apart from this, at orgs.rs:3710 failed, 0 refused, though two fronts came to a port and path an earlier one had within the ten minutes, each then loaded with its own key
With this Mac's own ports (handed out in turn, so one comes round only once the busy machine has used them all): one PDS, 12 runs, two at a time1 run failed on refused assertions: two ports came round within four minutes, and private_flow::private_repositories_made_and_shared_from_the_pages and private_repositories_on_the_pages got 502 at gateway.rs:15980 refused, though one port came round within the ten minutes

Two runs side by side on one PDS also fail themes_from_the_pages at gateway.rs:2717, on both builds: it lists everyone's themes the latest first and finds the other run's. The check never runs two gateway binaries on one PDS at once, so that is left out of the counts above (the MCP tests publish themes too, on the spaces net, so it could still meet one there).

The test check ran every binary again on both nets, as the harness changed, and all passed.

Shortcomings

A just pds left running from before this change has the old certificate and resolves only the bare name: restart it.

The rates above are from this Mac, with Linux's choice of port imitated in the fronts alone. The runner shares each PDS among more binaries than these runs did (pages, forge and MCP servers have fronts too), so it likely met the fault more often than 6.7% of runs.

Approved

This gives each test TLS front its own hostname, <tid>.gateway.knotless.internal. A reused port then no longer produces a client ID an earlier server had. The harness certificate now covers *.gateway.knotless.internal, and the PDS's DNS override resolves those names to 127.0.0.1.

I read the whole diff and found nothing that makes it wrong. knotless::keys::tid() exists and is public. A TID label is valid in DNS and in a certificate wildcard. The endsWith('.' + GATEWAY_HOST) check is anchored on the dot, so it can't match unrelated names. The other places that use the bare name (the e2e Playwright config and the video camera) still work, because the bare name still resolves and is still in the certificate.

I did not run it. The cause you describe (the PDS keeps client metadata for ten minutes by client ID) is consistent with the code change, and the reproduction you give with two gateways forced onto one port is convincing.

Approved

Since my approval at 877930d, the only change is that harness/quarantine.json drops the entry for racing_gateways_rebuild_on_the_moved_branch. That entry's until date (2026-11-14) has already passed, so removing it is consistent. If that test still fails on the spaces net, the test check will show it. What I approved still stands.

Merged as dc4585d
Merging this pull closed these issues