Test racing_gateways_rebuild_on_the_moved_branch fails a client_assertion signature

What happened

racing_gateways_rebuild_on_the_moved_branch (spaces net, gateway (test)) failed on A check's log is read ahead as it is filed, and many more are kept and passed in the logs of the other pulls:

knotless: UpstreamFailure (502): invalid_client: Validation of "client_assertion" failed: signature verification failed

thread 'racing_gateways_rebuild_on_the_moved_branch' panicked at crates/knotless-serve/tests/gateway.rs:721:9:
assertion `left == right` failed
  left: 1
 right: 0

Quarantined until 2026-11-14.

How to see it

knotless run list knotless://chadtmiller.com/knotless --pull at://did:plc:bcgltzqazw5tb6k2g3ttenbj/dev.knotless.pull/3mzqlrem6k5wr --log, search for racing_gateways. Seen once in four pulls' logs.

What you expected

Both gateways to log in to the PDS and rebuild, with the failure count at line 721 at 0.

Where

The runner (spaces net), main as of 2026-11-07.

Guess at the cause: the test starts two gateways that each sign OAuth client assertions for the PDS. The PDS rejected one signature, which points to a client key or JWKS it had cached from another gateway or an earlier test, so a gateway whose key the PDS has not refreshed fails its first token request. The two gateways should share one client key, or the test should retry the first call once.

chadtmiller.com/knotless Switch repository Loading…