~ knotless Search/
↑↓ move↵ open⇧↵ all results

socialweb.computer/socialweb-computer-kcp

socialweb.computer: a compute market on atproto (kcp)

1 collaborator
0
0
225 lines · 20.1 KB Raw History

fedproxy.com: what runs there and what depends on it

One DigitalOcean droplet (157.230.140.76). mush.at, fedproxy.com, *.fedproxy.com and irc.fedfork.com all resolve to it. Read live on 2026-11-10 over ssh root@fedproxy.com; the relay was at 39d519ea1747. Other hosts: inventory.md. How to deploy: infra/README.md.

Summary

  • Caddy (xcaddy v2.11.7 with the Cloudflare DNS module) is the only public listener besides sshd and atprp's port 2222. Ports 80 and 443 belong to one server, srv0. Its route list is built by three writers: the Caddyfile, atprp-ssh-relay and the Go relay.
  • atprp-ssh-relay (port 2222) turns an ssh -R <svc>:80:... from an atproto handle into <svc>--<handle-dashed>.fedproxy.com. It does this by adding Caddy routes over the admin socket. Every 30 s it adds back any that are missing.
  • The xrpc ingress relay (Go, xrpc-relay-server.service) serves xrpc.fedproxy.com, did-key-*.xrpc.fedproxy.com and the custom domains (mush.at). On 127.0.0.1:5556 it answers Caddy's on_demand ask, the check Caddy runs before it obtains a certificate for a new name.
  • rp.fedproxy.com is oauth-client-webapp. irc.fedfork.com is atirc. fedproxy.com is a static site in /var/www/html.
  • A Caddy restart drops everything that was added over the admin API. atprp and the Go relay put their parts back. On 2026-11-10 everything was back 44 s after systemctl restart caddy.

Processes, units, ports

UnitBinaryListensServesDeployed by
caddy.service (/usr/lib/systemd/system, Environment=CF_API_TOKEN)/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile:80, :443, admin unix /opt/caddy/caddy-admin.sockeverything belowby hand; the Caddyfile is infra/fedproxy-legacy/Caddyfile (same as live)
atprp-ssh-relay.service (CF_API_TOKEN, THIS_ENDPOINT=fedproxy.com, CADDY_SOCK)/usr/bin/atprp-ssh-relay (~/src/atproto-reverse-proxy/src/golang/cmd/atprp-ssh-relay):2222 (ssh), 127.0.0.1:6060 (pprof), one unix socket per tunnel /tmp/ssh-fwd-*/tcp.sock<svc>--<handle>.fedproxy.com, *.<svc>--<handle>.fedproxy.comby hand. Never run its scripts/deploy.sh: it prints CF_API_TOKEN
xrpc-relay-server.service + drop-in 30-go.conf (over the deno 20-socialweb-computer-kcp.conf)/opt/xrpc-relay-go/current/socialweb-computer internal ingress-relay --unix-socket /opt/caddy/xrpc-relay-server.sock --hostname xrpc.fedproxy.com --ingress-relay-ask-listen 127.0.0.1:5556 --ingress-relay-caddy-admin-socket /opt/caddy/caddy-admin.sock --ingress-relay-caddy-custom-domainsunix /opt/caddy/xrpc-relay-server.sock, 127.0.0.1:5556 (ask)xrpc, did-key hosts, custom domainsscripts/deploy-xrpc-relay.sh (infra/ingress-relay/deploy.sh wraps it)
oauth-client-webapp.service/usr/bin/oauth-client-webapp (atproto-reverse-proxy)unix /opt/caddy/web-ui.sock, sqlite /opt/caddy/oauth-client-webapp.dbrp.fedproxy.comby hand
atirc.service, socket-activated by atirc-irc.socket (127.0.0.1:6667), atirc-xrpc.socket (127.0.0.1:8080), atirc-ops.socket (127.0.0.1:6670); atirc-iroh.service/usr/local/bin/atirc serve ... / atirc iroh-proxy ... (atirc 0.1.0)those three, plus irohirc.fedfork.comthe atirc repo's CONTRIBUTING (atirc server --install / --upgrade)
caddy-check-dns-from-config.service (disabled)/usr/bin/caddy-check-dns-from-config127.0.0.1:5555 when startedthe old ask, used before the Go relaystarted only by scripts/deploy-xrpc-relay.sh --rollback
static files/var/www/html (index.html, main.js, oauth-client-metadata.json)-https://fedproxy.com/atproto-reverse-proxy/scripts/deploy.sh (src/javascript/fedproxy-ssh-public-keys), never run. Nothing current publishes it

Dead, but still configured: the xrpc-test.fedproxy.com and *.xrpc-test blocks point at a socket nothing serves, so they answer 502. /opt/xrpc-relay (pre-iroh) is left over.

How Caddy routes a request

Caddy goes through srv0's routes in order and stops at the first terminal match. If no route matches, Caddy answers 200 with an empty body (Server: Caddy, Content-Length: 0). That is not an error page. On :80, Caddy answers the ACME HTTP-01 challenges and redirects everything else to https with a 308.

#MatchHandlerWritten by
0xrpc-test.fedproxy.comproxy to the missing test socket (502)Caddyfile
1xrpc.fedproxy.comproxy to the relay socketCaddyfile
2rp.fedproxy.comproxy to /opt/caddy/web-ui.sockCaddyfile
3irc.fedfork.comproxy to 127.0.0.1:8080 (atirc)Caddyfile
4fedproxy.comfile_server /var/www/htmlCaddyfile
5*.xrpc-test.fedproxy.com502Caddyfile
6*.xrpc.fedproxy.com (one label: did-key-<multibase>)proxy to the relay socketCaddyfile
7..route-<svc>--<handle>.fedproxy.com, one per live tunnelproxy to /tmp/ssh-fwd-*/tcp.sockatprp (@id route-<fqdn>)
..route-wildcard-catchall-fedproxy.com: *.fedproxy.com404 no route configured for hostatprp; moved back to the end whenever it adds a tunnel
anyroute-xrpc-custom-domains: NOT (fedproxy.com, *.fedproxy.com, *.*.fedproxy.com, *.*.*.fedproxy.com)proxy to the relay socketthe Go relay

The custom-domain route matches no fedproxy.com name, so its position in the list does not matter. A Caddyfile site block without a host would instead sit before atprp's appended routes, and it would take every tunnel. Host classes:

HostRouteAnswer
fedproxy.com4the static site
rp., xrpc., xrpc-test.fedproxy.com, irc.fedfork.com0-3their upstream
did-key-*.xrpc.fedproxy.com6the relay: the subscriber of that did:key, or the relay's 404
<svc>--<handle>.fedproxy.com with a live tunnel7..the tunnel (502 when the far end has nothing listening, e.g. bob-bid-0001 while bidder.service on mini-cloud-0001 is inactive)
any other single label under fedproxy.comcatch-all404
deeper names under fedproxy.com that are not *.xrpc or a tunnel (a.b.fedproxy.com)noneTLS fails: no certificate covers them and the ask refuses them
a name outside fedproxy.com (mush.at)custom-domainthe relay: the live subscriber that its _xrpc.<host> TXT names, else 404
the same name while the custom-domain route is gonenonethe empty 200, served with the certificate Caddy already holds (see below)
a name outside fedproxy.com with no certificate and no TXT-TLS fails (the ask says 403)

Certificates

NamesIssuerHow
*.fedproxy.comLet's Encrypt, DNS-01 (Cloudflare)atprp's tls-policy-wildcard-fedproxy.com policy (no subjects: the default) and apps/tls/certificates/automate. The Caddyfile also loads the stored file as tag cert0 for rp, xrpc and xrpc-test
*.xrpc.fedproxy.com, *.xrpc-test.fedproxy.comDNS-01 (Cloudflare)the Caddyfile tls { dns cloudflare } blocks
fedproxy.com, irc.fedfork.comACME HTTP-01 / TLS-ALPN-01the Caddyfile's own sites
*.<svc>--<handle>.fedproxy.com (a *.x tunnel)on demand, DNS-01atprp's policy has on_demand: true; the ask decides
custom domains (mush.at)on demand: DNS-01 is tried first and fails fast (not in the Cloudflare zone), then the fallback plain ACME issuer (HTTP-01 / TLS-ALPN-01)the Go relay appends that fallback issuer to atprp's policy

The on_demand ask is http://127.0.0.1:5556/check (in the Caddyfile, so it survives a restart). The Go relay approves a name when any srv0 route has that host, either exactly or as a single-label *.suffix (the same rule as the old 5555 checker; it reads the routes over the admin socket). It also approves a name whose _xrpc.<name> TXT names a live subscriber. Certificates live in /root/.local/share/caddy/certificates/ and stay there after their route is gone.

Dynamic config and Caddy restarts

PartOwnerPut back
tls-policy-wildcard-fedproxy.com (DNS-01, on_demand), *.fedproxy.com in certificates/automateatprp ensureWildcardTLSPolicyevery 30 s, written only if missing
tunnel routes, the catch-allatprp reconcileLoop (ensureForwardRoute, ensureCatchAllRoute)every 30 s, from its live SSH sessions; GETs only when nothing is missing
route-xrpc-custom-domains, the fallback issuerthe Go relay --ingress-relay-caddy-custom-domains (internal/atproto/ingress/customdomains.go)at start and every 30 s, written only if missing; it waits for atprp's policy before adding the issuer
the ask endpointthe Caddyfileat every start

Each config write reloads Caddy and cancels in-flight ACME orders. That is why both reconcilers write only what is missing.

Measured on 2026-11-10 after systemctl restart caddy (06:59:35 UTC). Before the restart, the live config was saved as a rollback snapshot that could be POSTed to /load; it was not needed.

  • +12 s: the custom-domain route was back.
  • +16 s: atprp had put back its policy, the wildcard automation, the three tunnel routes and the catch-all.
  • +42 s: the fallback issuer was back. The relay's earlier tick had found no atprp policy yet.
  • Between 0 and +16 s, the tunnels and mush.at answered the empty 200.
  • After that, every probe below answered as before the restart. The mush.at subscriber's websocket reconnected by itself.

Registration

  • Only the holder of a key registers it. getRegistrationNonce answers 403 IssuerMismatch unless the service-auth token's iss is the key the nonce is asked for. subscribe answers 403 IssuerMismatch, before the upgrade, unless the token's iss is the did param.
  • In the registration, only a signature whose key is the registration's key counts; the others are ignored. Without one that verifies, the socket closes 1008 registration rejected: no signature by the registered key (or ... verifies over the nonce).
  • A did:plc or did:web key signs with its DID document's atproto key, and its token is issued by that DID (a PDS getServiceAuth). A did:key signs and self-issues with the same key.
  • A new subscription of the same key takes the route (a restarted client gets its host back). The relay closes the old socket with 4009 replaced by a newer subscription of the same key and drops its aliases. A 4009 is final for the Go subscriber instance: it reports the terminal state replaced and stops reconnecting (a restarted process registers again and wins), so two live instances of one key do not swap the route in a loop. An Exposure's subscriber (every Exposure in ingress mode: a requester's, a gateway's, a bidder's PDS) turns the Exposure's IngressRegistered condition False, reason Replaced, and the exposure publisher logs an error. The bidder's market relay subscriber (internal/services/bidder/routes.go) and the guest control plane relay (internal/guestcp/relay) have no condition to report into; each logs an error naming its DID. A client that reconnects at once after a 4009 and runs twice with one key loops.
  • Before deploying a relay commit that sends 4009 (6b38347a or later), check by hand; the deploy script does not refuse (its usage text has the same commands):
    1. The stable channel carries a9f7f109 (main lands squashed, so check the content): PREFIX=~/.tmp/stable-check scripts/install.sh stable, then ~/.tmp/stable-check/bin/socialweb-computer version gives the commit C, and git show C:internal/atproto/ingress/status.go | grep -c SubscriberReplaced is 1 or more.
    2. No live duplicate key on the relay (one subdomain connecting twice in a minute prints a line): ssh root@fedproxy.com 'journalctl -u xrpc-relay-server.service --since -1h -o cat | grep "\"msg\":\"subscriber_connected\""' | jq -r '[.time[0:16], .subdomain] | @tsv' | sort | uniq -d prints nothing.
    3. The non-Go clients treat 4009 as final. Done: typescript-helpers/lib/compute-spa-relay-client/mod.ts (status replaced, no reconnect). Not done (other repos, scheduled by the lead): atlibs-fedproxy modules/ingress/src/lib.rs:388-395 (the close branch re-registers up to MAX_RECONNECTS with no look at the close code); socialweb-computer-console scripts/browser/ingress_ws.js:354-368 (onCloseEvent reads the code, then scheduleReconnect for any code); publicdomainrelay compute-spa/lib/relay.js:310-325 (its copy of the TS client: onclose reconnects on any close).

Custom domains

  • A subscriber is served on host H when _xrpc.H TXT holds its did:key (or its did-key host). This is checked per request and cached 60 s, negative answers included.
  • A subscriber may also send com.fedproxy.temp.xrpc.alias {hosts}. Each host is checked against the same TXT, at most 16 per subscriber. A claimed host is exclusive, and the claim ends on disconnect. The relay answers alias#result {accepted, rejected}.
  • A rejected host is still served through the TXT once the record appears. The atlibs-fedproxy ingress module re-sends its aliases every 60 s while any host is rejected, so its result catches up (atlibs-fedproxy 9dc1fe2).
  • infra/fedproxy-legacy/xrpc-custom-domains.sh status prints the route, the issuers and the ask. remove deletes the route and the fallback issuer. While the Go relay runs with the flag, it adds them back within 30 s. remove sticks only after scripts/deploy-xrpc-relay.sh --rollback (which runs it once the deno relay serves) or a CUSTOM_DOMAINS=0 deploy.

Why mush.at answered 200 while the custom-domain route was removed

This was reproduced on 2026-11-10. With the route deleted, four requests to https://mush.at/ (and /xrpc/_health) answered HTTP/2 200, server: Caddy, content-length: 0, with no via header. With the route present, they answer 200 from the relay (via: 1.1 Caddy, the subscriber's 39-48 byte body). Nothing proxied the request: it matched none of srv0's routes, so Caddy wrote its default empty 200. TLS still worked because the mush.at certificate (Let's Encrypt YE1, valid until 2027-02-08) was in Caddy's cache and storage, and removing the issuer does not evict a certificate. A name with no certificate (example.invalid) failed the TLS handshake instead, because the ask refused it. apply brought the relay's answer back.

Deploy and rollback

ComponentDeployRollback
xrpc relayscripts/deploy-xrpc-relay.sh, or infra/ingress-relay/deploy.sh --apply. It builds linux/amd64 from HEAD and ships to /opt/xrpc-relay-go/<version>, checking the sha256. It flips current, writes 30-go.conf (with the flag unless CUSTOM_DOMAINS=0) and restarts. It checks that _health carries the commit and that did.json answers 200, and on a failed check it rolls back. Last, it points the ask at 5556, live and in the Caddyfile, and prints xrpc-custom-domains.sh status--rollback, in this order: 1) caddy-check-dns-from-config up on 5555; 2) ask -> 5555 (Caddyfile backup .bak-<ts>-pre-5555); 3) remove 30-go.conf and restart (the deno relay); 4) xrpc-custom-domains.sh remove. Back to a previous Go version: point /opt/xrpc-relay-go/current at $(cat /opt/xrpc-relay-go/previous) and systemctl restart xrpc-relay-server
Caddyby hand. Change config through the admin API only; a Caddyfile reload or restart drops the API-added parts until the reconcilers put them backPOST a saved /config/ snapshot to /load (the snapshot holds CF_API_TOKEN: keep it mode 600 and pipe it over ssh, never print it)
atprp, oauth-client-webapp, the checker, the static siteinfra/fedproxy-legacy is a stub: --status only-
atircthe atirc repothe atirc repo

The default ssh key of these scripts is ~/.ssh/id_ed25519_no_password, else ~/.ssh/id_ed25519.

What depends on fedproxy.com

ConsumerUsesFails how when fedproxy is downCheck
atlibs-fedproxy modules/ingressdid:key service-auth (aud did:web:xrpc.fedproxy.com), getRegistrationNonce, one kept-open subscribe websocket, #request frames, alias framesregistration fails; after 3 reconnects (2, 4, 8 s) the run endscargo wasm && node --test modules/*/test.mjs (relay faked)
atlibs-fedproxy modules/fedproxy, web/fedproxy-ssh-public-keyscom.fedproxy.sshPublicKey records on the user's PDS; OAuth client https://fedproxy.com/oauth-client-metadata.json (the static site)the key page cannot sign incurl -s https://fedproxy.com/oauth-client-metadata.json
atlibs-examples-spa-networking marque-fedproxy-ingress (the owner's tab on http://127.0.0.1:8765/)the ingress module in forward mode with aliases: mush.at; Marque writes the CNAME and the _xrpc.mush.at TXThttps://mush.at/ and its did-key host are deaddig +short TXT _xrpc.mush.at; curl -sI https://mush.at/ (via: 1.1 Caddy = relayed)
atlibs-agent-reference-spaPublish: the ingress module serves https://did-key-<key>.xrpc.fedproxy.com/mcp/<server>published MCP URLs are dead; the in-tab agent keeps workingclaude mcp add --transport http x https://did-key-<key>.xrpc.fedproxy.com/mcp/marque
market registry and bidder (poc, mini-cloud-0001)market-registry-tunnel.service, bidder-tunnel.service: ssh -NnT -p 2222 ... -R market-registry-0001:80:127.0.0.1:4031 johnandersen777.bsky.social@fedproxy.com (and bob-bid-0001 -> 4021), Restart=always; discovery.ts pins did:web:market-registry-0001--johnandersen777-bsky-social.fedproxy.com#pdr_temp_market; the registry also publishes did-key relay endpoints (DISPATCHER_HOST=xrpc.fedproxy.com)RFP and bid discovery fail; the ssh units retry every 5 scurl -s https://market-registry-0001--johnandersen777-bsky-social.fedproxy.com/.well-known/did.json
atlibs--johnandersen777-bsky-social tunnelan ad-hoc ssh -R atlibs:80:... from a dev sessionthat pagecurl -sI https://atlibs--johnandersen777-bsky-social.fedproxy.com/
socialweb-computer (this tree)xrpc.fedproxy.com is the default ingress (internal/app/compose/exposures.go, internal/stack/bidder, internal/stack/gatewaykcp/env.go, run requester)exposures and bidder ingress have no public URLscripts/deploy-xrpc-relay.sh --status
market requester atirc agentsdid:web:irc.fedfork.com#atirc mint/revoke (internal/market/requester/atirc.go)no atirc agent VM leasescurl -A curl -sI https://irc.fedfork.com/chat
the fedproxy skill (~/.claude/skills/fedproxy)documents the tunnel: port 2222, <svc>--<handle>, the PDS key record--

Verify

scripts/deploy-xrpc-relay.sh --status                    # relay unit, version, health, journal
infra/fedproxy-legacy/xrpc-custom-domains.sh status      # route 200, issuers [cloudflare, null], ask 5556
infra/hosts/fedproxy-com.sh --status                     # every component, read-only
for u in https://fedproxy.com/ https://rp.fedproxy.com/ https://xrpc.fedproxy.com/xrpc/_health \
  https://irc.fedfork.com/chat https://mush.at/ \
  https://market-registry-0001--johnandersen777-bsky-social.fedproxy.com/.well-known/did.json \
  https://nosuch--nobody.fedproxy.com/; do
  printf '%s %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 "$u")" "$u"
done                                                     # 200 401 200 200 200 200 404 (rp: 401 when signed out)
ssh root@fedproxy.com 'curl -s --unix-socket /opt/caddy/caddy-admin.sock http://127.0.0.1/config/apps/http/servers/srv0/routes |
  jq -r ".[] | .[\"@id\"] // .match[0].host[0]"'      # the route order above
ssh root@fedproxy.com 'journalctl -u xrpc-relay-server -o cat | grep ingress_caddy_custom_domains'  # adds after a Caddy restart
ssh root@fedproxy.com 'journalctl -u atprp-ssh-relay -o cat | grep reconciled'                      # atprp's re-adds

Never print /config/ whole: atprp's policy carries the Cloudflare token. Select fields with jq.

Open

  • atirc, the static site, atprp and oauth-client-webapp have no infra/ deploy; their sources are the atirc repo and ~/src/atproto-reverse-proxy.
  • xrpc-test.fedproxy.com and /opt/xrpc-relay are dead and still configured.
  • Nothing prunes /opt/xrpc-relay-go/<version>.