johnandersen777.bsky.social/knotless
git that lives in your Atmosphere account
Forked from chadtmiller.com/knotless
0
0
# Pages
A repository can have a website. The site is a set of static files, such
as HTML, CSS, scripts and images, that knotless-serve answers over HTTP at
an address of its own.
The difficulty is that anyone may publish a site, and the files live in
the publisher's own atproto account on a stock PDS. A PDS is the server
that holds an account's records and blobs. Serving what strangers publish
raises four questions. How do you keep their pages away from
knotless.dev's login and away from each other? How do you fit a site into
the limits of a PDS? How do you make sure a visitor gets what the owner
published, however many caches sit between? How do you let a site change
without a visitor ever seeing half of one deploy and half of another?
[Storage](storage.md) explains how a repository itself is kept, and
[serve](serve.md) the server that answers for it.
## What a site is
A repository has at most one live site. Its files are blobs in the owner's
account, and one record, `dev.knotless.site`, lists them. A record is a
JSON document in a collection, addressed by a key. A blob is binary data
stored beside records. A record references a blob by its CID, a content
hash, and the PDS frees any blob that no record references.
knotless-serve only serves a site. It holds nothing that a rebuild from
the PDS would lose.
A site is built the way a release is (see [packages](packages.md)). Each
file is a blob, and the signed record carries its `size` and `sha256`. A
file larger than the PDS accepts as one blob is split into parts by the
same code that splits packs. A part is at most `KNOTLESS_CHUNK_SIZE`
bytes, 4,000,000 by default and never above the PDS's `blobUploadLimit`.
Every blob is stored as `application/octet-stream`.
## What shapes the design
A site must not run next to knotless.dev's login. A page that runs on a
knotless.dev origin could act as the signed-in visitor.
Owners' sites must not reach each other. One owner's script must not read
or set another owner's cookies or storage.
A PDS limits the size of a blob, and some PDSes limit how many blobs an
account uploads in a day. A site has many files, and it is deployed again
and again.
A visitor reads through caches, such as the browser's, a proxy's and
serve's own. What arrives must be what the owner published.
A site changes, and a deploy must not leave a mix of old and new files.
## The record
One `dev.knotless.site` record exists per repository, in the owner's
account. Its record key is the repository's key, a TID, which is a
time-ordered key. Each deploy replaces the record with `putRecord` or
`applyWrites`.
Releases do the opposite. A release writes a record for every version,
because someone may install an old version again. A site is meant to
change, and nobody asks for last week's. Replacing the one record has
these effects.
- The PDS frees the old deploy's blobs once nothing references them, so
old deploys cost no storage.
- Serve reads the live site with one `getRecord` by key. It never lists.
- The swap is one write. A visitor sees the old site or the new one, never
a mix.
The cost is that a site has no deploy history. To roll back, deploy an
older commit's build again. The record names the `commit` it was built
from and the `check` run that built it, so what is live is always known.
Deleting the record takes the site down.
### Details
- `files` lists each file with `path`, `size`, `sha256`, an optional
`contentType`, and `parts`, the blobs that hold it. An empty file has no
parts.
- `notFound` is optional. It is the path of a page served with status 404
for a path no file has.
- `fallback` is optional. It is the path of a page served with status 200
for a page no file has, for single-page apps. For a page, it is taken
before `notFound`.
- `commit` and `check` are optional.
- `createdAt` is the time of the deploy.
- A site has at most 2,000 files.
- A path is relative and `/`-separated. It has no leading `/`, no `.` or
`..` segment, and at most 512 bytes. Paths are distinct within a site.
- `contentType` is what serve sends as `Content-Type`. Publishing picks it
from the file's extension, and serve does the same for a file without
one. Because the blob is `application/octet-stream`, the PDS's own guess
at its type never matters, and the PDS never serves the file as a page
from its own origin.
## Publishing a site
The `knotless repo site` command deploys a directory as a repository's site.
It runs as the owner, logged in as that account. In CI it runs as releases
do, with an app password that the repository's secrets give the step. The
flags are in the [CLI reference](../reference/cli.md).
The command works in this order.
1. It walks the directory and checks every path and size.
2. It opens the repository. A private one's site is kept in its space
(see [private sites](#private-sites)).
3. It reads the current `dev.knotless.site` record.
4. It uploads only the files whose `sha256` the current record does not
already list. For the files it does list, it reuses their blob
references.
5. It writes the new record. For a private repository, it then shows
the appview the space, as any write there does, since nothing written
in a space reaches the firehose.
Reuse matters because uploads are scarce. bsky.social, for one, allows
1,000 blob uploads a day. A site uploaded whole on every deploy would run
out. A build that names its assets by hash changes few files from one
deploy to the next, so most deploys upload little.
Inside a runner step, `commit` defaults to `KNOTLESS_RUN_SHA`, as it does
for releases.
## Deploying on a push
A site deploys on every push to a branch by a workflow step that runs
`knotless repo site` on `push` to that branch. The `site` shared step makes
this one line (see [reusable workflows](reusable-workflows.md)). The
[sites guide](../guide/sites.md#deploy-on-every-push) shows the
workflow, and the [workflow reference](../reference/workflows.md) lists
the keys.
There is no mode that serves a branch's tree directly, so there is one way
to store and serve a site. The reasons are these.
- Most sites are built. Serving the tree would commit the build's output
to the repository's history for good.
- The record names exactly which bytes are live and what built them.
- Serving a blob by its hash is simpler than reading a tree out of packs
on every page view. Nothing has to watch a ref to rebuild a map of
files.
## Where a site is served
Sites are served on `knotless.page`, a registrable domain of its own, and
never under knotless.dev. Browsers treat every `*.knotless.dev` name as
the same site as knotless.dev. A page published there could set cookies
that knotless.dev then receives. It could also make form posts that
`SameSite` cookies do not stop, because those cookies block only requests
from other sites.
`knotless.page` is to be on the Public Suffix List. This is the list
browsers use to decide where one site ends and the next begins, for
cookies and storage. With the domain on it, each `<name>.knotless.page`
is a site of its own, and one owner's pages cannot read or set another's
cookies. Until the list takes it, a page can set a cookie for all of
`knotless.page`, which another owner's pages then receive. Nothing there
signs anyone in, so such a cookie reaches only what the sites themselves
keep in cookies.
Each owner has a subdomain, and each repository's site is a path under it.
```
https://<owner>.knotless.page/<repo>/
```
All of one owner's sites share an origin, which is acceptable because they
share an owner. Different owners never share one.
## Names
### From a handle to a label
An account has a DID, its permanent ID such as `did:plc:…`, and a handle,
a changeable name such as `alice.example.com` that resolves to the DID.
The owner's subdomain is their handle written as one DNS label. Each `-`
becomes `--`, and then each `.` becomes `-`.
| Handle | Label |
| --- | --- |
| `chadtmiller.com` | `chadtmiller-com` |
| `alice.bsky.social` | `alice-bsky-social` |
| `a-b.c` | `a--b-c` |
| `a.b-c` | `a-b--c` |
No two handles give the same label. Serve reads the handle back from the
label, where a pair of `-` is a `-` and a single `-` is a `.`. It then
resolves the handle to a DID as it resolves any handle, and checks that
the DID's document names the handle back.
The label is a single one for a reason. A handle has any number of dots,
and a dot in a hostname starts a new level. The Public Suffix List can
only mark a boundary a fixed number of labels deep. With
`chadtmiller.com.knotless.page`, the boundary would make
`com.knotless.page` the site that every `.com` handle shared. A wildcard
certificate, `*.knotless.page`, also covers one level only.
The address uses the handle, not a name people claim, so nothing has to be
handed out or arbitrated, and no one must claim a name before a site
works. The cost is that the address follows the handle and moves when the
handle does. After a change, the old label resolves to whoever holds that
handle now, if anyone. A site's address is always its owner's current
handle.
### When the handle does not fit
A handle has no subdomain if its label would be over 63 bytes, or if it
would begin `xn--`, which browsers read as an internationalized name.
Every `did:plc` account also has a subdomain named by its DID, so
`did:plc:<id>` is served at `<id>.knotless.page`. A plc id has no `-`, and
every handle label has one because a handle has a dot, so the two forms
never collide. The DID name never changes.
### Renames
A repository rename moves the site to the new path, because the record
follows the repository's key. The old path does not redirect yet. It is
to redirect temporarily, as repository pages do (see
[identity](identity.md)), once the index keeps a rename's `redirect`
record, which a site's lookup reads rather than the owner's PDS.
The address `https://<owner>.knotless.page/`, with no repository,
redirects to the owner's page on knotless.dev.
## How paths are answered
Serve answers a request from the record's file map.
- `/<repo>` redirects to `/<repo>/`.
- `/<repo>/x/` serves the file `x/index.html`.
- `/<repo>/x`, where no file `x` exists but `x/index.html` does, redirects
to `/<repo>/x/`.
- A page no file has gets the `fallback` page with status 200 if the
record names one. A request is for a page when it asks for `text/html`
in its `Accept` header, or its path's last segment has no extension.
- Anything else no file has gets the `notFound` page with 404, or else a
plain 404.
A single-page app uses `fallback`, so that its router can answer paths the
server has no file for. Only pages fall back. A script or image that is
missing still gets a 404, so a broken build shows as one rather than as
the app's HTML where a script should be.
Because the site lives under `/<repo>/`, its build must use that as its
base path. In Vite that is `base`, and in SvelteKit it is `paths.base`.
Responses carry `X-Content-Type-Options: nosniff`, so a browser trusts the
record's `contentType`. They carry an `ETag` of the file's `sha256`. They
also carry a short `Cache-Control` max-age. The max-age is short because a
site changes and its paths are not content-addressed. A revisit
revalidates by `ETag`.
## How serve keeps a site
Serve indexes records from Jetstream, the public stream of every account's
record changes, as it does for releases. A new deploy is live within
seconds. The index row holds the record.
From the record, serve builds a map of path to file. It keeps the map
until the indexed record changes, so a redeploy needs no TTL and no
invalidation.
Serve reads each file from the owner's PDS the first time someone asks for
it. It joins the parts, checks the result against the record's `size` and
`sha256`, and keeps it by its hash for good, as it does for release files.
A file whose bytes differ from the record fails rather than being served.
What a visitor gets is therefore what the owner's account signed,
whichever cache it came through.
### Details
- Files are fetched, checked and kept as release files are, by the code
behind `Records::fetch` in `crates/knotless-serve/src/releases.rs`.
- The PDS signs the account's repo, so the record's hashes carry the
owner's signature.
## What is not served
A private repository's site is not served on its owner's subdomain.
[Private sites](#private-sites) says where it is served instead.
A taken-down account or repository is not served, in the same way its
pages are not. Moderation is described in [security](security.md).
Only `GET` and `HEAD` are answered. Anything else gets 405. Serve sets no
cookies on `knotless.page`, and it answers nothing else there, so there is
no XRPC and no login on that domain.
Custom domains are not served yet.
## Private sites
A private repository's site is read only by its members, as the
repository is (see [access](access.md)). A shared password would name no
one, and could not be taken back from one person, so membership is the
gate.
The site's record and blobs are kept in the repository's space, as its
packs are. Serve's index of the space keeps the record, as it keeps the
space's other records, and serve reads the blobs as the visitor, with
that member's credential, as it reads a private repository for its
pages.
A private site cannot share its owner's origin. Any script there, such
as one on the owner's public site, could fetch the private site with
the visitor's cookie, since a cookie the page's scripts cannot read is
still sent with their requests. So each private site has a host of its
own under `private.knotless.page`, named by the repository's key and its
owner's plc id, so that a rename does not move it:
```
https://<repo key>-<plc id>.private.knotless.page/
```
The site is served at the host's root. An owner whose DID is not a
`did:plc` has no such host, and publishing says so.
`knotless.page` has no login, so the visitor proves who they are to
knotless.dev instead.
```mermaid
sequenceDiagram
participant B as Browser
participant S as Private site's host
participant D as knotless.dev
participant P as Owner's PDS
B->>S: a page, with no pass
S-->>B: redirect to knotless.dev for a ticket
B->>D: ask for a ticket, logged in
D->>P: a credential to the space, as the visitor
P-->>D: given only to a member
D-->>B: redirect back with a ticket
B->>S: the ticket
S-->>B: a pass, as a cookie of the host's own
B->>S: the page, with the pass
```
1. A request with no pass for the host redirects to knotless.dev's
`/oauth/site`, naming the host and the page asked for.
2. There, a visitor who is not logged in is sent to log in, and back.
3. knotless.dev proves the visitor a member, as it does to show them the
repository, and redirects back to the host's `/.knotless/enter` with a
ticket. The ticket is signed by serve, names the visitor's DID and
that one host, lasts a minute, and is taken once.
4. The host trades the ticket for a pass, a cookie of its own, and goes
on to the page first asked for.
Each request after is checked against the pass and against membership.
Membership is asked again as the pages ask it, a minute after it was
last proven, so someone removed from the repository stops reading its
site within that minute. A pass signed before its visitor logged out of
knotless.dev no longer holds.
### Details
- The cookie is `__Host-knotless_site`, `HttpOnly`, `Secure`, signed,
and lasts eight hours.
- Responses are `Cache-Control: private, no-store`, so no shared cache
keeps them.
- Serve keeps a private site's files apart from every other site's, so a
record elsewhere that names the same hash is never served them.
- Paths are answered as a public site's are, with the host's root in
place of `/<repo>/`.
## Certificates and DNS
`knotless.page` and `*.knotless.page` point at the host that serve runs
on. One wildcard certificate, issued by a DNS challenge, covers every
owner's subdomain. The operations detail is in `deploy/README.md`.