~ knotless Search/
↑↓ move↵ open⇧↵ all results

chadtmiller.com/knotless
Switch repository
Loading…

git that lives in your Atmosphere account

knotless.dev
1 collaborator

knotless 0.13.0 Latest 2 packages, 1 image

1
10

Pages are designed, with the dev.knotless.site record

Merged chadtmiller.com asked to merge into feature/pages from chadtmiller.com/knotless:pages/design
docs/design/pages.mdnew+324−0
View file
@@ -0,0 +1,324 @@
# Pages
A repository can have a website. The site is a set of static files, such
as HTML, CSS, scripts and images, that knotless-serve answers over HTTP at
an address of its own.
The difficulty is that anyone may publish a site, and the files live in
the publisher's own atproto account on a stock PDS. A PDS is the server
that holds an account's records and blobs. Serving what strangers publish
raises four questions. How do you keep their pages away from
knotless.dev's login and away from each other? How do you fit a site into
the limits of a PDS? How do you make sure a visitor gets what the owner
published, however many caches sit between? How do you let a site change
without a visitor ever seeing half of one deploy and half of another?
[Storage](storage.md) explains how a repository itself is kept, and
[serve](serve.md) the server that answers for it.
## What a site is
A repository has at most one live site. Its files are blobs in the owner's
account, and one record, `dev.knotless.site`, lists them. A record is a
JSON document in a collection, addressed by a key. A blob is binary data
stored beside records. A record references a blob by its CID, a content
hash, and the PDS frees any blob that no record references.
knotless-serve only serves a site. It holds nothing that a rebuild from
the PDS would lose.
A site is built the way a release is (see [packages](packages.md)). Each
file is a blob, and the signed record carries its `size` and `sha256`. A
file larger than the PDS accepts as one blob is split into parts by the
same code that splits packs. A part is at most `KNOTLESS_CHUNK_SIZE`
bytes, 4,000,000 by default and never above the PDS's `blobUploadLimit`.
Every blob is stored as `application/octet-stream`.
## What shapes the design
A site must not run next to knotless.dev's login. A page that runs on a
knotless.dev origin could act as the signed-in visitor.
Owners' sites must not reach each other. One owner's script must not read
or set another owner's cookies or storage.
A PDS limits the size of a blob, and some PDSes limit how many blobs an
account uploads in a day. A site has many files, and it is deployed again
and again.
A visitor reads through caches, such as the browser's, a proxy's and
serve's own. What arrives must be what the owner published.
A site changes, and a deploy must not leave a mix of old and new files.
## The record
One `dev.knotless.site` record exists per repository, in the owner's
account. Its record key is the repository's key, a TID, which is a
time-ordered key. Each deploy replaces the record with `putRecord` or
`applyWrites`.
Releases do the opposite. A release writes a record for every version,
because someone may install an old version again. A site is meant to
change, and nobody asks for last week's. Replacing the one record has
these effects.
- The PDS frees the old deploy's blobs once nothing references them, so
old deploys cost no storage.
- Serve reads the live site with one `getRecord` by key. It never lists.
- The swap is one write. A visitor sees the old site or the new one, never
a mix.
The cost is that a site has no deploy history. To roll back, deploy an
older commit's build again. The record names the `commit` it was built
from and the `check` run that built it, so what is live is always known.
Deleting the record takes the site down.
### Details
- `files` lists each file with `path`, `size`, `sha256`, an optional
`contentType`, and `parts`, the blobs that hold it. An empty file has no
parts.
- `notFound` is optional. It is the path of a page served with status 404
for a path no file has.
- `fallback` is optional. It is the path of a page served with status 200
for a page no file has, for single-page apps. For a page, it is taken
before `notFound`.
- `commit` and `check` are optional.
- `createdAt` is the time of the deploy.
- A site has at most 2,000 files.
- A path is relative and `/`-separated. It has no leading `/`, no `.` or
`..` segment, and at most 512 bytes. Paths are distinct within a site.
- `contentType` is what serve sends as `Content-Type`. Publishing picks it
from the file's extension, and serve does the same for a file without
one. Because the blob is `application/octet-stream`, the PDS's own guess
at its type never matters, and the PDS never serves the file as a page
from its own origin.
## Publishing a site
The `knotless publish --site` command publishes a site from a directory.
It runs as the owner, logged in as that account. In CI it runs as releases
do, with an app password that the repository's secrets give the step. The
flags are in the [CLI reference](../reference/cli.md).
The command works in this order.
1. It walks the directory and checks every path and size.
2. It opens the repository and refuses a private one, as publishing a
release does.
3. It reads the current `dev.knotless.site` record.
4. It uploads only the files whose `sha256` the current record does not
already list. For the files it does list, it reuses their blob
references.
5. It writes the new record.
Reuse matters because uploads are scarce. bsky.social, for one, allows
1,000 blob uploads a day. A site uploaded whole on every deploy would run
out. A build that names its assets by hash changes few files from one
deploy to the next, so most deploys upload little.
Inside a runner step, `commit` defaults to `KNOTLESS_RUN_SHA`, as it does
for releases.
## Deploying on a push
A site deploys on every push to a branch by a workflow step that runs
`knotless publish --site` on `push` to that branch. A shared step makes
this one line (see [reusable workflows](reusable-workflows.md)). The
[workflow reference](../reference/workflows.md) lists the keys.
There is no mode that serves a branch's tree directly, so there is one way
to store and serve a site. The reasons are these.
- Most sites are built. Serving the tree would commit the build's output
to the repository's history for good.
- The record names exactly which bytes are live and what built them.
- Serving a blob by its hash is simpler than reading a tree out of packs
on every page view. Nothing has to watch a ref to rebuild a map of
files.
## Where a site is served
Sites are served on `knotless.page`, a registrable domain of its own, and
never under knotless.dev. Browsers treat every `*.knotless.dev` name as
the same site as knotless.dev. A page published there could set cookies
that knotless.dev then receives. It could also make form posts that
`SameSite` cookies do not stop, because those cookies block only requests
from other sites.
`knotless.page` is on the Public Suffix List. This is the list browsers
use to decide where one site ends and the next begins, for cookies and
storage. With the domain on it, each `<name>.knotless.page` is a site of
its own, and one owner's pages cannot read or set another's cookies.
Each owner has a subdomain, and each repository's site is a path under it.
```
https://<owner>.knotless.page/<repo>/
```
All of one owner's sites share an origin, which is acceptable because they
share an owner. Different owners never share one.
## Names
### From a handle to a label
An account has a DID, its permanent ID such as `did:plc:…`, and a handle,
a changeable name such as `alice.example.com` that resolves to the DID.
The owner's subdomain is their handle written as one DNS label. Each `-`
becomes `--`, and then each `.` becomes `-`.
| Handle | Label |
| --- | --- |
| `chadtmiller.com` | `chadtmiller-com` |
| `alice.bsky.social` | `alice-bsky-social` |
| `a-b.c` | `a--b-c` |
| `a.b-c` | `a-b--c` |
No two handles give the same label. Serve reads the handle back from the
label, where a pair of `-` is a `-` and a single `-` is a `.`. It then
resolves the handle to a DID as it resolves any handle, and checks that
the DID's document names the handle back.
The label is a single one for a reason. A handle has any number of dots,
and a dot in a hostname starts a new level. The Public Suffix List can
only mark a boundary a fixed number of labels deep. With
`chadtmiller.com.knotless.page`, the boundary would make
`com.knotless.page` the site that every `.com` handle shared. A wildcard
certificate, `*.knotless.page`, also covers one level only.
The address uses the handle, not a name people claim, so nothing has to be
handed out or arbitrated, and no one must claim a name before a site
works. The cost is that the address follows the handle and moves when the
handle does. After a change, the old label resolves to whoever holds that
handle now, if anyone. A site's address is always its owner's current
handle.
### When the handle does not fit
A handle has no subdomain if its label would be over 63 bytes, or if it
would begin `xn--`, which browsers read as an internationalized name.
Every `did:plc` account also has a subdomain named by its DID, so
`did:plc:<id>` is served at `<id>.knotless.page`. A plc id has no `-`, and
every handle label has one because a handle has a dot, so the two forms
never collide. The DID name never changes.
### Renames
A repository rename moves the site to the new path, because the record
follows the repository's key. The old path redirects, temporarily, as
repository pages do (see [identity](identity.md)).
The address `https://<owner>.knotless.page/`, with no repository,
redirects to the owner's page on knotless.dev.
## How paths are answered
Serve answers a request from the record's file map.
- `/<repo>` redirects to `/<repo>/`.
- `/<repo>/x/` serves the file `x/index.html`.
- `/<repo>/x`, where no file `x` exists but `x/index.html` does, redirects
to `/<repo>/x/`.
- A page no file has gets the `fallback` page with status 200 if the
record names one. A request is for a page when it asks for `text/html`
in its `Accept` header, or its path's last segment has no extension.
- Anything else no file has gets the `notFound` page with 404, or else a
plain 404.
A single-page app uses `fallback`, so that its router can answer paths the
server has no file for. Only pages fall back. A script or image that is
missing still gets a 404, so a broken build shows as one rather than as
the app's HTML where a script should be.
Because the site lives under `/<repo>/`, its build must use that as its
base path. In Vite that is `base`, and in SvelteKit it is `paths.base`.
Responses carry `X-Content-Type-Options: nosniff`, so a browser trusts the
record's `contentType`. They carry an `ETag` of the file's `sha256`. They
also carry a short `Cache-Control` max-age. The max-age is short because a
site changes and its paths are not content-addressed. A revisit
revalidates by `ETag`.
## How serve keeps a site
Serve indexes records from Jetstream, the public stream of every account's
record changes, as it does for releases. A new deploy is live within
seconds. The index row holds the record.
From the record, serve builds a map of path to file. It keeps the map by
the record's CID. A redeploy has a new CID, so the map needs no TTL and no
invalidation.
Serve reads each file from the owner's PDS the first time someone asks for
it. It joins the parts, checks the result against the record's `size` and
`sha256`, and keeps it by its hash for good, as it does for release files.
A file whose bytes differ from the record fails rather than being served.
What a visitor gets is therefore what the owner's account signed,
whichever cache it came through.
### Details
- Files are fetched, checked and kept as release files are, by the code
behind `Records::fetch` in `crates/knotless-serve/src/releases.rs`.
- The PDS signs the account's repo, so the record's hashes carry the
owner's signature.
## What is not served
A private repository has no site yet. Publishing refuses one, and serve
serves none for it. [Private sites](#private-sites) says how one is to
work.
A taken-down account or repository is not served, in the same way its
pages are not. Moderation is described in [security](security.md).
Only `GET` and `HEAD` are answered. Anything else gets 405. Serve sets no
cookies on `knotless.page`, and it answers nothing else there, so there is
no XRPC and no login on that domain.
Custom domains are not served yet.
## Private sites
A private repository's site is to be read only by its members, as the
repository is (see [access](access.md)). A shared password would name no
one, and could not be taken back from one person, so membership is the
gate.
The site's record and blobs are kept in the repository's space, as its
packs are. Serve reads them as the visitor, with that member's
credential, as it reads a private repository for its pages.
`knotless.page` has no login, so the visitor proves who they are to
knotless.dev instead. A request with no cookie for the site redirects to
knotless.dev. There the visitor logs in, if they are not already, and
knotless.dev checks that they are a member. It then redirects back to
the site's host with a ticket. The ticket is signed by serve, names the
visitor's DID and that one host, lasts a minute, and is taken once. The
host sets a cookie of its own from it, and each request after is
checked against the cookie and against membership, asked again as the
pages ask it.
A private site cannot share its owner's origin. Any script there, such
as one on the owner's public site, could fetch the private site with
the visitor's cookie, since a cookie the page's scripts cannot read is
still sent with their requests. So each private site has a host of its
own under `private.knotless.page`, which is on the Public Suffix List
too: `<repo key>-<plc id>.private.knotless.page`.
### Details
- The cookie is `__Host-` prefixed, `HttpOnly`, `Secure` and signed, and
lasts hours. Unsharing a member ends their access at the next
membership check.
- Responses are `Cache-Control: private, no-store`, so no shared cache
keeps them.
## Certificates and DNS
`knotless.page` and `*.knotless.page` point at the host that serve runs
on. One wildcard certificate, issued by a DNS challenge, covers every
owner's subdomain. The operations detail is in `deploy/README.md`.
docs/design/README.md+4−2
View file
@@ -139,6 +139,8 @@ what each one forces.
together in batches.
10. [Releases](packages.md): how a repository publishes versions of a
package, as records that never change.
11. [Integrity and trust](security.md): what each party can and cannot do.
12. [Prior art](prior-art.md): what came before, and what knotless takes
11. [Pages](pages.md): how a repository's website is kept in its owner's
account and served at `knotless.page`.
12. [Integrity and trust](security.md): what each party can and cannot do.
13. [Prior art](prior-art.md): what came before, and what knotless takes
from it.
docs-site/astro.config.mjs+1−0
View file
@@ -78,6 +78,7 @@ export default defineConfig({
page('Shared steps', 'design/reusable-workflows'),
page('The merge queue', 'design/merge-queue'),
page('Releases', 'design/packages'),
page('Pages', 'design/pages'),
page('Integrity and trust', 'design/security'),
page('Prior art', 'design/prior-art'),
],
lexicons/dev/knotless/site.jsonnew+44−0
View file
@@ -0,0 +1,44 @@
{
"lexicon": 1,
"id": "dev.knotless.site",
"description": "A repository's website: the static files served at `https://<owner>.knotless.page/<repo>/`, and the commit they were built from.",
"defs": {
"main": {
"type": "record",
"description": "Kept in the repository owner's account, keyed by the `key` of the dev.knotless.repo whose site it is, so a repository has one site. Each deploy replaces it, and the blobs only the old one named are freed. Deleting it takes the site down. Readers check every file against its `sha256`.",
"key": "tid",
"record": {
"type": "object",
"required": ["files", "createdAt"],
"properties": {
"files": { "type": "array", "minLength": 1, "maxLength": 2000, "items": { "type": "ref", "ref": "#file" }, "description": "What the site serves, by distinct paths." },
"notFound": { "type": "string", "minLength": 1, "maxLength": 512, "description": "The path of a file in `files` served, with status 404, for a path no file has." },
"fallback": { "type": "string", "minLength": 1, "maxLength": 512, "description": "The path of a file in `files` served, with status 200, for a page no file has, as a single-page app routes in the browser: a request asking for `text/html`, or for a path whose last segment has no extension. Taken before `notFound`; anything else missing gets `notFound`." },
"commit": { "type": "string", "minLength": 40, "maxLength": 64, "description": "The hex object ID of the commit the files were built from." },
"check": { "type": "string", "format": "at-uri", "description": "The dev.knotless.check, in its runner's account, of the run that built and deployed it. A reader trusts it only from a runner the repository names, and only once it has passed at `commit`." },
"createdAt": { "type": "string", "format": "datetime", "description": "When this deploy was made." }
}
}
},
"file": {
"type": "object",
"description": "One file of a site.",
"required": ["path", "size", "sha256", "parts"],
"properties": {
"path": { "type": "string", "minLength": 1, "maxLength": 512, "description": "Where it is served under the site's root: `/`-separated, with no leading `/` and no `.` or `..` segment. `x/index.html` is served at `x/` too." },
"size": { "type": "integer", "minimum": 0, "description": "Bytes in the file." },
"sha256": { "type": "string", "minLength": 64, "maxLength": 64, "description": "The lowercase hex SHA-256 of the file." },
"contentType": { "type": "string", "minLength": 1, "maxLength": 255, "description": "The `Content-Type` it is served with, such as `text/html; charset=utf-8`, or else the type its path's extension names. Its blobs are `application/octet-stream` whatever it holds." },
"parts": { "type": "array", "maxLength": 1000, "items": { "type": "ref", "ref": "#part" }, "description": "The file, split over blobs in order, each no larger than its PDS takes. None for an empty file." }
}
},
"part": {
"type": "object",
"description": "One blob of a file, in order.",
"required": ["blob"],
"properties": {
"blob": { "type": "blob", "accept": ["application/octet-stream"] }
}
}
}
}