~ knotless Search/
↑↓ move↵ open⇧↵ all results
0
0
315 lines · 16.8 KB Raw History
# Records against their lexicons

A write checks each record it makes against the lexicon of its type, when
the server knows one, and answers which it did. What every method shares,
and how the server is checked, is in [`PDS.md`](PDS.md); the writes
themselves are as [`REPO.md`](REPO.md) says, and blob refs as
[`BLOBS.md`](BLOBS.md) says. `atproto/lexicon.md` defines the lexicon
language; this page says exactly what the server holds a record to, which
is in places more or less than that.

## Which records

The server knows the record types whose lexicons are in `lexicons/`, and no
others: the 36 the reference knows.

- `app.bsky.actor`: `profile`, `status`, `link`,
  `contentVisibilityDeclaration`
- `app.bsky.feed`: `post`, `like`, `repost`, `generator`, `threadgate`,
  `postgate`
- `app.bsky.graph`: `follow`, `block`, `list`, `listitem`, `listblock`,
  `starterpack`, `verification`, `referencelistoptout`
- `app.bsky.labeler.service`, `app.bsky.notification.declaration`,
  `chat.bsky.actor.declaration`
- `com.atproto.label.record`, `com.atproto.lexicon.schema`
- `com.germnetwork.declaration`; `place.stream.livestream`,
  `place.stream.chat.profile`, `place.stream.chat.pinnedRecord`;
  `site.standard.document`, `site.standard.publication`,
  `site.standard.graph.subscription`, `site.standard.graph.recommend`,
  `site.standard.theme.basic`; `social.grain.gallery`,
  `social.grain.gallery.item`, `social.grain.photo`, `social.grain.favorite`

A lexicon there of anything but a record makes no type known: a record of
an object's type (`app.bsky.embed.images`), of a lexicon of definitions
(`app.bsky.feed.defs`) or of a method's (`app.bsky.feed.getTimeline`) is
of another type, as is one of an NSID in a known namespace that names
nothing (`app.bsky.feed.unknownthing`). Nor do the spaces' lexicons
(`com.atproto.space`, `memberspace`, `personalspace`, `managedspace`,
`SPACES.md`).

*Note:* their records (`com.atproto.space.policy`,
`com.atproto.memberspace.member`) are a space's, which the reference
knows nothing of as a repository's record types.

A record's type is its `$type`, which is its collection (`REPO.md`: one
without `$type` is given the collection's, and one with another is refused
before anything here).

## validate

`createRecord`, `putRecord` and `applyWrites` take `validate`, a boolean;
anything else, `null` too, is `400 InvalidRequest` with the input.

| `validate` | A known type | Another type |
| --- | --- | --- |
| left out | checked; `validationStatus: "valid"` | not checked; `validationStatus: "unknown"` |
| `true` | checked; `"valid"` | `400 InvalidRequest` |
| `false` | not checked; no `validationStatus` | not checked; no `validationStatus` |

A record that fails the check is `400 InvalidRequest`, and the write makes
nothing. `validationStatus` is in `createRecord`'s and `putRecord`'s answer,
and in each `#createResult` and `#updateResult` of `applyWrites`;
`#deleteResult`s have none, and deletes are not checked. `validate` applies
to every write of an `applyWrites`, which checks them in order, and answers
the first that fails; its key and record are as below, whatever the others.

The record is stored exactly as given: the check fills in nothing, not a
lexicon's `default`, so its CID is the CID of what was sent (with `$type`
added where it was left out).

A record not checked, with `validate: false` or of another type, is held
to nothing here, only to the data model (`REPO.md`): an object in it with a
`$type` that is not a string (`{"$type": {"a": 1}}`, `{"$type": 5}`), or a
union value without one, is stored as it is.

## Where in a write

The order `PDS.md` gives goes on: a write checks its token, its input (the
method's own lexicon, `validate` among it), its `repo`, its record's
`$type`, the record key's syntax (`REPO.md`), and then, for a known type
unless `validate` is `false`, the record key against the type's lexicon and
then the record against it. Only after these come the legacy blob shape
(`BLOBS.md`), a swap, a key that exists already (`createRecord`'s `500`),
`putRecord`'s write of exactly the record stored (`REPO.md`: still checked,
so an invalid record is refused even then), and the other blob refs.

`validate` itself is input, but what it asks for is not: the refusal of
`validate: true` on a type the server knows no lexicon for comes where the
record is checked, after the `repo`. So such a write to another account's
repository, by `createRecord`, `putRecord` or `applyWrites`, is `401
AuthenticationRequired`, as is one of an invalid record or record key
there; `validate: null` there is `400 InvalidRequest`, as input.

## The record key

A type's lexicon names its key:

| `key` | The record key must be | Types |
| --- | --- | --- |
| `tid` | a TID: 13 characters of `234567abcdefghijklmnopqrstuvwxyz`, the first one of `234567abcdefghij` (`atproto/tid.md`) | most |
| `literal:self` | `self`, in lower case | `profile`, `status` and the declarations |
| `any` | any record key | `app.bsky.feed.generator` |
| `nsid` | an NSID (`atproto/nsid.md`) | `com.atproto.lexicon.schema` |

A key that is not is `400 InvalidRequest`. `createRecord` without `rkey`, or
an `applyWrites` create without one, makes a TID, which is no `self`: such a
write of a `literal:self` or `nsid` type is `400 InvalidRequest` too, unless
`validate` is `false`.

## The record

The record is checked against its lexicon's `record` object, as
`atproto/lexicon.md` defines each type, with these rules. Any one broken is
`400 InvalidRequest`; which is found first is not compared.

- **Objects.** A `required` field must be there; one that is not, and a
  field that is there, must be of its type. A field set to `null` is
  there, and no field of a known type may be `null`. Fields the lexicon does
  not name are taken as they are, at any depth.
- **`$type` of a nested object.** An object a `ref` names (`reply` of a
  post, `app.bsky.feed.post#replyRef`; its `root`, a
  `com.atproto.repo.strongRef`) may leave out `$type`. If it has one, it
  must be that definition's: the NSID for a lexicon's `main`, `<NSID>#<name>`
  for another.
- **Unions.** Every union of a known type is open. Its value must be an
  object with a `$type` that is a string. A `$type` that is one of the
  union's `refs` (written as above) is checked as that definition; any
  other string is taken without a look inside, a definition the union does
  not name (`app.bsky.embed.images#view` for a post's `embed`), or a string
  that names nothing (`not a type!`).
- **Strings.** `maxLength` and `minLength` count UTF-8 bytes;
  `maxGraphemes` counts extended grapheme clusters (Unicode UAX #29), so a
  family emoji of five code points joined by ZWJs is one, and so are a flag
  and a letter with a combining accent. `knownValues` are not enforced: any
  string is taken. A `format` is as below.
- **Integers** are held to `minimum` and `maximum`. (Floats are outside the
  data model, refused before, `REPO.md`.)
- **Booleans** must be `true` or `false`.
- **Arrays**: each item as its lexicon says, and `maxLength` and
  `minLength` count items.
- **Bytes** must be bytes, `{"$bytes": …}` in JSON, of any length, empty
  too; a string or a link is not.
- **Blobs** must be a blob ref of exactly the keys `$type` (`blob`), `ref`
  (a link to a CIDv1 of the `raw` codec), `mimeType` (a string with a `/`)
  and `size` (an integer, 0 or more; `-1` is no blob here, though
  `BLOBS.md` takes it in other places), and nothing else: the legacy shape
  is no blob here either. Its `mimeType` must match `accept`: be one of its
  entries, or start with `<type>/` for an entry `<type>/*`, case and all
  (`image/*` takes `image/gif` and `image/`, not `IMAGE/PNG` or
  `imagex/png`); and its `size` must be at most `maxSize`. These are the
  ref's own `mimeType` and `size`, as the record says them; `BLOBS.md` then
  holds them to the uploaded blob.

No known type uses `enum`, `const`, `nullable`, `minGraphemes`, a closed
union, `cid-link` or `unknown`.

### String formats

`interop/syntax/` has atproto's test vectors for each format: a string in a
`*_syntax_valid.txt` file is valid, and one in a `*_syntax_invalid.txt` or
`*_parse_invalid.txt` file is not. The server is held to them all, but
where the reference differs:

- **`cid`**: a string that, parsed as a CID, is written the same way again:
  a CIDv0 in base58btc (`Qm…`, which the vectors call invalid), or a CIDv1
  in base32 in lower case (`bafy…`, `bafk…`). A CIDv1 in any other base
  (`z…`, `m…`, `f…`), upper case, or of digits, which the vectors call
  valid, is refused.
- **`uri`**: exactly the strings `^\w+:(?:\/\/)?[^\s/][^\s]*$` matches,
  `\w` being ASCII letters, digits and `_`, of any length. So
  `content-type:text/plan` and `microsoft.windows.camera:thing`, which the
  vectors call valid, are refused, and the very long URI they call invalid
  is taken.

The formats records use, and what each is:

- **`datetime`**: at most 64 characters, matching
  `^[0-9]{4}-(0[1-9]|1[012])-([0-2][0-9]|3[01])T([0-1][0-9]|2[0-3]):[0-5][0-9]:([0-5][0-9]|60)(\.[0-9]+)?(Z|[+-]([0-1][0-9]|2[0-3]):[0-5][0-9])$`,
  not ending `-00:00`, with seconds other than `60` (the regular expression
  lets `60` through, but no leap second is taken), and whose instant, in
  UTC, is in the years 0000 to 9999 (`0000-01-01T00:00:00+01:00` is not).
  The day is not held to its month: `2023-02-30T00:00:00Z` and
  `2023-04-31T00:00:00Z` are taken. Upper-case `T` and `Z` only.
- **`did`**, **`handle`**, **`nsid`**, **`at-uri`**: as `atproto/did.md`,
  `handle.md`, `nsid.md` and `at-uri-scheme.md` define them, and the
  vectors agree: a `did` is any DID by its syntax (`did:example:x` and
  `did:plc:abc` are valid DIDs), and an `at-uri` may stop after its
  authority or collection (`at://bob.test`).
- **`language`**: a `Language-Tag` of RFC 5646 §2.1's grammar, its
  letters in any case but where these say otherwise, with no variant or
  extension singleton repeated, compared in any case (§4.1):
  `de-DE-1901-1901` and `en-u-co-phonebk-U-ca-buddhist` are refused.
  - A tag with a primary language subtag starts with 2 or 3 lower-case
    letters: `ja`, `ban`, `ja-jpn`, but not `JA`, `Ja` or `ZH-HAKKA`, and
    not the 4 and 5 to 8 letter subtags the grammar allows (`jaja`,
    `abcde`). The subtags after it are in any case: `en-latn-us`,
    `zh-YUE-hk`, `en-X-A`.
  - A tag of private use alone starts `x` or `X`: `x-fr-ch`, `X-fr-CH`
    (and `x` alone is no tag).
  - Of the grandfathered tags, the irregular ones are taken only exactly
    as RFC 5646 writes them: `en-GB-oed`, `i-ami`, `i-bnn`, `i-default`,
    `i-enochian`, `i-hak`, `i-klingon`, `i-lux`, `i-mingo`, `i-navajo`,
    `i-pwn`, `i-tao`, `i-tay`, `i-tsu`, `sgn-BE-FR`, `sgn-BE-NL`,
    `sgn-CH-DE`. So `I-default`, `i-Klingon`, `en-gb-oed`, `sgn-be-nl`, and
    any other `i-` tag (`i-foo`) are refused. The regular ones (`zh-hakka`,
    `art-lojban`, `no-bok`, ...) are tags of the grammar too, and are taken
    as any such tag is: `zh-Hakka`, not `Zh-hakka`.

## Duplicates

`createRecord`, unless `validate` is `false`, replaces what it duplicates.
Its record links to a subject:

- an `app.bsky.graph.follow` or `app.bsky.graph.block`, its `subject` when
  that is a string that is a DID;
- an `app.bsky.feed.like` or `app.bsky.feed.repost`, its `subject.uri` when
  that is a string that is an AT URI.

The same commit then deletes every record of the account in the same
collection whose link is that same subject, compared as strings (so a like
of `at://bob.test/…` and one of `at://did:plc:…/…` are not the same), before
it creates the new one; `createRecord` answers as for any create, and the
deletes are in the commit, on the firehose and in the repository as any
delete is: the commit's ops are the deletes, in any order, then the create.
A record links to its subject whichever write made it, with `validate:
false` too, by `putRecord` or `applyWrites`; only `createRecord` deletes.
So two follows of one DID, both kept by `validate: false` or `putRecord`,
are both deleted by the next `createRecord` of a follow of that DID. The
other writes delete nothing: `putRecord` and `applyWrites` of a second
follow keep both.

The record already at the key the write creates is one of those it deletes
when it links to the same subject: that `createRecord` is no `500` (as it is
for a key that exists, `REPO.md`), but the record there deleted and the new
one created, in one commit, even when the two are the same. Its ops are a
delete of the key and a create of it, whose `prev` is the deleted record's
CID (`SYNC.md`). A record at the key that links to another subject, or to
none, is kept, and the write is the `500`; and so it is, whatever the
record there, with `validate: false`.

## Data

The record lexicons, and the definitions they use, are in `lexicons/`:

- `app.bsky.*` and `chat.bsky.*` from `lexicons/` of
  https://github.com/bluesky-social/bsky at `be148a09fb` (2026-11-01), and
  `com.atproto.*` from `lexicons/` of https://github.com/bluesky-social/atproto
  at `0849c24d49` (2026-11-09), both MIT or Apache-2.0, unchanged. They
  mean the same as the copies the reference is built with, at
  `@atproto/pds@0.5.55` (`31eb9a09d3`): those leave out `"lexicon": 1` and
  the descriptions, and nothing else differs.
- `com.germnetwork.*`, `place.stream.*`, `site.standard.*` and
  `social.grain.*` as the reference is built with them,
  `packages/pds/lexicons/` (and `lexicons-overrides/`, which some of those
  link to) of https://github.com/bluesky-social/atproto at `31eb9a09d3`,
  MIT or Apache-2.0, unchanged. They are other publishers' lexicons, which
  that repository holds a copy of.

`interop/syntax/` is the `syntax/` directory of
https://github.com/bluesky-social/atproto-interop-tests at `056e5741bb`,
CC0 as the rest of `interop/`.

## How it is checked

`harness/checks/validation.mjs` writes records of the known types, mostly
`app.bsky`, and some of other types (in no known namespace, an `app.bsky`
NSID that names nothing, and lexicons that are no record's), on both sides,
and compares the answers as `PDS.md` says (`validationStatus` among them,
and the record's CID): by `createRecord`, `putRecord` and `applyWrites` (of
one to four creates, updates and deletes), with `validate` left out,
`true`, `false` or not a boolean (`null`, `"true"`, `1`, `0`, `{}`). Each
record is drawn valid from its lexicon, then sometimes broken in one way:

- a field left out where it is `required`, or any field `null`, or of
  another type;
- a string too long in bytes, or in graphemes, with ASCII, accented
  letters precomposed and with combining accents, CJK, ZWJ emoji, flags and
  skin tones (or as long as it may be, valid); or too short;
- a string of each format drawn from `interop/syntax/`, valid and not, the
  reference's differences above among them, and datetimes and language
  tags of each rule above;
- an integer below its `minimum` or above its `maximum`, an array with
  more items than its `maxLength` or fewer than its `minLength`;
- a union value without `$type`, with a `$type` that is not a string, of
  a member broken inside, of a definition it does not name, or naming
  nothing; a nested object with a `$type` of another definition;
- a blob of a type `accept` refuses, larger than `maxSize`, with `size`
  `-1`, without `$type`, or in the legacy shape;
- bytes that are a string or a link;
- a record key against its type's `key`: a `tid` type with `self`, a TID
  of the wrong length, alphabet or case; a `literal:self` type with
  another key; an `nsid` type with a non-NSID; or no key at all, which
  `createRecord` answers with a TID of each side's own, taken out of the
  answers before they are compared, and the record, made, deleted again.

Fields the lexicon does not name are added at random, and `knownValues`
fields are given other strings. A valid blob ref names a blob the account
holds, as `BLOBS.md`'s check knows them, or one no account has uploaded.
Each record also carries a field of a number no other has, so that no two
writes send the same record, but where one sends again exactly the record
written last at its key. Follows, blocks, likes and reposts are drawn from
few subjects at few keys, so that duplicates are often made, by every
write and `validate`, at another key and at the same one. After such a
write the collection is listed on both sides, compared, and the
reference's listing held to the keys the harness knows to be there, by
this page's rules; the account's repository then holds the same records
on both sides (`PDS.md`'s repository check), deletes and all. With a
`--firehose` run, the deletes are on both firehoses. Writes of each kind
go to another account's repository too, with `validate: true` on another
type among them.

Not compared: which error a record broken in more than one way answers
first, and the error messages, as everywhere; the order of a commit's
deletes of duplicates (`SYNC.md`'s check compares a commit's ops as a
set). The harness does not draw grapheme clusters whose segmentation
changed in recent Unicode versions.