~ knotless Search/
↑↓ move↵ open⇧↵ all results
0
0
544 lines · 30.4 KB Raw History

Blobs

Files an account uploads, such as images, which its records reference by CID: com.atproto.repo.uploadBlob, com.atproto.sync.getBlob and com.atproto.sync.listBlobs, and what a blob ref in a record does. What every method shares, and how the server is checked, is in PDS.md; the records are as REPO.md says.

A blob is an account's: each account holds its own blobs, and a record can reference only blobs its own account holds. A blob is held from when it is uploaded until no record references it any more (below).

uploadBlob

uploadBlob takes the blob as the raw request body, its type as Content-Type, and the account's accessJwt as its bearer token. It checks, in this order, and answers the first error:

  1. The token. None is 401 AuthMissing. One the server did not give out is 400 InvalidToken. The reference answers 500 to a token that is not a JWT (three base64url parts, the first two of them JSON); that is not compared, and the harness sends a JWT.
  2. The Content-Type. Its media type is the header up to its first ;, as sent: neither lowercased nor trimmed, so TEXT/Plain is TEXT/Plain, text/plain; charset=utf-8 is text/plain, and text/plain ;q=1 is text/plain with its space. No Content-Type, an empty one, or one whose media type is empty is 400 InvalidRequest. Any media type with a / in it is taken, a/b/c and */* too. The reference looks one with no / up as a file extension (png is image/png); that is not compared, and the harness sends none.
  3. The size. A body of more than PDS_BLOB_UPLOAD_LIMIT bytes (LIMITS.md), 5242880 (5 MiB) unless it is set, is 413 PayloadTooLarge, whether it has a Content-Length or comes chunked. One of exactly the limit is taken, and so is an empty one.
  4. The account's quota: an upload that would take the blobs it holds past PDS_BLOB_QUOTA_BYTES is 400 BlobQuotaExceeded (LIMITS.md). The reference has no quota; the harness reaches it only in a run of its own.

It answers {blob}, a blob ref:

{"blob": {"$type": "blob", "ref": {"$link": "bafkrei…"}, "mimeType": "image/png", "size": 66}}
  • ref is the CID of the body's bytes: CID version 1, the raw codec (0x55), a SHA-256 multihash, in base32 (atproto/data-model.md).

  • size is the body's length in bytes.

  • mimeType is the type the body's first bytes show, if they show one, and otherwise the request's media type. The reference recognises the hundreds of formats its sniffing library knows; the harness sends only these, and bodies that recognise as none:

    The bodymimeType
    the PNG signature 89 50 4E 47 0D 0A 1A 0A, then an IHDR chunk of 13 bytes, then chunks up to an IDAT chunk, with no acTL chunk before it; nothing after that IDAT's type is readimage/png
    FF D8 FF, then a byte other than F7image/jpeg
    GIF89aimage/gif
    RIFF, any 4 bytes, then WEBPimage/webp
    %PDF-application/pdf
    any 4 bytes, ftyp, then the brand isomvideo/mp4
    starting spec-pds blob , or emptythe request's media type

describeServer answers blobUploadLimit, that limit, beside what ACCOUNTS.md lists.

Uploading bytes the account already holds answers the same ref and size, with this request's mimeType, but the blob keeps the mimeType it was first stored with: that is what getBlob serves, and what a record's ref must name (below). A blob the account no longer holds is stored afresh, with the new type.

The upload is held, but getBlob does not serve it and listBlobs does not list it until a record references it. An upload no record references is held for PDS_BLOB_TTL_SECS from when it was last uploaded, across restarts too, and then deleted, as the lexicon allows (LIMITS.md). The reference keeps it for good; the harness's runs are shorter than the default hour, and the run that waits for it is LIMITS.md's.

Blob refs in records

A record names a blob by a blob ref anywhere in it: in a map, in a list, at any depth. A map is a blob ref when it has exactly the keys $type, ref, mimeType and size, and:

  • $type is blob;
  • ref is a link ({"$link": …} in JSON), of any CID;
  • mimeType is a string with a / in it;
  • size is an integer, 0 or more, or -1.

Any other map is not a blob ref, though it looks like one, and is stored as it is, as any map of the data model is: it references nothing and is checked for nothing. The reference stores each of these, and so does the server:

  • a map missing any of the four keys;
  • a $type other than blob, such as Blob;
  • a size that is a string, null, or an integer below -1, such as -2;
  • a mimeType that is empty, has no /, or is not a string;
  • a ref that is a string, or a map other than a link, such as {"$link": <cid>, "x": 1}, {"$bytes": …} or {"a": 1};
  • a fifth key, beside the four.

interop/data-model/data-model-invalid.json calls a blob with a string size or a missing key invalid; records are not held to that, and such a map is not refused. A size that is a float is refused as any float is: 400 InvalidRequest, outside the data model (REPO.md). A ref that is a link to a CID that is not raw, such as a record's, makes the map a blob ref, of a blob no account holds: 400 BlobNotFound (below).

A blob ref is part of the record, and of what its CID covers, as any other map is: ref is a CID link in DAG-CBOR.

A record that a CAR file (MIGRATION.md, "importRepo") or a space repo's archive (SPACES-EXPORT.md, "importRepo") brings may hold, as a blob ref's ref, a map of the one key $link whose value is a CID string, in place of a link, as some servers wrote them. JSON writes that map as it writes the link, and it is taken as the link: the map of the four keys is a blob ref of that CID. The record keeps the map as it is, and the CID its block has. A write cannot make one: in its JSON, {"$link": …} is a link.

The legacy shape, a map of exactly cid, a CID string of the raw codec, and mimeType, a non-empty string, is refused: 400 InvalidRequest. It is checked with the record, right after its $type and validate, so before a swap, a key that exists, or any other blob ref. One whose cid is a CID of another codec the reference answers 500, which is not compared. A map of those keys whose cid is not a CID is no blob ref.

The other blob refs are checked last, after everything PDS.md and REPO.md say a write checks, so a write refused for its token, input, repo, $type, validate, swap, or a key that exists or does not, is refused for that whatever its refs. Then each ref, against the blobs the account held before the write:

  • a blob the account does not hold, whether never uploaded, another account's, or no longer held, is 400 BlobNotFound, and so is a ref that is not a raw CID, such as a record's;
  • a blob no record referenced before the write must match the ref: a mimeType other than the one it was stored with is 400 InvalidMimeType, then a size other than its size (-1 too) is 400 InvalidSize;
  • a blob some record referenced before the write is not checked against the ref at all: a ref with any mimeType and size is taken, and getBlob still serves the blob as stored.

Within one write every ref is checked against the blobs as they were before it, so in one record, or one applyWrites, a ref that matches a new blob does not let another that does not. A write refused for a ref makes no commit and changes nothing: a blob it named that no record referenced is still not served, still not listed, and a later ref to it is still checked against its stored mimeType and size. The reference, by a race of its own, sometimes serves such a blob from getBlob after the refusal (in about one run in ten), until a record references it; nothing else of it changes. That is not compared: the harness holds the server to the rule here for such a blob, and the reference to nothing. Which error a write answers when more than one of its refs is wrong is not compared: the harness sends at most one ref that may be wrong in a write.

A blob is referenced by every record of the account whose blob refs name it, each record counted once however many of its refs name it. When a write deletes or replaces the last record that references a blob, and puts no record referencing it in its place, the account stops holding it: getBlob answers it as not found, listBlobs leaves it out, and a ref to it is BlobNotFound until it is uploaded again. A write that replaces or deletes one record and references the blob from another, in one applyWrites or in the same record by putRecord, keeps it.

What a write leaves referenced is counted from the records as the commit leaves them. An applyWrites that writes one key more than once, a create then an update or a delete, or two updates, is where the reference does otherwise, as a fault of its index: it counts every blob any of those writes referenced as referenced by that key, though the record the batch leaves there, if any, references none of them. Such a blob stays served and listed until a later write replaces or deletes that record, and, where the batch deleted it, for good. This is not compared: the harness writes each key at most once in a batch.

The repository is unchanged by blobs: records holding refs are records, in the MST and commits as any other. On the firehose, a #commit's blobs is always empty.

Note: the lexicon says it soon will be.

getBlob

getBlob takes did and cid, and no token: one the server gave out, of any account, is the same as none (but the account's own, of an account that is not active, LIFECYCLE.md), and one it did not is 400 InvalidToken, after the parameters are checked. A did or cid that does not parse is 400 InvalidRequest; a did no account here has is 400 RepoNotFound; a blob the account does not hold, or holds but no record references, is 400 InvalidRequest (not BlobNotFound, though the lexicon lists it).

It answers 200 with the blob's bytes and these headers:

HeaderValue
Content-Typethe blob's stored mimeType; the reference adds ; charset=utf-8 to some types, such as text/* and application/json
Content-Lengththe blob's size
X-Content-Type-Optionsnosniff
Content-Dispositionattachment; filename="<cid>"
Content-Security-Policydefault-src 'none'; sandbox

listBlobs

listBlobs takes did, and optionally since, limit (1 to 1000, by default 500) and cursor, and no token, as getBlob. A parameter that does not parse, such as a since that is not a TID, is 400 InvalidRequest; a did no account here has is 400 RepoNotFound.

It answers {cids, cursor}: the CIDs of the blobs that records of the account reference, each once, in ascending order of the CID string, at most limit of them.

  • With cursor, only CIDs greater than it, comparing strings: it need not be a CID. An empty one is the same as none.
  • With since, a rev, only blobs referenced by a record whose last write, a create or an update, was in a commit with a rev greater than since. A record updated since lists every blob it references now, old or new.
  • cursor in the answer is the page's last CID, whenever the page has any, full or not, and is left out when it has none.

Where blobs are kept

By default the server keeps each blob's bytes in PDS_DATA_DIR, a file of their own. With PDS_BLOBSTORE_S3_BUCKET set, it keeps them in that bucket of an object store that speaks S3's API (Amazon S3, Cloudflare R2, MinIO, RustFS and the like) instead, and none of their bytes in the data directory. The reference offers both, by these variables:

VariableIsDefault
PDS_BLOBSTORE_S3_BUCKETthe bucket; unset, blobs are kept in PDS_DATA_DIRunset
PDS_BLOBSTORE_S3_REGIONthe region requests to the store are signed forus-east-1
PDS_BLOBSTORE_S3_ENDPOINTthe store's base URL, http or https, with no query or fragment, as https://<account>.r2.cloudflarestorage.com; unset, Amazon's, https://s3.<region>.amazonaws.comunset
PDS_BLOBSTORE_S3_FORCE_PATH_STYLEtrue: the bucket is the first segment of each request's path, <endpoint>/<bucket>/<key>; false: the first label of its host, <bucket>.<endpoint's host>/<key>false
PDS_BLOBSTORE_S3_ACCESS_KEY_ID, PDS_BLOBSTORE_S3_SECRET_ACCESS_KEYthe credentials requests are signed with; both unset, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from the environment, with AWS_SESSION_TOKEN if it is setunset
PDS_BLOBSTORE_S3_UPLOAD_TIMEOUT_MShow long storing one upload may take, in milliseconds20000
PDS_BLOBSTORE_S3_REQUEST_TIMEOUT_MShow long any other request to the store may take, in milliseconds10000

A value that does not parse (a FORCE_PATH_STYLE other than true or false, a timeout that is not a positive integer, an endpoint that is not an http or https URL or has a query or fragment), one of the two keys without the other, any of these variables set without the bucket, or the bucket set with no credentials from either place, makes pds serve exit 1 before it writes anything to standard output, with why on standard error, as LIMITS.md says of its variables. An empty variable is as if unset.

A data directory keeps its blobs where its first start put them: started again with the bucket set when it kept them in itself, or unset when it kept them in a bucket, pds serve exits 1 before it serves, with why on standard error. pds migrate-blobs moves them from one to the other (below). Which bucket, and the endpoint, may change between starts, as when a store's objects are copied to another: the server reads no object at its start, so a start takes no longer with a bucket, and a bucket it cannot reach shows only when a request needs it.

Requests to the store

Each request is signed with AWS Signature Version 4 for the service s3 and the region, its x-amz-content-sha256 the SHA-256 of its body, and with x-amz-security-token when there is a session token; over https, the store's certificate is verified. They are the operator's, to a store the operator names, often on a private address, so SSRF protection (PROXY.md) does not apply to them.

A request goes over a connection an earlier one left open, if the store keeps one open; the server keeps one unused for up to 90 seconds. Every request the server makes of the store may be made twice to the same effect (a PUT of the same bytes to the same key, a GET, a DELETE), so one that fails on its way, the connection closed before any answer came, as when the store closes a connection it held idle just as the request is sent, is sent once more over a connection of its own, within the same time limit. Only a request that fails so twice, or one the store answers and then fails, fails as "The objects" says below.

Note: a store across a network, as R2 is from a server outside Cloudflare, takes tens of milliseconds and a TLS handshake to open a connection: a server that kept none between requests paid that on most getBlobs and uploads.

The objects

Each blob an account holds is one object, keyed blocks/<did>/<cid>, its bytes the blob's: the reference's key for a blob a record references (it keeps an upload no record references yet under tmp/, and moves it when one does). The server reads, writes and deletes no key of another form, so the bucket may hold other things; but two servers holding the same accounts, as a server and another started on a copy of its data directory (DURABILITY.md), must not share a bucket, as each deletes what it stops holding.

  • uploadBlob answers once the store has answered the object's PUT with a 2xx. A PUT the store answers with another status makes it 500 InternalServerError; one it does not answer, the connection closed or no answer within PDS_BLOBSTORE_S3_UPLOAD_TIMEOUT_MS, 504 UpstreamTimeout, as the reference answers them. Either way the account holds nothing more, and its quota is as before.
  • getBlob, of a public blob and of a space's (SPACES-EXPORT.md), reads the object as it sends it. A store that fails the read before the first byte makes it 500 InternalServerError, as the reference answers: any status other than 200 (a 404 too, as of an object something other than the server deleted), the connection closed, or no answer within PDS_BLOBSTORE_S3_REQUEST_TIMEOUT_MS. A failure after the first byte ends the answer short. (The reference, given no answer, waits on, past its own timeout: that is not compared.)
  • A write never reads or writes the store: a record's first ref to an upload needs nothing of it, where the reference moves the object then, and answers 500 when the store fails that (not compared).
  • A blob the account stops holding, by a write, by outliving PDS_BLOB_TTL_SECS unreferenced (LIMITS.md), or by the account's deletion (LIFECYCLE.md), has its object deleted within a minute while the store answers, and its deletion tried again, across restarts, until the store has answered it. No write waits on it, or fails for it, as the reference's do not. An object is never deleted for a blob the account holds again by then, as one uploaded again at once.
  • A blob taken down keeps its object (LIFECYCLE.md).

So, within a minute of any change while the store answers, and of a start, the objects under blocks/<did>/ are those of the blobs the account did holds, but for: an upload on its way, or one the server was killed during, whose object the store may have kept while the server holds nothing of it (deleted within a minute of the next start); one deleted by something other than the server; and, after a power cut, an object whose upload the server had not yet made durable (DURABILITY.md).

Moving them

pds migrate-blobs --to bucket moves a data directory's blobs from it into the bucket, and pds migrate-blobs --to disk from the bucket into it. It is given the environment pds serve is, with the bucket's variables set either way, as the bucket is where the blobs go or come from, and runs with no server on the data directory (COMMANDS.md, "One binary"). The reference has no such command: its operator stops it, copies the files into the bucket, or the objects out of it, and starts it with the other variables.

  1. It checks the environment as pds serve does, and exits 1 on what pds serve would, before it writes anything to standard output. A --to other than bucket or disk, or none, is a usage error: exit status 2.
  2. A data directory a server, or another pds migrate-blobs, holds is refused: exit 1, with why on standard error, and nothing changed.
  3. A data directory that keeps its blobs where --to says, with no move unfinished, is left as it is: it prints {"moved":0,"bytes":0} and exits 0.
  4. Otherwise it first marks the data directory, durably, as moving its blobs. From then until it finishes, pds serve on it exits 1 before it serves, with why on standard error, naming the command to run.
  5. It copies each blob every account holds, referenced or not, taken down or not, to the other place: into the bucket, the object blocks/<did>/<cid> with the file's bytes; into the data directory, the object's bytes as the file, synced. Each blob's bytes must be its CID's, read where they are kept: a blob whose bytes are missing there, or are not its CID's, stops it, exit 1, with each such blob (DID and CID) on standard error, the data directory still moving. A store that fails a request, as pds serve would count a failure, stops it the same way, with why.
  6. With every blob copied, it makes the data directory keep its blobs where --to says, durably, and then deletes what is left where they were: into the bucket, the blobs' files; into the data directory, the blobs' objects, and those the server still owed a delete of (The objects, above). The objects of a deleted account the server owed are deleted either way.
  7. Then it marks the move finished, prints one line, {"moved":<blobs>,"bytes":<bytes>}: the blobs the accounts hold, every one now where --to says, and the sum of their sizes; and exits 0.

Killed at any moment, or stopped by a failure, the command is run again: with the same --to it carries on and finishes, copying again what it had; with the other --to, it moves the blobs back where they were, as a move that way. Either way, once it has finished, pds serve with the environment of the place it moved them to holds every blob it held before, each read answering as it did; the data directory, with the bucket, holds none of their bytes; and the bucket, with the data directory, holds none of the objects of the accounts it holds, as The objects says.

The server is down while the command runs, for as long as copying every blob takes.

Durability

Blobs are kept with everything else in PDS_DATA_DIR, or in the bucket (DURABILITY.md):

  • An upload the server has answered survives the server being killed, as does a blob a record references: after a restart, getBlob serves the same bytes, listBlobs lists the same CIDs, and a record written after the restart may reference an upload from before it, which still needs its first stored mimeType, while the upload is within PDS_BLOB_TTL_SECS.
  • An upload the server is killed during is stored whole or not at all: a ref to it afterwards is either BlobNotFound or served whole.
  • A blob that stopped being held before the kill stays gone.

Against a power cut, a blob is durable once a committed record references it, and an upload no record references yet may be lost, as DURABILITY.md says under Blobs. In a bucket the bytes are the store's before the upload is answered, so what a power cut may lose of such an upload is the server's entry of it, and its object may be left in the store (above).

How it is checked

harness/checks/blobs.mjs holds the server to the reference:

  • uploadBlob with bodies of each kind in the table, spec-pds blob bodies of random bytes (some sent again, by the same account or another, with another type), empty bodies, and rarely bodies of exactly 5242880 bytes and one more, chunked or not; with media types of mixed case, with parameters, a/b/c and */*, or none; with the account's token, none, or a JWT the server did not give out. The answers are compared whole. The reference's ref and size are also held to the body itself, which checks the check.
  • Records with blob refs, written by createRecord, putRecord and applyWrites in a collection of their own, com.example.blob: refs to the account's uploads as answered, and at most one that may be wrong, of another mimeType or size (-1 too), to another account's blob, to a CID never uploaded or a record's, in the legacy shape, or a map that is no blob ref. Records are deleted and replaced the same ways. Answers are compared as for any write, so record CIDs must be the reference's, and the repository checks (PDS.md) verify the records and their tree.
  • getBlob of the account's uploads, held or not, another account's, a CID never uploaded, a record's CID, parameters that do not parse and a DID no one holds; with no token, the account's, another's, or one never given out. The status, the error name, and on 200 the bytes and the headers above are compared. The harness asks for no compression, with Accept-Encoding: identity, and Content-Type is compared up to its first ;, trimmed: the charset parameter is not compared.
  • listBlobs of the account, paged with random limits to the end, from random cursors, and since revs of the collection's writes, each side given its own rev from the same write, a TID before every rev, or one after. The answers are compared whole, cursor too: it is a CID, the same on both sides.
  • describeServer's blobUploadLimit.
  • With --crash-every, an upload is sometimes on its way when the server is killed. After the restart its body is uploaded again with the same type, and every account's listBlobs, and getBlob of the blobs the harness has uploaded, must match the reference's; a record referencing an upload from before the kill is written on both sides.

--s3 is a run of the random steps (with --crash-every, and --lifecycle too) with both sides' blobs in buckets of an S3 store the harness runs (harness/lib/s3.mjs), a bucket each, path-style, with timeouts of 2 seconds on both sides. The store verifies every request's signature, and one it refuses, or of an operation it does not know, fails the run. Besides every check above, with the store answering, the steps have it fail, the same way on both sides, and compare the answers (harness/checks/blobstore.mjs):

  • uploadBlob while the store fails every write of an object: with an S3 500 or 503, the connection reset, or no answer. Then getBlob of it, which neither side holds. The server's log must have a blobstore failure line of op put, with the status when there was one.
  • getBlob of a blob a record references while the store fails its reads, the same ways, and again once it reads; and a blobstore failure line of op get.
  • getBlob of such a blob, of 2 bytes or more, on the server alone, while the store answers its read with the object's headers and the first half of its bytes and then closes the connection (the reference is not compared on a failure after the first byte): 200 with the blob's Content-Length, and the answer ending short of it ("The objects", above); its request line's bytes fewer than the blob's size (OPERATIONS.md, "Events").
  • getBlob of such a blob whose object the harness deleted from both buckets, as a backup brought back finds one, and again once it is put back.
  • getBlob of such a blob while the store resets the connection of each side's first read of it, once: on both sides 200, the blob whole.
  • On the server alone, two getBlobs of such a blob 3 s apart, the store keeping a connection open 5 s: the second read goes over the connection the first left open.
  • A record deleted, the last to reference a blob, while the store fails the server's deletes (the reference's go on): both answer alike, and the server logs a blobstore failure of op delete.
  • A blob that stops being held by its record's deletion, uploaded again at once and referenced again: read back whole.
  • The same blob uploaded again while the deletion lands: the store answers the server's PUT half a second late, the record is deleted on both sides meanwhile, then the reference is sent the upload. Both answer it alike, a record referencing it is taken, and it reads back whole.
  • With --crash-every, sometimes an upload to the server alone whose object the store keeps and never answers, the server killed then: after the restart a record referencing it is 400 BlobNotFound, and its object goes (below).

The reference keeps its uploads under keys of its own, so the server's bucket is held to this page alone, after each restart, after the lifecycle run and at the end, within a minute: every object keyed blocks/<did>/<cid>, of an account here, its bytes the CID's; an object for each blob the harness knows an account holds; and none of a blob the account stopped holding, of an account deleted, or of an upload the server was killed during. No file of the data directory may be the bytes of a blob an account holds, and pds_outbound_requests_total of target blobstore must count at least each upload the store kept since the server started (OPERATIONS.md).

node compare.mjs --migrate ... is a run of its own (harness/checks/migrate.mjs), with the S3 store --s3 runs. The server starts with its blobs on disk, the reference with its own on disk throughout. Blob steps of the run (uploads, records with blob refs, getBlob, listBlobs) are compared, then, the server stopped:

  • pds serve, and pds migrate-blobs, beside the running server must exit 1 (COMMANDS.md); migrate-blobs with no --to, or another, is a usage error; --to disk on a data directory keeping its blobs there prints {"moved":0,"bytes":0}.
  • --to bucket, with the store answering each request 100 ms late, is killed at a random moment within 600 ms of the store's first request of each run, and run again until it finishes. The first time it is killed, pds serve must refuse the data directory, and the move is turned back: --to disk, killed too, to its end, the server started on disk and every blob read as the reference's, and the bucket holding no object of the accounts' blobs; then --to bucket again. A kill, of either move, that came after the move was marked finished and before the command exited (7, above) leaves it finished: the next run may print {"moved":0,"bytes":0}; and the first time, pds serve with the environment of the place it moved them to may serve, --to that place then printing {"moved":0,"bytes":0}. Either way the move is taken as done. Each finished run must print the count and bytes of the blobs the harness knows the accounts hold, and --to bucket then {"moved":0,"bytes":0}.
  • Started with the bucket: every blob a record references read as the reference's, listBlobs and a record referencing an upload compared, and the bucket and data directory held as --s3 holds them. More blob steps, then the same refusals beside it.
  • One object deleted, --to disk must exit 1 naming its DID and CID, and pds serve refuse the data directory either way; the object put back, --to disk killed and run again to its end, then served as on disk, the bucket holding no object of the accounts' blobs, and more blob steps.

Not compared, as no client could rely on them: the other headers of getBlob (Cache-Control, Vary, CORS), compression of its answer, a Range request (the reference ignores it), HEAD, a Content-Encoding on an upload (the reference decodes gzip; the harness sends none), the reference's 500s above, a ref of {"$link": …} whose value is not a CID (the reference stores it as a plain map; whether the data model takes it is REPO.md's), the blobs an applyWrites writing one key more than once leaves referenced (above), and the order of the errors when one write has more than one wrong ref, or writes at once reference one new blob with different types; and the reference's getBlob of an unreferenced blob that a write refused for a ref named (above), which it serves or not as a race in it falls.