~ knotless Search/
↑↓ move↵ open⇧↵ all results

chadtmiller.com/knotless
Switch repository
Loading…

git that lives in your Atmosphere account

knotless.dev
1 collaborator

knotless 0.13.0 Latest 2 packages, 1 image

1
10

Records in spaces are read as written, with no public-form references left

Merged chadtmiller.com asked to merge into main from chadtmiller.com/knotless:remove-short-form
.changeset/space-short-form-moved.mddeleted+0−7
@@ -1,7 +0,0 @@
---
default: patch
---
# Records in a private repository that name another in the public form are written again in full
The first time knotless writes into a private repository's space for you, it rewrites the records you wrote there whose `subject`, `pull`, `reply`, `check` or pull head names another in the public form, `at://<author>/<collection>/<rkey>`, to the full form, through your own login, once. What you typed in a body is left as you wrote it.
crates/knotless/src/forge.rs+0−1
View file
@@ -988,7 +988,6 @@ impl Target {
.as_ref()
.map(|s| Space::owner(ctx, &s.uri, &me)),
};
crate::short_form::move_once(ctx, &mine, &me.auth()).await;
Ok((mine, me))
}
crates/knotless/src/lib.rs+0−1
View file
@@ -57,7 +57,6 @@ pub mod reviews;
pub mod sealed;
pub mod secrets;
pub mod session;
pub mod short_form;
pub mod social;
pub mod space;
pub mod space_sync;
crates/knotless/src/remote.rs+4−36
View file
@@ -258,24 +258,6 @@ impl Remote {
at.resolved(self.space_ref().as_ref()).to_string()
}
/// `raw`, read from here: in a space, with the references it holds in
/// the public form given the space.
fn parse<T: DeserializeOwned>(&self, raw: RawRecord) -> Result<Stored<T>> {
self.parse_as(raw, true)
}
/// [`Remote::parse`], or with `resolve` false, `raw` as it was written.
fn parse_as<T: DeserializeOwned>(
&self,
mut raw: RawRecord,
resolve: bool,
) -> Result<Stored<T>> {
if let Some(space) = self.space_ref().filter(|_| resolve) {
crate::short_form::resolve_refs(&mut raw.value, &space);
}
parse(raw)
}
/// The space this one points to, for a space named for a later name of
/// a private repository.
pub async fn home(&self) -> Result<Option<String>> {
@@ -347,7 +329,7 @@ impl Remote {
.read::<RawRecord>("getRecord", &[("collection", collection), ("rkey", rkey)])
.await;
match res {
Ok(raw) => self.parse(raw).map(Some),
Ok(raw) => parse(raw).map(Some),
Err(err) if XrpcError::of(&err).is_some_and(|e| e.error == "RecordNotFound") => {
Ok(None)
}
@@ -473,7 +455,7 @@ impl Remote {
return Ok(page);
}
page.next = Some(rkey.to_string());
page.items.push(self.parse(raw)?);
page.items.push(parse(raw)?);
}
if !full {
page.next = None;
@@ -704,25 +686,11 @@ impl Remote {
let limit = limit.to_string();
let params = [("collection", collection), ("limit", limit.as_str())];
let out: ListOut = self.read("listRecords", &params).await?;
out.records.into_iter().map(|r| self.parse(r)).collect()
out.records.into_iter().map(|r| parse(r)).collect()
}
/// Every record in `collection`, from the public repo or the space.
pub async fn list_all<T: DeserializeOwned>(&self, collection: &str) -> Result<Vec<Stored<T>>> {
self.list_with(collection, true).await
}
/// [`Remote::list_all`], each record as it was written, its references
/// in the public form left so.
pub async fn list_as_written(&self, collection: &str) -> Result<Vec<Stored<Value>>> {
self.list_with(collection, false).await
}
async fn list_with<T: DeserializeOwned>(
&self,
collection: &str,
resolve: bool,
) -> Result<Vec<Stored<T>>> {
let mut all = Vec::new();
let mut cursor: Option<String> = None;
let limit = PAGE.to_string();
@@ -734,7 +702,7 @@ impl Remote {
let out: ListOut = self.read("listRecords", &params).await?;
let full = out.records.len() == PAGE;
for raw in out.records {
all.push(self.parse_as(raw, resolve)?);
all.push(parse(raw)?);
}
match out.cursor {
Some(next) if full => cursor = Some(next),
crates/knotless/src/short_form.rsdeleted+0−252
@@ -1,252 +0,0 @@
//! Moves references written in the public form, `at://<author>/<collection>/<rkey>`,
//! for records in a private repository's space, to their full form there
//! ([`crate::aturi`]): as they are read ([`resolve_refs`]), and in the
//! records themselves, rewritten through their author's own session the
//! first time knotless acts for them in that space ([`move_once`]). This
//! module is removed, with knotless-serve's `index::short_form`, once the
//! appview counts no record holding one.
//!
//! A record in a space is about that space's repository, and the records
//! it names by these fields are too, so a reference there in the public
//! form names a record of that space. What is typed, as a command's
//! argument or a `Fixes` in a pull's body, is not moved: [`crate::aturi`]
//! reads it.
use crate::aturi::{self, AtUri, SpaceRef};
use crate::env::Ctx;
use crate::records::nsid;
use crate::remote::{self, Remote};
use crate::telemetry::{self, Fields, Level};
use crate::xrpc::Auth;
use anyhow::Result;
use serde_json::Value;
use std::collections::HashSet;
use std::sync::Mutex;
/// The collections an account writes into a repository's space that can
/// name another of its records.
const WRITTEN: [&str; 11] = [
nsid::PULL,
nsid::ISSUE,
nsid::COMMENT,
nsid::STATE,
nsid::HIDE,
nsid::BLOCK,
nsid::QUEUE,
nsid::LABEL,
nsid::REACTION,
nsid::CHECK,
nsid::RERUN,
];
/// The most writes one `applyWrites` of moved records carries.
const WRITES: usize = 10;
/// The accounts and spaces moved in this process, as `<did> <space>`.
static MOVED: Mutex<Option<HashSet<String>>> = Mutex::new(None);
/// The fields of knotless's records that name another of its records in
/// the same space: `(field, field of each item when it is a list or an
/// object)`.
const FIELDS: [(&str, Option<&str>); 9] = [
("subject", None),
("subject", Some("uri")),
("pull", None),
("reply", None),
("review", None),
("reverts", None),
("check", None),
("batch", Some("pull")),
("checks", Some("check")),
];
/// `value`, a record read from `space`, with each reference in the public
/// form that [`FIELDS`] names given the space. Says whether it moved one.
pub fn resolve_refs(value: &mut Value, space: &SpaceRef) -> bool {
let mut moved = false;
for (field, inner) in FIELDS {
let Some(found) = value.get_mut(field) else {
continue;
};
match (inner, found) {
(None, at) => moved |= resolve(Some(at), space),
(Some(inner), Value::Array(items)) => {
for item in items {
moved |= resolve(item.get_mut(inner), space);
}
}
(Some(inner), object) => moved |= resolve(object.get_mut(inner), space),
}
}
moved
}
/// A pull's head repository, when `value` names it in the public form.
pub fn head(value: &Value) -> Option<AtUri> {
let uri = value.pointer("/head/repo/uri")?.as_str()?;
AtUri::parse(uri).ok().filter(|at| at.space.is_none())
}
/// Rewrites the records `mine`, an account's repo in a space, holds with
/// a reference in the public form, in full, as `auth`: once for each
/// account and space, as this process and the ones before it noted in the
/// cache. A failure is logged, and tried again the next time. Says how
/// many it rewrote, when it tried and could.
pub async fn move_once(ctx: &Ctx, mine: &Remote, auth: &Auth) -> Option<u64> {
let space = mine.space_ref()?;
let key = format!("{} {space}", mine.did());
let path = ctx.env.cache_dir.join("short-form-moved");
{
let mut moved = MOVED.lock().unwrap();
let moved = moved.get_or_insert_with(|| {
let kept = std::fs::read_to_string(&path).unwrap_or_default();
kept.lines().map(str::to_string).collect()
});
if !moved.insert(key.clone()) {
return None;
}
}
let fields = |count, error| Fields {
did: Some(mine.did()),
source: Some(&space.skey),
count,
error,
..Fields::default()
};
match move_all(ctx, mine, &space, auth).await {
Ok(n) => {
telemetry::note(
Level::INFO,
"short-form references moved",
&fields(Some(n), None),
);
let _ = std::fs::create_dir_all(&ctx.env.cache_dir);
let kept = std::fs::read_to_string(&path).unwrap_or_default();
let _ = std::fs::write(&path, format!("{kept}{key}\n"));
Some(n)
}
Err(err) => {
let failed = fields(None, Some(format!("{err:#}")));
telemetry::note(Level::WARN, "short-form references not moved", &failed);
MOVED.lock().unwrap().get_or_insert_default().remove(&key);
None
}
}
}
/// [`move_once`]'s rewrite: how many records it rewrote.
async fn move_all(ctx: &Ctx, mine: &Remote, space: &SpaceRef, auth: &Auth) -> Result<u64> {
let mut writes = Vec::new();
for collection in WRITTEN {
for r in mine.list_as_written(collection).await? {
let mut value = r.value;
let mut moved = resolve_refs(&mut value, space);
if let Some(home) = head_space(ctx, &value).await {
let head = head(&value).expect("a head named");
value["head"]["repo"]["uri"] = head.resolved(Some(&home)).to_string().into();
moved = true;
}
if moved {
writes.push(remote::update(collection, &r.rkey, value));
}
}
}
for some in writes.chunks(WRITES) {
mine.apply(some.to_vec(), auth).await?;
}
Ok(writes.len() as u64)
}
/// The space of the head repository `value`, a pull, names in the public
/// form, when it is a private one this account reads.
async fn head_space(ctx: &Ctx, value: &Value) -> Option<SpaceRef> {
let head = head(value)?;
let url = crate::forge::repo_url(&head.to_string()).ok()?;
Remote::open(ctx, &url).await.ok()?.space_ref()
}
/// Gives `at`, a string in the public form, `space`.
fn resolve(at: Option<&mut Value>, space: &SpaceRef) -> bool {
let Some(at) = at else {
return false;
};
let Some(uri) = at.as_str() else {
return false;
};
let Ok(parsed) = AtUri::parse(uri) else {
return false;
};
if parsed.space.is_some() || !parsed.author.starts_with("did:") {
return false;
}
*at = Value::String(aturi::resolve(uri, Some(space)));
true
}
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
const S: &str = "at://did:o/space/dev.knotless.space/r";
#[test]
fn references_in_the_public_form_take_the_space() {
let space = SpaceRef::parse(S).unwrap();
let pull = "at://did:a/dev.knotless.pull/p";
let full = format!("{S}/did:a/dev.knotless.pull/p");
let mut comment = json!({
"repo": {"uri": "at://did:o/dev.knotless.repo/r", "key": "k"},
"subject": pull,
"reply": "at://did:b/dev.knotless.comment/c",
"body": format!("see {pull}"),
});
assert!(resolve_refs(&mut comment, &space));
assert_eq!(comment["subject"], full);
assert_eq!(
comment["reply"],
format!("{S}/did:b/dev.knotless.comment/c")
);
// A repository is matched by its owner and key; what is typed stays.
assert_eq!(comment["repo"]["uri"], "at://did:o/dev.knotless.repo/r");
assert_eq!(comment["body"], format!("see {pull}"));
// Moved, it moves no more.
assert!(!resolve_refs(&mut comment, &space));
let mut reaction = json!({"subject": {"uri": pull, "cid": "c"}});
assert!(resolve_refs(&mut reaction, &space));
assert_eq!(reaction["subject"]["uri"], full);
let mut check = json!({"pull": pull, "batch": [{"pull": pull}, {"commit": "x"}]});
assert!(resolve_refs(&mut check, &space));
assert_eq!(check["pull"], full);
assert_eq!(check["batch"][0]["pull"], full);
assert!(check["batch"][1].get("pull").is_none());
let mut queue =
json!({"subject": pull, "checks": [{"check": "at://did:r/dev.knotless.check/k"}]});
assert!(resolve_refs(&mut queue, &space));
assert_eq!(
queue["checks"][0]["check"],
format!("{S}/did:r/dev.knotless.check/k")
);
}
#[test]
fn what_is_no_reference_in_the_public_form_is_left() {
let space = SpaceRef::parse(S).unwrap();
// An organization's acceptance names the organization by DID.
for mut value in [
json!({"subject": "did:o"}),
json!({"subject": 4}),
json!({"subject": "at://alice.test/dev.knotless.pull/p"}),
json!({"subject": {"cid": "c"}}),
json!({"checks": "x", "batch": 1}),
json!({"pull": format!("{S}/did:a/dev.knotless.pull/p")}),
] {
let was = value.clone();
assert!(!resolve_refs(&mut value, &space), "{was}");
assert_eq!(value, was);
}
}
}
crates/knotless/tests/library.rs+0−96
View file
@@ -1009,99 +1009,3 @@ fn concurrent_commits_replay_onto_the_new_tip() {
assert!(clone.join(format!("f50-{i}")).exists());
}
}
/// What an account wrote into a private repository's space naming another
/// record there in the public form is written again in full, once, as it
/// next writes there; what it typed stays as it was.
#[test]
fn references_in_the_public_form_in_a_space_are_moved_once() {
if !net().spaces {
return;
}
let s = Sandbox::new();
let url = s.url("moving");
assert_eq!(s.cli(&["create", &url, "--private"], b"").0, 0);
let rt = rt();
let ctx = ctx(&s);
rt.block_on(async {
let r = Remote::open(&ctx, &RepoUrl::parse(&url).unwrap())
.await
.unwrap();
let space = r.space_ref().expect("a private repository");
let auth = auth(&ctx, &r).await;
let key = r.repo().await.unwrap().unwrap().value.key;
let did = r.did().to_string();
let named = json!({"uri": format!("at://{did}/{}/moving", nsid::REPO), "key": key});
let issue = format!("at://{did}/{}/i", nsid::ISSUE);
let now = records::now();
let written = [
(
nsid::ISSUE,
"i",
json!({"repo": named, "title": "t", "createdAt": now}),
),
(
nsid::PULL,
"p",
json!({"repo": named, "title": "t", "base": "main", "createdAt": now,
"head": {"repo": named, "branch": "b"}}),
),
(
nsid::COMMENT,
"c",
json!({"repo": named, "subject": issue, "body": format!("see {issue}"),
"createdAt": now}),
),
];
let writes = written
.iter()
.map(|(c, rkey, v)| remote::create(c, rkey, v.clone()))
.collect();
r.apply(writes, &auth).await.unwrap();
assert_eq!(
knotless::short_form::move_once(&ctx, &r, &auth).await,
Some(2)
);
let one = |c: &'static str| {
let r = &r;
async move { r.list_as_written(c).await.unwrap().remove(0).value }
};
let comment = one(nsid::COMMENT).await;
assert_eq!(
comment["subject"],
format!("{space}/{did}/{}/i", nsid::ISSUE)
);
assert_eq!(comment["body"], format!("see {issue}"));
let pull = one(nsid::PULL).await;
let home = format!("{space}/{did}/{}/moving", nsid::REPO);
assert_eq!(pull["head"]["repo"]["uri"], home);
// Once: not again in this process, nor in the next.
assert_eq!(knotless::short_form::move_once(&ctx, &r, &auth).await, None);
let kept = std::fs::read_to_string(ctx.env.cache_dir.join("short-form-moved")).unwrap();
assert_eq!(kept, format!("{did} {space}\n"));
// A public repository has nothing to move.
let public = Remote::connect(&ctx, &r.url).await.unwrap();
assert_eq!(
knotless::short_form::move_once(&ctx, &public, &auth).await,
None
);
});
// One that cannot be read is tried again the next time.
let other = s.url("unread");
assert_eq!(s.cli(&["create", &other, "--private"], b"").0, 0);
let refusing =
Router::new().fallback(|| async { axum::http::StatusCode::INTERNAL_SERVER_ERROR });
let addr = mock(&rt, refusing);
rt.block_on(async {
let mut r = Remote::open(&ctx, &RepoUrl::parse(&other).unwrap())
.await
.unwrap();
let auth = auth(&ctx, &r).await;
r.xrpc = knotless::xrpc::Xrpc::new(reqwest::Client::new(), &format!("http://{addr}/"));
assert_eq!(knotless::short_form::move_once(&ctx, &r, &auth).await, None);
let kept = std::fs::read_to_string(ctx.env.cache_dir.join("short-form-moved")).unwrap();
let space = r.space_ref().unwrap().to_string();
assert!(!kept.contains(&space), "{kept}");
});
}
crates/knotless-run/src/check.rs+0−4
View file
@@ -318,10 +318,6 @@ impl Reporter {
url: of.url.clone(),
space,
};
if mine.space.is_some() {
let auth = session::ensure(&self.ctx, &self.identity).await?.auth();
knotless::short_form::move_once(&self.ctx, &mine, &auth).await;
}
Ok(mine)
}
crates/knotless-serve/src/index/short_form.rsdeleted+0−335
@@ -1,335 +0,0 @@
//! Moves what the index keeps of records from spaces, and the references
//! they hold, from the public form of their URIs to their full form in
//! the space ([`knotless::aturi`]), and counts the records whose authors
//! wrote a reference in the public form still. This module, and
//! [`knotless::short_form`], are removed once that count is zero
//! ([`Index::short_form_left`]).
use super::*;
use knotless::telemetry::{self, Fields, Level};
/// The records from spaces that hold a reference in the public form as
/// their authors wrote them, by URI, with the author; when the index's own
/// rows were moved, once; and the space records by name, for a pull's head
/// repository's space to be found.
pub(super) const TABLES: &str = "CREATE TABLE IF NOT EXISTS short_form (
uri TEXT PRIMARY KEY, author TEXT NOT NULL);
CREATE TABLE IF NOT EXISTS short_form_moved (at TEXT NOT NULL);
CREATE INDEX IF NOT EXISTS space_record_key ON space_record (did, collection, rkey);";
/// The tables a record's value is kept in, whichever its collection: what
/// it is put again from, under its full URI.
const VALUED: [&str; 6] = ["forge", "check_run", "repo", "release", "minutes", "theme"];
/// The tables that keep a record by URI with no value to put it again
/// from: its row is renamed in place.
const RENAMED: [&str; 6] = ["reaction", "star", "follow", "block", "push", "moved"];
/// `value`, a record of `did`'s at `uri` from `space`, with the references
/// it holds in the public form given their spaces: those of its own
/// space, and a pull's head repository the one the index keeps that
/// repository's record in. Notes whether its author wrote one so. None
/// when it moves none, or `space` is empty.
pub(super) fn resolved(
db: &Connection,
space: &str,
uri: &str,
did: &str,
value: &Value,
) -> rusqlite::Result<Option<Value>> {
let Ok(space) = SpaceRef::parse(space) else {
return Ok(None);
};
let mut value = value.clone();
let mut moved = knotless::short_form::resolve_refs(&mut value, &space);
if let Some(head) = knotless::short_form::head(&value) {
let home: Option<String> = db
.query_row(
"SELECT space FROM space_record WHERE did = ?1 AND collection = ?2 AND rkey = ?3",
params![head.author, nsid::REPO, head.rkey],
|row| row.get(0),
)
.optional()?;
if let Some(home) = home.and_then(|s| SpaceRef::parse(&s).ok()) {
value["head"]["repo"]["uri"] = head.resolved(Some(&home)).to_string().into();
moved = true;
}
}
match moved {
true => db.execute(NOTED, [uri, did])?,
false => forgot(db, uri)?,
};
Ok(moved.then_some(value))
}
/// Notes that the record at `uri`, by its author, holds a reference in the
/// public form.
const NOTED: &str = "INSERT OR REPLACE INTO short_form VALUES (?1, ?2)";
/// Notes that the record at `uri` holds no reference in the public form,
/// as when it is gone.
pub(super) fn forgot(db: &Connection, uri: &str) -> rusqlite::Result<usize> {
db.execute("DELETE FROM short_form WHERE uri = ?1", [uri])
}
/// Whether the index's own rows were moved.
const DONE: &str = "SELECT EXISTS (SELECT 1 FROM short_form_moved)";
impl Index {
/// Moves every record from a space that the index keeps under the
/// public form of its URI to its full one, with the references it
/// holds: once, noted as done, before anything reads the index.
pub(super) fn move_short_form(&self) -> rusqlite::Result<()> {
let records: Vec<(String, String, String, String)> = {
let db = self.db.lock().unwrap();
let done: bool = db.query_row(DONE, [], |row| row.get(0))?;
if done {
return Ok(());
}
let mut stmt = db.prepare(
"SELECT space, did, collection, rkey FROM space_record
-- scans: space_record, once, as the index moves to full URIs",
)?;
let rows = stmt.query_map([], |row| {
Ok((row.get(0)?, row.get(1)?, row.get(2)?, row.get(3)?))
});
rows?.collect::<rusqlite::Result<_>>()?
};
for (space, did, collection, rkey) in &records {
self.move_one(space, did, collection, rkey)?;
}
let db = self.db.lock().unwrap();
db.execute(
"INSERT INTO short_form_moved VALUES (?1)",
[knotless::records::now()],
)?;
self.stamps.mark_all();
Ok(())
}
/// [`Index::move_short_form`], of one record.
fn move_one(
&self,
space: &str,
did: &str,
collection: &str,
rkey: &str,
) -> rusqlite::Result<()> {
let short = format!("at://{did}/{collection}/{rkey}");
let value: Option<String> = {
let db = self.db.lock().unwrap();
let mut found = None;
for table in VALUED {
let sql = format!("SELECT value FROM {table} WHERE uri = ?1");
found = db.query_row(&sql, [&short], |row| row.get(0)).optional()?;
if found.is_some() {
break;
}
}
found
};
// Put again, it is kept as one from the space is now; taken out
// under its old URI first, all its rows go with it.
if let Some(value) = value.and_then(|v| serde_json::from_str::<Value>(&v).ok()) {
self.delete_at("", did, collection, rkey)?;
return self.put_at(space, did, collection, rkey, &value);
}
let full = record_uri(space, did, collection, rkey);
let db = self.db.lock().unwrap();
for table in RENAMED {
let sql = format!("UPDATE OR REPLACE {table} SET uri = ?2 WHERE uri = ?1");
db.execute(&sql, [&short, &full])?;
}
// A new ref's first version is kept under the key its claim would
// have had.
if collection == nsid::REF {
let claimed = |uri: &str| format!("{uri}.0").replace(nsid::REF, nsid::CLAIM);
db.execute(
"UPDATE OR REPLACE push SET uri = ?2 WHERE uri = ?1",
[claimed(&short), claimed(&full)],
)?;
}
if collection == nsid::REACTION {
let space = SpaceRef::parse(space).ok();
let subject: Option<String> = db
.query_row(
"SELECT subject FROM reaction WHERE uri = ?1",
[&full],
|row| row.get(0),
)
.optional()?;
let resolved = subject.map(|s| knotless::aturi::resolve(&s, space.as_ref()));
let sql = "UPDATE reaction SET subject = coalesce(?2, subject) WHERE uri = ?1";
db.execute(sql, params![full, resolved])?;
}
Ok(())
}
/// How many records from spaces each author wrote that name another
/// in the public form still, the most first: what is left for
/// [`knotless::short_form`] to move.
pub fn short_form_left(&self) -> rusqlite::Result<Vec<(String, u64)>> {
let db = self.db.lock().unwrap();
let mut stmt = db.prepare(
// scans: short_form, a small table that only shrinks
"SELECT author, count(*) n FROM short_form GROUP BY author ORDER BY n DESC, author",
)?;
let rows = stmt.query_map([], |row| Ok((row.get(0)?, row.get(1)?)))?;
rows.collect()
}
/// Logs [`Index::short_form_left`]: a line for each author, and one in
/// all.
pub(super) fn say_short_form_left(&self) -> rusqlite::Result<()> {
let left = self.short_form_left()?;
for (author, n) in &left {
let fields = Fields {
did: Some(author),
count: Some(*n),
..Fields::default()
};
telemetry::note(Level::INFO, "short-form references left", &fields);
}
let fields = Fields {
count: Some(left.iter().map(|(_, n)| n).sum()),
..Fields::default()
};
telemetry::note(Level::INFO, "short-form references left in all", &fields);
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
/// What an index from before keeps of a private repository's records,
/// by their URIs in the public form, is moved to their full ones once,
/// as it opens, and what their authors wrote so is counted.
#[test]
fn an_index_from_before_is_moved_once() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("i.db");
let space = knotless::space::uri("did:o", "r");
let short =
|did: &str, collection: &str, rkey: &str| format!("at://{did}/{collection}/{rkey}");
let full =
|did: &str, collection: &str, rkey: &str| format!("{space}/{did}/{collection}/{rkey}");
let named = json!({"uri": "at://did:o/dev.knotless.repo/r", "key": "k"});
let pull = short("did:a", nsid::PULL, "p");
{
// As an index before kept them: by the public form, the
// references as written.
let index = Index::open(&path).unwrap();
let db = index.db.lock().unwrap();
db.execute("DELETE FROM short_form_moved", []).unwrap();
drop(db);
let repo = json!({"name": "r", "key": "k", "createdAt": "2026-01-01"});
let opened = json!({"repo": named, "title": "t", "base": "main",
"head": {"repo": named, "branch": "b"}, "createdAt": "2026-01-02T00:00:00Z"});
let said = json!({"repo": named, "subject": pull, "body": "b",
"createdAt": "2026-01-03T00:00:00Z"});
let reacted = json!({"subject": {"uri": pull, "cid": "c"}, "content": "heart"});
let made = json!({"repo": "k", "name": "refs/heads/main", "target": "a1",
"generation": 1, "updatedAt": "2026-01-02T00:00:00Z"});
for (did, collection, rkey, value) in [
("did:o", nsid::REPO, "r", &repo),
("did:a", nsid::PULL, "p", &opened),
("did:b", nsid::COMMENT, "c", &said),
("did:b", nsid::REACTION, "h", &reacted),
("did:o", nsid::REF, "k.m", &made),
("did:o", nsid::ORG_MEMBER, "did:x", &json!({})),
] {
index.put(did, collection, rkey, value).unwrap();
let db = index.db.lock().unwrap();
db.execute(
"INSERT INTO space_record VALUES (?1, ?2, ?3, ?4)",
params![space, did, collection, rkey],
)
.unwrap();
}
}
let index = Index::open(&path).unwrap();
let entries = index.forge("did:o", "k").unwrap();
let uris: Vec<&str> = entries.iter().map(|e| e.uri.as_str()).collect();
let (opened, said) = (
full("did:a", nsid::PULL, "p"),
full("did:b", nsid::COMMENT, "c"),
);
assert!(
uris.contains(&opened.as_str()) && uris.contains(&said.as_str()),
"{uris:?}"
);
assert!(
!uris
.iter()
.any(|u| u.starts_with("at://did:a/") || u.starts_with("at://did:b/"))
);
let pulled = entries.iter().find(|e| e.uri == opened).unwrap();
let home = full("did:o", nsid::REPO, "r");
assert_eq!(pulled.value["head"]["repo"]["uri"], home);
let comment = entries.iter().find(|e| e.uri == said).unwrap();
assert_eq!(comment.value["subject"], opened);
assert!(
index
.repo(&full("did:o", nsid::REPO, "r"))
.unwrap()
.is_some()
);
let db = index.db.lock().unwrap();
let one = |sql: &str| {
db.query_row(sql, [], |row| row.get::<_, String>(0))
.unwrap()
};
assert_eq!(
one("SELECT subject FROM comment -- scans: comment, a test's one row"),
opened
);
assert_eq!(
one("SELECT uri FROM reaction -- scans: reaction, a test's one row"),
full("did:b", nsid::REACTION, "h")
);
assert_eq!(
one("SELECT subject FROM reaction -- scans: reaction, a test's one row"),
opened
);
assert_eq!(
one("SELECT uri FROM push -- scans: push, a test's one row"),
full("did:o", nsid::CLAIM, "k.m.0")
);
drop(db);
// The pull's author wrote its head in the public form, and the
// comment's its subject.
let left = |n: &[&str]| n.iter().map(|d| (d.to_string(), 1)).collect::<Vec<_>>();
assert_eq!(index.short_form_left().unwrap(), left(&["did:a", "did:b"]));
index.say_short_form_left().unwrap();
// Moved once: opened again, nothing more is done, and the comment
// written again in full is counted no more.
drop(index);
let index = Index::open(&path).unwrap();
let again = json!({"repo": named, "subject": opened, "body": "b",
"createdAt": "2026-01-03T00:00:00Z"});
index
.put_from_space(&space, "did:b", nsid::COMMENT, "c", &again)
.unwrap();
assert_eq!(index.short_form_left().unwrap(), left(&["did:a"]));
// Nor is one deleted.
index
.put_from_space(
&space,
"did:b",
nsid::COMMENT,
"d",
&json!({"repo": named, "subject": pull}),
)
.unwrap();
assert_eq!(index.short_form_left().unwrap(), left(&["did:a", "did:b"]));
index
.delete_from_space(&space, "did:b", nsid::COMMENT, "d")
.unwrap();
assert_eq!(index.short_form_left().unwrap(), left(&["did:a"]));
index.say_short_form_left().unwrap();
}
}
crates/knotless-serve/src/index.rs+9−19
View file
@@ -29,7 +29,6 @@ use std::time::{Duration, Instant};
use tokio::sync::{broadcast, mpsc};
mod shared;
mod short_form;
pub use shared::{STEP, Shared, Version, kind_of};
/// The kinds of preference kept, in `pref`: how the pages are drawn, and
@@ -525,10 +524,8 @@ pub(crate) fn connect(path: &Path) -> rusqlite::Result<(Connection, bool)> {
DROP VIEW IF EXISTS temp.down;
CREATE TABLE IF NOT EXISTS down (subject TEXT PRIMARY KEY);
{shared}
{short_form}
{triggers}",
shared = shared::TABLES,
short_form = short_form::TABLES,
triggers = down_triggers(),
))?;
// What is down, from what makes it so, as it stands now: the
@@ -778,8 +775,6 @@ impl Index {
stamps: Stamps::new(STAMPS_MAX),
kept: Mutex::new(Kept::new(KEPT_ENTRIES)),
};
index.move_short_form()?;
index.say_short_form_left()?;
Ok(index)
}
@@ -807,8 +802,6 @@ impl Index {
) -> rusqlite::Result<()> {
let uri = record_uri(space, did, collection, rkey);
let db = self.db.lock().unwrap();
let resolved = short_form::resolved(&db, space, &uri, did, value)?;
let value = resolved.as_ref().unwrap_or(value);
let text = value.to_string();
// Whether an account says it is a bot, of every account, for the
// feed to leave out what bots say and push.
@@ -1165,7 +1158,6 @@ impl Index {
let uri = record_uri(space, did, collection, rkey);
let db = self.db.lock().unwrap();
self.tell(|| deleted_about(&db, &uri));
short_form::forgot(&db, &uri)?;
org_rows(&db, space, did, collection, rkey, None)?;
// Whose pulls and issues it changes, stamped once all is deleted.
let scope = |sql: &str| db.query_row(sql, [&uri], |row| row.get(0)).optional();
@@ -5365,14 +5357,17 @@ mod tests {
assert_eq!(index.name("did:a", "k").unwrap().as_deref(), Some("r"));
}
/// A record from a space is kept by its URI in full there, and the
/// references it holds in the public form are given their spaces.
/// A record from a space is kept by its URI in full there, with the
/// references it holds as written.
#[test]
fn records_in_a_space_are_kept_by_their_full_uri() {
let dir = tempfile::tempdir().unwrap();
let index = Index::open(&dir.path().join("i.db")).unwrap();
let space = knotless::space::uri("did:o", "r");
let forks = knotless::space::uri("did:f", "f");
let full =
|did: &str, collection: &str, rkey: &str| format!("{space}/{did}/{collection}/{rkey}");
let head = format!("{forks}/did:f/{}/f", nsid::REPO);
let fork = json!({"name": "f", "key": "fk", "createdAt": "2026-01-01"});
index
.put_from_space(&forks, "did:f", nsid::REPO, "f", &fork)
@@ -5381,20 +5376,18 @@ mod tests {
index
.put_from_space(&space, "did:o", nsid::REPO, "r", &repo)
.unwrap();
let named = json!({"uri": "at://did:o/dev.knotless.repo/r", "key": "k"});
let named = json!({"uri": full("did:o", nsid::REPO, "r"), "key": "k"});
let pull = json!({"repo": named, "title": "t", "base": "main",
"head": {"repo": {"uri": "at://did:f/dev.knotless.repo/f", "key": "fk"}, "branch": "b"},
"head": {"repo": {"uri": head, "key": "fk"}, "branch": "b"},
"createdAt": "2026-01-02T00:00:00Z"});
index
.put_from_space(&space, "did:a", nsid::PULL, "p", &pull)
.unwrap();
let said = json!({"repo": named, "subject": "at://did:a/dev.knotless.pull/p",
let said = json!({"repo": named, "subject": full("did:a", nsid::PULL, "p"),
"body": "b", "createdAt": "2026-01-03T00:00:00Z"});
index
.put_from_space(&space, "did:b", nsid::COMMENT, "c", &said)
.unwrap();
let full =
|did: &str, collection: &str, rkey: &str| format!("{space}/{did}/{collection}/{rkey}");
let entries = index.forge("did:o", "k").unwrap();
let uris: Vec<&str> = entries.iter().map(|e| e.uri.as_str()).collect();
let (pull_uri, comment_uri) = (
@@ -5408,10 +5401,7 @@ mod tests {
let comment = entries.iter().find(|e| e.uri == comment_uri).unwrap();
assert_eq!(comment.value["subject"], pull_uri);
let opened = entries.iter().find(|e| e.uri == pull_uri).unwrap();
assert_eq!(
opened.value["head"]["repo"]["uri"],
format!("{forks}/did:f/{}/f", nsid::REPO)
);
assert_eq!(opened.value["head"]["repo"]["uri"], head);
// A repository is found by its record's URI in either form.
let short = "at://did:o/dev.knotless.repo/r";
assert!(index.repo(short).unwrap().is_some());
docs/design/access.md+3−23
View file
@@ -114,29 +114,9 @@ form the space of the repository it is about (`Target::named`). A key made
from one, as a label's or a pull's checks', is made from the public form,
so it is the same for both (`crates/knotless/src/aturi.rs`).
Some records in spaces name what they refer to in the public form. Two
modules, `crates/knotless/src/short_form.rs` and
`crates/knotless-serve/src/index/short_form.rs`, exist to move them to the
full form, and are removed once nothing is left to move:
- Read from a space, by a member or by the appview's sync, each reference
in a field that names a record of the same space is given that space,
so a comment that names its pull so is still on it. The appview's index
also gives a pull's head repository the space it keeps that
repository's record in.
- The first time knotless writes into a space for an account (a command,
a page, the review or triage bot, the runner reporting a check), it
rewrites that account's records there that hold such a reference, in
full, through the account's own session or delegation, once, noted in
its cache (`short-form-moved`).
- The appview moves what its index keeps under a public-form URI to the
full one once, as it opens, noted in `short_form_moved`. Each start it
logs `short-form references left`, by author, and
`short-form references left in all`: the records it indexes whose
authors wrote one. Once that count is zero, the two modules go.
What people typed, as a URI in a body, is left as they wrote it, and read
by `aturi::resolve`, which stays.
A record knotless writes into a space names what it refers to there in
full, and is read as written. What people typed, as a URI in a body, is
left as they wrote it, and read by `aturi::resolve`.
Spaces are in the reference PDS's permissioned-data alpha. A PDS without
them answers the space methods as unknown, and every repository on it is