chadtmiller.com/knotless
git that lives in your Atmosphere account
1
10
A push no longer waits on the appview after it lands
.changeset/push-asks-the-appview-early.mdnew+7−0
| @@ -0,0 +1,7 @@ | ||
--- | ||
default: patch | ||
--- | ||
| ||
# A push no longer waits on the appview after it lands | ||
| ||
`git push` of a branch asks the appview for the pulls it may have landed while the push goes out, where it asked once the push had landed: git hears of the push as soon as the PDS answers it, four round trips sooner over https. A pull opened while the push is on its way is not marked merged by it. | ||
crates/knotless/src/forge.rs+72−15
| @@ -857,13 +857,28 @@ impl Target { | ||
/// The repository `snap` read of `remote`, as [`Target::open`] would | ||
/// have it from there, without reading it again. | ||
pub async fn read(ctx: &Ctx, remote: Remote, snap: Snapshot) -> Result<Target> { | ||
Target::read_with(ctx, remote, snap, None).await | ||
} | ||
| ||
/// [`Target::read`], with `members` those of the organization its owner | ||
/// is who may push to it, when they were asked for already | ||
/// ([`crate::org::public_writers`]). | ||
pub async fn read_with( | ||
ctx: &Ctx, | ||
remote: Remote, | ||
snap: Snapshot, | ||
members: Option<Vec<String>>, | ||
) -> Result<Target> { | ||
let url = &remote.url; | ||
let repo = snap | ||
.repo | ||
.as_ref() | ||
.with_context(|| format!("{url}: no such repository"))?; | ||
let mut record = repo.value.clone(); | ||
crate::org::fill(ctx, &remote, &mut record).await?; | ||
match members { | ||
Some(members) => record.members = members, | ||
None => crate::org::fill(ctx, &remote, &mut record).await?, | ||
} | ||
let previous = previous(ctx, &remote, &record, &repo.rkey).await?; | ||
Ok(Target { | ||
previous, | ||
| @@ -3211,16 +3226,59 @@ pub(crate) async fn push_as( | ||
accepted(report) | ||
} | ||
| ||
/// After a push to `remote` that landed, marks merged each open pull into a | ||
/// branch it moved whose head that branch now has and did not before: a | ||
/// pull fast-forwarded by hand, or a stack of them pushed at once. `before` | ||
/// is the repository as the push read it, `git` where it pushed from. | ||
/// Returns the pulls marked, each with the branch it landed on. A pull | ||
/// whose head commit `git` does not have is left as it is. | ||
pub async fn landed( | ||
/// What a push reads to find the pulls it lands: the repository as the push | ||
/// read it, and every record about it. Begun by [`watch`] as the push goes | ||
/// out, so that git waits for the slower of the two and not for both, and | ||
/// stopped when dropped. | ||
pub struct Watch(tokio::task::JoinHandle<Result<(Target, Vec<Entry>)>>); | ||
| ||
impl Drop for Watch { | ||
fn drop(&mut self) { | ||
self.0.abort(); | ||
} | ||
} | ||
| ||
/// Begins reading what [`landed`] goes by, for a push of `updates` to | ||
/// `remote` as `before` read it: none when the push moves no branch, or | ||
/// the repository is not there yet, or nothing says what its pulls are. A | ||
/// pull opened while the push goes out is not seen, as one the appview has | ||
/// not heard of yet is not. `members` is who may push to it besides those | ||
/// its record names, if that was asked already. | ||
pub fn watch( | ||
ctx: &Ctx, | ||
remote: &Remote, | ||
before: &Snapshot, | ||
updates: &[Update], | ||
members: Option<tokio::task::JoinHandle<Vec<String>>>, | ||
) -> Option<Watch> { | ||
let moves = |u: &Update| u.src.is_some() && u.dst.starts_with("refs/heads/"); | ||
// Pulls on a public repository are read from an appview: with none, | ||
// there are none to mark. | ||
let unread = ctx.env.appview.is_none() && remote.space.is_none(); | ||
if !updates.iter().any(moves) || unread || before.repo.is_none() { | ||
return None; | ||
} | ||
let (ctx, remote, before) = (ctx.clone(), remote.clone(), before.clone()); | ||
Some(Watch(tokio::spawn(async move { | ||
let asked = futures::future::OptionFuture::from(members).await; | ||
let members = asked.and_then(Result::ok); | ||
let t = Target::read_with(&ctx, remote, before, members).await?; | ||
let entries = t.entries(&ctx).await?; | ||
Ok((t, entries)) | ||
}))) | ||
} | ||
| ||
/// After a push to a repository that landed, marks merged each open pull | ||
/// into a branch it moved whose head that branch now has and did not | ||
/// before: a pull fast-forwarded by hand, or a stack of them pushed at | ||
/// once. `watch` is what [`watch`] began before the push, `before` the | ||
/// repository as the push read it, `git` where it pushed from. Returns the | ||
/// pulls marked, each with the branch it landed on. A pull whose head | ||
/// commit `git` does not have is left as it is. | ||
pub async fn landed( | ||
watch: Option<Watch>, | ||
ctx: &Ctx, | ||
before: &Snapshot, | ||
git: &Git, | ||
updates: &[Update], | ||
report: &push::Report, | ||
| @@ -3233,14 +3291,13 @@ pub async fn landed( | ||
moved.insert(branch.to_string(), git.rev_parse(src).await?); | ||
} | ||
} | ||
// Pulls on a public repository are read from an appview: with none, | ||
// there are none to mark. | ||
let unread = ctx.env.appview.is_none() && remote.space.is_none(); | ||
if moved.is_empty() || unread || before.repo.is_none() { | ||
// A push that moved no branch landed none, and waits for no answer. | ||
let Some(mut watch) = watch.filter(|_| !moved.is_empty()) else { | ||
return Ok(Vec::new()); | ||
} | ||
let t = Target::read(ctx, remote.clone(), before.clone()).await?; | ||
let items = assemble(&t.record, t.remote.did(), &t.entries(ctx).await?); | ||
}; | ||
let read = (&mut watch.0).await; | ||
let (t, entries) = read??; | ||
let items = assemble(&t.record, t.remote.did(), &entries); | ||
let mut found = Vec::new(); | ||
for item in items | ||
.iter() | ||
crates/knotless/src/helper.rs+47−4
| @@ -8,6 +8,7 @@ use crate::fetch; | ||
use crate::forge; | ||
use crate::gateway; | ||
use crate::git::Git; | ||
use crate::org; | ||
use crate::push::{self, Report, Update}; | ||
use crate::remote::{PAGE, RECENT, Remote, Snapshot}; | ||
use crate::session::{self, Session}; | ||
| @@ -140,6 +141,10 @@ pub struct Knotless { | ||
pub remote: Remote, | ||
pub git: Git, | ||
snap: Option<Snapshot>, | ||
/// Who may push to the account's public repositories besides those | ||
/// their records name, asked of the appview as git lists the refs to | ||
/// push, with the account asked about: see [`Knotless::ask_early`]. | ||
early: Option<(String, tokio::task::JoinHandle<Vec<String>>)>, | ||
} | ||
| ||
impl Knotless { | ||
| @@ -151,6 +156,7 @@ impl Knotless { | ||
remote, | ||
git, | ||
snap: None, | ||
early: None, | ||
}) | ||
} | ||
| ||
| @@ -232,11 +238,44 @@ fn said(done: Result<()>) { | ||
} | ||
| ||
impl Knotless { | ||
/// Asks the appview who may push to the repository's owner's public | ||
/// repositories, without waiting for the answer. A push that moves a | ||
/// branch goes by it to find the pulls it lands ([`forge::watch`]), and | ||
/// asked now, while the refs are read, it is in by the time the push is | ||
/// ready, over a connection the next question to the appview shares: | ||
/// git waits for neither. None for a private repository, whose members | ||
/// are in its space, and with no appview. | ||
fn ask_early(&self) -> Option<(String, tokio::task::JoinHandle<Vec<String>>)> { | ||
if self.remote.space.is_some() || self.ctx.env.appview.is_none() { | ||
return None; | ||
} | ||
let (ctx, owner) = (self.ctx.clone(), self.remote.did().to_string()); | ||
let about = owner.clone(); | ||
let asked = tokio::spawn(async move { org::public_writers(&ctx, &owner).await }); | ||
Some((about, asked)) | ||
} | ||
| ||
/// Begins reading the pulls a push of `updates` may land, as the push | ||
/// goes out: with what [`Knotless::ask_early`] asked, unless the | ||
/// repository has since been found in another account. | ||
fn watch(&mut self, before: &Snapshot, updates: &[Update]) -> Option<forge::Watch> { | ||
let early = self.early.take(); | ||
let members = early.filter(|(owner, _)| owner == self.remote.did()); | ||
let members = members.map(|(_, asked)| asked); | ||
forge::watch(&self.ctx, &self.remote, before, updates, members) | ||
} | ||
| ||
/// After a push that landed, marks merged the open pulls whose heads | ||
/// it brought into the branches they target, and says so. The push has | ||
/// landed either way, so what fails is said and left. | ||
async fn landed(&self, before: &Snapshot, updates: &[Update], report: &Report) { | ||
let found = forge::landed(&self.ctx, &self.remote, before, &self.git, updates, report); | ||
async fn landed( | ||
&self, | ||
watch: Option<forge::Watch>, | ||
before: &Snapshot, | ||
updates: &[Update], | ||
report: &Report, | ||
) { | ||
let found = forge::landed(watch, &self.ctx, before, &self.git, updates, report); | ||
match found.await { | ||
Ok(marked) => { | ||
for (pull, branch) in marked { | ||
| @@ -267,6 +306,9 @@ impl Knotless { | ||
impl Backend for Knotless { | ||
async fn list(&mut self, for_push: bool) -> Result<Listing> { | ||
self.snap = None; | ||
if for_push { | ||
self.early = self.ask_early(); | ||
} | ||
let snap = self.snapshot(!for_push).await?; | ||
Ok(Listing { | ||
refs: snap | ||
| @@ -293,6 +335,7 @@ impl Backend for Knotless { | ||
| ||
async fn push(&mut self, updates: Vec<Update>) -> Result<Report> { | ||
let snap = self.snapshot(false).await?; | ||
let watch = self.watch(&snap, &updates); | ||
if let Some(me) = session::reader(&self.ctx, &self.remote.identity).await | ||
&& me.did != self.remote.did() | ||
{ | ||
| @@ -300,7 +343,7 @@ impl Backend for Knotless { | ||
let report = | ||
gateway::push(&self.ctx, &self.remote, &snap, &self.git, &me, &updates).await?; | ||
self.claim_email(&me, &report).await; | ||
self.landed(&snap, &updates, &report).await; | ||
self.landed(watch, &snap, &updates, &report).await; | ||
return Ok(report); | ||
} | ||
let auth = session::ensure(&self.ctx, &self.remote.identity).await?; | ||
| @@ -318,7 +361,7 @@ impl Backend for Knotless { | ||
self.after_push(&auth).await?; | ||
} | ||
self.claim_email(&auth, &report).await; | ||
self.landed(&snap, &updates, &report).await; | ||
self.landed(watch, &snap, &updates, &report).await; | ||
Ok(report) | ||
} | ||
} | ||
crates/knotless/src/org.rs+9−4
| @@ -388,14 +388,19 @@ impl From<Answer> for Roster { | ||
pub async fn fill(ctx: &Ctx, remote: &Remote, record: &mut RepoRecord) -> Result<()> { | ||
record.members = match remote.space { | ||
Some(_) => pushing(ctx, remote).await?, | ||
None => { | ||
let roster = asked(ctx, remote.did(), None).await; | ||
roster.map(|r| r.writers(false)).unwrap_or_default() | ||
} | ||
None => public_writers(ctx, remote.did()).await, | ||
}; | ||
Ok(()) | ||
} | ||
| ||
/// The members who push to the public repositories of `owner`, the | ||
/// organization it is, as the appview has them: what [`fill`] names in | ||
/// one. None when it is no organization, or the appview does not answer. | ||
pub async fn public_writers(ctx: &Ctx, owner: &str) -> Vec<String> { | ||
let roster = asked(ctx, owner, None).await; | ||
roster.map(|r| r.writers(false)).unwrap_or_default() | ||
} | ||
| ||
/// The members who push to the private repository `remote` reads: those | ||
/// the record in its space names, who are members still to whoever is | ||
/// logged in, when they are inside the organization, which alone sees who | ||
crates/knotless-serve/tests/forge.rs+6−0
| @@ -2168,6 +2168,12 @@ fn moving_to_another_account() { | ||
// Archived, its pages say so, and nobody writes about it. | ||
ok(&heir, &["repo", "archive", &new]); | ||
fails(&heir, &["comment", "add", &new, &open, "too late"]); | ||
// Nor pushes to it, and a push that lands nothing looks for no pulls | ||
// it landed. | ||
heir.commit_file(&clone, "g", "too late\n"); | ||
let (pushed, _, err) = heir.git_raw(&clone, &["push", "-q", "origin", "main"]); | ||
assert!(!pushed && !err.contains("not marked so"), "{err}"); | ||
heir.git(&clone, &["reset", "-q", "--hard", "HEAD~1"]); | ||
let asked = format!( | ||
"{}/xrpc/dev.knotless.getRepo?repo=at://{}/dev.knotless.repo/moving", | ||
av.base, heir.account.did | ||
docs/design/collaboration.md+9−0
| @@ -659,6 +659,15 @@ names, as `knotless pull merge` would. It says so on git's stderr (`knotless: | ||
the pushing checkout has its head. The push has landed either way: a | ||
failure to read or mark the pulls is said and left. | ||
| ||
Reading the pulls takes two questions to the appview, who may push to the | ||
owner's repositories and what is said about this one, and git should not | ||
wait on them after the push has landed. So the first is asked as git | ||
lists the refs to push, and the second as the push goes out, on the | ||
connection the first opened: by the time the PDS has answered the push, | ||
the appview has answered both. A pull opened while the push is on its way | ||
is not seen, as one the appview has not heard of yet is not, and a push | ||
that moves no branch stops waiting for the answers. | ||
| ||
Details: | ||
| ||
- A squash commit keeps the author and date of the head's tip. The | ||
docs/reference/performance.md+8−6
| @@ -11,7 +11,7 @@ to the PDS, so every operation is planned around them. | ||
| `git fetch` | 1 when the new packs are among the newest 4 and under 192 KiB, as they arrive beside the refs (three reads in parallel); a client further behind adds one for the packs its refs name; larger packs add parallel blob reads | | ||
| `knotless repo commit`, no checkout | 2, plus 3 per lost race | | ||
| A ref's compare-and-swap | 0 extra: its claim record goes in the same `applyWrites` | | ||
| `git push` to a branch, with an appview | 2 more, to the appview, one after the other: who may push, and the pulls the push may have landed | | ||
| `git push` to a branch, with an appview | 0 more that git waits for: who may push, and the pulls the push may have landed, are asked of the appview [while the push goes out](../design/collaboration.md#merged-by-a-push) | | ||
| Connecting | 2 for each server a command reaches over https: TCP's handshake, then TLS's | | ||
| ||
"Warm" means after the first call: the identity, session and repository key | ||
| @@ -32,7 +32,7 @@ round on a repository of its own, with 5 clones a round. | ||
| --- | --- | --- | | ||
| `git push`, one small commit | 296 ms | 320 ms | | ||
| `git fetch`, one new commit | 259 ms | 309 ms | | ||
| `git push` to a branch, with an appview | 507 ms | 516 ms | | ||
| `git push` to a branch, with an appview | 299 ms | 306 ms | | ||
| `knotless repo commit`, no checkout | 435 ms | 469 ms | | ||
| `git clone`, 31 packs | 379 ms | 412 ms | | ||
| `git clone`, compacted | 263 ms | 299 ms | | ||
| @@ -42,9 +42,11 @@ round on a repository of its own, with 5 clones a round. | ||
| 16 agents committing to one branch at once | 8.9 s total, 91 pushes | | | ||
| ||
Every row but the third runs with no appview. A client set up as the | ||
installer leaves it has one, and a push of a branch asks it two things | ||
before git hears the push landed: on a connection of its own, four round | ||
trips more. The appview here is as far away as the PDS. | ||
installer leaves it has one, and a push of a branch asks it two things, on | ||
a connection of its own. It asks while the push goes out, so the push | ||
takes no longer for it; asked after the push had landed, as they were, the | ||
two cost four round trips more, 507 ms. The appview here is as far away as | ||
the PDS. | ||
| ||
Three things are kinder than a PDS on the network. Compaction is off. The | ||
harness's PDS does not wait for its writes to reach the disk. And the two | ||
| @@ -91,7 +93,7 @@ column, measured the same way. | ||
| --- | --- | --- | --- | --- | --- | --- | | ||
| `git push`, one small commit | 92 ms | 296 ms | 496 ms | 898 ms | 4.0 | 2.0 | | ||
| `git fetch`, one new commit | 105 ms | 259 ms | 406 ms | 710 ms | 3.0 | 1.0 | | ||
| `git push` to a branch, with an appview | 108 ms | 507 ms | 911 ms | 1.71 s | 8.0 | 4.0 | | ||
| `git push` to a branch, with an appview | 95 ms | 299 ms | 498 ms | 897 ms | 4.0 | 2.0 | | ||
| `knotless repo commit`, no checkout | 242 ms | 435 ms | 640 ms | 1.05 s | 4.1 | 2.0 | | ||
| `git clone`, 31 packs | 237 ms | 379 ms | 533 ms | 842 ms | 3.0 | 1.0 | | ||
| `git clone`, compacted | 114 ms | 263 ms | 414 ms | 716 ms | 3.0 | 1.0 | | ||
README.md+2−2
| @@ -162,8 +162,8 @@ proxy adding 50 ms to every round trip: | ||
| `knotless repo commit`, no checkout | 2 | 435 ms | | ||
| 16 agents pushing 16 branches at once | 2 each | 561 ms in total | | ||
| ||
With an appview, as the installer sets one, a push of a branch also asks it | ||
which pulls the push landed, and takes 507 ms. | ||
A client with an appview, as the installer sets one, takes no longer: what | ||
a push asks the appview goes out beside the push. | ||
| ||
A repository's size does not change these. With no latency added, | ||
nodejs/node, 49,240 commits and 1.53 GB on the PDS, pushes one small commit | ||