~ knotless Search/
↑↓ move↵ open⇧↵ all results

chadtmiller.com/knotless

git that lives in your Atmosphere account

knotless.dev
1 collaborator

knotless 0.13.0 Latest 2 packages, 1 image

1
10

A review agent's rounds count only its requests for changes

Merged
Merged chadtmiller.com asked to merge into main from chadtmiller.com/knotless:agent/review-rounds

What changes

Only a request for changes spends a round. knotless-agent review --rounds now limits how many times the agent asks for changes on a pull. Before, it limited how many times it reviewed the pull at all. A pull it approved is reviewed again at each later push. A pull it keeps asking changes of still stops at --rounds.

A review after an approval starts from the approved head. The command is given approved, the head its last review approved, and changes, how many times it has asked for changes. knotless-claude-review reads what changed since the approved head with repo_diff, and judges that.

An answered request is no longer carried forward. When the agent has asked for changes, the author then edits the title or body, and then merges the base in, the agent used to carry its old "changes" verdict to the merge head, because the pull's own diff was unchanged. Now it reviews that head, as it already did when the head stayed the same.

Why

This comes from every review comment on this repository: 759 pulls, 309 reviewed, 565 reviews, 74 requests for changes and 4 dismissals.

  • 21 pulls used all 5 rounds, 11 of them with nothing but approvals. 190 of the 565 reviews were an approval right after an approval. 3 of the 4 dismissals ever written were for a pull the reviewer had approved and then ran out of rounds on, where the change since was only coverage or tests. The cap is there to end a pull argued back and forth, and approvals were using it up.
  • On "Maintainers dismiss reviews", the body answered the review at 07:07. Three minutes later the agent carried its "changes" verdict to a merge of main, and the approval came 50 minutes later, at another push.

Re-reviewing when the author replies in a thread was considered and left out. In 74 requests for changes, an author answered without pushing only twice: once in the body, which already triggers a review, and once where the reviewer was wrong and was dismissed. Every other answer came with a push, which already triggers a review.

How it was checked

an_agent_reviews_pulls now checks that:

  • a pull pushed to after an approval is reviewed again, given approved and changes;
  • a pull asked for changes at every head stops at --rounds;
  • a request for changes followed by a body edit and a merge of the base is reviewed again at the merge head, not carried.

The rest is left to CI.

Shortcomings

Pulls that keep being pushed to after an approval now get one more review per push. Each of those reads only what changed since the approved head, so it should cost less than a full review, but nothing measures that yet.

🤖 Generated with Claude Code

Approved

This makes --rounds count only requests for changes. It passes the command approved and changes, and stops carrying an answered "changes" verdict to a merge-of-base head. I traced the lib.rs logic and found it sound. Carried reviews are dropped before counting, so a carried "changes" isn't counted twice. approved is read before that drop, so a carried approval still names its head. The new tests cover the three behaviours the body lists, and the docs, changeset and prompt are updated to match. One non-blocking gap is in the prompt, noted on the line below.

crates/knotless-agent/claude-reviewline 81
- Ask for changes only for what review alone would catch. What a check the pull must pass already enforces, as a failing test or a line its body must hold, the check fails on: mention it at most, and do not withhold approval for it.
- On a later round, say whether what your earlier reviews asked for was done, and do not ask again for what was.
- An earlier review at the same head commit as now means the author changed the pull's title or body since: judge whether what it says now answers what you asked.
- When "approved" names a commit, your last review approved the pull at it. Read what changed since with repo_diff from that commit to the head, and judge that: what you approved stands unless what came since breaks it. A merge of the base among it brings the base's changes, which are not the pull's: pull_diff shows what the pull itself changes.
review.knotless.dev

repo_diff is approved...head, which diffs from the merge-base. If the author force-pushed or rebased, the approved commit is gone or no longer an ancestor of the head. The diff then fails or shows more than what came since. Say what to do then, for example fall back to pull_diff and review the pull whole.

Merged as 115c4af