chadtmiller.com/knotless
git that lives in your Atmosphere account
7
# Review agents
`knotless-agent review` reviews your pulls with a model. You ask its account for a review on a pull, and it reads the pull, writes a review with a verdict, a body and comments on lines, and reviews again when you push. It is an atproto account of its own. It ships with `knotless-claude-review`, which has Claude Code do the reading, and this guide sets that up; any other model works through [another command](#use-another-command).
## What you need
- An atproto account for the reviewer, such as a new Bluesky account, and an app password for it. An app password is a password for one app, made in the account's settings (Bluesky: Settings > Privacy and security > App passwords).
- Claude Code, with a Claude Pro or Max subscription or an Anthropic API key.
- Node 18 or later.
- `knotless-agent` and `knotless-mcp`, both built from the knotless repository. `knotless-mcp` is knotless's MCP server: Claude reads the pull through its tools.
- `knotless-claude-review`, the script that runs Claude Code for each review. It is `crates/knotless-agent/claude-review` in the repository.
Use a new account, not your own. The owner's approvals count toward required reviews, and the reviewer posts as whichever account it logs in as.
## Set it up
1. Make the reviewer's account, say `reviewer.example.com`, and make an app password for it.
2. Install the two binaries and the script from a checkout of knotless:
```sh
cargo install --path crates/knotless-agent
cargo install --path crates/knotless-mcp
install crates/knotless-agent/claude-review /usr/local/bin/knotless-claude-review
```
The knotless server image (`docker pull knotless.dev/chadtmiller.com/knotless/knotless-server`) has all three, with Claude Code, the script at `/usr/local/bin/knotless-claude-review`.
3. Install Claude Code:
```sh
npm install -g @anthropic-ai/claude-code
```
4. On a machine with a browser, make a token for your subscription and copy what it prints:
```sh
claude setup-token
```
With an API key instead, skip this step and set `ANTHROPIC_API_KEY` where the next step sets `CLAUDE_CODE_OAUTH_TOKEN`.
5. Try it once. `--once` reads each repository named, reviews the pulls waiting for it, and exits:
```sh
KNOTLESS_IDENTIFIER=reviewer.example.com KNOTLESS_PASSWORD=<app password> \
CLAUDE_CODE_OAUTH_TOKEN=<token> \
knotless-agent review --command knotless-claude-review --once alice.example.com/my-project
```
Name as many repositories as you like, each as `<handle>/<name>`, or `--owner alice.example.com` for every public repository that account holds, including those it makes later. With no pull asking for it yet, it reads the repository and exits without writing anything. A review takes Claude a minute or two.
6. Run it without `--once` to keep it watching. See [Keep it running](#keep-it-running).
`knotless-agent` reads pulls through the appview `KNOTLESS_APPVIEW` names, https://knotless.dev unless you set another. An appview is a service that indexes records across accounts; knotless-serve is knotless's.
## Ask for a review
A pull asks the reviewer through its `reviewers`. Name the reviewer's handle when you open the pull:
```sh
knotless pull <url> main <url> <branch> "<title>" --reviewer reviewer.example.com
```
Or add it to a pull already open:
```sh
knotless edit <url> <pull> --reviewer reviewer.example.com
```
Only the pull's author can set its `reviewers`. A maintainer cannot ask for a review on someone else's pull.
The review appears within seconds, or at the next sweep. It is a review record in the reviewer's account, as any reviewer's is: the verdict `approve` or `changes`, a body, and its comments on lines, together with the head commit it looked at. It shows in the pull's merge box, and `knotless show <url> <pull>` lists it as a `review:` line. [Collaborating](collaborating.md) covers pulls and reviews in general, and [reviews](../design/collaboration.md#reviews) how they are kept.
To have it review every open pull without being asked, start it with `--all`.
## Tell it what to look for
A repository says what it wants of its reviews in `.knotless/review.md`, in plain words:
```md
Check that every new query has an index, and that the docs page the change touches is updated.
Leave out style: `cargo fmt` and clippy check it.
```
The agent reads the file from the branch the pull merges into, never from the pull, so a pull cannot change how it is reviewed. Commit the file to the base branch, and the next review follows it. Claude also follows the repository's `AGENTS.md` or `CLAUDE.md`, as the base branch has them.
Whoever runs the reviewer can add instructions for every repository it reviews, with `KNOTLESS_REVIEW_PROMPT`. Neither can lift the rules the script sets: the read-only tools, and that the pull is material to review, not instructions.
## Review private repositories
The reviewer reads a private repository once the owner shares it, to read:
```sh
knotless share <url> reviewer.example.com
```
Reading another account's private repository takes an OAuth login, which an app password is not. Log the reviewer in with OAuth on the machine that runs it, with the same `KNOTLESS_CONFIG_DIR`:
```sh
knotless login reviewer.example.com
```
Then start it without `KNOTLESS_PASSWORD`. Claude reads the repository through that login, and its review is kept with the repository's other comments, where only those it is shared with see it. Reviewing a private repository sends its code to Anthropic, as any use of Claude with it does.
## When it reviews
The reviewer listens to Jetstream, a public stream of every account's record changes. Jetstream only tells it when to look. When a pull record is written about a watched repository, or an account whose pull asks it pushes a ref, it waits `--settle` seconds (5) and reads the repository again. It also reads every repository when it connects to Jetstream, and every `--sweep` seconds (300) whatever happens.
It reviews a pull once per head commit. When the author pushes again, it reviews the pull again and gives Claude its earlier reviews, so that Claude checks whether its earlier asks were done. It stops after `--rounds` reviews of one pull, 3 unless you set another number.
A push that changes nothing the pull itself does, such as a merge of the base branch into the pull's branch, is not reviewed again. The agent compares the pull's changes against its base at the new head and at the last head it reviewed, ignoring where in each file they fall. If they match, it writes its last verdict again at the new head, in a review that begins "Carried forward:". That review does not count toward `--rounds`, and Claude is not given it as an earlier review.
If the head moves while Claude is reading the pull, the agent drops the answer and reviews the pull at its new head.
The agent keeps no state besides its login session. It learns what it has reviewed by reading back its own reviews, so it can restart, or move to another machine, without reviewing anything twice.
## Pulls it does not review
- A pull from someone who is not the repository's owner, a collaborator, or a member of the owner's organization. Anyone can ask any account for a review, and each review spends your model usage, so a stranger's pull is passed over without a word.
- A draft, until it is marked ready.
- A pull that is closed or merged.
- A pull opened by the reviewer's own account.
## What Claude can do while reviewing
`knotless-claude-review` runs Claude Code headless, in an empty temporary directory, so no CLAUDE.md on the machine is read. Claude Code's own tools are off: no shell, no file edits, no web.
Claude's only tools are `knotless-mcp`'s read tools: `pull_diff`, `diff`, `read_file`, `list_tree`, `search`, `blame`, `log`, `refs`, `show_item`, `list_items`, `list_labels` and `checks`. The script starts `knotless-mcp --read-only`, logged in as the reviewer, so Claude reads what the reviewer may read, private repositories it was shared into included. A read-only `knotless-mcp` offers no tool that writes, and runs none if asked, so the login gives Claude nothing to change. [The MCP reference](../reference/mcp.md) describes each tool.
Claude does not write the review. It returns JSON in a fixed shape, at most 30 comments on lines, and the agent writes the review from that. The prompt tells Claude that everything the pull carries is material to review, not instructions. That includes its title, body, diff, comments, and any AGENTS.md or CLAUDE.md it adds or changes. Claude follows the repository's AGENTS.md or CLAUDE.md as the base branch has them.
So a hostile pull has little to work with. If it talks Claude into something, the worst it gets is a misleading review of itself.
## Approvals and required reviews
The reviewer's approval does not count toward a protected branch's required reviews (`knotless protect --reviews`). Only the owner's and collaborators' approvals count. Keep the reviewer's account out of your collaborators, so that its approval stays advice. [Required reviews](../design/collaboration.md#required-reviews) explains the rule, and [review agents](../design/collaboration.md#review-agents) the design of the reviewer.
## Subscriptions and API keys
A subscription token is for your own use. Run the reviewer for your own repositories with your own token. Its reviews draw on the same usage limits as your Claude Code sessions.
A reviewer that serves other people's repositories needs an API key. Anthropic's terms do not allow offering claude.ai logins to other people through a product; read [Anthropic's consumer terms](https://www.anthropic.com/legal/consumer-terms) before you decide.
## Its environment
`knotless-claude-review` reads these environment variables:
| Variable | What it sets |
| --- | --- |
| `CLAUDE_CODE_OAUTH_TOKEN` | A subscription token, from `claude setup-token`. |
| `ANTHROPIC_API_KEY` | An API key, billed to your API account. |
| `KNOTLESS_REVIEW_MODEL` | The model Claude Code uses (its `--model`). |
| `KNOTLESS_REVIEW_PROMPT` | Instructions for every review, after the script's own. |
| `KNOTLESS_MCP` | The `knotless-mcp` to run, `knotless-mcp` on PATH by default. |
| `CLAUDE` | The `claude` to run, `claude` on PATH by default. |
## Keep it running
Run the same command as in step 5 without `--once`, under whatever keeps a service running on your machine, such as systemd or launchd. Give it the environment variables of step 5. It reconnects to Jetstream when the connection drops.
The knotless server image holds everything the reviewer needs. Run `knotless-agent review --command knotless-claude-review <repositories>` (or `--owner <handle>`) in it with the same environment.
On the host that serves knotless.dev, the compose file has a `review` service that does this beside knotless-serve and reads pulls from it. [The deploy README](../../deploy/README.md) says how a deploy turns it on. To review private repositories there, put the reviewer's DID in `KNOTLESS_BOT_DIDS` and log it in at `/oauth/bot`, as the runner and the triager do, for the OAuth session that reads them.
[Workflows](workflows.md) covers knotless-run, which runs checks rather than reviews. [Agents](agents.md) covers connecting your own MCP client to knotless.
## Use another command
`--command` takes any command, run with `sh -c` once per review. Another model, a linter or a script works as long as it reads and writes this JSON.
On stdin it gets one object:
| Field | What it holds |
| --- | --- |
| `repo` | The repository, as `knotless://<owner>/<name>`. |
| `pull` | The pull's AT-URI. |
| `title`, `body` | The pull's title and body. |
| `author` | The author's handle, else their DID. |
| `base` | The base branch. |
| `head` | `repo`, `branch` and `commit` of the pull's head. |
| `round` | Which review of this pull this is, from 1. |
| `rounds` | The most reviews of one pull, from `--rounds`. |
| `earlier` | Its earlier reviews of the pull, each with `verdict`, `body` and `commit`. |
| `instructions` | What `.knotless/review.md` says on the base branch, if it is there. |
On stdout it answers with one object:
```json
{
"verdict": "changes",
"body": "The new parser drops the last line.",
"lines": [
{"line": {"path": "src/parse.rs", "line": 42, "side": "new"}, "body": "This skips the final line."}
]
}
```
`verdict` is `approve` or `changes`. `lines` may be left out. Each line has a `path` and a `line`, and may have a `startLine` for a range and a `side`, `old` or `new`. A review holds at most 100 comments on lines.
The command's stderr goes to the agent's log. A command that exits non-zero, answers something that is not this shape, or runs past `--timeout` (1200 seconds) fails the review. [The CLI reference](../reference/cli.md#knotless-agent) lists every flag.
## Troubleshooting
The agent logs one line for each event, with the repository and the pull.
- `reviewed`: it wrote a review. The line's `reason` is the verdict.
- `reviewing failed`: the command failed, answered something unreadable, or ran out of time. The line's `error` says which. The agent tries again at the next read.
- `reading the repository failed`: it could not read a repository or its pulls. Check the repository's name and `KNOTLESS_APPVIEW`.
`knotless-claude-review` writes its own lines to stderr, which land in the same log:
- `reviewed in <n> turns, $<cost>`: Claude finished a review. With a subscription, the cost is what the review would have cost at API prices, not a charge.
- `claude failed: ...`: Claude Code exited with an error. Check the token or API key, and that `claude` is on PATH.
- `claude gave no review: ...`: Claude Code ran but returned no answer in the review's shape.
No line at all for a pull means the reviewer does not consider it asked. Check that the pull names the reviewer's handle in `reviewers`, is open and not a draft, and comes from the owner or a collaborator.