What a reviewer needs to see first
Reading a pull request from the top of the diff was always a slow way to find out what a change does, and agents make it worse. Put the behavior change first, and generate it from the code.
Say you work on a TypeScript service that serves orders over HTTP, and an agent opens a pull request titled “Rename fetchOrder to loadOrder.” fetchOrder is the helper that looks up an order by its ID, and renaming it touches 38 files. The description, which the agent also wrote, says it’s a rename with no change in behavior. For every file but one, that’s true. Each of them swaps one name for the other, and a reviewer can scroll through them quickly.
The remaining file is the helper itself, and while the agent was in there, it changed what the function returns when the order doesn’t exist:
// before
export async function fetchOrder(id: string): Promise<Order | null> {
const row = await db.order.findUnique({ where: { id } });
return row ? toOrder(row) : null;
}
// after
export async function loadOrder(id: string): Promise<Order> {
const row = await db.order.findUnique({ where: { id } });
return row ? toOrder(row) : ({} as Order);
}
The route that serves an order only picked up the new name:
app.get("/orders/:id", async (req, res) => {
const order = await loadOrder(req.params.id);
if (!order) {
return res.status(404).json({ error: "order not found" });
}
return res.json(order);
});
An empty object is truthy in JavaScript, so that if never fires again. A request for an order that doesn’t exist used to get a 404, and now it gets a 200 with an empty object, so every client that checked for the 404 shows a blank order page. The type checker is happy, because the cast told it to be. The tests pass too, unless somebody wrote one for a missing order.
A reviewer could catch it, since the line is right there in the diff. But it’s a single changed line in a 38-file pull request whose description promised nothing would change, and seeing the problem means connecting a return value in one file to a status code in another. Reading from the top at the end of an afternoon of reviews, I’d probably approve it, and I think most reviewers would.
Reviewing is mostly figuring out what changed
We know a fair amount about where review time goes. In 2013, Alberto Bacchelli and Christian Bird sat with 17 developers at Microsoft while they reviewed code and interviewed them afterwards. They also sorted hundreds of comments from Microsoft’s code review tool and surveyed more than a thousand managers and programmers. Finding defects was the top reason people gave for reviewing code, but comments about defects were a small share of what reviewers actually wrote.
What kept coming up, even though they never asked about it, was understanding. One senior developer told them the hardest part of a review is “understanding the reason of the change.” A tester said understanding the code takes up most of the review time. Nearly all of the programmers they surveyed (91 percent) said files they don’t know take longer to review. Some of the developers didn’t find the author’s description of a change much help either. As one of them put it, people “can say they are doing one thing, while they are doing many more of them.”
That was 2013, when a person wrote both the code and the description. Now an agent often writes both, and the description comes from the same place as the code it’s describing. I treat it as the agent’s opinion of its own work, because that’s what it is. Of everything in the pull request, it’s the part a reviewer has the least reason to believe, and the rename’s description was accurate about every file except the one that changed behavior.
Reading from the top wastes a reviewer’s attention
Working out what a change does takes attention, and a reviewer’s attention runs out. Even the companies selling review tools say so. SmartBear, which makes one, says a study it ran of a programming team at Cisco found that you shouldn’t review more than 200 to 400 lines at a time. A review that size, over 60 to 90 minutes, finds most of the defects. It’s a vendor’s study, so I wouldn’t put much weight on the exact numbers, but the general point is hard to argue with.
Google comes at it from the author’s side and tells its engineers to keep changes small. Small changes get reviewed sooner, because it’s easier to find five minutes than a 30-minute block. They also get reviewed more carefully, because large ones bury everyone in comments until important points get missed. The guide adds that a refactor usually belongs in its own change, apart from features and bug fixes.
The rename goes against all of that advice, because it’s 38 files long and hides a behavior change inside a refactor. An agent can produce one like it in a few minutes. I think most teams respond by telling reviewers to be more careful, which stops working the week the number of pull requests doubles. The other common response is to add an AI reviewer, but it reads the same diff in the same order, starting from the top with no better idea of where to look.
Splitting out the behavior change, as Google suggests, would have fixed this if anyone had known there was a behavior change to split out. Nobody did, and in any sense that matters for the review, that includes the agent.
Put the behavior change at the top
Before any of those 38 files, the reviewer needed a short list of what the change does differently when it runs. For this one, the list has a single entry:
GET /orders/:id
when the order does not exist
before: 404 { error }
after: 200 {}
With that list at the top, the review goes differently. The description says nothing changed, and the list says a route now returns something else. That mismatch tells the reviewer where to start: the helper, the cast, and a question for the author. The renamed files get a quick scroll, because the list says none of them changes what the code does.
The list should cover the places where other code depends on yours: which routes the change serves and what each one returns, which fields it reads and writes, which tables it queries and which messages it publishes. Those are the places where a change reaches someone who isn’t in the review, and they’re the hardest to piece together from a line diff. (If the list runs past a screen, the pull request probably needs splitting.)
I think every pull request needs this list, refactors and agent pull requests included. And when the list and the description disagree, the review stops, the same way a failing test stops a merge, until somebody explains the difference.
The list has to come from the code
If the author writes the list, you’re back to checking the author’s story against the diff, which is the whole job the list was supposed to save you. The agent that wrote ({} as Order) would never have listed the 404, because as far as it knew, it hadn’t touched one. A model asked to summarize the pull request has the same problem: it reads the same diff and has no more reason to spot the truthiness bug than the agent that wrote it.
So generate the list from the code on both sides of the change, with something that reads the code the same way every time. Tests can help. A test that asked for a missing order and expected a 404 would’ve failed, and that failure tells you about the behavior change too. But tests only cover what somebody thought to check. A description generated from the code covers the routes nobody wrote a test for.
That’s what we build suss to do. It reads TypeScript, Python and Ruby and writes down what each handler, consumer and query does, without using AI, and suss inspect --diff compares that description before and after a change.
Where your attention goes now
With the list on top, a review runs in a different order:
- Read the list before the diff, and check each entry against what the pull request says it does.
- Save the slow, line-by-line reading for the entries nobody mentioned, since those are the changes the author may not know about.
- If the list is empty, make sure the description agrees, then read the diff for structure and naming, which goes quickly.
- A refactor that turns out to change behavior goes back, with a request to put the behavior change in its own pull request.
You still read the code, but now you know what you’re looking for before you open the first file. Bacchelli and Bird saw something like this with reviewers who already knew the files: most of the programmers they surveyed (82 percent) said those reviewers give different feedback, and the paper calls it deeper and more detailed. The list gives a reviewer who doesn’t know the files some of that head start, and with agents writing more of the code, fewer reviewers are going to know the files they’re reviewing.
Whoever merges a change owns it, which means being able to say what it does when it runs. Starting from 38 files and a promise that nothing changed makes that much harder than it needs to be.