Agreement isn't verification

On one run in my personal agent experiments, eight reviewing agents looked at the same work and all eight signed off. Nobody had an open objection. The work still had three defects in it. One file was well over its size limit, a test depended on an entry that didn't exist, and a third problem sat in an unrelated part of the codebase. None of them were subtle. Any of them would have shown up the moment someone ran the checks.

The reviewers weren't bad. In that same run they caught four real design problems that none of the automated checks were built to catch. They were just answering a different question. Consensus among agents measures coherence, meaning whether the reasoning holds together. Execution is how you measure correspondence, meaning whether the work actually matches reality.

Why agents agree

It's tempting to treat AI reviewers like careful colleagues, where agreement means several people checked independently and came to the same answer. It doesn't work that way. Models can sincerely believe weak work is good. In a group, they drift toward agreeing with each other. And when a reviewer reads the author's confident summary first, it tends to adopt that framing and review the summary instead of the work.

Adding more reviewers doesn't fix it on its own. If they share the same assumptions and start from the same framing, you get more agreement, which feels like more confidence, and it's built on the same blind spots.

Let execution answer execution questions

The first fix is to stop asking reviewers questions that a machine can answer for certain. Does it build? Do the tests pass? Is the file under the limit? Those get settled before the work is ever offered for review. If the checks fail, it never reaches a reviewer at all. The author fixes it and tries again.

The tempting alternative is to tell every reviewer to run the checks themselves. That wastes the one thing reviewers are good for. Their attention is scarce, and it belongs on design, correctness and intent, which are the problems checkers aren't built to find. The system can do the mechanical part once, deterministically, for everyone. A reviewer may still run a targeted test to settle a specific question. That's investigation, and the shared checks still run.

A partial test run is limited evidence, and it shouldn't be treated as more than that. On that same run, the handoff reported a few thousand tests passing and one failing, from a run that had been narrowed to the parts of the code that changed. The full suite was more than twice as large, and it caught the third defect, which lived somewhere the narrowed run never looked. Narrowing is a fine way to go fast, but it can't prove the whole thing works. And even a full passing suite only vouches for the behavior its tests actually cover.

Make approval cost something

"Looks good to me" is cheap to say whether or not you looked. So an approval has to point at specific things, like the files, lines, commits or tests it's vouching for, and a generic approval is rejected outright. The same goes for objections. It's hard to fake specific evidence without doing the reading.

Judge first, then read the author's take

Reviewers can see the work as soon as it exists, but they can't see the author's self-assessment until they've given their own verdict. They're told that a proposal is waiting, so nobody sits idle, but the author's framing stays hidden until each reviewer has formed a view of their own.

The same idea applies to any shared material reviewers start from. It should carry evidence and leave the conclusions to the reviewers, in a plain, mechanical order with no "areas of concern" and nothing ranked by importance. Emphasis is a form of anchoring. If whoever gathers the evidence editorializes, every reviewer inherits that framing, and agreement between them stops meaning anything.

Grade every claim

The last piece is honesty about how sure a reviewer actually is. Every potential finding gets one of three verdicts. It's confirmed when the reviewer can point to the exact requirement that isn't met and the code that fails it. It's plausible when it looks wrong but the evidence is incomplete, and the reviewer says what would settle it. It's refuted when the requirement turns out to be met, and the reviewer quotes where.

Only a confirmed finding, with a concrete way it fails, can block the work. Everything plausible is kept but downgraded to advice, so the signal survives without forcing another round of revisions. And the burden of evidence runs both ways. A reviewer who drops a finding has to show why, just as much as one who keeps it.

Reviewing a contract

The same split matters when agents review knowledge work, where "running the tests" isn't obvious. Three agents agreeing that a contract has no risky clauses tells you very little. What tells you something is checking against the source. Does every cited case exist? Is every quotation verbatim? Does each flagged risk point to a specific clause? Those are the execution checks of legal work, and they should run before any reviewer weighs in. After that, reviewers should judge the document before they read the drafter's summary of it, cite the clause behind every finding, and say plainly what they're sure of and what they aren't.