A second look at our own merged PRs
Code is arriving faster than teams can check it.
Maintainers surveyed by the Linux Foundation estimate a quarter of incoming PRs are “AI slop.” Bain calls it a “reliability paradox,” software is getting cheaper to build but costlier to run.
Our Speed Trap report, covering 22,000 developers, found unreviewed merges up 76.3%, PR size up 71.8%, and time in QA up 300.6%. More code is getting through, and the work of checking it is piling up downstream.
We put 28 PRs that humans had already approved and merged through our new AI review system to see what had slipped through. Here’s what it caught, where the system went wrong, and how we changed our process.
Critical findings reviewers had missed
Our review harness, built on Open Code Review (OCR), runs two independent reviewers in parallel, one on an Anthropic model and one on an OpenAI model. Findings are rated P0 to P4. Before approval, every P0 and P1 must be fixed or explicitly dismissed.
On the 28 historical PRs, the harness found between 3 and 14 critical issues per PR that the human review had not raised.
Not one of the 28 came back clean.
Most of the original human reviews were a single word: LGTM.
These were not style nits. On a live PR to our routing-decisions backend, the review caught cached input being double-counted, reporting $5.15 in savings instead of $1.15. It also flagged a connection-handling error that could skip cleanup and trigger a full retry. A human merged the PR with that P1 still open. The fix landed a few days later because the finding was there in writing.
None of us were surprised that a careful reviewer found issues. What surprised us was how much confidence we had been placing in reviews that had quietly stopped catching anything.
The failure mode on the other side
An AI reviewer will always find something. Ask it to look for P0s and P1s and it will find P2s and P3s on the way. If the loop treats every finding as a reason to fix and re-review, it does not converge. One of our engineers had a review run across an entire night and into the next morning, until a VPN disconnect finally stopped it. A documentation PR went through 13 rounds with one reviewer, each round surfacing only P2 and P3 findings.
The cause was a helper skill that had been added to the flow to auto-apply findings marked "fix now," including the low-severity ones. Every small fix changed the diff, every changed diff earned a fresh review, and the fresh review found more small things.
The gate was still P0/P1, but the loop was not.
The fix was to make the gate and the loop the same thing. Once every reviewer clears P0 and P1, the PR is approved. Lower-severity findings and the author's disposition on each stay on the PR as a record and do not extend it.
Without a severity cutoff, an AI reviewer will keep finding things to fix and keep charging you for it.
The Speed Trap in our own code
Our Speed Trap report describes a pipeline moving faster while verification and rework pile up downstream. Our 28 PRs show how that happens.
Unreviewed merges rose 76.3%, compared with 31.3% six months earlier, while PR size grew 71.8%. Teams have more code to check and are increasingly letting it through unchecked. Restarts per developer rose 66.7%, meaning more work has to be revisited when the first attempt goes wrong.
The Linux Foundation survey adds to that picture: respondents estimate AI cuts code-authoring time by 52%, while 59% of maintainers report growing contribution volume, rising to 69% on large projects. Faster authoring is adding pressure to an already stretched review process.
There is a promising countermeasure. Teams in our report using agentic review heavily see faster first reviews and lower change failure rates. Those findings are correlational, but our experiment shows how it can help: examining changes in detail and leaving specific findings for authors to address. Better context during authoring remains the earlier opportunity to prevent those defects.
How we changed our reviews
We did five things, in this order.
- Ran the reviewer on history before trusting it on live PRs. The 28-PR replay turned "the review process is fine" from an assumption into a number.
- Set the gate on severity, not on a clean bill. P0 and P1 block. Everything else is recorded and dismissed by a person, with a reason on the PR.
- Kept two reviewers independent. Different model families, run in parallel, no shared context in the first pass.
- Made the loop stoppable and the failures loud. A human can halt a run, and infrastructure errors end the run instead of retrying into the night.
- Measured the review itself. Every round, finding, fix and disposition is logged with its cost.
Two things we have not solved: A merged P1 is still possible when a human approves over an open finding, and we are deciding whether that should be allowed at all or only with a written dismissal. And the cost of review scales with how much the code changes between rounds, which is really a statement about authoring quality. The long reviews were long because the code needed the work.
If you take one thing away from this, check whether your process is doing its job. We found missed defects and review loops that wouldn’t end, then changed the rules. Don’t wait for a production incident to find out what needs fixing
Taking a closer look at how we work
LGTM doesn’t tell you how closely anyone read the code. We went back through 28 approved PRs and found enough missed defects to rethink how we worked.
Adding AI review helped, but we also had to decide which findings should block a merge and when to stop asking for another pass. There’s more to do. Better context for the agents writing the code should mean fewer mistakes for anyone to catch.
