Where coding agents actually go wrong

We clustered ~4,000 agent failures across six models. The single biggest cause wasn't a capability gap. It was one sentence in the prompt.

TL;DR: Everyone quotes agent pass rates. Almost nobody can tell you why the failures happened, because a binary test verdict has nothing to say beyond "no." Our rubric-based judge does: every failed requirement comes with a specific reason. So we took six coding agents, ran each on the same 100 SWE-bench Pro tasks, and clustered every one of the ~4,000 failed rubric items by what was asked and why it failed. The headline finding was telling on where optimization often lies: the largest failure cluster across all six models is agents obeying a boilerplate instruction so literally that they refuse work the task explicitly requires. This won’t be fixed by a smarter model, but by reading your failures.

Why should you care?

If you've been following this series, you know the setup: real engineering work mostly can't be scored by running tests, so we score patches against per-task rubrics, which are weighted checklists of binary requirements, graded by a blinded LLM judge. We've shown that this graded score sees quality inside failures, and that when it disagrees with a benchmark's tests, the benchmark is sometimes the one that's wrong.

This post is about another valuable property of the same instrument. A test harness gives you one bit per task. A rubric gives you a verdict per requirement—did the right files change, was the spec satisfied, were tests weakened, does the behavior hold—and for every failed requirement, the judge writes down why. Multiply that across six models and 100 tasks and you get 1,858 scored requirements per model, 11,148 graded verdicts, and roughly 4,000 failures, each with an explanation attached.

That's not a leaderboard. That's a diagnosis. And when you cluster those explanations, the picture of "where agents go wrong" on your codebase becomes a gold mine.

The setup: six agents, 100 tasks, every failure explained

We ran six models (gemini-flash-lite-3.1, claude-haiku-4-5, gpt-5-mini, gemini-flash-3, gemini-pro-3.1, and claude-opus-4-7) on the same 100 SWE-bench Pro tasks under the same harness. Each patch was graded against the task's rubric, whose items fall into four categories:

• FC: file-change: did the right files get touched, and only those?
• SA: spec-alignment: does the change do what the PR asked?
• I:  integrity: no weakened tests, no unrelated churn, no broken compatibility.
• R: runtime: does the behavior actually hold?

Overall item success was 64.5%, ranging from 52.6% (gemini-flash-lite) to 75.3% (claude-opus-4-7). Those are the numbers a leaderboard would report. Everything below is what the leaderboard can't see.

Finding 1: The #1 failure mode is an instruction conflict, not a capability gap

The largest explanation cluster across all six models—247 failures—is agents misreading or over-literally applying task instructions, and one instruction dominates: the harness boilerplate that says "DO NOT MODIFY: Tests, configuration files."

That line exists for a good reason: it stops agents from deleting failing assertions to make their patch "pass." But many SWE-bench Pro rubrics—mirroring the actual merged PRs—require adding test cases or updating an assertion so a boundary test stays meaningful. The agent is now holding two contradictory instructions, and we can watch it choose, in its own words. Here's claude-haiku-4-5 on a Teleport task that required updating a payload-size boundary test:

The agent then shipped a patch that left a boundary test exercising nothing. 

On other tasks, agents wrote perfectly good test cases - into throwaway scripts in /tmp, where the rubric can't see them and no CI ever will. On future-architect/vuls, this single misreading—"do not modify tests" interpreted as "do not add tests"—is the dominant failure mode for the entire repository, 52 failures on its own.

Note this cluster is not a weak-model problem. It's the #1 explanation theme for claude-haiku-4-5 (56 failures) and for claude-opus-4-7 (63 failures)—the best model in the study. If anything, the stronger model reasons its way into the refusal more articulately. You cannot buy your way out of a contradictory prompt with a bigger model. The fix costs one sentence of prompt engineering—distinguishing "don't weaken existing tests" from "don't touch anything test-shaped"—and in this dataset that sentence is worth more than some model upgrades.

Finding 2: Agents fail like sloppy engineers, not stupid ones

The second-largest cluster, 141 failures, is pure change hygiene: temporary debug scripts, helper files, and unrelated edits swept into the final diff—very often by a single reflexive git add -A. One agent fixed a NodeBB function correctly, then committed its scratch test file alongside the fix and introduced quote-style inconsistencies because it edited the source with sed.

That sed detail generalizes. On the runtime axis, the top explanation cluster is unsafe automated text editing: brittle search-and-replace commands that target the wrong lines, overwrite whole files, or silently fail to apply - leaving half-modified implementations the agent never re-read. The code that results isn't wrong because the model couldn't reason about the problem. It's wrong because the edit never landed the way the agent believed it did, and nothing in its loop made it check.

Read enough of these and a pattern emerges: the failures look less like a junior engineer who doesn't understand the codebase and more like a rushed senior one who doesn't verify their own work—staging blindly, editing mechanically, submitting without a final read of the diff. That's actually good news. "Doesn't verify" is a harness problem with harness solutions: a mandatory git status review before submission, structured editing instead of sed, a self-check pass on the final diff. None of it requires a smarter model.

Finding 3: What separates strong models from weak ones (and what doesn't)

With failures broken out by rubric axis, you can see exactly where capability lives:

Three things jump out. Integrity is nearly a solved problem for everyone. Even the weakest model avoids weakening tests or breaking compatibility three times out of four; the spread across models is small. Agents have largely learned not to vandalize.

Spec-alignment and runtime correctness are where the money goes. These axes separate strong from weak models most sharply: flash-lite fails spec-alignment at more than twice Opus's rate. When you pay for a frontier model, what you're buying is follow-through: the change does what the PR actually asked, and the behavior actually holds.

File-change discipline is hard for everyone. Touching exactly the right files, and nothing else, is the highest failure rate on the board even for the best model, at 31.9%. Precision of scope, not raw problem-solving, is the frontier's remaining weakness.

Finding 4: Upgrading a model removes its quirks, not the hard tasks

Here's our favorite cut of the data. For every failed item, we counted how many other models also failed it. 297 requirements were failed by all six models, genuinely hard asks, from preserving an obscure end-of-role marker in Ansible's PlayIterator to handling a space-switch timing gap in Element's room list.

Now look at each model's failures through that lens. For gemini-flash-lite, 20.5% of failures are unique to it—items every other model handled. For claude-opus-4-7, that number is 2.2%. Ten failures out of 458. Meanwhile, roughly two-thirds of Opus's failures are the universal 297 that nobody solved.

That's a precise statement of what a model upgrade buys you: it makes the idiosyncratic failures disappear. What's left over is task hardness—under-specified requirements, deep repository context, genuinely subtle bugs—and no model on the market gets you past those. If your agent keeps failing a class of tasks, the first question isn't "which model is next"; it's "would any model pass this, or is the task the problem?" With per-item cross-model data, that stops being a philosophical question and becomes a lookup.

One more comparison drives it home. The spread between the best and worst model is 22.7 points of item success. The spread between the easiest and hardest repository in the study—qutebrowser at 77.6% versus tutao/tutanota at 35.5%—is 42.1 points. Where your codebase sits matters roughly twice as much as which frontier model you pick. Model selection isn't a global question; it's a per-repository one.

What we're not claiming

Honesty notes, in the tradition of this series.

The judge has a noise floor. 167 items (13%) split exactly 3/3 across the six models—the band most sensitive to judge variance—and near-boundary conclusions there deserve caution. The clusters we've highlighted are the largest ones, well clear of that band, but individual "why it failed" explanations are LLM judgments, not ground truth. We audit this judge continuously precisely because we make claims like these on top of it.

One harness, one run per model. All six models ran under the same scaffold, which is what makes the cross-model comparison clean. But it also means some failure modes (the one-command-at-a-time constraint that pushed agents toward sed, the boilerplate that created the test-file conflict) are properties of the harness-plus-model pair, not the model alone. That's not a weakness of the analysis; it's its point. Your agents also run inside a harness, and its fingerprints are all over their failures too.

Cluster themes are summaries, not measurements. The counts are exact; the prose descriptions of each cluster are model-written characterizations of its members, which we spot-checked against the underlying transcripts.

The takeaway: Stop counting failures and start reading them

Across ~4,000 failures, the recurring lesson is that a pass rate hides the one thing you can act on. "Opus scores 75%, Haiku scores 60%" tells you what a model tier costs. "Your harness prompt is vetoing required test work, your submission step is committing scratch files, and 297 of your tasks are unwinnable as written" tells you what to fix: this week, for free.

And every one of those findings came out of the same rubric scores we already produce for ranking. The explanations were sitting in the failure data; clustering them is cheap. If you're evaluating agents with any judge that produces reasons—and if it doesn't, that's worth fixing first—group your failures by what was asked and why it failed before you spend another dollar on a model comparison. You'll likely find, as we did, that your biggest "model problems" aren't.

This diagnostic layer is part of what Time Machine now produces. When we replay your engineering history to compare models and harnesses on your own repositories, the same clustering runs over your failures: which instructions your agents trip on, which repositories punish them, which tasks no model can pass as specified. You don't just learn which model wins on your work - you learn why the others lose, and how much of that is yours to fix. 

Contact us for a demo.