Why we built an AI model routing evaluation framework

In our recent model routing experiment, we compared how different AI coding models and harnesses performed on real engineering work. The goal was to determine which combinations could produce comparable results at lower cost, and where stronger models were still worth the premium.

Given how consequential these decisions are, the AI model routing evaluation framework behind them needs to be credible. If a company reroutes coding work based on a score, it is betting engineering time and budget on that score being meaningful and trustworthy.

Why not simply run tests to know whether a code change works?

When engineers want to know whether a code change works, the obvious approach is to run the test suite. Running tests requires two things:

1. A working environment with the correct dependencies, versions, services, and state.
2. Tests that cover the behavior the patch is intended to change.

When those conditions are met, tests produce a clear verdict: the patch passes or fails—and that binary result is useful when deciding whether code is ready to merge. [Note: In this article, we use the word patch to mean the code change produced for a task—the same kind of diff an engineer would review in a pull request.]

For model evaluation, however, a pass-or-fail result can be too coarse. A patch that solves nearly the entire problem but misses one edge case receives the same result as an empty or irrelevant patch. Both fail, even though they reflect very different levels of progress. When comparing models, we often want to know not only whether an attempt fully succeeded, but how close it came to solving the task.

There is also a more fundamental limitation: on real codebases, the conditions required to run tests often do not hold. Many tasks in our evaluations come from pull requests merged months or years ago. The original environment may no longer exist, and rebuilding it may be impractical or impossible. Test coverage may also be incomplete, and some tasks have no test that directly captures what the change was meant to accomplish.

Public benchmarks such as SWE-bench reduce these problems by including only tasks whose environments and tests can be reproduced. That makes them highly useful for research, but it also limits them to the subset of engineering work that can be evaluated through test execution.

How the framework evaluates model and harness routes

Our approach is different. Importantly, we replay a company's actual engineering history, including work for which the original environment or complete test coverage may no longer be available. Then, to evaluate that broader set of tasks, we use a method that does not depend on executing the code.

Instead of reducing every patch to a binary pass or fail, we score it against a rubric that captures the intended behavior of the original change. Rubric-based scoring preserves partial progress, distinguishes nearly correct solutions from irrelevant ones, and lets us evaluate work that cannot be reproduced through tests alone.

What is a rubric score?

For every task, we create a rubric before evaluating any candidate solutions. The rubric is a weighted checklist describing the important properties of a correct solution. Its criteria are specific to the task and repository. They may refer to the actual files that should change, the functions involved, the root cause of the issue, or the behavior the patch should introduce. We avoid vague criteria such as "the patch should be correct." Instead, the rubric asks concrete questions that can be answered by inspecting the proposed change.

Then, a separate LLM judge receives three pieces of information: the task description, the candidate patch, and the rubric. For each criterion, the judge answers yes or no. The final score is the weighted proportion of criteria the patch satisfies. A score of 0.70 means the patch satisfied 70% of the rubric by weight.

Two design choices are especially important:

1. The judge is blind. The judge does not know which model or harness produced the patch. It sees only the task, the proposed code change, and the rubric. This reduces the risk that brand, model reputation, or other irrelevant information will influence the result.
2. The judge is constrained. The judge is not asked to freely rate the code or assign an impressionistic score. Instead, it answers a fixed set of specific yes-or-no questions. This makes the evaluation more structured and helps produce consistent results across repeated runs.

How does a rubric score differ from test pass rates?

Rubric scores and test results measure different things.

Running tests executes the code, so it can catch real runtime failures. But it requires a working environment and relevant test coverage, and the result is usually binary. Tests can also be passed without solving the underlying problem—for example, by hard-coding the exact value a test expects.

A rubric score, on the other hand, does not require an environment or existing tests. Instead, it evaluates the proposed change itself: whether it addresses the root cause, stays within scope, and fits the codebase. It also provides a graded score rather than a simple pass or fail. The limitation is that the rubric never runs the code. A strong patch can score highly while still missing an edge case that causes the test suite to fail.

So we should expect the two methods to agree directionally, not perfectly. Higher scores should correspond to a greater likelihood of success, and lower scores to a greater likelihood of failure.

How we validate the rubric score against ground truth

A quality metric that never runs the code should not be accepted on faith alone. We validate the framework using public SWE-bench datasets, where candidate patches already have ground-truth results from a real test harness. Each patch is labeled either:

• Resolved: the patch passed the benchmark tests
• Unresolved: the patch failed them

We apply three main checks, and we repeat them for every new repository and each major generation of the system.

Check 1: Known-correct solutions must score well

Each benchmark task includes a gold patch: the real fix that was ultimately accepted. If our rubric gave low scores to these known-correct solutions, that would suggest it was asking for the wrong things.

In a recent evaluation, gold patches received a mean score of 0.80 and a median score of 0.85. They did not all receive a perfect 1.0. That is not necessarily a problem. The rubric is designed to grade the quality and completeness of a solution, not automatically approve any patch known to have shipped.

Check 2: The judge must give the same answer twice

For a score to be trustworthy, it should not change substantially simply because the judge was run again. We therefore score the same patches in three independent judge runs and compare the results.

In a recent experiment, the typical variation was approximately ±0.02, with a correlation of about 0.96 between repeated runs. This suggests that differences substantially larger are actual differences, and unlikely to be caused by judge randomness.

Check 3: The score must rank real successes above failures

This is the central validation test. For each task, we compare a patch that passed the benchmark tests with one that failed and ask: how often does the rubric assign the successful patch a higher score?

We summarize that result using AUC, or area under the curve. An AUC of 0.5 is no better than chance; 1.0 represents perfect ranking. Across the SWE-bench Verified repositories we have validated, per-task AUC ranges from 0.75 to 0.86. That means the rubric reliably, though not perfectly, ranks successful work above unsuccessful work.

Rankings can be seen in the image below. Patches that really passed cluster at high scores, while patches that failed are spread across the whole range.

The overlap between the two groups is expected. A patch may solve most of a task but miss one edge case, causing it to fail the test suite while still earning a high rubric score. The rubric is intended to preserve that graded information, not reproduce a binary test result exactly.

The strongest evidence: the score sees quality inside failures

The most revealing test looks only at patches that failed. If the rubric measures the quality of the work—not merely whether it passed—then stronger models should produce better failed attempts than weaker models. Their unsuccessful patches should, on average, come closer to a correct solution. For example, Claude-Opus-4.7 solves far more benchmark tasks than Claude-Haiku-4.5, so its failures should also land closer to the target, and the rubric should pick that up.

We tested this using six models with different benchmark success rates. For each model, we looked only at their failing patches and calculated the average rubric score of those failures. The failure scores closely tracked the models' actual success rates, with a Spearman correlation of +0.94. In other words, models that solved more tasks also tended to fail more productively.

This matters because a pass/fail benchmark assigns every patch in this analysis the same score: zero. The rubric still recovers meaningful differences in quality, distinguishing an attempt that was close to usable from one that made little progress.

What a trustworthy AI evaluation framework makes possible

Numbers need evidence. AI-judged scores now influence real decisions, including which model writes your code, so they deserve the same scrutiny as any other engineering control. Building this framework taught us two broader lessons.

First, in agentic engineering, generating code is not the scarce capability; evaluating it is. AI models can produce plausible patches at scale, but it's challenging to know which ones are actually good. Once you can determine that, everything else unlocks downstream.

Second, a trustworthy score compounds over time. The same number that judges one patch can pick the best of several attempts, choose between models, and serve as the feedback signal that makes agents better over time. All of this is possible only when that score is a measured instrument.

At Faros, that is how we use it. The score ranks multiple attempts at the same task, compares models and harnesses on real work, and tracks quality across repositories and over time. Time Machine applies this framework to your engineering history, helping answer which model should handle which work, when to escalate, and where cost savings hold up across your repositories, task mix, and review standards. Contact us to schedule a demo.

AI model routing evaluation framework FAQ

Does the judge execute the code?

No. Scoring is fully execution-free: no environment, no test runs.

What does the judge see?

The task description, the candidate diff, and the task's rubric. Nothing else.

Does the judge know which model produced the patch?

No. The judge is blind to the model, harness, and route; labels are kept as separate metadata.

Where do the rubrics come from?

They are generated per task, grounded in the actual repository, before any candidate is scored. Generation details are proprietary.

Can a rubric be gamed by a verbose patch?

Criteria are specific, weighted binary checks rather than holistic impressions. Padding a diff does not satisfy a check that the patch does not address.

What ground truth is the validation against?

Public SWE-bench benchmarks, where each patch has a real resolved or unresolved verdict from a test harness.

Why not require perfect agreement with tests?

Because the metric is a complement, not a mirror: good patches can fail tests, and poor patches can pass them. Perfect agreement would be a red flag. It could suggest the judge had somehow seen the answers—for example, that test contents or verdicts had leaked into scoring—rather than indicating a good metric.