Router quality judgments depend on locally constructed benchmarks

Model routers have a limited understanding of what “good” is, and what’s good in one environment is not necessarily good in another. 

Ramp Router, for example, routes 2.75 trillion tokens a month, tests each new model on real work, sends every request to the lowest-cost model that clears its quality bar, and applies over 100 optimizations across caching, compaction, and spend controls. Ramp reports about a 30% cut in its own LLM costs at roughly 30ms of added latency. 

But where did their router’s definition of quality come from, and what did they have to do to define it?

For coding, Ramp’s quality bar is Ramp SWE-Bench, a private benchmark of 80 tasks mined from their own production pull requests. They built it because public benchmarks saturate, leak into training data, and—their words—have “none quite resembling the work our engineers do every day.” If you read their methodology, you’ll see just how much work it took to build it: reconstructing repos at base commits, holding out merged patches as gold artifacts, sandbox-validating that tests flip from fail to pass, LLM judges auditing every task for fairness, a model ladder to discard tasks that carry no signal, and human review as the final gate.

Ramp’s router works for them because they did the hard work of defining a quality standard based directly on their own codebase. To get similar results, you would have to do that same work for your specific tasks. 

A generic router might be smart, but it will fall short if it doesn’t know what good looks like for your teams or your code. Furthermore, a standard router has an incomplete feedback loop, as it considers a job done the moment it generates a response, without ever knowing if that code was actually accepted, rewritten, or reverted by your engineers.

Local evaluation of 211 tasks exposes the limits of aggregate routing

We wanted to know how big this gap is in practice, so we measured it. We took 211 historical engineering tasks from Faros repositories, restored each repo to its state just before the accepted change, ran each task through six model-and-harness routes, and scored every patch 0–100% against the accepted implementation on functional correctness, solution approach, and integration with the surrounding code (methodology and full data).

One route won on aggregate: highest mean quality score, lowest cost per task, fastest runtime. Any reasonable router would send traffic there by default. However, that route was only the best choice on 84 of the 211 tasks. On the other 127 tasks—60% of the cohort—some other route did better (per-task results).

Furthermore, we also found: 

1. The optimal route varies by task, and misrouting carries a substantial quality penalty.

The intuitive routing policy is “cheap model for easy tasks, frontier model for hard ones.” Our data doesn’t support anything that clean.

The best route flipped depending on where the work lived. For example:  

• Our AI/agent tooling code favored Claude Code + Kimi K2.6 (41.9% mean score, everything else well behind). 
• Our data-graph platform favored OpenCode + GLM 5.2 (69.3%). 
• Our UI and reporting code favored Claude Code + GLM 5.2 (66%). 

Different domains, different leaders (slices by domain).

Best route by repository domain

It was the same story when we sliced by work type and by complexity: infra/devex work had a different leader than bug fixes and features, and the low-complexity leader wasn’t the high-complexity leader (intent, complexity).

Getting the routing right for each task is high-stakes because the outcomes fall into two extremes. There is a large cluster of tasks where the routing choice barely matters; any route works just fine. However, there is another large cluster of tasks where picking the right route creates a massive 70+ point advantage—literally the difference between a working patch and garbage. When you average the gap between the best and worst choices across all tasks, it comes out to a significant 43 points.

Spread between best and worst route per task

At the top of the aggregate table, our two leading routes finished in a statistical tie: 56.8% vs 56.6%, with a 51.9% head-to-head win rate (pairwise). No public leaderboard breaks that tie, but local data does: one of the two costs half as much ($0.92 vs $1.78), and the tie dissolves as soon as you segment by work type. 

A one-size-fits-all routing strategy is fundamentally flawed at the individual task level. To make accurate decisions, a router needs deep context—specifically the repository, the type of task, and its complexity. The problem is that this vital information isn’t included in a standard prompt; it has to be pulled directly from your broader engineering system.

2. Performance is a property of the route, not the model.

In our experiment, we never evaluated raw models. Instead, we evaluated routes: a model working inside a coding-agent harness, complete with a repository and tools. This harness—the software that loads context, exposes tools, runs the loop, and turns output into a patch—is half the variable.

The same model moved materially between harnesses. For example, Opus 4.8 scored better inside Claude Code than inside OpenCode. GLM 5.2 scored about the same in both, but took twice the wall-clock time in one (321s vs 620s per task). All of these findings can be found in the route summary.

Routers select among models. Look at any router’s catalog, and the units are model names with category descriptions (“everyday agentic coding,” “hardest, highest-value tasks”). Even Ramp SWE-Bench deliberately pins one lean harness (mini-swe-agent) across all models to isolate model behavior. This is a sound choice for ranking models, but it is exactly the variable an engineering team can’t hold fixed, because engineers ship through real harnesses and the harness moves the score.

Selecting a model while treating the harness as fixed is optimizing over the wrong set. And the harness is only one of the surrounding layers. A bug fix can fail because the agent loaded the wrong part of the repo, missed a convention, couldn’t run a required tool, or declared victory after compilation. None of that is visible at the routing layer, and none of it is fixable by a better model choice. 

A cheaper model with the right repository context and a verification loop can beat a stronger model working blind—which means context and harness work moves the quality-cost frontier itself. The router can only pick a point on whatever frontier you hand it.

3. Route rankings are non-stationary and require periodic re-evaluation.

Compare our two evaluation releases, two weeks apart (July 6 vs July 20). One route is labeled “cache-corrected” because we found a provider-side caching issue, got it fixed, and had to rerun. The fix changed the evidence for a production decision even though the model name and the tasks were identical. Task win rates shifted across the board with no change to the task set.

New models, harness updates, provider fixes, and a shifting work mix each invalidate the previous answer. A routing decision is a policy you have to keep rerunning, and the crux of it is who owns that loop and what data it reruns against.

Optimization must extend from the routing layer to the codebase

In conclusion, the intelligent model routing stays an important control point. But when the aggregate-best route is wrong on 60% of real tasks, intelligence at the routing layer isn’t a substitute for evidence from the engineering system. The optimization loop has to reach the codebase.