The cheapest model isn’t the cheapest route

When it comes to controlling AI spend, engineering leaders are being offered two bad options: cut off access, or accept unpredictably high bills.

Both options assume the same flawed premise, that the size of that bill needs to be made smaller. The better question: does each dollar of AI spend produce an outsized outcome worth paying for?

Answering that means rethinking what you're optimizing for and what you're optimizing across. A router optimizes one request at a time. The question above is about the whole system.

Token price is the wrong target

The obvious move is to route work to cheaper models. Token prices vary by an order of magnitude, so the arithmetic seems irresistible.

Then you meet the Model Tax. A cheaper model that fails on the first attempt costs you a retry. One that produces plausible-looking code costs you a review. Another one that ships something subtly wrong costs you rework, possibly weeks later, in senior engineering time. The token price was lower, but the route was more expensive.

The same error works the other way, too. Premium models get used for routine work because they're the default and nobody opted for a different one. That spend is invisible in aggregate but enormous in total.

Neither problem is detectable from a token price. Both are obvious from cost per outcome: what you paid, measured against work that actually shipped.

Leaderboards can't answer this

So you consult a benchmark. Find the best model, route the work there, and move on.

But there’s no universally best model. Model performance depends on your codebase, your task types, the agent harness wrapped around the model, the context it's given, and your quality bar. Change any one of those, and the ranking changes with it.

Public benchmarks are useful resources, but they cannot determine your routing policy, because they weren't run on your work.

They also carry more noise than most people assume. In July 2026, OpenAI audited SWE-bench Pro and found roughly 30% of its public tasks were broken, a finding we had reached independently. When a third of a widely cited benchmark's tasks are defective, a routing decision resting on that leaderboard rests on very little.

Session-based routing leaves output on the table

Model routers exist, and they do their job well. Session-based routing means the route is chosen for one session at a time, from what can be seen before the work starts.

A good router weighs more than difficulty. It considers task complexity, the repository and task type, latency and availability requirements, how much context has to fit, and the cost ceiling it has been given, then picks a route that satisfies all of it. Faros does this too. 

Notice that every one of those inputs is knowable before the session runs. That is the ceiling on session-based routing. If the session fails, gets retried, or produces code that is reverted in review, the cheap choice becomes the expensive one, and a session-based router never finds out, because it decided before the outcome existed. 

Outcome-based routing closes that gap. Routes are chosen based on what comparable sessions actually shipped. That evidence lives downstream of the request, in the pull request, the review, and sometimes in production. That is what lets you ask the questions session-based routing structurally cannot:

• Are these routes improving the quality of work that ships?
• Are they creating more review, rework, or failures downstream?
• Is the organization spending more efficiently across teams, repositories, and task types?
• Where should the next AI dollar go?

None of that data lives in the router. It lives in your engineering systems, which is why outcome-based routing needs a view of both.

The route, not the model

Faros tested this across 211 real engineering tasks, demonstrating that performance depends on the model, the harness, the repository context, and the task together. Any one of them alone under-determines the result.

The winning route landed in the top quality band, ran faster than the alternatives, and cost roughly half as much per task as the next closest. Different types of work resulted in different winners. There was no single answer, only a set of task-specific ones.

All this means the unit of optimization is the route: the model + harness + context + configuration, evaluated on a specific kind of work.

Prove it before you trust it

Here is the part that most changes how this works in practice. There are three levers available to any router:

• Upgrade: This repository is complex, so upgrade it to a more expensive model
• Downgrade: This class of task is simple, so downgrade it to a cheaper model
• Reconfigure: This work needs more reasoning, so change the configuration

And they're all applied on the basis of a quick assessment made before the work happens.

The Faros Time Machine tests the hypothesis instead of trusting it. It takes work that already shipped from your own history and runs it again under a different route, then scores what comes back. Same task, different route, and a result you can check against what your team actually merged. A recommendation you can trust, because it already came true.

That does two things.

1. It validates a routing change before you make it. Instead of wondering whether a cheaper model would have been sufficient for last quarter's refactors, you're looking at what it actually produced on them.
2. It replaces the public leaderboard with yours. The Time Machine builds a benchmark from your own merged code, and that benchmark can be granular, broken out by task complexity, by repository, and by team. The result frequently looks quite different from the public leaderboard, which is the whole point: the public leaderboard was measuring somebody else's work.

What optimization looks like with Faros

In practice, optimizing AI spend with Token Engineering comes down to five concrete moves:

• Classify the spend. Token usage sorts into productive, inefficient, and wasteful, so the conversation moves from "spend is up 40%" to "this portion produced shipped work, this portion produced retries."
• Measure cost per verified outcome, broken out by team, model, and workflow, rather than cost per token.
• Compare routes on three axes at once: quality, runtime, and cost. A route that wins on one and loses on the others isn't a win, and a single-number comparison hides that.
• Evaluate in the context of the full lifecycle. Faros looks at the task, the model and harness that handled it, the code produced, the review it required, and the outcome it achieved. That's what makes the system-wide measure possible. You can't compute cost per verified outcome without seeing the verification.
• Turn findings into recommendations for model selection, routing, and context improvement. And when you want to apply one, it goes through your gateway or through ours. If you've already built a routing layer, Faros drives yours; if you haven't, Faros provides one. Either way the evaluation stays independent of any single vendor's stack.

Build your own benchmark

The fastest way to understand any of this is to stop reading about someone else's tasks and look at yours.

Faros evaluates models, harnesses, and configurations against your own merged pull requests, under controlled and comparable conditions, and shows you which routes would have produced the same result for less, and which cheap routes would have cost you more downstream.

Book a demo and we'll build a private benchmark against your repositories.

{{cta}}