Cost vs quality vs speed: you usually get two out of three

Most companies continue to exceed their AI token spend budgets every month, and the trend is accelerating: the FinOps Foundation put 73% of enterprises over their AI cost projections last year. The fix is rarely as simple as switching to the highest rated cheaper model. There are always trade-offs. 

My rule of thumb has always been that with cost, quality, and performance, you tend to get only two out of three every time: pick a quality-rated, cheaper model and it does not perform as well; choose quality and performance for a model and it will not be cheap.

A public leaderboard will not tell you which two you are getting, because it ranks models on tasks that are not yours, and it knows nothing about your codebase, what your team accepts as done, or the rates you actually pay. All three have to be measured together, on real work. So we tested that assumption on our own repositories. The outcome surprised us.

We took 30 tasks straight from our own merged pull requests, including real features, bug fixes, and refactors across 12 services, and asked three current coding routes (an agent, a provider, and a model, which is what a team actually deploys) to solve each one from the exact commit our engineers started from. Every attempt was graded against a rubric based on the solution our team actually shipped. We scored them using a separate judge—kept constant across all three setups—and priced it based on observed token and cache usage.

The finding that mattered was not "model X wins." It was that the expected trade-off did not appear: the cheapest and fastest of the three gave up nothing measurable in quality. "Two out of three" remains the rule of thumb, and here we got those two decisively in cost and runtime while paying almost nothing on the quality axis. At the task volume a large engineering organization actually runs, that efficiency gap is worth a conservative $900k a year. Therein lies the real argument: the benchmark driving your model-selection decision should be built from your own work, not an abstract public score.

Six reasons a public leaderboard can’t pick your AI model

Public benchmarks, such as SWE-bench, HumanEval, the other arena leaderboards, are useful priors and poor decisions, for reasons that are common though not universal. No single benchmark carries every limitation below:

• They may be in the training data. Public tasks can leak into model training, inflating scores in ways that may not transfer to your private codebase. On SWE-bench Verified, OpenAI's own analysis found frontier models reproducing exact gold patches and verbatim problem details.
• They are someone else's code. Curated open-source puzzles do not reflect your stack, your languages, your frameworks, or the shape of your work.
• They usually grade against hidden unit tests, not against what your team accepts as a correct, shippable change. OpenAI's own audit of SWE-bench Verified found test cases that reject functionally correct submissions.
• Most do not price the workload. A quality rank with no cost axis—or an abstract list price—cannot tell you the quality-per-dollar tradeoff on your prompts.
• Most score a model in isolation. In production you do not run a model; you run a model inside a harness (an agent). The harness often matters as much as the model.
• Most give you one number. A single aggregate rank cannot tell you to route bug fixes to one model and refactors to another.

None of these are fixed by choosing a “better” leaderboard. They are fixed only by changing what you evaluate on. OpenAI has made a version of this argument publicly, recommending privately authored, uncontaminated benchmarks in place of SWE-bench Verified.

And there is a sharper reason the score column cannot carry a decision on its own: per-task rubric scores are too noisy to rank frontier routes at any cohort size a company would actually run, which is precisely why you should be choosing on cost, latency, and reliability rather than a quality leaderboard number.

What changes when the benchmark is built from your own PRs

Every task is one of your merged pull requests. The model is given the ticket (the linked issue description), never the merged code, and must produce a patch from the same base commit your engineer did. The grading rubric is generated from the human-accepted solution and validated to separate that solution from wrong or empty ones before any model is judged.

Here is what that buys you that a generic benchmark typically cannot:

How we built the benchmark: 30 PRs, 3 routes, 1 held-constant judge

The method is deliberately auditable. Every headline number re-derives from raw artifacts.

• Tasks from real merged PRs. The prompt is the linked issue description; the gold patch (the diff our engineer merged) is withheld from every model. Every route starts from the same pinned base commit. This gives lower contamination risk than a public benchmark, since we withheld the answer and used no known public benchmark tasks, but we cannot prove a model never encountered this repo or issue in training, and we won't claim we can.
• A rubric grounded in the human fix. For each task we generate a weighted checklist from the accepted solution, then validate that it discriminates (it must score the human solution near the top and clearly separate it from empty or wrong patches) before any model is graded. The same rubric grades every route, so no model shapes its own test.
• A single judge, separate and held constant. Claude Sonnet 4.6. Scoring is a separate stage, held constant across all routes, applying the same rubric to each patch and returning a weighted 0–1 score. No route judges itself. One caveat we won't bury: Sonnet 4.6 shares a vendor and model family with Claude Code + Fable 5, a route under test. Fable 5 did not win, but same-family judging is a known bias risk, and a multi-vendor judge panel is on the roadmap.
• Scores and completion rate, reported separately. Scoring a failure as zero penalizes the route that broke; excluding it flatters a route that is unreliable. Neither is the whole picture, so we report both. This run had zero failures (30/30 valid, scored patches on every route), so the choice moves nothing here. It would on a messier cohort.
• Honest economics. Cost is computed from real per-task token and cache usage, normalized so cached tokens are not double-counted across providers that report them differently, a correction that, in this run, moved one route from an apparent 49% cache / $8.32 to a corrected estimate of 93.8% cache / $1.14 per task.
• Routes, not models. Each route is a harness x provider x model, because that is what a team actually deploys.

The results: quality was a tie, cost and speed were not

Three current frontier routes, 30 tasks, max effort, judged by a single separate, held-constant model:

Read it the way it should be read. On quality, the three routes are not separable in a run this size. The top two are 4.6 points apart, the highest score was shared by at least two routes on 20 of the 30 tasks, and outright wins split 4-3-3. A 30-task, single-trial design cannot resolve quality gaps below about 15 points, and the 95% interval bounds any Fable 5 quality advantage over GPT-5.6 at ≤6.4 points. Read the quality column as a tie.

Cost and runtime are not a tie. Fable 5 costs $3.66 per task against $1.14: 3.2x comparing cohort means, 2.6x on paired per-task ratios (95% CI 1.9–3.7x), and it is the more expensive route on 25 of the 30 tasks. Both figures are projected from observed token and cache usage against list rates, so the comparison is apples-to-apples; separately, Anthropic’s billed cache-write premium makes Fable 5’s observed cost $4.60 per task, 4x the leader, which is what a team would actually pay. On runtime, GPT-5.6 finishes in 250 seconds against 589 and 643, meaning it is 2.4 to 2.5x faster (95% CI 1.8–3.3x and 1.9–3.5x), and faster on 26 and 27 of the 30 tasks. At quality we cannot distinguish, one route costs a quarter as much and finishes in 40% of the time. That is the decision, and it does not rest on the quality column.

It is worth being precise about what the quality column shows, because the means mislead on their own. Median scores are 92.4%, 90.6% and 93.3%, which are effectively identical, and in a different order than the means. The spread comes almost entirely from the left tail: the number of tasks a route simply whiffed, scoring under 50, was 3, 7 and 9 respectively. On the tasks where all three routes cleared that bar, they sit within two points of each other, with Kimi K3 nominally highest. So Kimi K3 is not the lower-quality route (when it lands, it scores as well as anything); it is the less reliable one. GPT-5.6 whiffed on no task Kimi K3 handled, while Kimi K3 whiffed on six that GPT-5.6 handled (exact McNemar p = 0.031; across three comparisons, suggestive rather than established). Reliability, not score, is the right word for that difference, and Kimi K3’s four-cent price advantage is itself inside the noise.

One caveat on reading this as a verdict: Fable 5 is not a weaker model than GPT-5.6 in general. It lost on this cohort, on our repositories, against rubrics written from our engineers' solutions, at one effort setting. Another company’s tasks could reverse the order entirely, and that is precisely the point: on a benchmark like this the ranking is a property of the workload, not a verdict on the models.

Key takeaway: Paying ~3.2x for an option that ran slower and scored no better is hard to justify. In this cohort, the default is the setup that ran fastest at quality indistinguishable from the alternatives, for about four cents more per task than the cheapest option—and you can only say that because all three axes were measured on real work.

Two surprises: a cache-accounting error, and a trade-off that never appeared

There were two big surprises: 

The first was the cache accounting. Our initial pass had GPT-5.6 at 49% cache and $8.32 per task, which would have made it the most expensive option on the board and flipped the entire conclusion. Providers report cached tokens differently and we were double-counting. Corrected, the same raw usage data gives 93.8% cache and $1.14. A benchmark that gets cost wrong is worse than one that ignores cost, because it looks authoritative.

The second was that the fastest of them was also the cheapest of the top two, and getting there cost nothing in measured quality. I expected a tradeoff and went in ready to write about it. There wasn’t one in this cohort. That is a less interesting story and a much easier decision.

From one default model to a routing policy per work type

Because every task is a real piece of work from a specific service, the same method extends to a routing policy rather than a single default: with work-type labels on your tasks, you can send each kind of work to whichever route wins on quality-per-dollar, per work type and per repo. That is a capability of the method, not a result of this run.

This run does not report per-work-type results; the cohort is sized to separate routes, not work types, and the instances carry no work-type labels. The value here is the mechanism. With your full task history, the routing policy is yours to compute and to keep current as models change.

What it is worth at scale

The per-task numbers become a budget conversation quickly. At a directional, illustrative 1,000 coding tasks per month, Codex + GPT-5.6 runs roughly $1.1k against ~$3.7k for the premium route, about $2.5k/month, or ~$30k/year (95% CI $16k–$48k), and not a quality tradeoff, since the cheaper one did not score lower. That is a pilot-sized number. Scale it to a large engineering organization (say 5,000 engineers each completing six agent-run tasks a month, or roughly 30,000 tasks), and the same per-task delta is about $75k/month, or ~$900k/year (95% CI $0.5M–$1.4M). Measured against Fable 5’s observed billed cost, the same volume is closer to $1.2M (95% CI $0.8M–$1.9M). Those intervals are paired bootstrap over the 30 tasks and capture task-to-task cost variation only, not pricing, discount, or volume risk. These are list-price figures before enterprise discounts, and they exclude the cost of running the evaluation itself.

We are deliberate about what that figure is and is not:

• It is projected list cost, before enterprise discounts, committed-use tiers, and cache economics at scale. Where a provider reports billed cost we cite it separately as observed.
• It excludes the cost of running the evaluation itself (judge spend, infrastructure, router overhead).
• Every savings claim is conditional on the cheaper option not regressing shipped quality, which our rubric score is a proxy for, not yet a production-outcome measurement.

Key takeaway: Treat this as the shape of the opportunity, not a signed business case: at enterprise task volumes, a routing change that costs nothing in measured quality is a six- to seven-figure annual line item. Faros makes that number measurable on your workload rather than assumed.

The value of continuous benchmarking

A benchmark you run once is a study. The value comes from running it continuously, and that is what the platform is for. Faros learns from your code, collaboration and delivery systems to maintain a living view of how software actually gets built, then puts AI usage and cost on the same usage data as the outcomes. So the cohort, the rubrics and the cost accounting behind this post a benchmark you keep, as opposed to a one-off exercise. When a new model ships you drop it in and read the result against your existing baselines, alongside adoption, throughput and spend for every team. The routing policy stops being a slide and becomes a number leadership can watch.

The limits of this run, stated plainly

This run is a directional demonstration, not a statistical verdict, so we publish the limits because a customer-grounded benchmark's whole value is that you can check it.

• Single judge, single trial. We report paired-bootstrap intervals on the cost, runtime and quality differences, but there are no repeated trials, no multi-vendor judge panel, and no pre-registered primary endpoint. Therefore, “a tie on quality” is a bounded interval plus a tie count, not a passed significance test. The rigorous version adds a multi-judge panel, repeated trials, and a pre-declared non-inferiority margin.
• Quality is a rubric-alignment score, not a shipped-outcome measure (merge rate, review iterations, rework). Outcome validation is the next lift.
• The tail matters more than the mean. The score spread between routes is driven by how often a route whiffed a task, not by typical performance; medians are within three points of each other. Our look at the tasks where every route cleared a usable score is diagnostic only: it conditions on the outcome, so it explains what the mean is measuring rather than establishing a ranking.
• Cost is projected list where the provider reports none, observed where it does; it excludes enterprise discounts and the cost of the eval loop.
• A 30-task, single-org cohort is chosen to discriminate routes, not to be an unbiased population estimate.
• Contamination is reduced, not eliminated. We withheld the gold patch and used no public benchmark tasks; we cannot prove no model saw these repos.
• Every route ran at maximum effort. A different effort setting could change the ordering, and we did not sweep it.

Each of these is a concrete next step, and should illustrate the difference between a benchmark you operate and a leaderboard you read.

The deliverable is a defensible decision

Notice what did not matter: whether the cheapest one happened to top the table. If the premium model had won outright, the exercise would be exactly as valuable, because the deliverable is a defensible decision. We can see quality, cost, and speed on our own work, choose one for reasons we can show a skeptic, and back that choice with raw artifacts anyone can re-check.

And because the cohort and rubric are frozen, the decision stays current: when the next model ships, we drop it into the same benchmark and measure it against our existing baselines in an afternoon—no re-litigating the setup, no waiting for a public leaderboard to catch up.

How to run a benchmark evaluation on your own repos in five steps

The framework is our product. To run your own frontier evaluation:

1. Sample 20–50 of your merged PRs with linked tickets: bug fixes, features, refactors, your real mix. 
2. Run 2–3 routes (the harness x model combinations you would actually deploy) from each PR's base commit. 
3. Score against rubrics built from your shipped solutions, with a separate judge held constant across routes. 
4. Track the real economics ($/task, cache share, runtime) on your prompts and your rates.
5. Turn it into a routing policy, and re-run it when the next model ships on the same frozen cohort, apples-to-apples with every prior result.

Faros can help you easily run your own benchmark experiment. Contact us for more information.