Overview

An AI coding model that looks cheap on a per-million-tokens basis can actually turn out to be the more expensive choice once you count what it actually costs to get usable work out of it: failed attempts, verification, review, and rework. 

The solution is to stop optimizing for cost per token and start optimizing for cost per verified engineering outcome: spend divided by the number of results that actually met a standard you defined before the work began. 

Call the gap between those two numbers the model tax: the accumulation of hidden costs that a cheap model ends up billing you over time.

Introduction

Most engineering organizations can report what they spend on AI coding tools. Considerably fewer can report what that spend produced.

The gap between those two statements is the subject of this post. The measure that closes it is cost per verified outcome: total attributable spend divided by the number of engineering outcomes that met an acceptance standard defined in advance.

The value of the cost per verified outcome is that it surfaces the concept of a model tax that until now has been mostly glossed over. An AI model that looks cheap on a per-million-token basis can still be the more expensive choice once its rework, reviews, and failed attempts are counted. Cost per outcome surfaces that tax and adds it to the overall cost of model selection.

Cost per outcome is not a novel concept. The FinOps Foundation identifies use-case economics, such as the total cost of achieving a business outcome, measured per unit of that outcome, as the most important cost concept for AI systems, and names cost per code review completed among its examples. What follows applies that principle to software engineering, where the acceptance standard can be unusually precise.

The measurement problem is documented

Faros's 2025 AI Engineering study, The AI Productivity Paradox, analyzed telemetry from more than 10,000 developers across 1,255 teams. Teams with high AI adoption completed 21% more tasks and merged 98% more pull requests. Over the same period, pull request review time rose 91%, pull request size grew 154%, and bug counts rose 9%. At the organizational level, DORA delivery metrics showed no measurable improvement.

Faros's 2026 AI Engineering report, The Acceleration Whiplash, covers two years of telemetry from 22,000 developers across more than 4,000 teams, measuring the change within each organization between its lowest and highest AI-adoption periods. Task throughput per developer rose 34% and epics completed rose 66%. Against that: bugs per developer rose 54%, incidents per pull request rose 242.7%, median time in review rose 441.5%, and 31.3% more pull requests merged with no review at all.

Both of our reports are observational. Organizations that adopted AI heavily may differ from those that did not in ways the data cannot isolate, so they establish correlation, not causation.

With that caveat, the direction is consistent across two independent samples. Code generation accelerated. Review, testing, and deployment did not. The cost did not disappear; it moved from a vendor invoice into engineering hours that no system bills for.

DORA's 2026 ROI report names this directly, identifying the verification tax (the additional effort required to establish that AI-generated code is reliable, secure, and architecturally sound) as one of three causes of an early productivity decline it models as a J-curve, alongside the learning curve and pipeline adaptation. The report characterizes that decline as the tuition cost of transformation.

A dashboard denominated in tokens registers the invoice falling and is silent on where the cost went.

Why cost per token uses the wrong denominator

Let’s look at a hypothetical example to illustrate what the math would look like:

Measured per token cost alone, the second model is 68% cheaper. However, measured across 100 tasks:

Higher-cost: 100 × 1 × $25.00 = $2500, yielding 80 verified outcomes: $31.00 per outcome

Lower-cost: 100 × 2 × $8.00 = $1600, yielding 50 verified outcomes: $32.00 per outcome

The two are nearly equivalent. The lower-cost model has an additional cost: accounting for the additional thirty failures that it leaves behind. The apparent saving was an artifact of the denominator. 

This is a classic example of vibe-spending, where a lower invoice provides a sense of fiscal discipline but in fact masks the reality that the higher-cost route may actually be cheaper when rework is accounted for.

The market has moved, but only partway

On July 22, 2026 Cursor released Cursor Router, a classifier that selects a model for each request. Notably, Cursor reported results in cost per commit rather than cost per outcome, stating explicitly that engineering leaders care whether savings appear in shipped work. The published figures: $4.63 per commit in Balance mode and $6.76 in Intelligence mode, against $7.34 for Opus 4.8 and $12.69 for Fable 5 used alone. Across two weeks of early access, three high-volume enterprise accounts reported savings of 30 to 50% versus routing all traffic at Opus 4.8 rates, with no reported quality decrease; online A/B tests across millions of requests reported up to 60%. The classifier was trained on more than 600,000 live requests.

Two limitations bear on how much weight these figures can carry.

Every number was designed, produced, and scored by Cursor, against Cursor's own satisfaction signal and keep rate, the proportion of agent-generated code that remains in the codebase. Keep rate is a reasonable proxy, but it is a proxy, and no external team can reproduce the methodology.

More significantly, a commit is an intermediate artifact. Commits fail continuous integration, are superseded, require rewriting, remain unmerged, or are reverted. Cost per commit improves materially on cost per token. It stops one step short of an accepted result.

Cursor’s reporting of cost per commit is still an outlier in enterprise AI tooling. Most major platforms have not yet adopted anything like it. Databricks Unity AI Gateway, for instance, provides essential governance and routing for models, letting teams manage and control costs at the token level. But such systems leave the model tax invisible to both developers and engineering leadership.

Defining the acceptance standard

The events that instrumentation captures most easily (a model response, a process exit, a file modification, a pull request opening, an acceptance click) record that an interaction occurred. None establishes that the output was correct.

An acceptance standard specifies what evidence constitutes success. For example, for test generation on a low-risk component, a reasonable standard would require that CI checks pass, that no tests were skipped or disabled, that the tests exercise the behavior they claim to cover, that the pull request was accepted, that no substantial rewrite was required, and that no revert occurred within fourteen days.

The standard must be written before measurement begins. Written afterward, it tends to accommodate the results obtained, at which point the exercise describes an outcome rather than testing one.

The depth of verification materially changes the calculated cost per verified outcome, and should be stated alongside it:

In short, “$7.28 per verified test-generation outcome, verified through review and merge” is actionable. Total AI spend is not.

Two cost-per-verified-outcome calculations, reported separately

The direct calculation comprises vendor charges (model usage, retries, evaluation calls, gateway and sandbox costs) divided by verified outcomes. It derives from an invoice.

The fully loaded calculation adds the review and correction time attributable to that work, valued at a loaded engineering rate. The denominator is unchanged.

Two aspects of the calculation are commonly handled incorrectly.

Spend on unsuccessful attempts belongs in the numerator. Dividing the spend on successful attempts only is survivorship bias: that money was disbursed, so excluding it makes a route that fails half the time appear comparable to one that rarely fails. Since distinguishing between those routes is the purpose of the exercise, the exclusion defeats it.

The two calculations should not be combined into one. Vendor charges are precise. Correction time is an estimate assembled from review timestamps and judgment. Summing them transfers the authority of the invoice to the estimate. Reported separately, the reliability of each component remains visible:

An example

The numbers below are hypothetical.

The lower-cost route reduces the vendor invoice by 54% and increases the true cost of a verified outcome by 95%. The 54% reduction appears on a budget line. However, the 95% increase appears as senior engineering time, which is not invoiced and therefore not reviewed.

Faros's finding of review time rising against flat delivery performance suggests this is not an unusual configuration.

The market price of code review

AI code review is the workflow where per-unit pricing diverges most sharply, and here real figures are available.

GitLab prices its Code Review Flow at a flat $0.25 per merge request, and observes in its launch material that competing token-based review tools can cost $15 to $25 per review depending on the size and complexity of the change. Anthropic's Code Review for Claude Code is token-priced in that $15–25 range; Anthropic reports it flagged issues on 84% of large pull requests, averaging 7.5 findings each.

A sixty-to-hundredfold spread in list price cannot be evaluated on price. The relevant question is cost per accepted finding, and the variable that determines it is precision. A tool generating 7.5 findings per pull request produces value only in proportion to how many of those findings a reviewer acts upon, net of the time spent dismissing those they do not.

Consider the arithmetic. At a $100 loaded hourly rate, ten minutes of triage per pull request costs $16.67, exceeding the entire list price of the more expensive tool and dwarfing the cheaper one. Precision, not price, determines the economics of AI code review, and precision is not disclosed on any pricing page. When a reviewer spends ten minutes dismissing false-positive findings, that time is the model tax paid on a low-precision tool, rendering its low 'per-request' cost deceptive.

This is why "cost per code review completed," while a reasonable FinOps unit, remains too coarse here. The acceptance standard needs to incorporate accepted findings, false-positive burden, severity, duplicate suppression, and measured triage time.

The stakes are significant. GitLab reports that the median engineer at a large company waits thirteen hours for a merge request to merge, and that 44% of engineering teams identify slow code review as their single largest delivery constraint.

Published results do not transfer between organizations

The research literature appears to conflict. It is measuring materially different conditions.

Peng, Kalliamvakou, Cihon and Demirer (2023) ran a controlled experiment in which developers recruited through Upwork implemented an HTTP server in JavaScript. The group with GitHub Copilot access completed the task 55.8% faster than the control group.

GitHub's own randomized trial recruited 243 Python developers with at least five years of experience and obtained valid submissions from 202 — 104 with Copilot, 98 without — on a task writing API endpoints for a web server assessed by ten unit tests. Developers with Copilot access had a 53.2% greater likelihood of passing all ten. Blind peer review across 1,293 reviews found improvements in readability, reliability, maintainability and conciseness of between 2.94% and 4.16%. The trial was designed and run by GitHub, the quality effects are small, and reviewers were drawn from the participant pool.

METR (July 2025) conducted a randomized controlled trial with 16 experienced developers completing 246 real tasks in open-source repositories averaging 23,000 stars and a million lines of code, on which they had roughly five years of prior experience. Tools were those at the February–June 2025 frontier, primarily Cursor Pro with Claude 3.5 and 3.7 Sonnet. Developers forecast that AI access would reduce completion time by 24%. Measured completion time increased by 19%, with a confidence interval of +2% to +39%. After participating, developers still estimated a 20% reduction.

METR (February 2026) revised the picture. Working with 57 developers across 143 repositories and more than 800 tasks (ten from the original cohort, the remainder newly recruited from smaller and less mature repositories) METR now estimates an 18% speedup for the returning subset, with a confidence interval spanning a 38% speedup to a 9% slowdown, and a 4% speedup among newly recruited developers, with a confidence interval spanning a 15% speedup to a 9% slowdown. METR states that selection effects render this only very weak evidence for the magnitude of the change, noting that developers who believed AI helped them substantially were reluctant to enroll in a trial that would prohibit its use half the time.

DORA (2025) surveyed nearly 5,000 technology professionals alongside more than 100 hours of qualitative research. Its central finding is that AI functions as an amplifier, magnifying the strengths of high-performing organizations and the dysfunctions of struggling ones. Adoption is near-universal at 90%, more than 80% believe it has increased their productivity, and 30% report little or no trust in AI-generated code. DORA added rework rate to its delivery metrics that year. Its conclusion on where returns originate is unambiguous: from the quality of internal platforms, the clarity of workflows and the alignment of teams, rather than from the tools.

Four observations follow:

1. The strongest single result in the field (a within-subject randomized trial) reversed its own direction within seven months, because the tools under test changed. Any figure has a short shelf life.
2. The results vary systematically with conditions: task type, codebase maturity, developer familiarity with the code, and the strength of automated verification.
3. The largest reported gains come from bounded tasks in unfamiliar or greenfield code. The measured slowdown came from mature codebases the developers knew well. Most production engineering resembles the latter.
4. Consequently, a figure drawn from a vendor announcement or a research abstract describes a different codebase, a different population, and frequently an earlier generation of tooling.

The task matters as much as the model

Correcting a typo, adding unit tests, restructuring a shared library, remediating an authentication vulnerability, and executing an irreversible database migration are not comparable units. Averaged together, the resulting figure principally reflects which category of task happened to predominate in the measurement period. A model that looks decisively better or worse might have simply been given an easier, or harder, set of problems.

Comparisons therefore require boundaries: task type, repository, risk tier, verification mode, and exact tool revision. Without these, we are left with vibe-routing, the belief that a model is superior simply because it seems cheaper or better, without the data to confirm it actually ships verified outcomes more cost-effectively and reliably. “Model A against Model B on low-risk test generation under deterministic CI verification" is answerable. "Model A against Model B across the organization" is not. The question is too coarse for the arithmetic to mean anything.

This is precisely the condition a replay-based approach is built to satisfy. Such an approach holds the task fixed (the same historical work, the same repository, the same risk tier) and varies only the model or configuration under test. The comparison becomes valid because the boundary is enforced by construction, not by hoping the sample happened to be balanced.

Potential objections

While the proposed approach offers a rigorous framework for evaluating AI investments, it does come with implementation challenges:

Instrumentation has a cost. Connecting spend to pull requests, test runs and downstream outcomes requires engineering effort. This argues for beginning with a single workflow rather than constructing a measurement platform, and for excluding work that does not recur frequently enough for the measurement to repay itself.

Not all work admits automated verification. The measure suits repeated work with deterministic acceptance criteria: test generation, dependency updates, documentation, routine defect classes, mechanical refactoring, static-analysis remediation and CI repair. It suits architectural decisions, novel product definition, exploratory research and long-horizon rewrites poorly, where the acceptance standard reduces to expert sign-off and measurement becomes slower and less precise. This is a genuine boundary and should be stated rather than obscured.

Potential limitations

Cost per verified outcome can improve while conditions deteriorate in ways it cannot detect: increased latency, unpredictable routing, interrupted concentration, reviewer distrust of automated output, and exploratory work discouraged because it scores poorly.

It is a unit-cost measure. It belongs alongside delivery throughput, change failure rate, rework rate, review latency and developer satisfaction, the SPACE framework's central argument being that productivity cannot be represented by a single dimension. A decision report is more useful when it carries the full set:

Implications by function

Engineering leadership. Whether a lower-cost route is generating correction work; which categories of work are economically suited to AI; which repositories have verification too weak to support any conclusion; and whether an observed saving survived review or was transferred onto reviewers.

Finance. The governing distinction is that reduced consumption is not necessarily cash recovered. Under per-seat pricing, reduced token usage produces no cash saving in the period. It may still hold value, such as headroom under an existing contract, reduced overage exposure, or improved renewal position, but that value should be recorded as capacity rather than savings. Savings are best tracked in stages: modeled, tested, realized, realized net of measurement cost, and reconciled against the invoice. For self-hosted inference, hardware amortization, idle capacity, energy and operations belong in the cost base, since marginal token cost is uninformative at low utilization. DORA's ROI report also models the early productivity decline explicitly, which makes it a useful instrument for establishing expectations before a rollout rather than after one.

Chief executive. The relevant question is not total AI expenditure but what proportion converts into work that ships and remains in service, and whether that proportion is improving. DORA's amplifier finding carries the strategic implication: a more capable model applied to a weak verification process amplifies the weakness. DORA further argues that the principal return comes from reducing rework to recover engineering capacity and reinvesting it, rather than from headcount reduction.

Chief of staff. This is a cadence problem. One workflow, one acceptance standard, one baseline, one candidate at a time, with exposure caps and stop conditions agreed in advance, and one report in a fixed format reviewed at a fixed interval by a consistent group. The failure mode is not selecting the wrong model. It is thirty teams producing thirty savings claims that cannot be compared with one another or reconciled against a bill.

Implementation

Step 1: Select one recurring category of work, for instance, test generation across three low-risk services. Define the acceptance standard for it: CI passes, no tests disabled, pull request accepted, no substantial rewrite, no revert within fourteen days.

Step 2: Instrument the full cost of each attempt: route, model revision, tokens and cache, retries, evaluation calls, terminal state, and links to the pull request and CI run. This produces a baseline that includes attempts, verified outcomes, vendor charges, estimated correction time, and both cost figures.

Step 3: Test one candidate route on a small randomized share of the eligible work, judged against the same standard, under a fixed exposure cap and predefined stop conditions. Compare on both cost figures, the change in verification rate, the change in correction burden, severe outcomes, pending outcomes, and reconciliation against the invoice.

Step 4: Then hold steady. A favorable result across the first dozen tasks is not evidence, and expanding on it converts a measurement exercise back into an assumption.

Conclusion

Tokens record consumption. Requests record activity. Commits record output. None establishes that an engineering result was obtained.

The unit worth managing is the verified outcome: work meeting a standard defined before the work began. A lower-cost model may require more attempts, fail verification more often, generate more review burden, or produce more instability in production. That’s model tax: the hidden cost of rework a cheap model costs your organization after the invoice looks small. None of it is visible in a token dashboard. All of it is visible in the cost of one verified outcome.

Contact Faros to learn more

Faros traces your AI spend to shipped outcomes and tests model and configuration changes against your own historical work under controlled, comparable conditions. To see what cost per verified outcome looks like for your organization, contact us to learn more.