TL;DR: Open models are now strong enough for real software engineering work, and in this Faros cohort, they challenged expensive frontier defaults directly. Ron Meldiner, Field CTO at Faros, ran 211 real engineering tasks through seven AI coding routes to compare quality, cost, runtime, and consistency on real repos.

Claude Code + GLM-5.2 and Claude Code + Kimi K2.6 landed in the top quality band, while GLM-5.2 was meaningfully cheaper and faster. Claude Code + Opus 4.8 and Codex + GPT-5.5 did not buy their way into the top quality band in this run.

The key takeaway is not to pick a model from public benchmarks alone. Instead, measure routes against your own codebase, workflows, costs, and review standards, then route work based on what actually performs.

Open models are catching up. The advantage is knowing when to use them.

I keep hearing the same question in customer and partner conversations: If open models are getting this good, why are so many software teams still paying frontier-model prices for every AI coding task?

At Faros, we spend a lot of time connecting engineering activity to outcomes: which work shipped, which reviews churned, which CI runs failed, and where teams lost time. Managing and optimizing AI coding spend is now part of that picture. A token bill by itself tells you almost nothing. It’s more useful to know where the spend produced a patch, whether the patch was good enough to move work forward, and whether a cheaper model could have handled the same job.

So I ran Time Machine, our proprietary Faros analysis for evaluating historical engineering work (we will write more about it separately), against our own engineering work in order to answer one question: Where can an open model handle real software development well enough that the expensive route stops being the default?

The idea is simple: Take real tasks and historical PR context, run the same task cohort through selected AI coding harnesses, and compare quality, projected cost, runtime, cache behavior, and invalid/no-diff outcomes.

The short answer surprised me. Claude Code + GLM-5.2 was statistically close to Claude Code + Kimi K2.6 on quality, while being meaningfully cheaper and faster in this cohort. The Opus and Codex routes lagged behind on both quality and economics in this cohort. That gave me a more specific operating rule: test GLM-5.2 and Kimi as the practical default for broad production-like throughput.

The rest of this post walks through what we tested, what changed my mind, and how I would run a similar test inside another engineering organization.

Why public benchmarks are useful but incomplete

The public signal is strong enough that software leaders should pay attention. Open models are closing benchmark gaps quickly, and recently released open models such as GLM-5.2, Kimi K2.7, Qwen3-Coder, and DeepSeek V4 are explicitly aimed at long-context, tool-using, agentic coding workflows.

Those benchmarks set priors. They cannot see your repositories, test setup, review standards, cache behavior, CI failure modes, permission boundaries, or the mix of work your engineers actually need help with. A model can look strong on a leaderboard and still be the wrong default for your environment if the harness wastes context, the cache economics break, or the model struggles on your common task shapes.

I wanted a Faros-specific answer. Which model and harness should we trust for our repositories, our task mix, and our review standards?

This is related to the new wave of model routers, including Sakana's Fugu, which dynamically orchestrates multiple models behind a single API. I like that direction. The limit is that a generic router still needs local evidence: the judge and routing policy have to reflect your codebase, tests, permission boundaries, review burden, cache behavior, and the work slices your engineers actually do.

The experiment we ran: Comparing model plus harness combinations

The experiment compared model plus harness combinations. I care about the pairing because engineers do not use a raw model in isolation. They use a model through a harness that decides how context is loaded, how tools are called, how diffs are produced, and how failures show up. I wanted the methodology visible before the result, because this experiment is only useful if the reader knows what kind of evidence it is.

Our main finding: GLM and Kimi were in the top quality band; GLM had better economics

GLM-5.2 and Kimi both landed in the top quality band. Claude Code + GLM-5.2 scored 0.568 and Claude Code + Kimi scored 0.566, close enough that I would not treat the difference as a quality win by itself. The operational difference was economics: in this cohort, GLM-5.2 delivered comparable quality while running faster and costing meaningfully less per task. That makes GLM-5.2 the route I would test first for broad production-like throughput, with Kimi still worth keeping in the routing pool where local data shows an edge.

Because cache share was nearly identical after the rerun, 89.7% for Claude Code + GLM-5.2 and 90.0% for Claude Code + Kimi, the cost gap was not a cache artifact.

The expensive frontier baselines did not buy their way into the top quality band in this cohort. Claude Code + Opus 4.8 trailed both Claude Code + Kimi and Claude Code + GLM-5.2 on mean score; its projected cost was roughly in the same range as Claude Code + Kimi, but materially higher than Claude Code + GLM-5.2. Codex + GPT-5.5 is included here as a high-reasoning current-default comparison, not as a Fireworks route.

The most efficient routes changed by work type

The aggregate table tells me which routes are efficient overall. The work-type slices tell me how I would route real work. I would not turn the aggregate winner into a one-size-fits-all rule.

Mean score tells me average quality. Average rank tells me which route stayed consistently near the top across tasks, even when it did not win outright.

For this cohort, the routing policy would start here:

The point is not to memorize these labels. The operating rule is to start with Claude Code + GLM-5.2 where local data says it clears the bar, escalate to Claude Code + Kimi when ambiguity, larger diffs, complexity, or review risk make quality worth paying for, and keep OpenCode + GLM-5.2 as the OpenCode route with the strongest consistency signal.

GLM-5.2 shows why the default has to keep moving

GLM-5.2 was announced while we were running this experiment, so we added it to the loop. The GLM-5.2 release post emphasized long-horizon engineering, larger context, and stronger coding benchmark performance. That made it worth testing. The completed run made it practical: Claude Code + GLM-5.2 landed in the top quality band, ran faster than Claude Code + Kimi, and cost roughly half as much per task.

This is what I want to emphasize: The best default changed while the experiment was still running. Kimi still remains the route I would keep for quality-sensitive escalation, especially where the local slices show an edge: feature work, very-high-complexity work, larger diffs, and agent-tooling tasks. But GLM-5.2 cleared the bar often enough—and cheaply enough—that it changed what I would test first.

OpenCode + GLM-5.2 also earned its place. It had the best average rank signal, strong cache behavior, and the best data graph / schema-heavy profile. If a team is standardizing around OpenCode, that route deserves serious evaluation rather than being treated as a fallback.

The point is not to pick a winner and freeze it. You need the ability to rerun the evaluation when the market moves, when a provider fixes caching, when a new model ships, or when your own workload changes. This has to become an operating loop and not a one-time bakeoff.

How to run a model routing experiment in your organization

Model routing is becoming an important engineering control. If I were doing this experiment with another engineering team, I would start small and make the decision useful within a week. Running your own small experiment could look like this:

Focus your model routing experiment on real tasks like test selection, deploy gates, and review policies. The process should reflect your actual workflows, adapt to changing conditions, and define exactly when to escalate. The goal is not to crown a winning model. The goal is to make model choice repeatable.

Your default today may not be your default tomorrow

Defaults need evidence. Open models now deserve real engineering evaluations on actual company work, turning model choice into an engineering control like test selection or deploy gates. The market simply moves too quickly for static defaults. When a new model ships, caching improves, or your workload shifts, the right answer changes. Therefore, your most durable capability is continuously measuring your own work to adapt. Model selection must be an operating loop, not a one-time bake-off.

Faros can produce model-routing policies from the engineering systems you already use, drawing on issues, PRs, CI, code ownership, review flow, repo metadata, and delivery outcomes. Time Machine turns that history into a clear guide for which route should handle which work, when to escalate, and where the economics actually survive contact with your codebase. Contact us for a demo today.