Published June 25, 2026 · Updated July 8, 2026
TL;DR: Open models are now strong enough for real software engineering work, and in this Faros cohort, they challenged expensive frontier defaults directly. Ron Meldiner, Field CTO at Faros, ran 211 real engineering tasks through seven AI coding routes to compare quality, cost, runtime, and consistency on real repos.
Claude Code + GLM-5.2 and Claude Code + Kimi K2.6 landed in the top quality band, while GLM-5.2 was meaningfully cheaper and faster. Claude Code + Opus 4.8 and Codex + GPT-5.5 did not buy their way into the top quality band in this run.
The key takeaway is not to pick a model from public benchmarks alone. Instead, measure routes against your own codebase, workflows, costs, and review standards, then route work based on what actually performs.
Are open source AI models good enough to replace frontier models?
I keep hearing the same question in customer and partner conversations: If open models are getting this good, why are so many software teams still paying frontier-model prices for every AI coding task?
At Faros, we spend a lot of time connecting engineering activity to outcomes: which work shipped, which reviews churned, which CI runs failed, and where teams lost time. Managing and optimizing AI coding spend is now part of that picture. A token bill by itself tells you almost nothing. It’s more useful to know where the spend produced a patch, whether the patch was good enough to move work forward, and whether a cheaper model could have handled the same job.
So I ran Time Machine, our proprietary Faros analysis for evaluating historical engineering work, against our own engineering work in order to answer one question: Where can an open model handle real software development well enough that the expensive route stops being the default?
The idea is simple: Take real tasks and historical PR context, run the same task cohort through selected AI coding harnesses, and compare quality, projected cost, runtime, cache behavior, and invalid/no-diff outcomes.
The short answer surprised me. Claude Code + GLM-5.2 was statistically close to Claude Code + Kimi K2.6 on quality, while being meaningfully cheaper and faster in this cohort. The Opus and Codex routes lagged behind on both quality and economics in this cohort. That gave me a more specific operating rule: test GLM-5.2 and Kimi as the practical default for broad production-like throughput.
The rest of this post walks through what we tested, what changed my mind, and how I would run a similar test inside another engineering organization.
Why public AI benchmarks don't predict real coding performance
The public signal is strong enough that software leaders should pay attention. Open models are closing benchmark gaps quickly, and recently released open models such as GLM-5.2, Kimi K2.7, Qwen3-Coder, and DeepSeek V4 are explicitly aimed at long-context, tool-using, agentic coding workflows.
Those benchmarks set priors. They cannot see your repositories, test setup, review standards, cache behavior, CI failure modes, permission boundaries, or the mix of work your engineers actually need help with. A model can look strong on a leaderboard and still be the wrong default for your environment if the harness wastes context, the cache economics break, or the model struggles on your common task shapes.
I wanted a Faros-specific answer. Which model and harness should we trust for our repositories, our task mix, and our review standards?
This is related to the new wave of model routers, including Sakana's Fugu, which dynamically orchestrates multiple models behind a single API. I like that direction. The limit is that a generic router still needs local evidence: the judge and routing policy have to reflect your codebase, tests, permission boundaries, review burden, cache behavior, and the work slices your engineers actually do.
How we tested 7 AI coding models across 211 real tasks
The experiment compared model plus harness combinations. I care about the pairing because engineers do not use a raw model in isolation. They use a model through a harness that decides how context is loaded, how tools are called, how diffs are produced, and how failures show up. I wanted the methodology visible before the result, because this experiment is only useful if the reader knows what kind of evidence it is.
GLM-5.2 vs. Kimi K2.6 vs. Claude Opus: Quality and cost results
GLM-5.2 and Kimi both landed in the top quality band. Claude Code + GLM-5.2 scored 0.568 and Claude Code + Kimi scored 0.566, close enough that I would not treat the difference as a quality win by itself. The operational difference was economics: in this cohort, GLM-5.2 delivered comparable quality while running faster and costing meaningfully less per task. That makes GLM-5.2 the route I would test first for broad production-like throughput, with Kimi still worth keeping in the routing pool where local data shows an edge.
Because cache share was nearly identical after the rerun, 89.7% for Claude Code + GLM-5.2 and 90.0% for Claude Code + Kimi, the cost gap was not a cache artifact.
The expensive frontier baselines did not buy their way into the top quality band in this cohort. Claude Code + Opus 4.8 trailed both Claude Code + Kimi and Claude Code + GLM-5.2 on mean score; its projected cost was roughly in the same range as Claude Code + Kimi, but materially higher than Claude Code + GLM-5.2. Codex + GPT-5.5 is included here as a high-reasoning current-default comparison, not as a Fireworks route.
AI coding model routing policy by work type
The aggregate table tells me which routes are efficient overall. The work-type slices tell me how I would route real work. I would not turn the aggregate winner into a one-size-fits-all rule.
Mean score tells me average quality. Average rank tells me which route stayed consistently near the top across tasks, even when it did not win outright.
For this cohort, the routing policy would start here:
The point is not to memorize these labels. The operating rule is to start with Claude Code + GLM-5.2 where local data says it clears the bar, escalate to Claude Code + Kimi when ambiguity, larger diffs, complexity, or review risk make quality worth paying for, and keep OpenCode + GLM-5.2 as the OpenCode route with the strongest consistency signal.
GLM-5.2 for coding: Why it changed our default model
GLM-5.2 was announced while we were running this experiment, so we added it to the loop. The GLM-5.2 release post emphasized long-horizon engineering, larger context, and stronger coding benchmark performance. That made it worth testing. The completed run made it practical: Claude Code + GLM-5.2 landed in the top quality band, ran faster than Claude Code + Kimi, and cost roughly half as much per task.
This is what I want to emphasize: The best default changed while the experiment was still running. Kimi still remains the route I would keep for quality-sensitive escalation, especially where the local slices show an edge: feature work, very-high-complexity work, larger diffs, and agent-tooling tasks. But GLM-5.2 cleared the bar often enough—and cheaply enough—that it changed what I would test first.
OpenCode + GLM-5.2 also earned its place. It had the best average rank signal, strong cache behavior, and the best data graph / schema-heavy profile. If a team is standardizing around OpenCode, that route deserves serious evaluation rather than being treated as a fallback.
The point is not to pick a winner and freeze it. You need the ability to rerun the evaluation when the market moves, when a provider fixes caching, when a new model ships, or when your own workload changes. This has to become an operating loop and not a one-time bakeoff.
How to run an AI model routing experiment (step-by-step)
Model routing is becoming an important engineering control. If I were doing this experiment with another engineering team, I would start small and make the decision useful within a week. Running your own small experiment could look like this:
Focus your model routing experiment on real tasks like test selection, deploy gates, and review policies. The process should reflect your actual workflows, adapt to changing conditions, and define exactly when to escalate. The goal is not to crown a winning model. The goal is to make model choice repeatable.
Make model selection a continuous operating loop
Defaults need evidence. Open models now deserve real engineering evaluations on actual company work, turning model choice into an engineering control like test selection or deploy gates. The market simply moves too quickly for static defaults. When a new model ships, caching improves, or your workload shifts, the right answer changes. Therefore, your most durable capability is continuously measuring your own work to adapt. Model selection must be an operating loop, not a one-time bake-off.
Faros can produce model-routing policies from the engineering systems you already use, drawing on issues, PRs, CI, code ownership, review flow, repo metadata, and delivery outcomes. Time Machine turns that history into a clear guide for which route should handle which work, when to escalate, and where the economics actually survive contact with your codebase. Contact us for a demo today.
Frequently asked questions when comparing open source vs. frontier AI models for coding
Are open source AI models as good as GPT-5.5 or Claude Opus for coding?
Yes, for certain engineering work. In Faros's test of 211 real coding tasks, open models GLM-5.2 and Kimi K2.6 scored in the top quality band, while frontier routes using Claude Opus 4.8 and GPT-5.5 scored lower and cost more. GLM-5.2 matched top quality at roughly half the cost per task.
What is AI model routing?
AI model routing is the practice of directing each engineering task to the model best suited for it, instead of sending every request to one expensive frontier model. Faros found this can cut AI coding costs by roughly 50% at equal quality, because cheaper open models handle routine work while costlier models are reserved for high-risk tasks.
How much can teams save by switching from frontier models to open models for coding?
Roughly 50% per task. In Faros's test, the best open-model route (GLM-5.2) cost $0.92 per task versus $1.76–$2.06 for frontier routes using Claude Opus 4.8 and GPT-5.5—at comparable quality. Actual savings depend on task mix and how much work can safely use the cheaper route.
What are the best open source AI models for coding in 2026?
The leading open source coding models in 2026 include GLM-5.2, Kimi K2.6, Qwen3-Coder, and DeepSeek V4, all built for long-context, agentic workflows. In Faros AI's testing of 211 tasks, GLM-5.2 and Kimi K2.6 performed strongest on real engineering work, with GLM-5.2 offering the best quality-to-cost ratio.
Can you trust public AI benchmarks when choosing a coding model?
Not fully. Public benchmarks are useful priors but can't account for your codebase, review standards, cache economics, or CI failure modes. Faros's test showed a top-benchmark model can still be the wrong default for a given environment. Validate candidate models against a sample of your own tasks first.
.webp)
.webp)