Local versus hosted
Running a coding model on your own laptop has obvious appeal. Source code never leaves the machine, and there is no per-token bill. What you give up is harder to say, because public benchmarks look little like the work a team does day to day.
We tried to find out with two real tasks from Faros’s Kaleidoscope repository. Flint, Recursive’s downloadable coding model, ran locally in LM Studio on a Mac, and we tested it at 64K and 128K, two limits on how much text it can work with at once. Opus 5.5 and GPT-6 Astra ran through hosted services. Time Machine v2 recreated each task from Faros’s repository history, so every model got the same starting code and the same instructions.
Opus and Astra finished each task in under three minutes. Flint at 128K matched their tool-search score, but it took about 26 minutes, and its cache pricing attempt was still unfinished when the hour ran out.
Keep the sample size in mind. We ran one attempt per configuration, eight in all, and one of our two judges, Opus 5.5, was also a contestant. These results describe this setup and don’t rank the model
Tasks and scoring
The tasks came from Faros's Kaleidoscope repository:
- Tool search: enable tool search in the Claude Code gateway installer while preserving existing settings and removal behavior.
- Cache pricing: correct one-hour cache-write billing and add two regression tests.
Scores average two independent assessments of weighted requirements by Opus 5.5 and GPT-6.1 Sol. We ran the repository’s build and tests separately. Opus was both a candidate and a judge in this experiment.
Requirement scores
The tool-search task covered installation, removal and directory-specific shell settings. Flint at 128K, Opus and Astra all scored 90.9% and passed the repository checks. Opus awarded full credit to all three; Sol could not confirm the same removal edge case in their patches.
That case involved a file containing the tool-search flag but no other Faros settings. Removal was supposed to leave the file untouched and avoid creating a backup solely because the flag existed. Sol’s “not demonstrated” finding does not establish that the code was broken.
The 64K Flint patch scored 68.2%. Installation was required to add the flag only when absent and preserve existing values such as “auto” or “false.” Sol marked that requirement as unmet, in addition to the removal concern. The repository checks passed.
Elapsed time
Every agent could work for up to 60 minutes and 400 turns. None reached the turn limit. The timings below include setup and waiting for outstanding model requests, but exclude the subsequent judging and repository checks.
Time spent on each attempt
Opus and Astra finished tool search in 53 and 48 seconds. Flint at 128K took 25 minutes and 43 seconds to reach the same requirement score. On pricing, Opus finished in 2 minutes and 45 seconds and Astra in 1 minute and 46 seconds; both scored 100% and passed the checks.
Flint’s 64K pricing attempt ended after 6 minutes and 17 seconds without a patch. Its shorter runtime does not represent a completed job. At 128K, Flint reached the 60-minute working limit with an incomplete patch. Outstanding requests took about seven more minutes to finish, bringing the recorded elapsed time to 67 minutes and 3 seconds.
All eight results: scores, times and code checks
Tool search
Cache pricing
All eight attempts, one per task and configuration. Score is the judges’ average weighted requirement score. Checks are the repository build and tests, including any test edits. Flint 64K’s pricing attempt left no patch; Flint 128K’s pricing time includes about seven minutes waiting for outstanding requests after its 60-minute working limit.
The pricing patch
The pricing bug could bill a one-hour cache write at zero when a special rate was unavailable. The requested change had to resolve model names against the rate catalog, preserve applicable catalog rates, and fall back to the base cache-write rate when no one-hour rate existed. Two new regression cases were required: one for a provider-prefixed model name and one for the no-catalog fallback.
Opus and Astra met every scored requirement and added both cases. Flint’s 128K patch implemented the rate changes accepted by both judges, but omitted the tests. It also increased the verifier’s minimum case count from 27 to 29 while leaving only 27 cases in the test file.
Pricing requirements met by each model
The two judges agreed on every scored pricing requirement for each nonempty patch. Flint 128K met seven of nine weighted points. The empty 64K patch received zero without judging.
The missing tests accounted for two of the nine available points, leaving Flint with 7/9, or 77.8%. Its build, Go tests and Python tests passed. The pricing verifier failed because it found fewer cases than the new minimum required. A developer receiving this patch would still need to add the cases and rerun validation.
These checks ran on the code each model produced, including its changes to tests. They were not hidden acceptance tests. For the empty 64K pricing attempt, a passing check on unchanged starting code was not counted as a solution.
Agent turns and tool use
A turn is a model response the agent receives and uses. That response can request several tool actions, such as reading files, running commands or editing code.
Agent turns and tool actions
On pricing, Flint at 128K used 69 turns and 97 tool actions. Its actions included 84 shell commands, 10 file reads and three calls to the edit tool. Opus used 15 turns and 17 actions; Astra used 11 turns and 19 actions. A shell command can also modify files, so the edit-tool count is not a count of all code edits.
The 64K pricing attempt made 56 tool calls and left no patch. Those calls comprised 50 shell commands and six file reads.
Flint’s 128K pricing run has 75 model requests in the service logs but only 69 responses used by the agent. Five requests hit the client’s five-minute timeout, and another was interrupted when the agent stopped. Those six responses cannot be counted as completed agent turns.
Text generated
The services report output in tokens, the small pieces of text models process. These totals include reported reasoning as well as tool requests, code and final answers. They are not counts of words or lines of code.
Text generated by each model
Service-reported output, including reported reasoning. Both panels use the same zero-based scale. Flint 128K’s pricing total includes responses the agent never received. Tokenization and reporting differ between models.
Flint’s 128K pricing run reported 107,134 output tokens, of which 105,887 were labeled as reasoning. Opus reported 13,072 output tokens and Astra 4,305. Flint’s larger total accompanied a patch still missing the two required tests.
The Flint total also includes output from requests whose responses never reached the agent. The model could continue generating after the client had timed out; that text could not help the agent finish the patch.
Models split text into tokens differently, and their services expose different reasoning and usage accounting. We used service-reported output consistently across all models. Flint’s native engine logs contain different totals; mixing those into the comparison would change the measurement. No dollar-cost comparison was calculated.
Context use
Each request carries the task, relevant code, tool results and conversation accumulated so far. Longer sessions can therefore require more input even when they are working on the same original task. We disabled automatic conversation shortening for these runs. The plots show the input sent with each successive model request; the horizontal axis counts requests, not minutes.
Input text over the course of each attempt
× Flint 64K ● Flint 128K ▲ Opus 5.5 ■ GPT-6 Astra
Flint at 128K reached 85,221 input tokens on tool search and 117,182 on pricing. Opus’s largest inputs were 18,910 and 47,314; Astra’s were 19,135 and 25,235. The hosted attempts finished after fewer turns and used less accumulated context on both tasks.
Both 128K Flint runs used more input than the 64K setting could hold. They also scored higher and took longer than the 64K attempts. With one attempt per setting, we cannot separate the effect of the larger window from variation between runs or differences in machine load.
Response limits and stopping
The context window, maximum response length and working-time limit are different constraints. Each response could initially use up to 32,768 tokens, but the runner reduced that allowance when the conversation left less room. It was a maximum for one response, not a budget for the entire task.
On Flint’s 64K pricing attempt, the final response was allowed 1,265 tokens and the engine used all of them. The service labeled the stop as normal, but the engine log suggests the response was cut short. No visible final answer or patch was produced. The tool-search attempt at 64K also used its entire final response allowance, after producing a patch.
At 128K, Flint’s final tool-search response could use 32,768 tokens and generated 715 according to the engine log. It finished with room remaining. The 128K pricing agent stopped at the 60-minute working limit. We evaluated the patch captured then; output arriving later could not change it.
The five-minute request timeout also affected that pricing run. We did not repeat it with a longer timeout, so the evidence does not show how much the client behavior contributed to the incomplete result.
Local hardware
Flint used a compressed Q4_K_M model in LM Studio 0.4.24+1 on an Apple M5 Pro with 48 GiB of memory. Its weights were stored on an external drive. The engine’s average generation rates at 128K were 32.3 tokens per second on tool search and 30.5 on pricing. These rates cover text generation; the elapsed times also include processing input, tool use and other waiting.
The Mac was running other workloads, and memory measurements covered the whole machine. We did not isolate an external-drive effect or measure the hardware behind Opus and Astra. Recursive also supplies an optimized inference engine that we did not test. These results describe the LM Studio configuration we ran.
Opus and Astra returned patches that passed the checks on both tasks within three minutes each. Flint at 128K matched their requirement score on tool search after a longer wait, but its pricing patch still needed tests after an hour.
Experiment details
Scope and task selection
The runs took place on 22 October 2026, local time. This is a preliminary comparison of two tasks from one repository, with one attempt per model and setting. Eight attempts are reported, including the empty patch and timeout. The local machine constrained the sample size; these observations do not establish a general model ranking.
Tasks came from an ordered candidate list. One candidate was excluded during preparation because its required Jira metadata lookup failed authentication, before any model attempt. The next eligible task was used. Original solutions and scoring checklists were withheld from the agents, although historical task descriptions can contain implementation guidance.
Runtime and limits
All candidates used the same coding-agent harness, pi. Flint used Q4_K_M weights in LM Studio 0.4.24+1 on an Apple M5 Pro with 48 GiB of memory. Its context settings were 65,536 and 131,072 tokens. Opus 5.5 and GPT-6 Astra used hosted services with medium reasoning effort. Flint retained its native reasoning behavior.
The agent could work for 60 minutes and up to 400 turns. Each response could use at most 32,768 output tokens; that allowance shrank when context was nearly full. Automatic conversation shortening was disabled. The client retained a five-minute timeout for each model request. No run reached the 400-turn limit. Total elapsed time includes setup and waiting for outstanding requests, but excludes the separately run repository checks and judging.
Flint’s weights were stored on an external drive. Memory measurements covered the whole machine. The experiment did not isolate an external-drive effect or establish equivalent hardware conditions for local and hosted inference.
Scoring and code checks
Opus 5.5 and GPT-6.1 Sol judged the patches independently against a fixed, calibrated checklist. Each headline score averages their weighted requirement scores. Preservation checks are reported separately and contribute no points. Opus was both a candidate and a judge. The tool-search score of 90.9% for Flint 128K, Opus and Astra includes the same removal edge case marked “not demonstrated” by the Sol judge.
Build, Go tests, Python tests and pricing checks ran on the resulting code, including model edits to tests. They were not hidden acceptance tests. A green check on unchanged code was not treated as a solution. The experiment used the same provisional local Time Machine v2 integration throughout; it was not a production-service evaluation.
Token counts
The output figures sum the token counts reported by the model services. Reported reasoning is included, not added a second time. Flint’s engine recorded different generation totals from those exposed by its API; the figures consistently use API counts for all models. Missing cache-write accounting prevented an official reconciled Flint usage total. No monetary comparison was calculated.
The follow-up changed Flint’s context setting while retaining the model weights, task inputs, checklists, judges and agent limits. One new stochastic attempt per task cannot isolate a causal effect of context. All figures in this article come from the existing archive; no new model experiments were run for the article.
About Faros
Faros connects AI spend to engineering outcomes. Its Token Engineering platform combines AI session and software delivery data so teams can see where tokens go, evaluate model choices against their own work, and govern AI usage.
Book a demo to discuss evaluating models on your team's engineering work.
