How to evaluate coding agents on your own codebase

Learn how to build a benchmark from your own engineering tasks, compare models and harnesses, and find the setup that fits your stack and standards.

A graphic showcasing sample scores for various coding models.

How to evaluate coding agents on your own codebase

Learn how to build a benchmark from your own engineering tasks, compare models and harnesses, and find the setup that fits your stack and standards.

A graphic showcasing sample scores for various coding models.
Chapters

Evaluating coding agents against your codebase

Your organization’s code, tools, and engineering standards give you a practical way to evaluate AI agents within your specific environment.

In this article, we’ll walk you through how to capitalize on work your team has already shipped to see how different setups handle your stack, follow your conventions, and meet your quality bar. Public benchmarks can help you find candidates, but your own context shows which ones fit the work you need to do.

There are 4 steps:

  1. Start by choosing broad, representative tasks that will form the basis for the benchmark.
  2. Then, set up steady, well-defined assessment environments for a fair comparison.
  3. Score each agent’s output and compare results.
  4. From there, further evaluate instances with differences to determine the best model and harness combinations.

How to choose tasks for an AI agent benchmark

Start with the software your engineers have shipped. At Faros, our Engineering World Model connects work from a task through its pull request and deployment to production. That context helps identify examples of the work an engineering organization needs to deliver.

Each benchmark instance pairs an assignment with its corresponding code change. You can inspect what was requested, what changed, and what happened after release.

Choose instances across several dimensions:

  • Business importance. Include critical repositories and services, such as payments or access control. Cover important but infrequent work.
  • Size and scope. Mix contained fixes with changes across files, services, or repositories. Lines changed alone don’t show difficulty.
  • Risk. Include core business behavior and lower-risk work, such as a small UI adjustment.
  • Technology and constraints. Cover languages, framework generations, internal dependencies, compatibility, and hardware limits.
  • Type of work. Include debugging, features, refactoring, tests, and maintenance.

Favor breadth in early comparisons. Choose the mix before seeing which candidate wins, and document historical work you can’t reproduce.

Figure 1. Build a representative set from production tasks and changes. Cover importance, scope, risk, technology, and type of work.

How to set up assessment environments for comparisons

Each instance needs a clear assignment, reproducible starting environment, and agreed assessment. Keep them consistent across candidates so you can understand what caused a difference.

When comparing models, hold the setup steady: starting code, instructions, tools, and resource limits. The software that gives an agent tools and manages its execution is often called a harness. If you’re evaluating a new harness or workflow, compare the complete setups and record what changed.

For historical tasks, give the agent only the code and context available when the task began. Keep the solution, private grading material, and later references out of reach. Give every attempt a clean start, and record model versions, settings, tools, time limits, and usage.

Separate unusable code from runs that couldn’t be graded because an environment failed. Repeat runs when needed to distinguish a consistent improvement from variation.

Figure 2. Keep code, instructions, tools, limits, and the quality bar consistent. Give each candidate a clean start.

How to evaluate AI-generated output

Use the production change as a reference, but don’t require candidates to reproduce it. A different implementation may solve the task just as well. Assess every candidate against the same standard.

Consider several kinds of evidence:

  • Code and behavior. Does the change meet requirements, preserve existing behavior, and fit the architecture? Is it maintainable?
  • Tests. Which relevant tests pass or fail? Do they cover the reported problem, likely regressions, and key constraints?
  • The agent’s approach. What did it investigate, attempt, and abandon? Failed commands, unnecessary edits, or missed dependencies can mean more work for your team.
  • Production context. Regressions, incidents, or follow-up fixes may reveal requirements missing from the task description. Use that history to improve evaluation without assuming a candidate would have caused the same outcome.

An LLM judge can assess qualities tests miss. Calibrate it with engineer-reviewed examples, explicit criteria, and checks against human judgment. Keep critical correctness requirements as gates; a high average score shouldn’t compensate for a serious defect. Faros describes its approach in this evaluation framework for AI model routing.

Compare results by repository and task category, as well as overall. An agent that improves UI work but struggles with a core service may suit a particular route without becoming your default. Track quality, elapsed time, review effort, and cost per accepted change, including failed attempts and retries.

Uber’s internal SWE benchmark uses thousands of real pull requests across its monorepos to compare models by task type. For its code-review agent, Uber measures precision and recall against known bugs, alongside cost, latency, and noise. These evaluations guide model selection for particular workloads.

How to drill down further for greater clarity

After several broad comparisons, identify instances that reliably reveal differences between candidates. Tasks that nearly every setup solves, or that defeat them all, may be less useful for screening.

Build a smaller screening set from those findings while retaining coverage of important work and risks. Run new agents or routes through it first. Promising improvements, mixed results, or changes affecting critical services can justify a broader evaluation.

Check the set against broad runs and refresh it as your codebase and tools change. Periodic full evaluations help ensure it still reflects your work.

Make evaluation part of development: test each new model, harness, or routing change against a consistent baseline. Over time, you build a record of what works for your software and a way to revisit that judgment as the options change. Faros’s Time Machine evaluates model and harness combinations against work your teams have already delivered.

See how Faros can run agent evaluations in your org

Public benchmarks give you a shared way to compare coding agents, but their tasks can’t reflect every part of your environment. Your frameworks, dependencies, engineering conventions, and business rules all shape what good work looks like. A benchmark built from your own engineering history gives you a more relevant basis for choosing a setup.

Faros brings that history together in our Engineering World Model, linking tasks and agent sessions to pull requests and production outcomes. The Time Machine uses that context to compare coding setups on work your team has delivered, so you can see where each one fits.

Faros is a Token Engineering platform that connects AI usage and spend to software outcomes. It helps teams evaluate coding setups against their own work. Book a demo to discuss what this could look like for your codebase.

Ron Meldiner

Ron Meldiner

Ron is an experienced engineering leader and developer productivity specialist. Prior to his current role as Field CTO at Faros, Ron led developer infrastructure at Dropbox.

Graduation cap with a tassel over a dark gradient background.
AI ENGINEERING REPORT 2026
The Acceleration 
Whiplash
The definitive data on AI's engineering impact. What's working, what's breaking, and what leaders need to do next.
  • Engineering throughput is up
  • Bugs, incidents, and rework are rising faster
  • Two years of data from 22,000 developers across 4,000 teams
AI Industry
10
MIN READ

Best AI gateways for enterprise software development

Compare 8 AI gateways for enterprise software teams, including Faros, Kong, and Cloudflare, plus five questions to help you choose one.

AI Industry
8
MIN READ

What is an AI gateway? Functions, limitations, and use cases

An AI gateway centralizes access, reliability, cost, and security controls for AI models. Learn how it works, when you need one, and what it can’t solve.

AI Industry
7
MIN READ

OpenAI Codex pricing explained

Codex pricing, usage limits, and ROI: how to compare plans and API costs and track AI coding spend across tools to see what’s worth it.