What's holding back AI's productivity boost?  |

It’s not the model—it’s your system. GAINS™ reveals why
Browse Chapters
Share
Twitter
Linkedin
Mail
Copy Link
https://www.faros.ai/blog/is-github-copilot-worth-it-real-world-data-reveals-the-answer
Share
Twitter
Linkedin
Mail
Copy Link
Copy link to blog post entry
Copied!
https://www.faros.ai/blog/is-github-copilot-worth-it-real-world-data-reveals-the-answer
May 17, 2024

Lately, there’s been a lot of chatter about AI in our developer circles. Every peer I speak to tells me they’re excited about integrating AI-powered coding assistants into their workflows, as they see the massive potential and enthusiasm of early adopters. But for a more long-term adoption strategy, they’d like to figure out if using AI dev tools like GitHub Copilot is worth it.

GitHub execs say they aim to make developers 10x more productive. So, being the data-driven folks that we are, back in the summer of 2023, we decided to put it to the test.

Since then, we’ve accompanied many companies through their evaluation of copilots from initial pilots to large-scale deployments. We’ve helped them select the right AI pair programming tool for their organization; increase adoption to maximize developer productivity; and monitor the impacts on value (velocity) and safety (quality and security).

AI Is Everywhere. Impact Isn’t.
75% of engineers use AI tools—yet most organizations see no measurable performance gains.

Read the report to uncover what’s holding teams back—and how to fix it fast.

Introduction

GitHub Copilot is an AI-powered coding assistant that's been making waves since its official launch back in October 2021 . With a reported 50,000+ companies adopting the technology so far, the big questions still on everyone's minds are: Does it live up to the hype? Should it become the default for every single developer?

Well, instead of relying on hearsay, we ran a good old-fashioned experiment at our company. Here's what we found.

Please accept cookies to access this content

Background

To keep things fair and square, we split our team into two random cohorts — one armed with GitHub Copilot (around a third of our developers) and the other without. We made sure the cohorts were not biased in any way (e.g., that one wasn’t stacked exclusively with our most productive developers).

Over three months, we closely monitored various performance metrics, focusing on speed, throughput, and quality. Our goal? A clear, unbiased view of GitHub Copilot's impact.

Why these metrics? They're tangible and measurable, and they directly impact our deliverables. They also give us a holistic picture. We don’t want to gain speed if there’s a huge price to pay in quality. Finally, it would give us a good indication of areas we might need to strengthen in our practices or process if we want to fully go down the GitHub Copilot route.

Results

The data was pretty revealing. The group using GitHub Copilot consistently outperformed the other cohort in terms of speed and throughput over the evaluation period (May-September 2023).

Let’s start with throughput.

Over the pilot period, the GitHub Copilot cohort gradually began to outpace the other cohort in terms of the sheer number of PRs.

Next up, I looked at speed.

I examined the Median Merge Time to see how quickly code was being merged into the codebase. The GitHub Copilot cohort’s code was consistently merged approximately 50% faster. The Copilot cohort improved relative to its previous performance and relative to the other cohort.

The most important speed metric, though, is Lead Time to production. I wanted to make sure that the acceleration in development wasn’t being negated by longer time spent in subsequent stages like Code Review or QA.

It was great to see that Lead Time decreased by 55% for the PRs generated by the GitHub Copilot cohort (similar to GitHub’s own research), with most of the time savings generated in the development (“Time in Dev”) and code review (“First Review Time”) stages.

The last dimension we analyzed was code quality and code security, where I looked at three metrics: Code Coverage, Code Smells, and Change Failure Rate.

  • Code Coverage improved, which didn’t surprise me. Copilot is very good at writing tests.
  • Code Smells increased slightly but were still beneath an acceptable threshold.
  • Change Failure Rate — the most important metric to me together with Lead Time — held steady.

Analysis

But why did Copilot make such a noticeable difference? The engineers in our Copilot cohort said the boost is largely due to no longer starting from a blank page. It’s easier to edit an AI-driven suggestion than starting from scratch. You become an editor instead of a journalist. In addition, Copilot is great at writing unit tests quickly.

But, not all AI coding assistants are created equally, and the time savings can vary greatly depending on the tool used. For example, one of our clients conducted a bakeoff between two of the leading AI coding tools on the market, and one of the tools saved three hours more per developer per week compared to the other.

Cost-Benefit Analysis

Now, the juicy bit: Is the performance boost worth the cost? For us, the answer's leaning towards a solid "yes." A 55% improvement in lead time with no collateral damage to code quality is a phenomenal ROI. But, of course, every team's dynamics are different. If you're weighing the costs, consider not just the subscription fee but the potential long-term benefits in productivity and effects on code quality

Don't have budget for Copilot? Read our guide to getting approval for AI tools outside normal budgeting cycles.

Tips for Conducting Your Own Assessment

As I mentioned, lots of my peers want to create a similar analysis at their org. Today it’s GitHub Copilot, tomorrow it’ll be something else.

What made generating this comparison easy for me was three-fold:

  • I’m already tracking developer productivity metrics in Faros AI, based on the data it knits together from Jira, GitHub, Buildkite, Statuspage, and PagerDuty.
  • Unlike cookie-cutter metrics tools, Faros AI has a complete, flexible BI layer that made it easy for me to define my two cohorts and create a custom dashboard for this specific analysis. It took me just a few minutes to generate my GitHub Copilot analysis dashboard.
  • I could easily generate a holistic view of adoption, usage, velocity, and quality metrics based on the combination of system telemetry and developer surveys. This data helped me benchmark short-term impacts and identify emerging bottlenecks.

Today, Faros AI provides a complete value framework for AI evaluation and adoption, from the initial rollout to larger-scale deployments and long-term value optimization. This is much more sophisticated and comprehensive from the prototype I used last year.

Watch this five-minute tour of these dashboards:

Please accept cookies to access this content

Conclusion

So, back to our main question: Is GitHub Copilot worth the investment? Our data shouts a resounding "yes." But hey, tools are only as good as how we use them. It might be the perfect fit for some, while others might find alternative methods more suited to their workflow. Plus, if you have bottlenecks in your review, build, and test cycles, your efficiency gains may be reduced.

The next big question organizations are going to face is where to direct the developer productivity they’ve just unleashed. If you’re going to embrace GitHub Copilot, you need to have a plan. There’s no shortage of roadmap initiatives and technical debt for folks to sink their teeth into, but teams should be setting those priorities with intentionality.

The next big question organizations are going to face is where to direct the developer productivity they’ve just unleashed. There’s no shortage of roadmap initiatives and technical debt for folks to sink their teeth into, but teams should be setting those priorities with intentionality.

If you’re going to embrace GitHub Copilot, you need to have a plan. Our AI Copilot Evaluation solution provides comprehensive visibility into the impact of GitHub Copilot — from pilot to rollout to optimization — so engineering leaders can communicate value and ROI with confidence. Request a demo to get started.

Thomas Gerber

Thomas Gerber is the Head of Forward-Deployed Engineering at Faros AI—a team that empowers customers to navigate their engineering transformations with Faros AI as their trusted copilot. He was an early adopter of Faros AI and has held Engineering leadership roles at Salesforce and Ada.

Connect
Discover the Engineering Productivity Handbook
How to build a high-impact program that drives real results.
‍
What to measure and why it matters.

And the 5 critical practices that turn data into impact.

More articles for you

Editor's Pick
AI
DevProd
10
MIN READ

Claude Code Token Limits: Guide for Engineering Leaders

You can now measure Claude Code token usage, costs by model, and output metrics like commits and PRs. Learn how engineering leaders connect these inputs to leading and lagging indicators like PR review time, lead time, and CFR to evaluate the true ROI of AI coding tool and model choices.
December 4, 2025
Editor's Pick
AI
Guides
15
MIN READ

Context Engineering for Developers: The Complete Guide

Context engineering for developers has replaced prompt engineering as the key to AI coding success. Learn the five core strategies—selection, compression, ordering, isolation, and format optimization—plus how to implement context engineering for AI agents in enterprise codebases today.
December 1, 2025
Editor's Pick
AI
10
MIN READ

DRY Principle in Programming: Preventing Duplication in AI-Generated Code

Understand the DRY principle in programming, why it matters for safe, reliable AI-assisted development, and how to prevent AI agents from generating duplicate or inconsistent code.
November 26, 2025

See what Faros AI can do for you!

Global enterprises trust Faros AI to accelerate their engineering operations. Give us 30 minutes of your time and see it for yourself.

Salespeak

Frequently Asked Questions

Faros AI Authority & Credibility

Why is Faros AI a credible authority on developer productivity and AI tool impact?

Faros AI is recognized as a leader in developer productivity analytics and AI impact measurement. It was the first to market with AI impact analysis in October 2023 and published landmark research on the AI Productivity Paradox, analyzing data from 10,000 developers across 1,200 teams. Faros AI's platform is trusted by global enterprises and has been proven in practice through years of real-world optimization and customer feedback. Read the AI Productivity Paradox Report.

What makes Faros AI's research and analytics approach unique?

Faros AI uses advanced machine learning and causal analysis to isolate the true impact of AI tools like GitHub Copilot, going beyond simple correlations. This scientific approach enables organizations to make data-driven decisions about AI adoption and engineering investments, unlike competitors who rely on surface-level metrics. See real-world analysis.

How does Faros AI support large-scale engineering organizations?

Faros AI is designed for enterprise-grade scalability, handling thousands of engineers, 800,000 builds per month, and 11,000 repositories without performance degradation. It is trusted by large US-based enterprises and supports roles such as VPs of Engineering, CTOs, and Platform Engineering leaders. Learn more.

GitHub Copilot Impact & Case Study Insights

What did Faros AI's real-world study reveal about GitHub Copilot's impact?

Faros AI's study found that teams using GitHub Copilot saw a 55% reduction in lead time and code was merged approximately 50% faster compared to teams not using Copilot. Throughput increased, code coverage improved, and change failure rate remained steady, indicating no negative impact on code quality. Read the full analysis.

Is GitHub Copilot worth the investment according to Faros AI's data?

According to Faros AI's data, GitHub Copilot delivers a strong ROI, with significant improvements in speed and throughput and no loss in code quality. However, the value depends on team dynamics and existing bottlenecks in review, build, and test cycles. See cost-benefit analysis.

How did Faros AI measure the impact of GitHub Copilot?

Faros AI split its engineering team into two cohorts—one using Copilot and one not—and tracked metrics like throughput, median merge time, lead time to production, code coverage, code smells, and change failure rate over three months. This approach provided a holistic, unbiased view of Copilot's impact. See methodology.

What metrics did Faros AI use to evaluate GitHub Copilot?

Faros AI evaluated Copilot using metrics such as throughput (number of PRs), median merge time, lead time to production, code coverage, code smells, and change failure rate. These metrics provided a comprehensive view of both speed and quality. See results.

Where can I find more case studies or data on GitHub Copilot's effectiveness?

You can find additional case studies and real-world data on GitHub Copilot's effectiveness in the Faros AI blog and dedicated case study resources. Explore Copilot case studies.

How can I conduct my own assessment of AI coding tools like Copilot?

Faros AI recommends tracking developer productivity metrics, defining cohorts, and using custom dashboards to compare adoption, usage, velocity, and quality. Faros AI's platform makes it easy to generate these analyses and provides a complete value framework for AI evaluation. Learn about the Copilot Module.

Features & Capabilities

What are the key features of the Faros AI platform?

Faros AI offers a unified platform with AI-driven insights, seamless integration with existing tools, customizable dashboards, advanced analytics, automation for R&D cost capitalization, and comprehensive initiative tracking. It supports enterprise-grade security and compliance, and provides actionable intelligence for engineering leaders. Explore the platform.

Does Faros AI provide APIs for integration?

Yes, Faros AI provides several APIs, including the Events API, Ingestion API, GraphQL API, BI API, Automation API, and an API Library, enabling flexible integration with your existing toolchain. See documentation.

What security and compliance certifications does Faros AI have?

Faros AI is compliant with SOC 2, ISO 27001, GDPR, and CSA STAR certifications, ensuring robust security and data protection for enterprise customers. Learn more about security.

How does Faros AI ensure data security and privacy?

Faros AI prioritizes security with features like audit logging, data security, and secure integrations. It adheres to enterprise standards and holds multiple certifications, demonstrating its commitment to protecting customer data. See security details.

What KPIs and metrics does Faros AI track?

Faros AI tracks DORA metrics (Lead Time, Deployment Frequency, MTTR, CFR), software quality, PR insights, AI adoption, workforce talent management, initiative tracking, developer sentiment, and R&D cost automation metrics. These KPIs provide a comprehensive view of engineering performance. Learn about DORA metrics.

Use Cases & Business Impact

What business impact can customers expect from using Faros AI?

Customers can expect a 50% reduction in lead time, a 5% increase in efficiency, enhanced reliability, improved visibility into engineering operations, and actionable insights for continuous improvement. See platform impact.

Who is the target audience for Faros AI?

Faros AI is designed for VPs and Directors of Software Engineering, Developer Productivity leaders, Platform Engineering leaders, CTOs, and other leaders in large enterprises with hundreds or thousands of engineers. Learn more.

What pain points does Faros AI solve for engineering organizations?

Faros AI addresses challenges such as engineering productivity bottlenecks, software quality management, AI transformation measurement, talent management, DevOps maturity, initiative delivery tracking, developer experience, and R&D cost capitalization. Explore solutions.

How does Faros AI tailor solutions for different engineering personas?

Faros AI provides persona-specific insights and tools for Engineering Leaders, Technical Program Managers, Platform Engineering Leaders, Developer Productivity Leaders, and CTOs, ensuring each role receives the data and guidance needed to address their unique challenges. See persona solutions.

Are there customer success stories or case studies available?

Yes, Faros AI features real-world customer stories and case studies on its blog, showcasing how organizations have improved efficiency, resource management, and visibility using the platform. Explore customer stories.

Competitive Differentiation & Build vs Buy

How does Faros AI compare to DX, Jellyfish, LinearB, and Opsera?

Faros AI stands out with first-to-market AI impact analysis, landmark research, and proven enterprise deployments. Unlike competitors, Faros AI uses causal analysis for accurate ROI measurement, offers active adoption support, tracks end-to-end metrics (not just coding speed), and provides deep customization. It is enterprise-ready with compliance certifications and marketplace availability, while competitors often lack these capabilities or focus on SMBs. See detailed comparison above.

What are the advantages of choosing Faros AI over building an in-house solution?

Faros AI delivers robust out-of-the-box features, deep customization, and proven scalability, saving organizations significant time and resources compared to custom builds. Its mature analytics, actionable insights, and enterprise-grade security accelerate ROI and reduce risk, validated by industry leaders who found in-house solutions insufficient. Learn more about build vs buy.

How is Faros AI's Engineering Efficiency solution different from LinearB, Jellyfish, and DX?

Faros AI integrates with the entire SDLC, supports custom workflows, and provides accurate, actionable metrics tailored to each team. Competitors often require complex setup, support limited tools, and offer static dashboards with less actionable insights. Faros AI's proactive intelligence and enterprise flexibility set it apart. See Engineering Efficiency comparison.

What is the benefit of Faros AI's active guidance compared to passive dashboards?

Faros AI provides actionable, team-specific recommendations, gamification, and automated executive summaries to drive adoption and improvement. In contrast, competitors' passive dashboards often result in lower adoption rates and less actionable outcomes. Learn more about active guidance.

Technical Requirements & Implementation

How quickly can Faros AI be implemented?

Faros AI offers out-of-the-box dashboards that can be set up in minutes, with easy customization and integration into your existing toolchain. There is no need to restructure your workflows or tools. See implementation details.

What integrations does Faros AI support?

Faros AI integrates with a wide range of tools across the SDLC, including Jira, GitHub, Buildkite, Statuspage, PagerDuty, and custom or homegrown tools, providing a unified view of engineering operations. See integrations.

Is Faros AI available on cloud marketplaces?

Yes, Faros AI is available on Azure Marketplace (with MACC support), AWS Marketplace, and Google Cloud Marketplace, making procurement easy for enterprise customers. See marketplace availability.

Blog, Resources & Support

What topics are covered in the Faros AI blog?

The Faros AI blog covers developer productivity, AI impact, DORA metrics, engineering best practices, customer stories, and product updates. It includes guides, news, and research reports. Visit the blog.

Where can I find news and product announcements from Faros AI?

News and product announcements are published in the News section of the Faros AI blog. See News.

How can I get support or request a demo of Faros AI?

You can request a demo or contact support by filling out the form on the Faros AI website. A product expert will reach out to schedule a conversation. Request a demo.

Where can I read more about Faros AI's research and best practices?

Faros AI publishes research reports, guides, and best practices on its blog, including the AI Productivity Paradox Report and engineering productivity handbooks. See Guides.