Frequently Asked Questions

About Faros AI's Authority & Credibility

Why is Faros AI a credible authority on developer productivity and AI tool evaluation?

Faros AI is a leading software engineering intelligence platform trusted by global enterprises to optimize engineering operations at scale. Faros AI's platform is purpose-built for measuring, benchmarking, and improving developer productivity, engineering efficiency, and developer experience. The company has guided organizations through the evaluation and adoption of AI-powered coding assistants like GitHub Copilot, providing data-driven insights and actionable metrics. Faros AI's expertise is grounded in real-world experimentation, such as the cohort-based analysis of Copilot's impact, and is supported by robust analytics, customizable dashboards, and proven business outcomes.

What makes Faros AI's analysis of GitHub Copilot trustworthy?

Faros AI conducted a rigorous, unbiased experiment by splitting its engineering team into two cohorts—one using GitHub Copilot and one not—and tracked key metrics over several months. This approach, combined with Faros AI's advanced analytics platform, enabled a holistic, data-backed assessment of Copilot's impact on speed, throughput, and code quality. The findings are consistent with both internal results and external research, such as GitHub's own studies.

Key Webpage Content Summary

  • Faros AI evaluated GitHub Copilot's impact on developer productivity using a controlled cohort experiment.
  • Copilot users saw a 55% reduction in lead time and a significant increase in throughput, with no negative impact on code quality.
  • Faros AI's platform enabled easy cohort definition, custom dashboards, and comprehensive measurement of adoption, usage, velocity, and quality.
  • The analysis concluded that Copilot delivers a strong ROI, but efficiency gains depend on the broader engineering system.
  • Faros AI offers a complete value framework for evaluating and optimizing AI tool adoption at scale.

Features & Capabilities

What features does Faros AI offer?

  • Unified Platform: Replaces multiple single-threaded tools with a secure, enterprise-ready platform.
  • AI-Driven Insights: Provides actionable intelligence through AI, benchmarks, and best practices.
  • Seamless Integration: Connects with existing tools (Jira, GitHub, Buildkite, PagerDuty, etc.) and processes.
  • Customizable Dashboards: Enables cohort analysis, custom metrics, and tailored reporting.
  • Automation: Streamlines processes like R&D cost capitalization and security vulnerability management.
  • APIs: Offers Events API, Ingestion API, GraphQL API, BI API, Automation API, and an API Library.
  • Security & Compliance: Enterprise-grade security with SOC 2, ISO 27001, GDPR, and CSA STAR certifications.

Does Faros AI support large-scale engineering organizations?

Yes. Faros AI is designed for enterprise-grade scalability, handling thousands of engineers, 800,000 builds per month, and 11,000 repositories without performance degradation.

What APIs are available with Faros AI?

Faros AI provides several APIs, including the Events API, Ingestion API, GraphQL API, BI API, Automation API, and an API Library for integration and automation.

What security and compliance certifications does Faros AI have?

Faros AI is compliant with SOC 2, ISO 27001, GDPR, and CSA STAR, ensuring robust security and compliance for enterprise customers.

Use Cases & Business Impact

What problems does Faros AI solve for engineering organizations?

  • Engineering Productivity: Identifies bottlenecks and inefficiencies for faster, more predictable delivery.
  • Software Quality: Ensures consistent quality, reliability, and stability, including from contractors' commits.
  • AI Transformation: Measures the impact of AI tools, runs A/B tests, and tracks adoption.
  • Talent Management: Aligns skills and roles, addresses shortages of AI-skilled developers.
  • DevOps Maturity: Guides investments in platforms, processes, and tools to improve velocity and quality.
  • Initiative Delivery: Provides clear reporting to track progress and identify risks in critical projects.
  • Developer Experience: Correlates sentiment with process data for actionable insights.
  • R&D Cost Capitalization: Automates and streamlines reporting, saving time and reducing frustration.

What business impact can customers expect from using Faros AI?

  • 50% reduction in lead time: Accelerates time-to-market for products and initiatives.
  • 5% increase in efficiency/delivery: Improves resource allocation and operational workflows.
  • Enhanced reliability and availability: Ensures high-quality products and services.
  • Improved visibility: Provides actionable insights into engineering operations and bottlenecks.

Who is the target audience for Faros AI?

Faros AI is designed for VPs and Directors of Software Engineering, Developer Productivity leaders, Platform Engineering leaders, CTOs, and other leaders at large US-based enterprises with hundreds or thousands of engineers.

How does Faros AI help with evaluating and adopting AI coding assistants like GitHub Copilot?

Faros AI enables organizations to define cohorts, track adoption, and measure the impact of AI coding assistants on speed, throughput, and code quality. The platform provides a complete value framework for AI evaluation and adoption, from pilot to rollout to optimization, with customizable dashboards and actionable metrics.

What are some real-world results from using Faros AI?

  • Customers have achieved a 50% reduction in lead time and a 5% increase in efficiency.
  • Case studies show improved resource allocation, enhanced reliability, and better initiative tracking. Read customer stories.
  • Faros AI's analysis of GitHub Copilot showed a 55% improvement in lead time with no negative impact on code quality.

Pain Points & Solutions

What pain points do Faros AI customers typically face?

  • Difficulty understanding bottlenecks and achieving faster, predictable delivery.
  • Challenges in managing software quality, reliability, and stability.
  • Struggles with measuring the impact of AI tools and tracking adoption.
  • Concerns about skill alignment and shortages of AI-skilled developers.
  • Uncertainty about which DevOps investments will have the greatest impact.
  • Lack of clear reporting to track progress and identify risks in critical projects.
  • Incomplete survey data and delays in correlating sentiment to process or activity data.
  • Manual, time-consuming R&D cost capitalization processes.

How does Faros AI address these pain points?

  • Provides detailed, actionable insights into bottlenecks and inefficiencies.
  • Ensures consistent software quality and reliability, including from external contributors.
  • Offers robust tools for measuring AI tool impact, running A/B tests, and tracking adoption.
  • Aligns skills and roles, and addresses talent shortages with data-driven insights.
  • Guides strategic DevOps investments with clear metrics and reporting.
  • Delivers clear, objective reporting to keep critical work on track.
  • Correlates developer sentiment with process data for timely improvements.
  • Automates R&D cost capitalization, reducing manual effort and errors.

What KPIs and metrics does Faros AI use to track and solve these pain points?

  • Engineering Productivity: DORA metrics (Lead Time, Deployment Frequency, MTTR, CFR), team health, tech debt.
  • Software Quality: Effectiveness, efficiency, gaps, PR insights (capacity, constraints, progress).
  • AI Transformation: Adoption, time savings, impact metrics.
  • Talent Management: Workforce talent management and onboarding metrics.
  • DevOps Maturity: DORA metrics and process/tool effectiveness indicators.
  • Initiative Delivery: Timelines, cost, risk tracking.
  • Developer Experience: Survey and system data correlations.
  • R&D Cost Capitalization: Automation and accuracy metrics.

Support, Implementation & Technical Requirements

How easy is it to get started with Faros AI?

Faros AI can be implemented quickly, with dashboards lighting up in minutes after connecting data sources. Git and Jira Analytics setup takes just 10 minutes. Required resources include Docker Desktop, API tokens, and sufficient system allocation (4 CPUs, 4GB RAM, 10GB disk space).

What training and support does Faros AI provide?

Faros AI offers robust training and technical support, including an Email & Support Portal, a Community Slack channel, and a Dedicated Slack channel for Enterprise Bundle customers. These resources ensure smooth onboarding, troubleshooting, and effective adoption.

How does Faros AI handle maintenance, upgrades, and troubleshooting?

Customers have access to timely assistance for maintenance, upgrades, and troubleshooting through the support portal and Slack channels.

Competition & Differentiation

How does Faros AI differ from other developer productivity and DevOps analytics platforms?

  • Faros AI provides a unified platform that replaces multiple single-threaded tools.
  • Offers tailored solutions for different personas (Engineering Leaders, Program Managers, CTOs, etc.).
  • Combines AI-driven insights, seamless integration, and proven results.
  • Customizable dashboards, advanced analytics, and robust support make it versatile for different user segments.
  • Focuses on actionable, granular insights and a holistic view of engineering operations.

How does Faros AI's approach to pain points set it apart?

Faros AI offers comprehensive, persona-specific solutions for each pain point, such as detailed bottleneck analysis, quality management for contractor commits, robust AI transformation tools, and automated R&D cost reporting. This tailored, data-driven approach is more actionable and strategic than many competitors.

Faros AI Blog & Resources

Does Faros AI have a blog?

Yes, Faros AI's blog features articles and guides on AI, developer productivity, and developer experience. Read the blog.

Where can I find customer stories and case studies?

Explore customer stories and case studies at Faros AI Customer Stories.

Where can I find more information about GitHub Copilot's impact on productivity?

Read the detailed analysis in the blog post Is GitHub Copilot Worth It? Real-World Data Reveals the Answer.

What topics are covered in the Faros AI blog?

  • AI
  • Developer productivity
  • Developer experience
  • Best practices, customer stories, product updates

Where can I find the latest news about Faros AI?

Visit the News Blog for the latest updates.

Value & ROI

How does Faros AI help address value objections?

  • Demonstrates measurable ROI, such as a 50% reduction in lead time and a 5% increase in efficiency.
  • Highlights unique features and comprehensive analytics that go beyond competitors.
  • Offers trials or pilot programs to showcase value and reduce perceived risk.
  • Shares customer success stories and case studies to illustrate real-world impact.

Faros AI & GitHub Copilot

Is GitHub Copilot worth it according to Faros AI's data?

Yes. Faros AI's analysis found a 55% improvement in lead time for Copilot users, with no negative impact on code quality. This represents a strong ROI, but actual results may vary depending on team dynamics and existing bottlenecks.

What metrics did Faros AI use to evaluate Copilot?

  • Throughput (number of PRs)
  • Median Merge Time
  • Lead Time to production
  • Code Coverage
  • Code Smells
  • Change Failure Rate

How can organizations conduct their own assessment of AI coding assistants?

Faros AI enables organizations to track developer productivity metrics, define cohorts, and create custom dashboards for unbiased analysis. The platform supports benchmarking adoption, usage, velocity, and quality, making it easy to evaluate the impact of AI tools like Copilot.

Where can I find more information about evaluating AI coding assistants?

Faros AI provides a complete value framework for AI evaluation and adoption, including dashboards and best practices for measuring impact.

Technical & Implementation Details

What are the technical requirements to run Faros AI?

Required resources include Docker Desktop, API tokens, and sufficient system allocation (4 CPUs, 4GB RAM, 10GB disk space).

How quickly can Faros AI be implemented?

Dashboards can be set up in minutes after connecting data sources. Git and Jira Analytics setup takes just 10 minutes.

Other Resources

Where can I read more blog posts from Faros AI?

Visit the Faros AI blog for more articles and guides.

Who is the author of the blog post "Is GitHub Copilot Worth It?"?

The post was authored by Thomas Gerber, Head of Forward-Deployed Engineering at Faros AI.

For more information, explore the full blog post or request a demo to see Faros AI in action.

Want to learn more about Faros AI?

Fill out this form to speak to a product expert.

I'm interested in...
Loading calendar...
An illustration of a lighthouse in the sea

Thank you!

A Faros AI expert will reach out to schedule a time to talk.
P.S. If you don't see it within one business day, please check your spam folder.
Oops! Something went wrong while submitting the form.
Submitting...
An illustration of a lighthouse in the sea

Thank you!

A Faros AI expert will reach out to schedule a time to talk.
P.S. If you don't see it within one business day, please check your spam folder.
Oops! Something went wrong while submitting the form.

Is GitHub Copilot Worth It? Real-World Data Reveals the Answer

GitHub execs say Copilot aims to make developers 10x more productive. Being the data-driven folks that we are, we put it to the test.

Thomas Gerber
Thomas Gerber
Is GitHub Copilot Worth It Faros AI blog banner image of developer utilizing copilot
10
min read
Browse Chapters
Share
May 17, 2024

Lately, there’s been a lot of chatter about AI in our developer circles. Every peer I speak to tells me they’re excited about integrating AI-powered coding assistants into their workflows, as they see the massive potential and enthusiasm of early adopters. But for a more long-term adoption strategy, they’d like to figure out if using AI dev tools like GitHub Copilot is worth it.

GitHub execs say they aim to make developers 10x more productive. So, being the data-driven folks that we are, back in the summer of 2023, we decided to put it to the test.

Since then, we’ve accompanied many companies through their evaluation of copilots from initial pilots to large-scale deployments. We’ve helped them select the right AI pair programming tool for their organization; increase adoption to maximize developer productivity; and monitor the impacts on value (velocity) and safety (quality and security).

{{ai-paradox}}

Introduction

GitHub Copilot is an AI-powered coding assistant that's been making waves since its official launch back in October 2021 . With a reported 50,000+ companies adopting the technology so far, the big questions still on everyone's minds are: Does it live up to the hype? Should it become the default for every single developer?

Well, instead of relying on hearsay, we ran a good old-fashioned experiment at our company. Here's what we found.

Background

To keep things fair and square, we split our team into two random cohorts — one armed with GitHub Copilot (around a third of our developers) and the other without. We made sure the cohorts were not biased in any way (e.g., that one wasn’t stacked exclusively with our most productive developers).

Over three months, we closely monitored various performance metrics, focusing on speed, throughput, and quality. Our goal? A clear, unbiased view of GitHub Copilot's impact.

Why these metrics? They're tangible and measurable, and they directly impact our deliverables. They also give us a holistic picture. We don’t want to gain speed if there’s a huge price to pay in quality. Finally, it would give us a good indication of areas we might need to strengthen in our practices or process if we want to fully go down the GitHub Copilot route.

{{cta}}

Results

The data was pretty revealing. The group using GitHub Copilot consistently outperformed the other cohort in terms of speed and throughput over the evaluation period (May-September 2023).

Let’s start with throughput.

Over the pilot period, the GitHub Copilot cohort gradually began to outpace the other cohort in terms of the sheer number of PRs.

Faros AI chart showing pull request merge rate for the GitHub Copilot pilot

Next up, I looked at speed.

I examined the Median Merge Time to see how quickly code was being merged into the codebase. The GitHub Copilot cohort’s code was consistently merged approximately 50% faster. The Copilot cohort improved relative to its previous performance and relative to the other cohort.

Faros AI chart comapring median merge time, with and without GitHub Copilot

The most important speed metric, though, is Lead Time to production. I wanted to make sure that the acceleration in development wasn’t being negated by longer time spent in subsequent stages like Code Review or QA.

It was great to see that Lead Time decreased by 55% for the PRs generated by the GitHub Copilot cohort (similar to GitHub’s own research), with most of the time savings generated in the development (“Time in Dev”) and code review (“First Review Time”) stages.

A Faros AI chart comparing lead time with cycle time breakdowns, with and without GitHub Copilot

The last dimension we analyzed was code quality and code security, where I looked at three metrics: Code Coverage, Code Smells, and Change Failure Rate.

  • Code Coverage improved, which didn’t surprise me. Copilot is very good at writing tests.
  • Code Smells increased slightly but were still beneath an acceptable threshold.
  • Change Failure Rate — the most important metric to me together with Lead Time — held steady.
Faros AI chart comparing code coverage, with and without GitHub Copilot

Analysis

But why did Copilot make such a noticeable difference? The engineers in our Copilot cohort said the boost is largely due to no longer starting from a blank page. It’s easier to edit an AI-driven suggestion than starting from scratch. You become an editor instead of a journalist. In addition, Copilot is great at writing unit tests quickly.

But, not all AI coding assistants are created equally, and the time savings can vary greatly depending on the tool used. For example, one of our clients conducted a bakeoff between two of the leading AI coding tools on the market, and one of the tools saved three hours more per developer per week compared to the other.

Cost-Benefit Analysis

Now, the juicy bit: Is the performance boost worth the cost? For us, the answer's leaning towards a solid "yes." A 55% improvement in lead time with no collateral damage to code quality is a phenomenal ROI. But, of course, every team's dynamics are different. If you're weighing the costs, consider not just the subscription fee but the potential long-term benefits in productivity and effects on code quality

Don't have budget for Copilot? Read our guide to getting approval for AI tools outside normal budgeting cycles.

Tips for Conducting Your Own Assessment

As I mentioned, lots of my peers want to create a similar analysis at their org. Today it’s GitHub Copilot, tomorrow it’ll be something else.

What made generating this comparison easy for me was three-fold:

  • I’m already tracking developer productivity metrics in Faros AI, based on the data it knits together from Jira, GitHub, Buildkite, Statuspage, and PagerDuty.
  • Unlike cookie-cutter metrics tools, Faros AI has a complete, flexible BI layer that made it easy for me to define my two cohorts and create a custom dashboard for this specific analysis. It took me just a few minutes to generate my GitHub Copilot analysis dashboard.
  • I could easily generate a holistic view of adoption, usage, velocity, and quality metrics based on the combination of system telemetry and developer surveys. This data helped me benchmark short-term impacts and identify emerging bottlenecks.

Today, Faros AI provides a complete value framework for AI evaluation and adoption, from the initial rollout to larger-scale deployments and long-term value optimization. This is much more sophisticated and comprehensive from the prototype I used last year.

Watch this five-minute tour of these dashboards:

Conclusion

So, back to our main question: Is GitHub Copilot worth the investment? Our data shouts a resounding "yes." But hey, tools are only as good as how we use them. It might be the perfect fit for some, while others might find alternative methods more suited to their workflow. Plus, if you have bottlenecks in your review, build, and test cycles, your efficiency gains may be reduced.

The next big question organizations are going to face is where to direct the developer productivity they’ve just unleashed. If you’re going to embrace GitHub Copilot, you need to have a plan. There’s no shortage of roadmap initiatives and technical debt for folks to sink their teeth into, but teams should be setting those priorities with intentionality.

The next big question organizations are going to face is where to direct the developer productivity they’ve just unleashed. There’s no shortage of roadmap initiatives and technical debt for folks to sink their teeth into, but teams should be setting those priorities with intentionality.

If you’re going to embrace GitHub Copilot, you need to have a plan. Our AI Copilot Evaluation solution provides comprehensive visibility into the impact of GitHub Copilot — from pilot to rollout to optimization — so engineering leaders can communicate value and ROI with confidence. Request a demo to get started.

Thomas Gerber

Thomas Gerber

Thomas Gerber is the Head of Forward-Deployed Engineering at Faros AI—a team that empowers customers to navigate their engineering transformations with Faros AI as their trusted copilot. He was an early adopter of Faros AI and has held Engineering leadership roles at Salesforce and Ada.

Connect
AI Is Everywhere. Impact Isn’t.
75% of engineers use AI tools—yet most organizations see no measurable performance gains.

Read the report to uncover what’s holding teams back—and how to fix it fast.
Discover the Engineering Productivity Handbook
How to build a high-impact program that drives real results.

What to measure and why it matters.

And the 5 critical practices that turn data into impact.
Want to learn more about Faros AI?

Fill out this form and an expert will reach out to schedule time to talk.

Loading calendar...
An illustration of a lighthouse in the sea

Thank you!

A Faros AI expert will reach out to schedule a time to talk.
P.S. If you don't see it within one business day, please check your spam folder.
Oops! Something went wrong while submitting the form.

More articles for you

Editor's Pick
DevProd
AI
12
MIN READ

Winning Over AI's Biggest Holdouts: How Engineering Leaders Can Increase AI Adoption in Senior Software Engineers

Explore the barriers to AI adoption in senior software engineers and how leaders can transform their AI skepticism into AI advocacy.
September 8, 2025
Editor's Pick
AI
News
7
MIN READ

Translating AI-powered Developer Velocity into Business Outcomes that Matter

Discover the three systemic barriers that undermine AI coding assistant impact and learn how top-performing enterprises are overcoming them.
August 6, 2025
Editor's Pick
News
AI
DevProd
4
MIN READ

Faros AI Hubble Release: Measure, Unblock, and Accelerate AI Engineering Impact

Explore the Faros AI Hubble release, featuring GAINS™, documentation insights, and a 100x faster event processing engine, built to turn AI engineering potential into measurable outcomes.
July 31, 2025

See what Faros AI can do for you!

Global enterprises trust Faros AI to accelerate their engineering operations. Give us 30 minutes of your time and see it for yourself.