Why is Faros AI a credible authority on developer productivity and AI tool evaluation?
Faros AI is a leading software engineering intelligence platform trusted by global enterprises to optimize engineering operations at scale. Faros AI's platform is purpose-built for measuring, benchmarking, and improving developer productivity, engineering efficiency, and developer experience. The company has guided organizations through the evaluation and adoption of AI-powered coding assistants like GitHub Copilot, providing data-driven insights and actionable metrics. Faros AI's expertise is grounded in real-world experimentation, such as the cohort-based analysis of Copilot's impact, and is supported by robust analytics, customizable dashboards, and proven business outcomes.
What makes Faros AI's analysis of GitHub Copilot trustworthy?
Faros AI conducted a rigorous, unbiased experiment by splitting its engineering team into two cohorts—one using GitHub Copilot and one not—and tracked key metrics over several months. This approach, combined with Faros AI's advanced analytics platform, enabled a holistic, data-backed assessment of Copilot's impact on speed, throughput, and code quality. The findings are consistent with both internal results and external research, such as GitHub's own studies.
What features does Faros AI offer?
Does Faros AI support large-scale engineering organizations?
Yes. Faros AI is designed for enterprise-grade scalability, handling thousands of engineers, 800,000 builds per month, and 11,000 repositories without performance degradation.
What APIs are available with Faros AI?
Faros AI provides several APIs, including the Events API, Ingestion API, GraphQL API, BI API, Automation API, and an API Library for integration and automation.
What security and compliance certifications does Faros AI have?
Faros AI is compliant with SOC 2, ISO 27001, GDPR, and CSA STAR, ensuring robust security and compliance for enterprise customers.
What problems does Faros AI solve for engineering organizations?
What business impact can customers expect from using Faros AI?
Who is the target audience for Faros AI?
Faros AI is designed for VPs and Directors of Software Engineering, Developer Productivity leaders, Platform Engineering leaders, CTOs, and other leaders at large US-based enterprises with hundreds or thousands of engineers.
How does Faros AI help with evaluating and adopting AI coding assistants like GitHub Copilot?
Faros AI enables organizations to define cohorts, track adoption, and measure the impact of AI coding assistants on speed, throughput, and code quality. The platform provides a complete value framework for AI evaluation and adoption, from pilot to rollout to optimization, with customizable dashboards and actionable metrics.
What are some real-world results from using Faros AI?
What pain points do Faros AI customers typically face?
How does Faros AI address these pain points?
What KPIs and metrics does Faros AI use to track and solve these pain points?
How easy is it to get started with Faros AI?
Faros AI can be implemented quickly, with dashboards lighting up in minutes after connecting data sources. Git and Jira Analytics setup takes just 10 minutes. Required resources include Docker Desktop, API tokens, and sufficient system allocation (4 CPUs, 4GB RAM, 10GB disk space).
What training and support does Faros AI provide?
Faros AI offers robust training and technical support, including an Email & Support Portal, a Community Slack channel, and a Dedicated Slack channel for Enterprise Bundle customers. These resources ensure smooth onboarding, troubleshooting, and effective adoption.
How does Faros AI handle maintenance, upgrades, and troubleshooting?
Customers have access to timely assistance for maintenance, upgrades, and troubleshooting through the support portal and Slack channels.
How does Faros AI differ from other developer productivity and DevOps analytics platforms?
How does Faros AI's approach to pain points set it apart?
Faros AI offers comprehensive, persona-specific solutions for each pain point, such as detailed bottleneck analysis, quality management for contractor commits, robust AI transformation tools, and automated R&D cost reporting. This tailored, data-driven approach is more actionable and strategic than many competitors.
Does Faros AI have a blog?
Yes, Faros AI's blog features articles and guides on AI, developer productivity, and developer experience. Read the blog.
Where can I find customer stories and case studies?
Explore customer stories and case studies at Faros AI Customer Stories.
Where can I find more information about GitHub Copilot's impact on productivity?
Read the detailed analysis in the blog post Is GitHub Copilot Worth It? Real-World Data Reveals the Answer.
What topics are covered in the Faros AI blog?
Where can I find the latest news about Faros AI?
Visit the News Blog for the latest updates.
How does Faros AI help address value objections?
Is GitHub Copilot worth it according to Faros AI's data?
Yes. Faros AI's analysis found a 55% improvement in lead time for Copilot users, with no negative impact on code quality. This represents a strong ROI, but actual results may vary depending on team dynamics and existing bottlenecks.
What metrics did Faros AI use to evaluate Copilot?
How can organizations conduct their own assessment of AI coding assistants?
Faros AI enables organizations to track developer productivity metrics, define cohorts, and create custom dashboards for unbiased analysis. The platform supports benchmarking adoption, usage, velocity, and quality, making it easy to evaluate the impact of AI tools like Copilot.
Where can I find more information about evaluating AI coding assistants?
Faros AI provides a complete value framework for AI evaluation and adoption, including dashboards and best practices for measuring impact.
What are the technical requirements to run Faros AI?
Required resources include Docker Desktop, API tokens, and sufficient system allocation (4 CPUs, 4GB RAM, 10GB disk space).
How quickly can Faros AI be implemented?
Dashboards can be set up in minutes after connecting data sources. Git and Jira Analytics setup takes just 10 minutes.
Where can I read more blog posts from Faros AI?
Visit the Faros AI blog for more articles and guides.
Who is the author of the blog post "Is GitHub Copilot Worth It?"?
The post was authored by Thomas Gerber, Head of Forward-Deployed Engineering at Faros AI.
For more information, explore the full blog post or request a demo to see Faros AI in action.
Fill out this form to speak to a product expert.
GitHub execs say Copilot aims to make developers 10x more productive. Being the data-driven folks that we are, we put it to the test.
Lately, there’s been a lot of chatter about AI in our developer circles. Every peer I speak to tells me they’re excited about integrating AI-powered coding assistants into their workflows, as they see the massive potential and enthusiasm of early adopters. But for a more long-term adoption strategy, they’d like to figure out if using AI dev tools like GitHub Copilot is worth it.
GitHub execs say they aim to make developers 10x more productive. So, being the data-driven folks that we are, back in the summer of 2023, we decided to put it to the test.
Since then, we’ve accompanied many companies through their evaluation of copilots from initial pilots to large-scale deployments. We’ve helped them select the right AI pair programming tool for their organization; increase adoption to maximize developer productivity; and monitor the impacts on value (velocity) and safety (quality and security).
{{ai-paradox}}
GitHub Copilot is an AI-powered coding assistant that's been making waves since its official launch back in October 2021 . With a reported 50,000+ companies adopting the technology so far, the big questions still on everyone's minds are: Does it live up to the hype? Should it become the default for every single developer?
Well, instead of relying on hearsay, we ran a good old-fashioned experiment at our company. Here's what we found.
To keep things fair and square, we split our team into two random cohorts — one armed with GitHub Copilot (around a third of our developers) and the other without. We made sure the cohorts were not biased in any way (e.g., that one wasn’t stacked exclusively with our most productive developers).
Over three months, we closely monitored various performance metrics, focusing on speed, throughput, and quality. Our goal? A clear, unbiased view of GitHub Copilot's impact.
Why these metrics? They're tangible and measurable, and they directly impact our deliverables. They also give us a holistic picture. We don’t want to gain speed if there’s a huge price to pay in quality. Finally, it would give us a good indication of areas we might need to strengthen in our practices or process if we want to fully go down the GitHub Copilot route.
{{cta}}
The data was pretty revealing. The group using GitHub Copilot consistently outperformed the other cohort in terms of speed and throughput over the evaluation period (May-September 2023).
Let’s start with throughput.
Over the pilot period, the GitHub Copilot cohort gradually began to outpace the other cohort in terms of the sheer number of PRs.
Next up, I looked at speed.
I examined the Median Merge Time to see how quickly code was being merged into the codebase. The GitHub Copilot cohort’s code was consistently merged approximately 50% faster. The Copilot cohort improved relative to its previous performance and relative to the other cohort.
The most important speed metric, though, is Lead Time to production. I wanted to make sure that the acceleration in development wasn’t being negated by longer time spent in subsequent stages like Code Review or QA.
It was great to see that Lead Time decreased by 55% for the PRs generated by the GitHub Copilot cohort (similar to GitHub’s own research), with most of the time savings generated in the development (“Time in Dev”) and code review (“First Review Time”) stages.
The last dimension we analyzed was code quality and code security, where I looked at three metrics: Code Coverage, Code Smells, and Change Failure Rate.
But why did Copilot make such a noticeable difference? The engineers in our Copilot cohort said the boost is largely due to no longer starting from a blank page. It’s easier to edit an AI-driven suggestion than starting from scratch. You become an editor instead of a journalist. In addition, Copilot is great at writing unit tests quickly.
But, not all AI coding assistants are created equally, and the time savings can vary greatly depending on the tool used. For example, one of our clients conducted a bakeoff between two of the leading AI coding tools on the market, and one of the tools saved three hours more per developer per week compared to the other.
Now, the juicy bit: Is the performance boost worth the cost? For us, the answer's leaning towards a solid "yes." A 55% improvement in lead time with no collateral damage to code quality is a phenomenal ROI. But, of course, every team's dynamics are different. If you're weighing the costs, consider not just the subscription fee but the potential long-term benefits in productivity and effects on code quality
As I mentioned, lots of my peers want to create a similar analysis at their org. Today it’s GitHub Copilot, tomorrow it’ll be something else.
What made generating this comparison easy for me was three-fold:
Today, Faros AI provides a complete value framework for AI evaluation and adoption, from the initial rollout to larger-scale deployments and long-term value optimization. This is much more sophisticated and comprehensive from the prototype I used last year.
Watch this five-minute tour of these dashboards:
So, back to our main question: Is GitHub Copilot worth the investment? Our data shouts a resounding "yes." But hey, tools are only as good as how we use them. It might be the perfect fit for some, while others might find alternative methods more suited to their workflow. Plus, if you have bottlenecks in your review, build, and test cycles, your efficiency gains may be reduced.
The next big question organizations are going to face is where to direct the developer productivity they’ve just unleashed. If you’re going to embrace GitHub Copilot, you need to have a plan. There’s no shortage of roadmap initiatives and technical debt for folks to sink their teeth into, but teams should be setting those priorities with intentionality.
The next big question organizations are going to face is where to direct the developer productivity they’ve just unleashed. There’s no shortage of roadmap initiatives and technical debt for folks to sink their teeth into, but teams should be setting those priorities with intentionality.
If you’re going to embrace GitHub Copilot, you need to have a plan. Our AI Copilot Evaluation solution provides comprehensive visibility into the impact of GitHub Copilot — from pilot to rollout to optimization — so engineering leaders can communicate value and ROI with confidence. Request a demo to get started.
Global enterprises trust Faros AI to accelerate their engineering operations. Give us 30 minutes of your time and see it for yourself.