But not without risks...

Despite its skyrocketing popularity and undeniable benefits, just like other tools leveraging generative AI technology, GitHub Copilot and similar tools such as Tabnine, Amazon CodeWhisperer, Replit Ghostwriter or FauxPilot, among others, have their limitations and should be rolled out with care.

For one, it might generate code that is not optimal or even sometimes downright buggy. Or it could run fine but not produce the expected output or follow product requirements. Code quality can vary and security or compliance issues can be introduced. Developers leveraging the tool excessively might not understand the code enough to debug it and answer questions coming up during code reviews, lengthening the code review process, or making code harder to maintain. Generated code could also potentially have copyright/plagiarism issues.

Gain visibility into your engineering operations

To get an accurate view of the impact, benefits and unintended consequences of rolling out tools like GitHub Copilot in your organization, you need full visibility across your entire software development lifecycle. Fortunately you can leverage existing frameworks and tools to do just that.

DORA metrics will help you keep an eye on BOTH velocity and quality. Monitoring Lead Time is a much better way than Ticket Cycle Time to measure actual improvements in what matters: delivering code to your customers in production. And an increase in Change Failure Rate is a red flag that there might be an issue with auto generated code. Engineering Productivity should be carefully analyzed and not reduced to the number of tickets closed in a sprint: pull request merge rates, planned vs unplanned work and team health among others should all be taken into account.

At Faros AI, we work with some of the largest organizations in the world, like Salesforce, Box and Coursera. Many of them are rolling out tools like GitHub Copilot with a mix of excitement and concerns. With teams of thousands or even tens of thousands of engineers, the stakes are high.

Faros AI provides a “single-pane” view across a software engineering team’s work, goals and velocity. You can connect key data sources to the platform (Jira, GitHub and many others) and leverage out-of-the-box modules such as our DORA metrics solution or customize and build your own analytics.

It is the perfect tool for these large organizations to monitor the impact of rolling out tools like Github Copilot and we ran an initial study with a subset of our customers to get some early signals.

Study results

For this first study we proceeded in two steps: we conducted interviews of developers using Github Copilot, then used Faros AI’s DORA module to explore metrics for teams using the tool more heavily to see if key delivery metrics were impacted.

The first learning from this study is that some organizations did not really have a good sense of how much tools like Github Copilot had actually penetrated their organizations. It had grown organically and somewhat below the radar. Some groups were heavy users, while others were not using them at all.

In terms of actual usage, the key way Copilot is used today is for code autocomplete. Most developers we talked to praised that functionality and were heavy users. Key use cases mentioned were writing boilerplate code, skeleton code, code comments and tests. All these amount to micro-savings, basically saving keystrokes, but accumulate throughout the day, and developers we talked to cited productivity gains upwards of 20% on coding work from this alone.

In terms of code suggestions, opinions varied. Some developers complained about them being too noisy/chatty, although they noted recent improvements from what they had experienced a few months ago. Hit rate was deemed low (~25%), especially on more complex code, but could sometimes be helpful as a starting point. For this task, another tool was actually preferred: chatGPT itself. Several developers we talked to used it actually even more than Copilot. Common examples given included generating code snippets from specs, translating from one programming language to another (for programmers starting on new languages), as an alternative to writing a script for tasks such as search and replace to write similar pieces of code, and as a tutor for debugging. Some developers cited time savings of over 1h per day leveraging chatGPT in this way.

A key theme throughout these interviews was that developers don’t really trust these solutions yet, describing them as “a junior assistant that is very energetic but often wrong”. All of them indicated using them on small chunks of code to be able to verify, as errors were expected and would be harder to find if too much code was generated at once. The main concern expressed was around introducing quality issues in edge case scenarios. While we talked mostly to senior developers, concerns were expressed around potential impact of these tools in the hands of more novice programmers who might lack the experience to spot such issues.

The next step was looking at the data. Once information was collected on which teams were using the tools more heavily, it was easy using Faros to conduct an A/B analysis and a before/after comparison, as the DORA metrics can be filtered down to the team level and charted over time.

When doing so, we observed, at this point in time and with a limited sample, that overall velocity for teams using copilot was not significantly different from those not using it, and had not changed that much before and after using it. Diving deeper was even more interesting and started to explain why: as Faros provides a breakdown of the lead time to change steps, it was clear that often the biggest bottlenecks were actually in the First Review Time, Merge Time and Time in QA parts of the cycle. In other words, potential gains in dev time were dwarfed by time spent in other stages of the pipeline, and as a result lead time to production, which is what really mattered, barely moved. This in itself was a powerful insight and several of our customers implemented PR review policy changes as a result.

Our second study will be looking at additional aspects, including quality and productivity metrics and we will be looking forward to sharing those results with you soon!