AI Productivity Checklist for Engineering Teams Using ChatGPT and Coding Copilots
A simple checklist can help engineering managers achieve net positive gains in team productivity, lead time, and quality.
Thierry Donneau-Golencer
Browse chapters
Share
July 17, 2023
AI Productivity Checklist for Engineering Teams Using ChatGPT and Coding Copilots
Github Copilot has been activated by more than one million developers in over 20,000 organizations, generating a staggering three billion accepted lines of code.
So it’s likely your team is using it.
While your developers may be thrilled with the shortcuts and time savings, as their manager do you know the net impact AI is having on your KPIs for team productivity, quality, and lead time?
Do you know how to have a conversation with your team about using AI for a net positive outcome?
We've created a checklist on how to have those conversations and what you should be tracking.
But, first...
Make sure you know how engineers are using AI in coding
Several enterprises have been monitoring the impact of rolling out new tools like Github Copilot and developers' unofficial adoption of chatGPT.
An initial study of enterprise usage shows the potential for tremendous time savings:
- Copilot Code Autocomplete is widely adopted for writing boilerplate code, skeleton code, code comments, and tests. It can save developers up to 20% coding time.
- Copilot Code Suggestions are deemed less valuable and helped in only 25% of the cases. For this use case, developers prefer chatGPT over Copilot to create code snippets from specs, translate from one programming language to another, or as a tutor for debugging. Estimated savings are over 1hr per day per developer.
But fascinatingly, Lead Time to Production has yet to improve despite personal productivity gains. Even with faster dev times, the time spent in code review, merging, and testing is still long.
That's where the AI Productivity Checklist comes in: To ensure AI helps your team realize overall productivity improvements in speed and velocity.
The AI Productivity Checklist
Given that you want to see net gains in lead time and productivity for the team, below is a checklist to guide your conversations with the team and ensure you monitor important KPIs for adverse effects.
The checklist has two parts — questions to ask your developers and metrics you should track as a manager.
Combined, the checklist will help create awareness around the impacts of introducing sub-optimal code generated by AI. You'll be able to ensure the efficiency gains for the individual aren’t dwarfed by the negative impacts on the team, your customers, and the business.
Here you go:
1) Questions to ask your developers:
☑ Do you have good test coverage for generated code?
☑ Do you have a way to assess the code quality of generated code?
☑ Are you able to identify potential security and compliance issues introduced by generated code?
☑ Is documentation for generated code clear and sufficient?
2) Metrics to track as a manager:
☑ Code Review Cycle Time: Are code reviews taking longer?
☑ QA Cycle Time: Is there an uptick in bugs and incidents? Is more time being spent on rework?
☑ Change Failure Rate: Are failures increasing?
☑ MTTR: Is incident resolution getting slower?
☑ Lead Time: Has overall lead time to production gotten faster or slower?
Need metrics?
Metrics that analyze the impact of new technology and practices on engineering processes and performance have become business-critical.
Faros AI specializes in visibility and analytics across any environment and stack. We know all about non-standard tool implementations, highly customized pipelines, homegrown systems, and proprietary data sources.
Talk to us about our extensible, customizable software engineering intelligence platform.
More articles for you
Editor's pick
Ben Cochran, VP of Developer Enablement at Autodesk, sat down with Vitaly Gordon, Co-founder and CEO of Faros AI, at the San Francisco Engineering Leadership Council annual event, for a conversation about Autodesk’s developer productivity case study and data-driven approach to engineering.
Editor's pick
GitHub Copilot is one of the fastest adopted tools in the history of software development. One year after its release, over 1 million developers and 20,000 organizations are using the tool. But how to measure its impact on your engineering operations? Read on..
Editor's pick
I am delighted to announce that we have raised $20M in Series-A financing, led by one of the most successful VC investors in Silicon Valley and the first board member in iconic companies like Gitlab, Splunk, Fastly, and Bill.com, David Hornik of Lobby Capital...
Get started with Faros AI today!
Start your free trial now and get the full picture in minutes.
No credit card required.