AI Productivity Checklist for Engineering Teams Using ChatGPT and Coding Copilots

Github Copilot has been activated by more than one million developers in over 20,000 organizations, generating a staggering three billion accepted lines of code.

So it’s likely your team is using it.

While your developers may be thrilled with the shortcuts and time savings, as their manager do you know the net impact AI is having on your KPIs for team productivity, quality, and lead time?

Do you know how to have a conversation with your team about using AI for a net positive outcome?

We've created a checklist on how to have those conversations and what you should be tracking.

But, first...

Make sure you know how engineers are using AI in coding

Several enterprises have been monitoring the impact of rolling out new tools like Github Copilot and developers' unofficial adoption of chatGPT.

An initial study of enterprise usage shows the potential for tremendous time savings:

• Copilot Code Autocomplete is widely adopted for writing boilerplate code, skeleton code, code comments, and tests. It can save developers up to 20% coding time.
• Copilot Code Suggestions are deemed less valuable and helped in only 25% of the cases. For this use case, developers prefer chatGPT over Copilot to create code snippets from specs, translate from one programming language to another, or as a tutor for debugging. Estimated savings are over 1hr per day per developer.

But fascinatingly, Lead Time to Production has yet to improve despite personal productivity gains. Even with faster dev times, the time spent in code review, merging, and testing is still long.

That's where the AI Productivity Checklist comes in: To ensure AI helps your team realize overall productivity improvements in speed and velocity.

The AI Productivity Checklist

Given that you want to see net gains in lead time and productivity for the team, below is a checklist to guide your conversations with the team and ensure you monitor important KPIs for adverse effects.

The checklist has two parts — questions to ask your developers and metrics you should track as a manager.

Combined, the checklist will help create awareness around the impacts of introducing sub-optimal code generated by AI. You'll be able to ensure the efficiency gains for the individual aren’t dwarfed by the negative impacts on the team, your customers, and the business.

Here you go:

1) Questions to ask your developers:

☑ Do you have good test coverage for generated code?

☑ Do you have a way to assess the code quality of generated code?

☑ Are you able to identify potential security and compliance issues introduced by generated code?

☑ Is documentation for generated code clear and sufficient?

2) Metrics to track as a manager:

☑ Code Review Cycle Time: Are code reviews taking longer?

☑ QA Cycle Time: Is there an uptick in bugs and incidents? Is more time being spent on rework?

☑ Change Failure Rate: Are failures increasing?

☑ MTTR: Is incident resolution getting slower?

☑ Lead Time: Has overall lead time to production gotten faster or slower?

Need metrics?

Metrics that analyze the impact of new technology and practices on engineering processes and performance have become business-critical.

Faros AI specializes in visibility and analytics across any environment and stack. We know all about non-standard tool implementations, highly customized pipelines, homegrown systems, and proprietary data sources.

Talk to us about our extensible, customizable software engineering intelligence platform.