In this guest series, we’ve had the opportunity to introduce the challenges of measuring developer productivity, to uncover that productivity delivers for the organization. We then explored how software development safety and velocity don’t need to be at odds or create undue risk.

Still, in modern development and deployment environments, it seems like human oversight alone will never be able to get teams of developers ahead of the rate of change.

To reach our destination at high velocity, all hands on deck should not only row faster but pull in the same direction—all while aligning their efforts with a regular cadence.

The practice of AI-augmented DevOps can optimize the pace of software delivery, by measuring work outputs and correlating signals with the intentions and goals of developers and teams.

A history of misaligned incentives and goals

Remember 10–15 years ago when pundits were promoting the concept of “bi-modal IT”—in which software delivery responsibilities would be segregated into two software delivery groups working at different paces?

Such pace layering represented the reality on the ground for many large enterprises. There would be one ‘Innovation Team’ tasked with prototyping new functionality and pushing the interface edge—totally disconnected from everyone else struggling with waterfall development dependencies, DBA requests, draconian change controls, and quarterly or annual release windows.

As analysts we relentlessly mocked bi-modal IT on several occasions. So let’s not allow the advent of AI-based development tooling create another such pace separation and throw off the cadence of our organization.

Software 2.0: Developing with AI

In this prescient 2017 article, Andrej Karpathy categorizes the whole of software development as we knew it—human developers writing code without AI assistance—as Software 1.0.

Thus, Software 2.0 would represent the next kind of development, one where much of the work of building software is handled by intricate AI models providing coding assistance and integration help, while human “developers” aren’t coding so much anymore. Instead, the ‘2.0 developer’ identifies desirable behaviors for the system, by curating and tagging the massive machine learning datasets needed to train the AI.

Weighting parameters for AI models, instead of coding application logic, would be a new paradigm for development. However, most organizations are likely not going to be able to completely remove developer knowledge and human oversight from the logical loop.

Take Air Canada, they recently had a court order to make good on a refund offer suggested to customers by their AI-powered chatbot. Nobody was sure how the chatbot’s large language model came up with the offer, but LLMs are notorious for occasionally ‘hallucinating’ an answer that will seem plausible or pleasing to end users.

What we really need is an AI that augments the developer’s capabilities for understanding how the application they are building will fit within both integration and business contexts, so they can get into the flow of development by eliminating tedious or repetitive tasks.

Can DevEx surveys improve developer experience?

Developer surveys can be incredibly valuable in determining the quality of developer experience (or DevEx). Thought leaders at ACM recently put out an extensive study boiling down DevEx into three logical dimensions of Flow State, Feedback Loops, and Cognitive Load.

All three dimensions point to developers’ natural desire to have engineering systems that allow them to move forward with fewer constraints, delays, and distractions. However, results of a DevEx survey are only as good as the timing of the survey, the exact wording of the questions, and the readiness of survey participants to provide accurate responses.

Time is the most constrained resource for developers. Time to finish each sprint, make that pull request, prepare a dataset, fix a hot Sev1 issue. Time to learn new skills, explore new technologies, and still have a life away from work.

No surprise, developers are unlikely to complete surveys. Further, many survey questions can deliver ambiguous conclusions from responses.

For instance, a survey might ask: “What is your satisfaction level with our current testing platform?” The organization’s average response could be 3 (on a 1–5 scale).

Digging deeper into that average satisfaction level, it turns out a development team doesn’t really engage with the test platform too much other than running sets of prescribed checks at each release window. If cursory tests don’t fail builds very often, they might like the platform well enough, and rate it a 4 or 5.

Meanwhile, an Operations team rates the testing platform a 1 or 2, because they are dealing with resulting production failures!

Continuously measure DevEx at the source

To improve, we need to marry less cumbersome survey touchpoints with real development metrics that allow advanced algorithms to determine developer sentiment and point out morale issues.

If sentiment questions are introduced subtly, perhaps as a single thumbs-up-or-down during work, that would seem much less daunting than an extensive survey. But still, what does a thumbs-up really mean?Non-obvious data points from the DevOps toolchain and non-verbal clues from developer actions would provide better indicators of causal patterns that represent poor DevEx, as it is concentrated down to the team and individual level. built a module specifically for developer experience, providing a prebuilt, curated set of data for analyzing the most relevant metrics, KPI benchmarks, activities, and events alongside survey data. For development managers and executives, this provides a great starting point for understanding the developer experience in light of system telemetry and tool usage.

Tuning a DevOps toolchain with AI provides a much faster correlation of data related to developers productively staying in a flow state, getting faster feedback loops, and having enough data and the right tools on hand to reduce cognitive load.

The correlations between surveys and telemetry increase the likelihood that future investments will deliver the desired improvements. Then, the team can set targets for DevEx success levels and identify paths forward for improvement from there, whether the development activity is coding, or tuning AI models to augment development.

Tracking toward outcomes at Coursera

Coursera grew rapidly over the last decade into one of the world’s leading online learning resources. While the engineering team was busy modernizing their application estate to a more open-source-based and scalable microservices architecture, the company’s culture was also heavily concerned with improving DevEx.

They established a dedicated developer productivity team to hone in on the DORA and SPACE frameworks, using platform engineering to enable new developer onboarding, end-to-end testing, and faster release cycles.

After experimenting with creating their own error-prone dashboards using Sumo Logic (a SecOps log management tool not intended for development teams), Coursera selected Faros AI to understand activity happening within several DevEx-related tools and platforms at once, from repositories to incident management to their CI/CD pipeline activity and OKR tracking.

"For measuring developer productivity, it’s important to not look at just one signal but rather have a holistic view that looks at developer activity but also other important metrics like developer satisfaction and the efficiency of flow of information in the organization," said Mustafa Furniturewala, SVP of Engineering at Coursera.

The Intellyx Take

To survive in a software-driven world, we must constantly transform and change paradigms, or fall behind. How can we keep pace, when the rate of change is too fast for humans to comprehend?

With AI-augmented DevOps, organizations can dynamically observe developer workload and tasks, and reorder work around multiple toolsets to identify the optimal times and task assignments for more productive team design meetings, coding, and testing.

Even the best developers can leverage enhanced intelligence and timely guidance, to make the whole team better than the sum of its parts.

©2024 Intellyx B.V. Intellyx retains editorial control of this document. At the time of writing, Faros.ai is an Intellyx client. No AI was used in the writing of this story.