How to Get a Holistic Understanding of the Developer Experience

The Prevalent Approach to DevEx Surveys

Employee surveys are a staple for organizations aiming to gauge workforce satisfaction, identify areas for improvement, and foster a positive workplace culture. About 80% of companies conduct engagement surveys according to the Society for Human Resource Management (S.H.R. M), an increase from 62% in 2010.

Done right, surveys serve as invaluable tools for gathering feedback directly from employees, providing insights into their perspectives on various aspects of the workplace, such as organizational culture, leadership, communication, processes, and job satisfaction.

In engineering organizations, surveys can be leveraged to capture developers’ perceptions of how their team delivers, insights into points of friction in the software delivery process, and feedback on what can be improved at the team or organizational level.

A growing number of engineering organizations are practicing “Agile Health” methodologies:

• Regularly running pulse-check surveys to catch emerging issues through early signals
• Monitoring the impact of operational or technical changes
• Tracking changes and trends over time
• Staying attuned to evolving employee needs and concerns.

Employee surveys contribute to fostering a culture of open communication, demonstrating to employees that their opinions are valued and considered. They can help foster a sense of ownership and commitment among the workforce, ultimately leading to increased productivity, employee retention, and the creation of a positive and supportive workplace culture.

But they also create expectations.

Employees, who took the time to voice their opinions and sentiments, now expect the organization to take their POV into account and to see some things change as a result.

Are Existing Employee Surveys Enough?

For many organizations, acting on employee surveys is challenging, due to problems in the survey itself or the partial picture it paints.

Let’s start with problems in the survey itself.

Common problems with the survey itself

In dozens of conversations with engineering leaders, a few common issues were surfaced:

• Surveys are not conducted frequently enough, in which case the information can be stale or biased by recent events.
• Surveys are too high level (e.g., at the organizational level and not the team level).
• Surveys provide inaccurate results due to the way questions are worded.

The other key challenge is that surveys only provide part of the picture.

Challenges acting on partial data

Surveys are essential to capturing the voice of developers — their perceptions and feelings. However, this feedback is highly contextual and can be easily misinterpreted if not complemented by data about engineering systems and processes (activity- or process-based metrics).

Here are some of the issues senior engineering leaders we’ve talked to face when dealing with survey data:

• Looking at survey results in aggregate when the situation varies considerably across teams. As an example, poor survey results on velocity could be due to slow build processes for one team and lots of dependencies for another. Investing purely on improving the build process won’t help the latter team.
• Fighting yesterday’s battles. Because they typically don’t run continuously, surveys can be lagging (and sometimes leading) indicators of issues, and can be heavily influenced by a specific recent event (e.g. fire drills around severe incidents, reorgs, etc.) — a.k.a. recency bias. It is essential to put the survey results in context.
• Not knowing, not asking. You don’t get answers to questions you don’t ask. Leaders find it hard to validate whether the survey questions are providing good insights into what is really going on and the most important issues and opportunities the company is facing.
• Understanding areas of friction and potential areas to improve. While surveys typically point in the general direction of an issue (e.g., concerns around quality), system metrics would help in understanding the contributing factors to this issue (e.g., poor code coverage).
• Keeping track of impact. As mentioned earlier, developers expect things to change and improve after sharing their thoughts in a survey. At the organizational level or team level, there currently isn’t a way to measure the current state and demonstrate the progress made based on the developers’ feedback.

Issues engineering leaders face when dealing with developer survey data on its own

How Can Surveys Become More Impactful?

Considering these issues, it would appear that augmenting survey results with system data, collected from engineering systems, could significantly help.

Powerful insights come when blending qualitative insights from surveys with data and metrics from systems, processes, and workflows, an approach that Google, for one, has used very effectively with its People Analytics, with an average of 90% participation rate in surveys.

Matthew Runkle, Director of Cloud Engineering at SmartBear, a Faros customer, shared an example. “We’ve always had this vision of correlating developer sentiment with the concrete process and outcome metrics we’re measuring on Faros to understand how the two are linked. For instance, one of the frequent pieces of feedback we got from our surveys was that developers wanted better tests. It was helpful to look at system data and correlate a team’s relative investment in product quality with its members’ satisfaction in this regard.”

Here’s another example. Below is a chart that correlates survey responses on “goals and alignment” to a team’s ratio of unplanned work. It helps leaders understand whether lower scores on alignment correlate to higher levels of unplanned work. If corroborated, managers can take corrective action faster, by implementing measures to limit or address the amount of unplanned work that floods into the team.

Correlating survey responses on goals and alignment with a team’s unplanned work ratio sheds light on developer feedback

A Novel Approach to Blended Visibility

To give engineering organizations the insights they need to monitor and improve the developer experience, we are delighted to introduce our new Developer Experience module.

What is a module? Modules are prebuilt analytics libraries — inclusive of all the data sources, metrics, dashboards, widgets, and customizations you need — that run on top of the Faros AI platform.

Infused with domain expertise, benchmarks, and best practices, modules provide rapid insight immediately upon connecting to your data sources. From there, you can build upon the module’s foundation by creating your own custom metrics, views, and reports.

The Developer Experience module centralizes developer satisfaction survey data in one place and intersects the sentiment data from employee responses with telemetry-based data from engineering operations.

Faros AI provides novel blended visibility into the complete developer experience

This novel blended visibility into the complete developer experience provides actionable insights that allow engineering leaders and their HR partners to take corrective measures faster and observe their impact on engagement, retention, and operational excellence over time.

Engineering leaders and their HR partners are now able to ingest survey data from any source into the Faros AI platform and overlay engineering data and metrics on the survey responses around alignment and goals, developer productivity, quality, speed and agility, and more.

Like everything in Faros, survey data can be analyzed over time and sliced and diced by team or other dimensions of choice.

Watch a 2-minute demo of the Developer Experience module

How It Works

Because every organization is unique and each team is different, the Developer Experience module is designed to be completely configurable:

1. Pull data from any survey tool you work with.
2. Configure your survey themes or categories based on what makes sense for your teams.
3. Select the system metrics you want to overlay on survey data, based on your team and organizational goals.

To get you up and running quickly, you can also leverage pre-packaged survey templates from Faros, that include categories and metrics based on industry benchmarks and best practices. Our Lighthouse AI engine will be running behind the scenes to provide you with actionable insights to help you analyze and act upon survey insights.

Want to see it in action? Request a demo of Faros AI today.