The Engineering Productivity Handbook

Step 2: Identify Your Operating Model

Below are typical operating models that indicate the additional lenses through which you’ll want to analyze your metrics.

Step 3: Identify Your Engineering Culture

Corporate and engineering culture will also influence the smallest unit of measurement, whether the individual or the team and how the metrics are applied. The table below illustrates a few examples.

Step 3: Expand

Don’t lose sight of the KPIs that act as checks and balances within your initiative. According to the authors of the SPACE framework, “productivity cannot be reduced to a single dimension… Only by examining a constellation of metrics in tension can we understand and influence developer productivity.”

That’s why, at this stage, you’ll want to expand to data sources related to quality and reliability to prevent over-focusing on velocity. You’ve likely identified several of these metrics in Chapter 1, and you should be ready to generate them now that the basics are in place.

• Collect data to form a holistic view of quality, reliability, and security. Pre-production sources provide a view into deployments, bugs, code smells, vulnerabilities, code and test coverage, test stability, and flakiness. Post-production sources are the systems of record for customer issues and tickets, production incidents, and system outages.
• Collect service-level metrics to correlate “code production" metrics with the stability and efficiency metrics of supporting systems and platforms. This includes metrics like CI reliability, build cache hit ratio, development environment usage statistics, and more.

‍

Step 4: Align

The C-Suite expects the engineering department, like every other corporate function, to demonstrate its impact on corporate objectives. To that end, the next step is collecting business results data in support of quarterly and annual planning and OKR tracking.

Some of that information may be readily available in the task management systems you’ve connected, which will allow you to measure say/do ratios and on-time delivery of product roadmap.

That said, there is a lot more opportunity to illuminate costs and impacts by intersecting engineering data with business metrics pertaining to product usage, customer satisfaction, and financial performance.

• Collect data from systems that record targeted business results, like cloud costs, uptime stats, customer satisfaction, and efficiency stats for business processes supported by engineering.
• For engineering teams that operate in lower layers of the stack, consider proxy metrics that represent value to those who consume their output.

‍

The following table summarizes the data sources to connect at each of stage of your program, with examples of specific tools.

In the next sections, we’ll provide guidance on the practicalities of collecting data from different data sources, including:

• Standard and custom data sources
• Sources for reporting structure
• Sources for data attribution and ownership
• Sources that require schema extension

Standard and Custom Data Sources

During the data collection phase, the data will be stored in a canonical data model, where it is normalized into a single, connected data set that can be queried efficiently across the entire organization.

Inevitably, your data sources will include a combination of vendor tools and homegrown sources. Faros AI handles these sources as follows:

• Popular vendor tools. Faros AI leverages optimized open-source connectors for common task management tools like Jira and Asana, source control tools like GitHub and GitLab, and static code analysis tools like SonarQube and Codacy. Connectors typically pull historical data from the source once and then pull dataset changes on a cadence. When possible, we recommend also using Webhooks to push data to Faros AI in real-time.
• Custom homegrown tools and sources (including spreadsheets). Two easy options exist to ingest data from custom sources.
• ~Connector Development Kit: Faros AI leverages an open-source framework called Airbyte, which has connectors to common sources like S3 and Postgres to ease development. You’ll simply have to map the data from the custom source to the schema.
• ~Events CLI: For builds and deployments, it is easy to instrument the tool and send events to Faros AI. This is accomplished by adding a line of code to a preexisting CI/CD script.

Sources for Reporting Structure

Engineering productivity data is often examined based on reporting structure, with leaders tracking metrics for the sub-org and teams they manage.

For example, if available, Faros AI ingests this data from an HR data source like Workday, which will update us when people join or leave the company or upon a major re-org.  Within Faros AI, the reporting structure is used for rollups and drill-downs because it represents how your teams, teams of teams, and groups are organized. It also enables comparisons, outlier identification, and team-tailored insights.

Sources for Data Attribution and Ownership

Beyond the formal reporting hierarchy, your metrics will rely on accurate data attribution, which rarely has an evergreen source of truth. That’s why Faros AI infers the mapping between teams, apps, and repos, and auto-selects the best attribution method based on our experience and domain expertise.

As with any other metric behavior in Faros AI, you can change the default team attribution method to utilize the people whose data contributed to the work or the specific app or repo that the data touches.

Extending the Schema for Special Use Cases

The canonical data model is designed to represent all the relevant SDLC data in a cohesive, interconnected manner. To use it, you simply need to plug in your data. For example, Faros AI’s standard schema will be sufficient in 99% of cases. In rare edge cases where a certain data set does not fit perfectly within the schema, the schema can be extended by leveraging tags, custom columns, and custom metrics.

Multiple Data Sources for the Same Data

It is quite common for different teams or sub-orgs to use different tools. For example, one team might manage their tasks in Jira, another uses Asana, and a third uses GitLab. Another example is a company with multiple instances of the same tool.

Normalization is very simple in these cases. The Faros AI connectors normalize the data upon ingestion, automatically mapping corresponding data types to the right place in our canonical schema.

Same Tool, Different Workflows

Another common scenario is projects within a single tool, like Jira, that use different workflows, as expressed by statuses. Faros AI automatically deals with status transitions and provides the desired breakdowns based on the level of analysis:

• Each team's particular workflow will be represented in its metrics, so team members can understand their bottlenecks, learn, and affect change where needed.
• At the leadership level, where we’re zoomed out to team-of-teams or much larger groups, metrics will be abstracted to the common statuses of To Do, In Progress, and Done. This is sufficient to see the bottom line metrics leaders care about, like task cycle time and amount of work in progress.

Same Tool, Different Usage

Every team in your organization might be using Jira, but they’re using it very differently. Normalization is required to report effectively across this variance in tool usage.

The Faros AI approach is to be compatible with how people work today, especially at the very beginning of your program. To that end, the data normalization can be handled in a couple of ways:

1. By building conditions into the chart queries. For example, let’s imagine you want to look at all high-priority unassigned issues. One team may use P0 and P1, while another uses Critical and High. A custom query can bake these different definitions into the chart.  
2. By using the platform’s data transform capabilities. For example, one group uses epics to track initiatives, while another group uses tags on tasks, and a third group uses a custom issue type. Faros AI transforms the data to the initiative portion of our schema, so you can then query all the initiatives in a single way.

At some point, if the maintenance of the queries or transforms becomes too complex and error-prone, Faros AI recommends introducing a few standard options. You don’t force everyone to comply to the same behavior, rather to select one of a handful of approved ways of doing things. This should cover the majority of team preferences while keeping the in-tool configurations manageable.

Different Goals for Different Teams

Let’s face it: Good and bad are relative.

Consider one product under active development and another product that is in maintenance mode. While you may want to measure the same things for these teams — for example, throughput — they will have different definitions of good. “Good” is also relative to a baseline, and their starting points may be wildly different.

The Faros AI approach is to make it easy for every role to easily understand how teams are performing relative to contextual goals.

• Teams can customize their thresholds for great, good, medium, and bad. These custom thresholds will be utilized for their personalized dashboards featuring team-level metrics and insights.
• Leaders will get a bird’s-eye view at the organizational level that takes all the personalized thresholds into account and visually identifies hotspots. It will also call out areas of improvement or decline.

Note: Popular frameworks like DORA publish annual benchmarks, but the way the metric is defined might not be applicable to how you work. For example, Deployment Frequency measures how often you deploy code changes to production. If your organization has a major product release four times a year, strict adherence to that definition won’t give you the insight you seek. In this example, Faros AI recommends measuring deployment frequency to your pre-prod environments, with the caveat noted and internally understood.

Basic Data Validation

As Faros AI begins to ingest and normalize data, it will identify gaps in data collection and mapping. Through troubleshooting and cleanup, you can address and fix these errors. In addition, charts can be tweaked, for example, to use median instead of average.

During this phase, you may also discover places where processes are not being followed internally and need to address the issue with the relevant teams.

• Faros AI produces a report that shows data gaps, including boards without an owner and teams without data.
• Faros AI highlights anomalies and outliers, primarily found in human-curated data, like tasks that have been open for hundreds of days or tasks that moved directly from ToDo to Done.

Once your data has been validated, you’ve paved the way to the next stage of analyzing it.

After you’ve gotten the initial read from the data (and triggered some data cleanup, if warranted), it’s time to go deeper. In this section, we’ll provide guidance on how to:

• Quickly gauge the current state with benchmarks
• Counteract data overload with visualization
• Counteract data overload with AI
• Validate and contextualize findings
• Perform ad-hoc, impromptu analysis

Quickly Gauging the Current State with Benchmarks

Organizations like to reference industry benchmarks to gain a better perspective on their comparative strengths and weaknesses. These popular benchmarks are often born of extensive research that ties high performance to better financial performance, which can help:

• Set priorities: Where do we start?
• Set goals: What should we aim for?
• Justify investments: How do we incrementally become world-class?

Faros AI incorporates many software engineering industry benchmarks and best practices for velocity, quality, reliability, predictability, security, and organizational composition to help you quickly evaluate your situation. These metrics are provided at every level of analysis.

Benchmarks include but are not limited to the DORA 4 (lead time, deployment frequency, change failure rate, and mean time to restore), cycle times, velocity, say/do ratios, planned vs. unplanned work, AI coding assistant impact, and staffing ratios. Use these benchmarks to understand your relative standing and surface performance areas to focus on.

Counteracting Data Overload with Visualization

Senior leaders typically benefit from effective summarization by a single powerful visualization. For example, the Faros AI scorecard consolidates the top 5–10 KPIs the organization has chosen as the most impactful leading and lagging metrics. It highlights which areas of the organization are performing well and which require attention. From there, it’s easy to drill down into sub-orgs and more detailed views, if required.

Counteracting Data Overload with AI

A lot of data can be both a blessing and a curse. Engineering leaders don’t have the time or capacity to spend hours sifting through dashboards and digging into the details. Like any leader, they prefer the key takeaways and interesting insights delivered to them. AI is like a group of expert data analysts to augment your staff and provide a daily or weekly briefing.

For example, Faros AI utilizes AI layer upon layer, from the simple to the complex, to shorten the time between identifying an issue and taking corrective action. Lighthouse AI, its built-in AI engine, applies statistical analysis and machine learning to pinpoint problem areas in specific sub-orgs, repos, applications, or stages of the SDLC. It automates very difficult and time-consuming analysis if done manually.

Lighthouse AI uses a proprietary machine-learning workflow to analyze key engineering metrics against 250+ factors that can impact them. It then presents personalized team-tailored insights into what’s inhibiting or improving performance. It also leverages LLMs to summarize and explain the findings and recommend solutions clearly.

Here are the different ways Lighthouse AI works.

Validating and Contextualizing Insights

Now that you’ve baselined your data, you’ve identified hotspots, bottlenecks, or areas of friction. You’ve also begun to uncover their contributing factors — with or without AI. Next, it’s time to validate and contextualize the information.

No one knows your business like your developers and managers, so any statistical finding should be validated by the people involved. Below is a tried-and-true approach.  

1. Measure: Gather the data points indicative of an issue.
2. Understand: Deepen your understanding of the issue by adding context.
3. ~Talk to your team. Speak to your team members to get their perspective. Learn what they think is happening and why. This is where you tie the telemetry to all your institutional knowledge and business context.
4. ~Consult developer survey data. Developer surveys should ask developers what’s causing the most friction in their daily work. If you’re not yet importing your qualitative developer survey data into Faros to correlate it with quantitative data, consider doing that now. If not possible, manually review recent developer surveys to see what they say about the issue.
5. ~Confirm the priority. How important and consequential is this issue? Arrive at a consensus on whether this issue is worth pursuing now.
6. Decide: Formulate a hypothesis and customize visibility to accompany your analysis.  
7. ~Formulate a hypothesis, time-box it, set a goal, and consider the different levers you have to take action.
8. ~Set up granular metrics for the workflow or process in question, if required. For example, if the issue pertains to incident resolution, configure a flow in Faros AI that will allow you to examine every step of the process across tools and interactions like Detect > Create > Triage > Resolve > Restore. This will help you measure the impact of your changes throughout its cycles.
9. ~Create new charts or dashboards that will enable you to examine the factors at play. For example, if your hypothesis is that incident resolution times are influenced by cross-geo delays, customize a chart to incorporate the geographical location of the team members involved.
10. ~Configure alerts or notifications that will be helpful in tracking or addressing the issue.
11. Act.
12. ~Enact the change. Designate someone to be focused on the issue and the change management process. They should enable the teams, monitor the impact, and share the learnings.
13. ~Monitor the impact. After acting, keep your eyes on the metrics. Is the change helping you achieve your goal? Are there unintended consequences you need to address?

Performing Ad-Hoc, Impromptu Analysis

Looking at data frequently prompts additional questions, like:

• Why was March such an outlier month?
• Did the new process we implemented in June impact our performance?
• What does last year’s holiday season tell us about this year’s planning?

Now that you have data at your fingertips, you can answer those questions faster.

Last quarter, our SVP asked me for information and walked away, expecting it to take a couple of days. I was back in five minutes with all the data.
‍
— Matt Runkle, Sr Director of Engineering, SmartBear

Faros AI recommends your organization designates at least one data analyst for the engineering productivity initiative. This individual should be deeply familiar with your business, priorities, and ways of working. During the implementation and customization phase, Faros AI will train them on the Faros AI data model so that they develop the expertise to produce meaningful metrics and dashboards efficiently and independently.

In addition, platform end-users can leverage wizards and natural language prompting to find answers. When relevant, Faros AI will suggest existing charts that meet the bill and guide the user to apply custom filters, aggregations, and visualizations to get the precise chart they want. When a custom metric is in order, a chat interface helps users learn how to build a custom query, including which tables to use, key information about the data, and a variety of tips and tricks. It can also simply build the query for them.

Delivery Cadence Metrics

Delivery is most commonly managed through agile methodologies, with work segmented into short, iterative cycles such as sprints or development iterations. Regular stand-ups, sprint planning, and retros help ensure projects remain on track and deliverables meet quality standards. Monthly product and tech reviews provide a structured forum for assessing progress, addressing issues, and aligning priorities.

Outcomes Cadence Metrics

Organizations operate on a quarterly cadence to set, review, and adjust their objectives and key results (OKRs). This process aligns teams with strategic goals, measures progress, and ensures accountability. Regular check-ins and quarterly reviews facilitate continuous alignment and goal achievement. Cross-functional QBRs provide a comprehensive evaluation of the quarter’s performance against the set OKRs.

In our QBRs, metrics in Faros AI help map engineering’s work to business value. The excellence with which our engineering teams deliver and operate can be tied directly to the lagging results, like helping the business acquire, retain, upsell, or increase customer satisfaction.

— Shai Peretz, SVP Engineering, Riskified

Budget Cadence Metrics

Engineering organizations begin their annual strategic planning towards the end of the fiscal year. This process involves forecasting financial needs, allocating resources, and setting financial goals for the upcoming year. Budget reviews and adjustments are made throughout the year to ensure alignment with evolving priorities and market conditions.

Talent Cadence Metrics

Talent reviews and performance evaluations are conducted twice a year to assess employee performance, identify high-potential talent, plan for career development, and identify skill gaps. Compensation reviews, which are typically implemented in March or April, ensure that employee remuneration reflects performance and market standards, thus attracting and retaining top talent.

Incorporating objective data in these processes reduces subjectivity, speeds up preparation, and increases confidence in decision-making. It also ensures feedback is evidence-based, leading to clear and actionable development goals.