The latest market correction has been a long time coming. For over a decade now, low interest rates and easy access to capital fueled a period of unprincipled growth in Silicon Valley. “Cash flow positive” seemed to have become a distant memory of a bygone era. But as Edward Abbey famously put it, growth for the sake of growth is the ideology of the cancer cell. He was referring to the erosion of wilderness at the hands of uncontrolled urban expansion in his beloved Arizona, but the analogy applies just as well to companies.

Software engineering organizations in particular, experienced rapid growth over this past decade, disproportionate to other functions. Headcount has always been the primary lever for engineering leaders to substantially increase output. The naive belief being that more engineers will mean more software delivered faster. Every problem has the same magical cure — hire more people! Need more features? Hire more engineers. Engineers are complaining? Hire more infrastructure people. Things are moving slowly? Hire more engineering managers, product managers, project managers, recruiters to fill these positions, and so on. It’s time to grow up.

The truth is, adding more headcount to an organization is an expensive band-aid fix that substantially increases the complexity of the system and often slows it down. The Mythical Man-Month talks about exactly this phenomenon. More engineers means more teams, more meetings, more dependencies, more resources spent on interviewing and onboarding, more process, more analysis-paralysis, more tech debt, more feature creep, and most debilitatingly, less focus on the truly important. Austen Allred, CEO of the Bloom Institute of Technology, calls it the “death spiral of bullshit”.

On the flip side, headcount has also been the primary lever for engineering leaders when it comes time to cut costs, and we are witnessing the fall-out now.

So why have engineering leaders only had such a blunt tool at their disposal? The answer lies in the lack of visibility into software engineering operations. If you were to ask a sales or marketing leader about their metrics – funnel conversion rates, channel efficiency, sales cycle lengths, forecasted revenues — the answers would be ready. In contrast, ask engineering leaders for a breakdown of monthly spend, forecasts for the next month, or the impact in terms of dollars of an unresolved incident — the answers would require weeks of effort, gathering data from different sources, digging through logs, writing ad hoc scripts, and more. The ironic result is that for an organization teeming with analytical minds, decisions are often based on incomplete data, and guesswork or intuition is a frequent substitute. The cobbler’s children are the worst shod indeed.

Lack of visibility into software engineering operations

It’s not the fault of the engineering leader. They’ve never been held accountable. Most other functions don’t know enough to challenge the almighty engineering leader. An engineering lead could go through an entire hour of content in a board meeting without being asked any questions. But just because they haven’t been held accountable thus far, doesn’t mean they shouldn’t do their jobs better.

So why is visibility into software engineering operations so poor? There are two main reasons for this. First, it's just plain hard. Engineering data sources are incredibly fragmented and silo-ed. Most organizations use dozens of systems to manage their engineering processes — from task and incident management to continuous integration and delivery, to cloud operations, budgeting, procurement, HR, and more. For the most part, none of these systems talk to each other or to any central system, yet many of the questions that engineering organizations need to answer involve querying data across these different sources.

The second reason is fear — fear of alienating a volatile and rare resource — the software engineer. Software engineering is a creative craft. Certain kinds of operational metrics can be viewed as “big brotherly”, and would stifle the creativity that leads to innovation.

But the result of tip-toeing around is that most software engineering organizations today are flying blind. Engineering leaders have only one way to grow — hire people, and only one way to cut costs — fire people. They have a poor grasp of their operations with bloated teams — many overwhelmed with dependencies, others with tech debt — and not enough visibility to provide the support that teams need when they need it. Constant reorgs are a typical symptom of this dysfunction, and very little of substance actually gets done between the upheavals.

It’s time to grow up! In the interests of keeping the peace, engineering leaders have forgotten that while organizations are made of people, they need to function like well-oiled machines. Especially in these times. Being an ostrich and sticking your head in the sand may be a good short term way to avoid “upsetting” engineers with “metrics”, but it’s a terrible way to know what the business actually needs, what a team’s pain points are, and how to best help them. Constant reorgs and layoffs do not make for happy engineers.

Engineering teams can do more with less

So where do we go from here? The good news is that while visibility into engineering operations is hard due to the fragmentation and diversity of data sources, software teams don't need to build the necessary instrumentation themselves. There are now platforms and tooling out there to provide this much-needed visibility out-of-the-box. Simultaneously industry benchmarks and frameworks such as DORA and SPACE have emerged and gained traction, enabling teams to get a sense of how they’re doing and the room for improvement.

So now, envision a world where engineering organizations had all their operational data at their fingertips. The velocity and quality of software delivery could actually be measured. Bottlenecks in processes could be uncovered and continuously improved on. Leaders would know exactly how much time and resources are being spent on major initiatives, and whether these align with overall business priorities. Teams could be supported with the resources they need, when they need it — a junior-heavy team, flooded with tech debt could be supported with a couple of senior engineers and more time to pay down tech debt and get back to treading water again.

More generally, growth could be methodical — driven by need and informed by data — what areas actually need investment, what areas would really move the needle. Course correction could be timely and incremental, avoiding big bang reorgs and layoffs. The focus on velocity and quality would usher in the right practices and technical capabilities that would allow engineering organizations to do a lot more with a lot less.

The ongoing technology revolution is changing our world more rapidly than ever before. It has given us the internet, smartphones, artificial intelligence, and will give us, in the near future, self-driving cars, private space exploration, and more. The technology industry employs some of the brightest minds of our generation, and yet we are nowhere close to realizing their full potential because of the immaturity of our engineering practices. Engineering leaders are winging it and rely too much on instinct. It is time to grow up.

This is why we built Faros AI

Faros AI is the connected engineering operations platform that gives engineering leaders a single-pane view of their entire software development lifecycle. Leading enterprises such as Box, Coursera, GoFundMe, and more are leveraging Faros AI to accelerate their EngOps journey.

‍Request a demo/trial, and we’ll be happy to set you up.

(An abridged version of this post was originally published earlier on Forbes under the title: It's Time for Software Engineering to Grow Up)