Log4Shell: The Quintessential Example of Five Unrealistic Expectations of Engineering Leaders
Adrianna Gugel
·
Chief Product Officer & Co-founder
September 5, 2024
About this blog
  • Engineering leaders are expected to provide immediate answers during crises (like Log4Shell) even when it’s unrealistic to fully understand the situation right away.
  • They face immense pressure to deliver solutions and timelines quickly, even when they lack all the necessary information, placing an unrealistic burden on them during high-stakes situations.
  • They must effectively represent both technical and business concerns to stakeholders, despite being removed from daily technical work.
  • Engineering leaders act as a shield for the business, managing crises, customer concerns, and internal pressures while making critical decisions.
  • They are expected to prioritize their teams’ morale and well-being, often at the expense of their own, while maintaining confidence and stability to protect both the team and the business.

Last week Aaron provided an engineering leader’s perspective on what it’s like to be expected to represent the company’s codebase.  But as an evergreen partner and stakeholder to my engineering counterparts, in this blog I want to talk about everyone else’s perspective of the engineering leader. To be frank: it’s not just unfair, it’s close to insane. While I could regale you with my observations of 1,000 papercuts worth of engineering leaders' pain, there is a single chain of events that encapsulates it all.

The day the internet was set ablaze 

December 9, 2021 will forever be etched in the memories of software engineering leaders worldwide. That afternoon at 2:25pm EST, the Log4Shell vulnerability was publicly disclosed on Twitter, setting off a chain of events that many in the tech industry would later describe as one of the most significant cybersecurity crises in history. This exact moment is burned into my mind because despite being a product management leader, I found myself coordinating my previous company’s incident response. The scale, impact, and urgency of the situation placed an unprecedented level of pressure on engineering leaders, perfectly exemplifying the often unrealistic expectations placed on them by their organizations. Throughout my career I had always worked closely with engineering leadership but this was the first time I truly had unfiltered visibility into the demands placed on these leaders as well as the realities that they deal with, day-in and day-out.

#1 - The assumption of omnipresent knowledge

In the hours and days following the disclosure of the Log4Shell vulnerability, it became immediately apparent that engineering leaders were expected to have all the answers—instantly. The moment the news broke, every corner of the company exploded with questions.  "How exposed are we?" "Which systems are vulnerable?" "What is our plan of action?" “What should I tell my customer?”. It was mid afternoon on a Friday a few weeks before Christmas… and our core engineering team, including our CTO and SVP of Engineering, were in Europe where their weekend had started hours earlier. The engineers on call quickly understood the complexity and gravity of the incident and insisted they needed to hear from their boss before making any statements. Their bosses said they needed to hear from their boss. After what seemed like eternity—but in reality was less than 30min—our CEO got in touch with our SVP of Engineering. The expectation was he could provide immediate, directionally correct information off the top of his head.

The reality, however, is that no one person can instantly understand, evaluate, and respond with the nuance required to represent the full scope of such an impact across a complex system. Yet, engineering leaders are expected to provide answers at any moment, regardless of the time of day or where they are. It is a near-impossible task, but failure to meet this expectation can lead to frustration and panic among other executives and stakeholders. During the Log4Shell crisis, I had a front row seat to witness the intense pressure on our engineering leaders to deliver instant knowledge, even as they worked tirelessly to gather the necessary information from their teams who were now working around the clock.

#2 - The demand for immediate solutions

When engineering leaders don’t have the answers, they are expected to know exactly how to find them and to provide a timeline—often on the spot—for when those answers will be available. This expectation was evident during the Log4Shell crisis, where our engineering team was scrambling to assess the extent of our exposure while simultaneously devising a plan to mitigate the risk. Despite the complexity and novelty of the situation, our SVP of Engineering was repeatedly asked to estimate when we would have a complete understanding of our vulnerabilities, which customers were impacted, and when a fix would be in place. 

Sadly and regrettably, in my temporary role as incident response coordinator, I was often the person amplifying the demands and reminding engineering leadership of the “date for a date” timelines they had previously provided that the entire company was dependent on for customer communications, given I was working in parallel with the office of the CISO, legal council, comms team, and Chief Customer Officer. Across every level, from the C-suite to employees on the front-line, folks expected the SVP of Engineering to predict the future, providing estimates that were not only accurate but also aligned with the business’ need for reassurance, and mitigate potential legal risk. This pressure to deliver immediate solutions, often without all the necessary information, is a common yet unrealistic burden placed on engineering leaders.

#3 - The mandate of complete representation

Another significant challenge faced by engineering leaders is the expectation that they can accurately represent the intricate details of their team’s work at any level, from technical deep dives to high-level business discussions. During the Log4Shell incident, our CTO and SVP of Engineering were involved in every major decision-making meeting. They were expected to not only understand the technical nuances of the situation but also to communicate these complexities in a way that made sense to non-technical stakeholders, including the CEO and Chief Customer Officer.

This expectation is particularly daunting because the engineering leaders are often removed from the day-to-day technical work carried out by their teams. Yet, they are required to be the voice of truth, delivering information that is both accurate and digestible for the broader organization. The decisions made based on this information are often seen by every part of the business as final and irrevocable, leaving little room for error.

#4 - The burden of exhaustive protection

If all that wasn’t bad enough, perhaps the most challenging aspect of being an engineering leader is the role they play as a shield for the rest of the business. During the height of the Log4Shell incident, our customers were understandably anxious, demanding immediate answers and assurances. The responsibility to communicate answers to our largest and most important customers fell squarely on the shoulders of our SVP of Engineering as he was rushed onto customer calls with little warning. As the top of the technical totem pole, he had to field hard-hitting questions while also avoiding potential legal landmines and favorably representing the company in the process. This position as the final line of defense is not only stressful but also isolating. Engineering leaders are expected to protect the business from the fallout of a crisis while simultaneously managing the internal chaos that such events inevitably bring. They must maintain composure, provide clear communication, and make critical decisions, all while knowing that their actions will be scrutinized long after the situation has passed.

#5 The pressure to put the whole team first 

Amidst all these demands, there is an unspoken expectation that engineering leaders must sacrifice their own well-being to protect and uplift their teams, simultaneously bending over backwards to support business stakeholders. During the Log4Shell crisis, I saw firsthand how our SVP of Engineering and CTO acted as buffers, absorbing the external pressures and shielding their teams from the panic and urgency that permeated the organization. They were the ones working at all hours while they instituted structured shifts for their engineers to prevent burnout and combat plummeting morale. They were the ones that took on the business risk—at risk to their personal careers—to skip one of the patches, predicting correctly an even newer patch would be released in 24-48hrs that would be better to focus on, so that their teams could get a couple nights of good quality sleep.

The pressure to maintain team morale is immense. Engineering leaders are expected to keep their teams motivated, calm, and driven, even when they themselves are under extreme stress. They must project confidence and stability, often hiding their own anxieties and fears to avoid unsettling the team and business. This self-sacrifice, though rarely acknowledged, is a significant and unrealistic expectation placed on engineering leaders.

The bottom line: don’t throw stones, help

The Log4Shell vulnerability was a defining moment for many engineering leaders, highlighting the often unrealistic expectations placed upon them by their stakeholders. Our engineering leaders are burdened with immense responsibility. No one can know everything at all times, make zero mistakes, communicate complexity at all levels of granularity, always know the right thing to say, protect their teams around the clock, and exude an aura of confidence that steadies the whole organization. That’s bonkers. Yet it’s just a day in the life of our engineering leaders. It’s high time we acknowledge these expectations and provide our engineering leaders with tangible support to empower them to succeed, not just in moments of extreme crisis, but every day.

Helping to ease these burdens for engineering leaders is important to us at Flux. Built by engineers for engineering leaders and their teams, we’ve designed our platform with their unique challenges in mind. For instance, when a vulnerable library is discovered, our platform almost instantly identifies the repos in which it is used, as well as how it is used. This empowers engineering leaders to quickly assess their exposure, take decisive action to mitigate risks, and confidently communicate their progress to stakeholders—allowing them to focus on what matters most: protecting customers, their teams, and the business. 

What do you think? What other unrealistic expectations do you think organizations have of their engineering leaders? Reach out on LinkedIn and let me know what I’ve missed!

Adrianna Gugel
Chief Product Officer & Co-founder
About
Adrianna

Adrianna Gugel is the CPO and Co-Founder of Flux. With 15+ years of product management experience and a proven history of launching new products and strategic partnerships, Adrianna’s unique blend of business acumen and technical understanding allows Flux to bridge the gap between ideas and achievable results.

About Flux
Flux empowers engineering leaders to triage, interrogate, and understand their team's codebase. Connect with us to learn more about what Flux can do for you, and stay in Flux with our latest info, resources, and blog posts.