Learning From Failure

Welcome to the psychological safety newsletter and thanks for subscribing. You are amazing. This week discusses dealing with learning from incidents, plus Elon Musk, share of voice, and Extreme Negative Feedback.

We have some new stickers too! These glow in the dark, so even if the lights go out, you can still Stay Safe!

A story of an incident and learning from failure (Long Read)

Indulge me, if you will. Reflecting on last week’s issue on guardrails and failure, I’m going to tell a story of an incident: what happened, what we did and how we felt, and how we laid the foundations for learning from it even before it happened. It’s rather techy and long, so I wouldn’t blame you if you wanted to skip to the good part – the “key points” section at the end!

This story is about an infrastructure related incident; I have many other stories, probably more interesting in the actual incident (at least one involves flying pterodactyls), but this story focusses on the troubleshooting process and incident management practice, and how we set ourselves up to best learn from an incident.

Around ten years ago, I was heading up the technology function at a large venue in the UK. We provided and supported technology operations for retail, bars, cafes, two public ice rinks, live 10,000+ attendee events, all alongside the usual office functions. We were a small but tight-knit and high performing team, and everyone on the team I considered an expert in their field. 

Complex Infrastructure

This was a complex network comprising multiple VLANs (these are “virtual” networks, separated logically, but not physically, from each other), many network cabinets containing multiple network switches, hundreds of wireless Access Points, and tens of thousands of devices, over a very large site. Some parts of the network were physically difficult to get to, sometimes through dark crawl spaces, or up on high gangways accessible only with safety harnesses. And you definitely couldn’t be scared of heights.

The infrastructure was heavily monitored. Some of the systems we ran and supported were safety-critical, especially on event days when there could be over 10,000 people in the venue. In case you’re wondering, security, access, fire, and CCTV systems were all on physically separate, airgapped (where possible) networks, and were unaffected. Our monitoring included multi-layer double-knocks, recursive checks, cascade detection and priority-based alerts. It was monitored from different parts of the network as well as externally to protect against partial failure that knocked out the monitoring systems themselves. The monitoring systems themselves were also monitored closely and any critical component had at least one redundant counterpart.

Incident Notification

One day, we’d finished work and I’d headed into town for dinner. Just as I sat down and took a sip of beer, an alert pinged on my phone, then another, then lots more. Then the serious alerts came in, and then it went silent. The automated on-call system kicked in, notifying engineers that stuff was going down. Getting the odd alert that something had failed was pretty normal, even day-to-day, and often didn’t require urgent attention. In large, complex systems where change is frequent, a small amount of failure is a constant reality. But this incident was impacting everything. Every device inside the network was down. My first thought was that there must have been a complete power outage or a major fire, but either way I had to get to site to find out, so left my beer (almost) untouched, and headed back.

I was greeted by lots of people telling me that nothing worked. All applications, systems, computers, phones, basically anything that required network access, was down. My team were shortly on site, and we convened as the response team. One person acted as the single point for communications, enabling us to communicate with the rest of the organisation whilst allowing the team to focus on problem solving. The alerting system had automatically created a Slack channel specifically for the incident, where we would communicate with each other, record actions, save photographs, and share funny gifs.

Diagnosis

Everything was down. And remember, at this point, we had no idea what was happening. It could have been switch failure, power, an application gone wild, virus, or maybe we were being attacked. It looked like a broadcast storm (which happens when a loop occurs in a layer 2 network), but that felt unlikely. We had mechanisms in place to protect and mitigate against that (spanning tree protocol – STP – and well architected routing amongst others). We’d tested our defences against network loops, and they’d worked every time. 

The cooling for the ice rinks was computer controlled and there was system redundancy in place (a backup machine), but there wasn’t redundancy for the network. Without the network, we couldn’t properly control the ice temperature across the rink. So the ice operations teams had to manually check the ice temperature at different points and manually control the system. It was like constantly rebalancing a huge system of radiators. Not only was this a lot of work, but there was an important ice hockey championship game the following day, and it would be very bad to have poor ice quality for that.

We got on the network to see what we could see. There were blinky lights everywhere, and any lights not blinking were just solid lit. Wireshark analysis showed exactly what looked like a broadcast storm, swamping the switch fabric on every switch, but how would we find the source of a potential loop, in a huge physical network, when there should be no possibility of a loop anyway? 

Our first thought, as experienced techs, was “What have we changed?” But nothing had changed on the network that day, other than the usual devices being connected or moved around. We’d planned for all sorts of outages, we had procedures, checklists, protocols, we carried out simulations and disaster recovery scenarios. But “things that have never happened before happen all the time” and we had to go back to basics.

After ruling out other possibilities, the only one left was a broadcast storm, as unlikely as it was, and the only way to stop it was to find the cause – most likely a cable patched back into the same switch – and disconnect it. But this was a huge place, and searching for it would take days or weeks. So we had to find a way to quickly narrow down where the cross-patch was. A team member suggested we go to each cabinet and power it down, taking a whole batch of switches off the network in one go – if the cross patch was on one of those, the network should recover, and we’d know we were close. We’d still need to find it, but we’d have narrowed down the search significantly. 

So, two people went off to power down each cabinet, working from east to west and staying in radio contact. It could be a long night, and we had no idea if it would work, but it was worth a go. 

All the while we were still communicating with senior stakeholders, security, operations, and other departments. By this time, it was late at night – there were no members of the public in the building and all the safety systems were still operational. Every time we shut down a cabinet, we’d wait to see the effect on the network, using Wireshark, live switch logs, and honestly, just looking at all the blinky lights. As a network engineer, you get good at seeing whether a network is healthy just by seeing the patterns of blinking lights in a room of switches.

Unfortunately, every switch we shut down had no effect. Until the final one. The old backstage cabinet. The engineer turned it off, reported back over the radio, and within seconds, the network looked healthier. It took 5 minutes for the traffic to die down, but we’d isolated it! As there were no critical systems passing through that cabinet, we left it turned off whilst remediating the impact and getting systems operational again. 

The “Root Cause”.

After further hunting, we found the causes: a misconfigured trunk port on a switch that allowed network traffic to traverse from one specific VLAN to another. Normally not even in use, the configuration wasn’t unusual enough to highlight a problem, and it wasn’t a security risk. The issue had been lurking for years, not misconfigured enough to become visible in day-to-day operations or be flagged in config management. This was a system that had drifted into failure.

What had triggered the incident? It turned out that evening, a well-intentioned cleaner had plugged an errant ethernet cable into the nearest socket, thinking it had been disconnected. In fact, the other end was plugged into a socket behind a fridge. Both those sockets connected to said cabinet, and to the same network switch. It was a particularly old switch, with an old implementation of Spanning Tree Protocol, and did not prevent a broadcast storm that traversed VLANs through a trunk port. As a result, within seconds of the loop being created, traffic began to build uncontrollably, first knocking out switches closer to the source, and then spreading rapidly through the network until it was overloaded. Once we knew the cause, it was as simple to fix as it had been to trigger.
 

“All ambiguity is resolved by actions of practitioners at the sharp end of the system.”
Dr. Richard I. Cook, How Complex Systems Fail
 

Key Points

– Everyone on the team was empowered and psychologically safe to come up with ideas to try. Without this, we’d have been troubleshooting for a lot longer, and may have made the situation worse before it got better. We’d never usually shut down vast swathes of the network, but necessity is the mother of invention not because it encourages ideas, but because it makes all ideas safe to try.

– Practitioners had to make difficult calls in tough conditions. It was explicit in the team that nobody would ever face consequences for making a “bad” call, particularly when under pressure. We had limited information available to us, intense time pressure, and everyone had already spent a full day at work.

– We had to be conscious of managing cognitive load. Concerned stakeholders were constantly (justifiably) asking what was happening and there was pressure to resolve the issue quickly. Other issues were also emerging and we couldn’t be sure if they were related, so it was difficult to know even what to investigate. It’s hard to focus on a single, highly complex problem under these circumstances – which is precisely why one person was allocated to external communications, sheltering the team who were working hard to resolve it.

– We recorded everything we could. Using the Slack Channel for the incident, we were able to look back with time stamps at what we discussed, actions we took, and how people were feeling and communicating at the time. This included silly side conversations, funny gifs, audio recordings, photos and videos. The intention behind this was threefold: 

1. To help us actually solve the problem in the moment, and make sure that all essential information was in one place.
2. To be able to evidence our actions and decisions to stakeholders after the case, should there be an audit or investigation.
3. To generate as much qualitative data as possible for an incident review and retrospective. The more high quality data we have, the better we were able to reflect and maximise the learning potential of the incident. The technological learning was the easy bit, but the “messy details” of the incident were the human factors, and provided the most valuable aspect of learning.

– There was never a “root cause”. The idea of a “root cause” is seductive. It suggests that if we only ask “why” enough times, we’ll find the smoking gun that caused the incident. Except that’s a fallacy. In reality, there were multiple causes –  the misconfigured switch port, the errant ethernet cable that should have been tidied away, a mismatched implementation of STP, and a myriad of other factors. Each cause was necessary but not sufficient. 



Note:

If you’re reading this and thinking “Why didn’t they do X? That would have solved it quickly!” or “They should have had the network set up like Z, then it wouldn’t have happened.” (A) This was over ten years ago, and I certainly remember some of the details wrongly. (B) Hindsight is 20/20, and we would have done some things differently in retrospect. (C) There existed multiple constraints not described here that influenced the choices and configurations we made. 

Further note. I’ve oversimplified certain aspects of networking, switching, VLANs and broadcast storms to aid readability. If you’re a network engineer reading this, please accept my sincere apologies!

This newsletter is sponsored by Conflux

Conflux is the leading business consultancy worldwide helping organizations to navigate fast flow in software. We help organizations to adopt and sustain proven, modern practices for delivering software rapidly and safely.

‘The Fearless Organization’ by Amy Edmondson is considered by many to be essential reading on the topic of psychological safety. In this article, Sophie Weston, Principal at Conflux, has put together some key takeaways from the book.

Psychological Safety In the Workplace:

Following on the incident theme from above, this is a fantastic podcast episode from On-Call Me Maybe: A podcast about DevOps, SRE, Observability, On-Call, and everything in between. In this episode, Nora Jones, CEO, and founder of Jeli.io, talks to Adriana and Ana about “Finding Humanity in Incidents“. I particularly love this point from Nora:

“…I will earnestly say incidents are kind of fun here. They’re never expected, but they are… we really enjoy working together and collaborating with them. And they’re just considered a regular part of work and not a very big deal.“

With Elon Musk’s purchase of Twitter, there’s almost too much to highlight in respect to psychological safety, from creating a “them and us” culture between Twitter and Tesla/SpaceX, to the way that layoffs were managed and communicated, or his insistence that everyone only works from the office. But one of the most interesting stories I came across about Musk this week is this old one from 2018, which alleges that Elon dislikes “the colour yellow, too many signs, or the warning beeps of forklift trucks“, which apparently have affected the Tesla factory’s safety for the worse. I’ll be diving further into Elon, Twitter, Tesla, management and culture in a later issue.


Here’s a great episode of the Secrets of Product Management podcast by Nils Davis, with Mehmet Baha, talking about psychological safety. Follow Nils on Twitter here.
 

I love Joan Cornellà’s nihilistic cartoons, and this one is true, though maybe not motivational! I’m really looking forward to including this in a conference slide deck.
 

Thanks so much to Nancy Staples for sharing this with me. How to handle “Extreme Negative Feedback”: a clip from That Mitchell and Webb Look. 
 

Here are some actual suggestions on how to receive and deliver feedback whilst building and maintaining psychological safety. I don’t suggest throwing a typewriter, though to be fair, I don’t specifically say not to, either.

Things to do and try:
Here’s a great piece from Liz Fosslein about the share of voice in a conversation. Ineffective and psychologically unsafe teams exhibit unequal shares of voice, with certain people dominating the conversation. In psychologically safe teams, the share of voice is more equal, and Liz presents a few tips to help encourage a balanced conversation.

1. Kick off with an equitable prompt. We know that people are more likely to engage in conversation if they’ve already spoken, even if it’s just to say “sandwich”. 
2. Break into duos. Many people feel safer speaking in smaller groups. I’d also suggest the 1-2-4-all practice for this. 
3. Acknowledge and appreciate perspectives. People are taking an interpersonal risk by sharing their idea, questions, concerns and mistakes, so make it a point to thank them for doing so.

This week’s poem:

The Power of Silence, by Ginnie Bale

My silence
holds truths
my voice is not
strong enough
to carry.

Mission statement

The mission of this newsletter is to make the world of work a better, safer, more inclusive and equitable place. We work towards this by sharing, as widely as possible, new research and opinion, case studies, news, practices and methods related to psychological safety. 

This includes topics such as inclusion, human factors, facilitation, safety critical environments, systems thinking, leadership and management, diversity and equity, mental health, organisational dynamics, culture, ethics, and more. 

A core goal is to amplify the voices of minoritised groups and those who are less represented in this field. This includes (but is not limited to) voices of people of colour, neurodiverse people, LGBTQ+ people, women in leadership and tech, and more.

Another goal is to share insights and best practices across different domains: for example, so that those of us in tech may learn from healthcare, and those of us in education can learn from aviation.

I believe knowledge is worthless unless it’s accessible. A key principle of this newsletter is to only share content that everyone can access. This excludes content behind paywalls, research papers that are not open-access, and proprietary tools that don’t open source their algorithms (preventing us from observing how systems make decisions about us). There may be some exceptions to this principle in practice, for example, where authors are able to share original texts of papers.

Whilst this statement will evolve this over time, the core principles of equity, amplification, and progressive & humanistic leadership shouldn’t change.