Qualitative Measurement of Psychological Safety

What the survey can’t tell you

A few months back, I was preparing for a workshop with a client, and they shared, anonymised, their staff engagement survey data. We both sat on our respective screens, peering at a heat map of red, orange and the occasional shiny green rectangles, and I realised the data itself told me nothing useful for planning the session.

I could see from the data why they had identified psychological safety as an area for improvement, but beyond that the numbers themselves gave me no context, nuance or insight. So I started to ask questions – how long have the team been working together? What are their challenges? How do they communicate with each other? Really quickly, a picture started to take shape. The team had previously worked under a tyrannical manager who had since left, but in whose shadow they were still dwelling. Communication only happened in one-to-one conversations or private slack channels – the whole group chat was largely silent. Lack of clarity about what the team’s real purpose was meant that prioritisation was always contentious and conflict-ridden, and decisions were often undermined by other team members. From a relatively short conversation, I had a much stronger starting point for planning my session. I still didn’t know everything, of course, more always emerges in the workshop, but I had a much clearer picture of where to start.

Psychological safety is a felt phenomenon. It’s something we experience, in context, for reasons that are unique to us and the team. Yes, we might be able to measure headline figures about psychological safety, and we can even dig into some of its different ‘flavours’ – are people more likely to feel safe sharing an idea than asking a question for example? But we still don’t know why until we actually start talking to people.

Hands holding a survey clipboard — Photo by Kindel Media

The limitations of ‘hard numbers’

There’s an assumption baked into a lot of workplaces that numbers are rigour and words are just impressions. People are often apologetic about offering their observations and might caveat them with, “This is just anecdotal,” or “Obviously this is only my take on it.” Which is good practice, of course, to acknowledge the limitations of our perspective and the biases in our take on the world. But I notice we tend to be less inclined to do the same with quantitative survey data. We run the risk of treating an engagement survey completed by only half the team at 5pm on a Friday, while they’re rushing and vaguely resentful of yet another survey, as if it’s more truthful than a set of thoughtful observations and reflections from someone who has worked in the team for years. It’s not, it’s just different, and ultimately it’s just more countable.*

Terry Pratchett put it best: “The best research you can do is talk to people.” Talking to people, and listening to them, with openness and curiosity, gives you things numbers can’t. You might discover that no one really speaks up because they know only the boss’s opinion counts. This means it’s not necessary that anything ‘bad’ will happen as a result of speaking up, but that it’s futile, so why bother. That’s a very useful insight into how the team works. Perhaps someone else tells you they’ve always taken responsibility for speaking up even when it’s hard or unpopular, and when you press a little, it turns out what they actually mean is that they don’t think they’d be fired anyway. This tells us that for this person, the stakes aren’t as high as they might be for others, which is quite different to either the whole team feeling safe or that person being particularly ‘psychologically brave’. None of this gets captured in survey scores, but it’s invaluable for planning next steps for a team.

The point, I think, is that we’re often scared of qualitative measurement because, unlike surveys, which have become accepted as standard practice across organisations, we don’t always know how to do qualitative measurement of psychological safety – and whether it “counts”.

What qualitative measurement looks like in practice

There are rigorous ways we can go about formal qualitative measurement of psychological safety if that’s what we want to do. However, I think it’s also important to acknowledge that we’re already doing it. In every conversation we have with someone about how they’re getting on with colleagues, in every team meeting we’re part of and every interaction we observe, we’re gathering information, rich qualitative data, about the team. If our focus is psychological safety, we can find out a lot this way. All of this helps build our picture of what the organisation is like. It’s not a complete picture, but neither is a survey score.

“Trying to measure the traditional way with metrics is a suboptimal (at best) approach for complex properties or phenomena like safety or culture. A metric will provide you limited information and more often than not mislead you. A better way of ‘knowing’ is to aim for rich descriptions.” – Carsten Busch

While many organisations speak the language of ‘numbers’, and while an alarming statistic might open the door, research suggests it’s qualitative data – people’s voices and stories – that actually shifts attitudes. Numbers justify decisions; stories make us believe in them. In my experience, even with the most quant-driven folks, what really brings a data story to life is actual quotes – using what people have actually said to build a picture of what’s behind the numbers.

If we do want to more formally gather some qualitative data about psychological safety, what could we try? One of the major constraints in qualitative measurement or research is that it takes time – and to put it bluntly, if we get loads of qualitative data, loads of words, it’s going to take a lot of time to analyse well. But there are manageable ways we can integrate more qualitative exploration into our measurements. A few things come to mind, from the ‘easiest’ to the more involved:

If we’re going to do a psychological safety survey anyway, we can at least add an open text comment field – pop a box in to ask people why, or give them an opportunity to share some qualitative nuance behind their answers.
Observe what’s already happening. When we have the opportunity to join team meetings or join sessions, what do we notice? Who’s speaking? Who isn’t? What sorts of questions, ideas and concerns get raised? What happens to them? What kinds of opportunities and channels do people have to speak up? Do they have “think time”? Can they share in writing instead of verbally, or after the meeting as well?
Set up short interviews, being mindful that this will likely be less effective if we’re interviewing people we directly work with or whose career trajectories we have power over. We also need to think carefully from an ethical standpoint about what we will do with the data and make sure we’re clear with our interviewee – can we guarantee anonymity? But if we can navigate all that, we will find a lot out, through asking people questions like:
1. How safe do you feel speaking up in this team?
2. What might stop you from sharing your ideas?
3. Can you tell me about a time that you held something back? etc.

– As an aside on this one – rather than having to start from scratch with interviews, there are also often wonderful rich sources of interview data held by organisations in exit interviews – what have people shared in those that might relate to psychological safety? And if your org doesn’t do them, why not? They’re an ideal opportunity to gain really candid feedback about what it’s like to work for the organisation – from people who are leaving, so don’t have so much to lose.

Focus groups. Again, these take some setting up, and a neutral and skilled facilitator, but on a group phenomenon like psychological safety, bringing together small groups to talk about their experiences of psychological safety in the organisation could generate some really rich data.

The most useful thing you can do before trying to change the conditions in a team is understand what they actually are. A chart or heat map won’t give you that. Talking to people, and listening to what they say, just might. When we listen to understand, rather than to try to close a gap on a dashboard, it’s an act of curiosity and an act of care. And if we don’t care, we shouldn’t ask.

*As a slightly tongue in cheek aside, I’d like to point out the fist of five – where people hold up fingers to indicate how psychologically safe they feel – is considered a valid measure in some quarters. If that clears the bar, I’ll take our qualitative observations with some confidence.