Choosing a psychometric tool

Measuring psychological safety? Here are some tips on choosing the right tool for the job.

As awareness of the importance of psychological safety in the workplace increases, there is a corresponding increase in the number of psychometric tools, applications and services that attempt to measure psychological safety. Some try to derive a direct metric, for example by asking “How psychologically safe do you feel on a scale of 1-5?” whilst others may indirectly measure it via proxy behaviours or patterns that occur more frequently in teams with high psychological safety, such as asking “How often do members of your team admit and discuss mistakes?”

Full disclosure here – in the psychsafety.com team, we don’t sell a proprietary tool or software application to measure psychological safety, and in fact we don’t really believe in it. Every organisation is different and we believe the most appropriate approach depends on the multitude of cultures, org structures, languages, and domains in the organisation, as well as the measurement aims and budget. Instead of providing a tool, we work with organisations to help them design bespoke research methods and surveys. Not only does this mean they get the actionable feedback they need, it also has the added benefit of empowering organisations to do the work themselves, rather than forever rely on us. 

However, if you are going to use a psychometric tool to measure psychological safety, what factors should you consider? How do you decide which one to use? How do you know if it’s effective, safe, reliable and secure? 

If you do opt for an off-the-shelf tool, you’ll want to make sure it is transparent, culturally sensitive, accessible to all, and secure. It should also be valid and reliable, upholding both data rights and human dignity. Let’s dive into these key considerations to take into account before using or buying a tool to measure psychological safety, or any other social phenomenon for that matter. 

In choosing a psychometric tool, we should ensure that:

• We understand the methods and algorithms the tool uses. Avoid any tool that doesn’t allow you to see how it works and how it makes decisions about people or categorises them. Whilst commercial pressures may encourage “black boxing” software to avoid competitors seeing inside it, it also means that the people who use it can’t see inside either. Personally, I believe if a system makes decisions that affect you, you should be allowed to see how those decisions are reached. Look for “Open Source” software options – these allow everyone to see what they are doing and how they uses any algorithms.
• It’s usable and accessible for everyone. An effective and equitable tool should allow people with different languages, contexts, neurodiversities and accessibility needs (such as visual impairment) to access and use it as fully as everyone else. Can a visually impaired person access the system? Is it designed with the needs of people with dyslexia and dyscalculia in mind? Is it only available in English? 
• It’s secure. The data in the system should be strongly secured against breaches and leaks. Data should be encrypted at rest and in transit, only kept for as long as necessary, and proper AAA access controls should determine who can see the data. You can ask vendors to show you their information security accreditations such as ISO 27001, the international standard for information security.
• People retain ownership of their own data. People should be able to remain anonymous if they wish to, and measures must be in place to ensure their data cannot be “de-anonymised” through examining metadata such as team, department, language or location. Anyone should be able to request that their data be removed from the system at any time, including after they’ve left the organisation. Data rights are human rights. In Europe, most of this is covered by GDPR, but not always well implemented, and rules outside Europe aren’t always so comprehensive. Similarly, if you want to move away and use a different tool, do you retain the data and can you exit the contract without hidden fees?
• The questions and statements actually correlate with psychological safety. The survey tool needs to be a valid (measures the thing it’s supposed to measure), reliable (consistently elicits the same results each time it is repeatedly used in the same way), and tested way to measure psychological safety in teams. If people are asked to self-report how psychologically safe they feel, do they actually feel safe enough to be honest? If we’re not measuring the thing we think we’re measuring, any changes we make as a result may well have no effect and could even damage psychological safety further.
• It doesn’t make assumptions based upon majority culture. If the majority culture in the organisation is American, ensure that the survey tool is as relevant for people with different cultures such as South East Asian or Scandinavian. We want to avoid concluding that certain cultures are less psychologically safe within an organisation simply because the survey tool has been designed for and tested with a different culture.
• The tool doesn’t create perverse incentives. Ensure that people aren’t just telling you what they think you want to hear, or what they’re incentivised to say. It can be easy to unintentionally skew data from surveys and measurement tools by rewarding high scores, criticising low scores, using Red/Amber/Green indicators, comparing different teams, functions or managers, or even just using the word “scores.” The gamification and incentivisation of psychological safety “scores” will very likely cause more harm than good. 
• Use of the tool is optional. Forcing people to use a tool or respond to a survey will result in bad data and decreased psychological safety in the teams. 

There are other commercial aspects too, such as Total Cost of Ownership (does it get more expensive over time?), but they’re beyond the scope of this piece. 

The complexity of all these considerations is why we tend to advocate for a tailored approach that considers the unique contexts and variables at play in different organisations. Of course, you may disagree with some of the above criteria, or agree with them and decide to use a tool anyway because the benefit outweighs the costs or risks. That’s awesome – the key is ensuring you have the knowledge and criticality to make informed decisions about these important tools. 

As a final word on the topic, a tool can only ever be as good as the care and integrity with which we use it and learn from it; the emphasis should always be on empowering individuals and teams to foster a culture of psychological safety and openness, rather than narrowing our focus to metrics or scores. After all, psychological safety can’t really be reduced down to a number – ultimately it needs to be seen, felt and experienced.

Measuring Productivity

Related, in a sense, to my piece above about psychometric tools, it was interesting this week when McKinsey published “Yes, you can measure software developer productivity.” More interesting than the article itself, which frankly provides very little in terms of actionable insight but does make a number of serious missteps, was the backlash to it. This 2-part piece by Kent Beck and Gergely Orosz critiques McKinsey’s piece and doesn’t pull any punches, particularly in respect to the dangers inherent in measuring intangibles and sharing that information with others.

I’ll attempt to summarise why approaches to measuring individual “productivity” of team members are flawed, whether we’re measuring software developers, doctors, teachers, or sportspeople. 

• “Productivity” is near impossible to define in a meaningful way, unless you choose some arbitrary output such as lines of code written or widgets assembled, which don’t necessarily correlate with the outcome you’re hoping to achieve. Goodhart’s Law is strong.
• We don’t all perform at the same pace at the same time. We all have off days, good days, great months, awful months. A core reason a team exists is to even out that bumpy line. If we got rid of every person who was less productive for a while, we’d eventually have none left.
• Teams are greater than the sum of their parts. Together, they achieve more than people do alone. Measuring individual productivity creates an incentive to “go it alone”, making the individual look productive, but damaging the team as a whole. 
• Not everyone needs to be “productive”. Some people in teams are assisters, glue people, unblockers, enablers and trainers. Are we going to penalise or get rid of them because they’re “unproductive”?
• It’s the “unproductive” members who often contribute most to success. Some team members are highly productive precisely due to the efforts of those assisting members who don’t get the glory.
• Measuring individual productivity can introduce unhealthy competition in a team, and damage psychological safety as well as overall performance. 
• Increasing “Productivity” can incentivise decreasing slack time, or buffer time, which reduces resilience and capacity to adapt to incidents and unforeseen changes.

This isn’t intended to suggest that accountability is bad, just that we can’t delegate good management to metrics, and we should be conscious that every measurement interferes with a social system. Measuring individual productivity invariably leads to suboptimal results. If we were to measure each person on a football team by how many goals they scored, we’d have given two of them huge bonuses and sacked the rest of them by the end of the first match.

Whenever we measure people in any way, we change their behaviour, often in subtle and unpredictable ways.

Communication in Low Trust Environments

I love this piece by the always-awesome Charity Majors, writing about communication in low-trust environments, and emphasising the idea that while conversations may feel like a minefield in low-trust situations, they are essential for working together and rebuilding trust. I really like this point: Acknowledge it’s hard beforehand, by saying:

• “I want to say something, but I am having a hard time with it.”
• “I have something to say, but I don’t know how you’ll take it.”
• “I need to tell you something and I am anxious about your reaction.”

There’s so much good advice in this piece, and whether you’re in a low-trust environment or not, a lot of these approaches will help foster psychological safety. Definitely worth a read!

More Psychological Safety Workshops!

We have places left on our upcoming workshops in October and November – you can choose from:

• Intermediate: essential theory and practice to help you level up your knowledge and skills.
• In-depth: examining elements such as cultural impacts on psychological safety, complexity, sociotechnicality, organisational scale and more.
• Practices: From Empathy Mapping to TRIZ. You learn practices and take away canvases and templates too.
• Leadership: Specifically for managers and senior leaders (or folks on that journey): we cover effective management practices, strategic leadership, reporting, 1-1s, work design and flow, managing neurodiverse people and more. 

 Find out more and register here.

Here’s a fab article by Amy Edmondson in The Guardian, highlighting how “Intelligent Failures” provide valuable insights, and can help us adopt a healthier perspective on failure, especially in an age dominated by social media perfectionism.

In her book The Fearless Organisation, Amy describes co-founder Ed Catmull’s approach to institutionalising candour. Here’s a great short piece by Allen Pike, about Pixar’s Michael B. Johnson who through his 25 years at Pixar has helped teams there “fail as quickly as possible”, and illustrates this with an excellent principle for getting unstuck – “just do something, so we can change it!”

Psychological Safety vs Safe Spaces

Finally, this tweet (it’ll always be Twitter to me) from Adam Grant caught my eye. In it, Adam states that “safe spaces treat people as fragile“, which I don’t think is correct or helpful. For example, an AA meeting, a domestic abuse survivors group, or a PTSD group for war veterans are “safe spaces”, and I certainly wouldn’t call the people in those spaces fragile. Many “safe spaces” comprise very resilient and strong people, and using “safe space” as a pejorative term only serves to harm those who need them most.

I don’t think Adam intended to cause harm – after all, we all make mistaks – I think it’s simply unfortunate wording, but essential to point out. For those people with large influence and for whom many people take as experts, it’s really important to communicate true and helpful information, and demonstrate the humility to acknowledge when we get it wrong.

There are also some excellent points made in this LinkedIn thread, by people far more eloquent than myself.

This week’s poem:

Today, by Jean Little

TODAY I will not live up to my potential.
TODAY I will not relate well to my peer group.
TODAY I will not contribute in class.
  I will not volunteer one thing.
TODAY I will not strive to do better.
TODAY I will not achieve or adjust or grow enriched
     or get involved.
  I will not put my hand up even if the teacher is wrong
     and I can prove it.

TODAY I might eat the eraser off my pencil.
     I’ll look at clouds.
     I’ll be late.
     I don’t think I’ll wash.

I NEED A REST.


From the anthology “She Is Fierce” by Ana Sampson