We love to measure stuff, don’t we? 

Maybe it’s human nature, but we seem to have a strong desire to make the intangible tangible, and we tend (or at least many of us do) to believe in the robustness of numbers over the nuance of abstract concepts and sentiment. Numerical data is perceived as the “exclusion of judgement, the struggle against subjectivity” (Porter, 1997) and numbers tend to be regarded as hard (Muller, 2018), whilst sentiment, conversations and experience are regarded as soft. This is echoed in the way we sometimes use the term “hard skills” to describe technical work, and “soft skills” to describe human work. In fact, as many of us have discovered, this does “soft skills” a huge disservice: the soft stuff is the hard stuff.

Typically, when we see metrics used in an organisation, it’s because they’re considered significant and robust. We often assume that the numbers directly reflect some real-world phenomena, and there’s not just a correlation between the number and the thing, but that the number is the thing. When we do that we may find ourselves trusting metrics more than our own observations, experience or conversations.

There are some real advantages to using quantitative metrics. They are particularly suited to dashboards and visualisation, so can help with sharing information. Quantitative data helps us spot patterns and trends, and can often tell us where to look when investigating issues. An awareness of “the data” can also help us to identify biases and step outside our own perspective to see other ways of looking at things. 

Where metrics mislead

However, an over-reliance on quantitative measures can lead to overconfident impressions of causality between factors. This can contribute to poor decisions, or even misinformation (Nash, Trott and Allen, 2022). For example if we notice an improvement in sales in the same quarter that we introduced free cake in Friday team meetings, we might be tempted to conclude that more cake leads to more sales, without exploring the many other factors influencing sales performance. We might even take a “data-driven” stance and introduce cake into every meeting. Now this isn’t necessarily a bad idea (who doesn’t love cake?), but to do it because you believe it improves sales is probably misguided. 

“What gets measured gets managed — even when it’s pointless to measure and manage it, and even if it harms the purpose of the organisation to do so”

Simon Caulkin in 2008, summarising VF Ridgway 1956.

There’s an additional danger lurking for us with regard to the accuracy of the metrics we use. The more decimal points there are (within reason) behind a metric, the more precise it appears and the more likely we are to believe it. Conversely, we tend to find big, simple numbers more attention grabbing and seductive. A big 100 on a dashboard satisfies managers that all is well, whilst a figure of 8.73 out of 10 suggests that the measure is very accurate (which is not necessarily the case). It’s possible to get people’s attention with the simple numbers, and convince them with the detailed ones – even if neither actually reflect reality.

Another challenge is the lack of critical thinking that can occur when metrics are presented as an objective truth. An organisation we recently worked with had suffered a very high profile failure in the past, and psychological safety is generally very low. People were afraid to stick their necks out, innovate, or challenge existing practices because they were so fearful of things going wrong again. One of the behaviours we observed in this environment was a heavy reliance on metrics and targets. People focused on hitting the numbers rather than achieving a positive outcome, because they felt that since no one could argue with the numbers, “following the data” was an effective way to avoid personal consequences for failure. In practice, this led to incrementally harder to achieve targets, and reinforced a normalisation of deviance, because hitting the targets and milestones took priority over everything else. Everyone knew that the metrics were near meaningless, but they could shelter underneath them, safe in the knowledge that they could point at the numbers to justify their actions.

“There’s beggary in the love that can be reckoned”
Shakespeare

Looking beyond metrics

Peter Drucker, the renowned management theorist is often quoted to have said “You can’t manage what you don’t measure”, but in fact he never said that. Indeed, Deming himself addressed this fallacy, describing it as “a costly myth” that you can only manage things you measure. Metrics can be misleading, hide subtler truths and outliers, mask change or trends, and lead to perverse incentives – even if those metrics aren’t actually used or intended to incentivise people. The sheer act of measuring behaviour can change the behaviour itself.

This is one of the reasons we advise teams and organisations to proceed with caution when measuring psychological safety. It’s a powerful thing to do, but it’s rarely as straightforward as we imagine it will be. 

As well as being less ambiguous and precise than they might seem, metrics are also less persuasive than we tend to assume. Whilst quantitative data such as big ROI numbers in a business case may comfort leaders in justifying their business decisions, it is in fact qualitative data we need to win over hearts and minds in the long term. Research has shown that quantitative data is actually less effective than qualitative data in persistently changing attitudes (Kazoleas, 1993). Numbers might help us justify a decision, but it’s qualitative data  – people’s voices and stories – that convinces us, deep down, that it’s the right decision.

So where does this leave us with measurement?

Of course, we’d be pretty stuck without metrics at all. Whether we’re measuring revenue or airspeed, we need that data in order to be able to do our jobs and keep people safe. But let’s keep in mind that the superpower of metrics is that they cause us to ask the “why” questions. Our revenue is falling – why? Our airspeed is dropping – why? Even great data visualisations beg us to ask “why” – and that’s where the real insight lies. A metric is rarely the answer, but it can pose an incisive question, like in the two images below: 

The renaissance of qualitative approaches

Personally, I believe we’re seeing the renaissance of qualitative approaches again in industry, particularly in the tech world, which for years was rather doggedly positivist and reductionist about human nature, arguably leading to some unwanted results. This change is, in my opinion, a good thing.

Measurement is not a substitute for judgment; it demands judgment. Decisions must be made about what to measure, whether to measure at all, how to interpret the significance of the measurements, how to publicise the results, and whether to associate rewards and penalties with them.

Metrics are simply one small part of the discovery and learning process. A good metric makes you think “Oh, that’s interesting, I wonder what’s going on there.” That’s where conversations begin, and the real work starts.

References and links:

Kazoleas, D. C. (1993). A comparison of the persuasive effectiveness of qualitative versus quantitative evidence: A test of explanatory hypotheses. Communication Quarterly, 41(1), 40-50.

Morwitz, V. G., & Fitzsimons, G. J. (2004). The mere-measurement effect: Why does measuring intentions change actual behavior? Journal of Consumer Psychology, 14(1-2), 64–74. https://doi.org/10.1207/s15327663jcp1401&2_8

Nash, K., Trott, V., & Allen, W. (2022). The politics of data visualisation and policy making. Convergence, 28(1), 3-12.

Porter, T. M., & Haggerty, K. D. (1997). Trust in numbers: the pursuit of objectivity in science & public life. Canadian Journal of Sociology, 22(2), 279.Muller, J. (2018). The tyranny of metrics. Princeton University Press.

Muller, J. (2018). The tyranny of metrics. Princeton University Press.

How to Measure Anything: Finding the Value of Intangibles in Business. Douglas Hubbard

If You Can’t Measure It… Maybe You Shouldn’t: Reflections on Measuring Safety, Indicators, and Goals Paperback. Carsten Busch 

Management tools, the intended and unintended consequences: James Pomeroy

Psychsafety.com related reading:

Preparing to measure psychological safety

Measure Psychological Safety

Amplifying Weak Signals

Data Visualisation

Zero Defects

Choosing a psychometric tool

Psychological Safety in Practice

Threats to New Safety Movements

I love this piece from Stephen Shorrock on 7 threats to New Safety Movements. Highlighting Egotism and Leaderism, as well as others, Stephen describes how some of these new paradigms (which include ideas such as psychological safety, “New View or Safety II”, HOP, Resilience Engineering and more) may, or already do, suffer from some of these symptoms. The risk is that they become watered down, lose credibility, suffer from semantic diffusion, or get reinvented by lots of people who want their own model or framework – when in fact all it needs is for people to practise it and share learning.

“Incident” shouldn’t be a four-letter word

Nora Jones wrote this piece on incident investigation back in 2021, but we came back to it this week in one of our Feedback workshops. In our workshops, we learn a number of principles, tools, approaches and techniques, and then come to a few scenarios. What’s fascinating is how a simple mistake or error, which initially appears to require a short feedback conversation, becomes a deeper exploration of context, environment, situation and understanding. Nora addresses that complexity well in this piece.

If AI harms a patient, who gets sued?

Ignoring a certain obsession with “accountability” in healthcare and suing someone whenever something goes wrong, this is an interesting piece reflecting on physician error and misdiagnosis, and how AI tools can improve diagnostics and treatment. But it comes with a risk if AI tools are used without appropriate human judgement.