The Streetlight Effect

Measuring the really important stuff is hard. How do we measure inclusion, or safety, or happiness? We can probably all agree that these things matter, but when it comes to measuring them, where do we start? In fact, some would go as far as to suggest that what matters can’t be measured.

As sociologist Willam Bruce Cameron explained in 1963:

“It would be nice if all of the data which sociologists require could be enumerated because then we could run them through IBM machines and draw charts as the economists do. However, not everything that can be counted counts, and not everything that counts can be counted.” 

And yet, much of the world of work remains fixated on measurements, on pinning everything down to quantitative metrics, as if the only way to prove something matters is indeed to measure it. But because measuring the important stuff is either very hard, or impossible, I’m increasingly convinced that a lot of the time (maybe most of the time?) what we are doing is not measuring what matters, but measuring what we can, and making it matter. 

And of course this is not just a technical problem of how to measure the difficult stuff, but a moral one – who gets to decide what’s important? When we flatten people’s lived realities into metrics, who decides what we quantify and what we disregard? Who has that institutional power? Measurement choices are about power. They determine whose experience is rendered visible, whose risks are legitimised, and whose realities remain structurally invisible.

“Measurement is a technology with power, harnessed by those able to use it.”
Sally Engle Merry, The Seductions of Quantification

There are so many small and large examples of this. We know there are organisations who measure the number of story points completed by software developers as a way to monitor their productivity. Or they record how “active” someone is whilst at their desk by measuring their mouse movement. Consultancies use consultant “utilisation” as a measure of both productivity and efficiency (and nobody records it honestly anyway). Even the typical annual employee engagement survey is an example – we ask what is convenient to ask and will fit into neat bar charts ready for reporting and year-on-year comparison. We measure what we can, or what we already have the data for, but this is no guarantee that the numbers we generate are a good reflection of what actually matters. Customers, patients, and service users don’t care about hours spent on a task or what time someone arrived at the office. They care about good outcomes. Am I being well looked after? Am I treated as a whole person, not a collection of symptoms? Does this product meet my needs, maybe even the needs I didn’t know I had? But these are much harder to measure, so we tend to look for other things to measure instead.

Activity does not necessarily equal value, just as compliance does not necessarily equal safety. 

In safety-critical contexts, we often count incidents and audit compliance, while the more significant determinants of risk such as time pressure, understaffing, misaligned incentives, fear of blame and more, remain largely unmeasured. As Carsten Busch reminds us: 

“An absence of accidents is not the same as the presence of safety.” 

But what is easier to count becomes what is taken most seriously. We look for answers where it’s easiest to look, not where the answers are most likely to be.

Looking Where the Light Is: The Streetlight Effect

This idea is known as the Streetlight Effect – a term that dates back to at least 1942, when it appeared in a Mutt and Jeff comic strip by Bud Fisher, published in the Florence Morning News. The idea however predates 1942 considerably, with a story of the Islamic folklore character Nasreddin, who loses a ring in a dark room of his house but instead looks for it in the yard because there is “much more light” out there.

A police officer is patrolling a neighbourhood when he sees a drunk man crawling around underneath a streetlight. The officer asks if there is a problem. The man explains that he dropped a quarter, and is trying to find it. The officer has a look, and after finding no quarter, he asks the man where exactly he dropped it. The drunken man replies that he dropped it two blocks away. When the confused police officer asks him why he is searching here instead of there, the man replies, “Because the light is better here.”

It’s hard to look in the right place, so he looks in the easy one instead – even though it’s the wrong place. The drunk man chooses convenience. Organisations, more often than not, institutionalise it.

From bias to design principle

The drunken man chooses where to look based upon convenience, whilst in organisations, the Streetlight Effect becomes almost a measurement design principle based upon four “streetlight” criteria:

1. Visibility (we’ll measure what we can see)
2. Legibility (we’ll measure what easily fits into reports and dashboards)
3. Defensibility (we’ll measure what can be justified to management, regulators, or exec teams)
4. Appearance (we’ll measure what makes us look good – compared to other people, teams, or organisations)

Together, these criteria determine not just what is measured, but what is believed, funded, and prioritised. Anything that falls outside them risks becoming organisationally invisible.

And these criteria often distract us from the  much more important question of “is this information useful?” Does this data actually tell us anything? More often than not, data simply begs more “but why?” questions. Were we even looking in the right place to begin with?

Organisations fall into this trap all the time – we gather the data that we can, not the data that actually matters. Psychological safety is far from immune to this – just because we can run a psychological safety survey, should we? Even if we successfully measure how safe people feel in their team, at a certain point in time, for any given value of “team”, what are we going to do with that information? At best, surveys give us more questions than answers; at worst, they can lead us down the wrong path altogether or distract us from the real issues at play. 

Measurement Theatre

One reason for our tendency to measure what we can, and then make it matter, is the seductive nature of “measurement theatre” – it feels good, productive, busy and (importantly) highly visible to measure stuff. It can feel like we’re doing something valuable and having an impact. Nobody ever got sacked for generating more data. We can create impressive looking dashboards, slide decks, reports and analyses, and use these to elevate our own status and visibility within the organisation by presenting it to senior leaders or sharing the findings with the whole organisation. Yet if we never do anything meaningful with this information, or if the measurement was pointing us in the wrong direction anyway, measurement can ultimately allow organisations to appear proactive and responsible without actually confronting harm, inequality, or imbalanced power.

This happens in academic research too, not just organisational research. As Battaglia and Atkinson argue in their analysis of medical research, the pressure to publish, secure funding, and advance careers increasingly drives people toward “safe” work: studies that affirm existing paradigms rather than challenge them. They describe this explicitly as a form of the streetlight effect: studies look where it is easiest and most defensible to gather data, not where the most important questions actually lie. 

What gets measured is often what careers can be built on. It influences what can get funded, resourced, and noticed, and it risks distracting us from the things that really matter. 

“It is wrong to suppose that if you can’t measure it, you can’t manage it – a costly myth.”
– W. Edwards Deming, The New Economics.

Methods for seeing in the darkness

What’s harder, but often far more instructive than looking where the light is, is going to the places where the answers are more likely to reside. This usually means actually talking to people, and inviting them to speak and share what’s really going on for them in the messy, lived complexity of their work. We need to explore the dark, as well as check what we can see in the light. 

And there are tools that can help us here. In our work, it often involves approaches like action research (learning whilst doing the work), listening carefully to stories (sometimes described as narrative capture), and qualitative sense-making, where people surface meaning together, rather than having it extracted from them by analysts. Underpinning all of this is a simple belief: that organisational knowledge should be participatory instead of merely observational – measurement should be in service of learning, not the other way around.

“What gets measured gets managed — even when it’s pointless to measure and manage it, and even if it harms the purpose of the organisation to do so”
Simon Caulkin in 2008, summarising VF Ridgway 1956.

Measurement doesn’t just describe reality (or our idealised version of it), but it shapes reality. And if metrics shape reality, then choosing what to measure is an act of governance. So by only measuring what is easy – the things that we can see under the streetlight – we don’t just notice those things, we make them important. We alter the very substrate of the system by influencing attention, behaviour and decision making. 

In that sense, the Streetlight Effect becomes a form of organisational self-deception. It preserves the status quo, reinforces hierarchies, and legitimises “power over”, all while appearing neutral and “objective”  And if we take that seriously, then the begging question isn’t what we can measure, but what kind of organisation and what kind of world are our measurements shaping?

“There’s beggary in the love that can be reckoned.”
Shakespeare; Antony and Cleopatra.

References

Fisher, B. Mutt & Jeff Florence Morning News, 3 June 1942, p. 7

1963, Informal Sociology, a casual introduction to sociological thinking by William Bruce Cameron, Page 13, Random House, New York. (Google Books snippet view) 

Battaglia M, Atkinson MA. The streetlight effect in type 1 diabetes. Diabetes. 2015 Apr;64(4):1081-90. doi: 10.2337/db14-1208. PMID: 25805758; PMCID: PMC4375074. https://pubmed.ncbi.nlm.nih.gov/25805758/ 

Caulkin, S., 2008. The rule is simple: Be careful what you measure. The Guardian. 
https://www.theguardian.com/business/2008/feb/10/businesscomment1

Deming, W. E. (1994). The New Economics for Industry, Government, Education (2nd ed.). Cambridge, MA: MIT Center for Advanced Engineering Study. https://amzn.to/4swn0YX 

Ridgway, V.F., 1956. Dysfunctional consequences of performance measurements. Administrative science quarterly, 1(2), pp.240-247. https://www.jstor.org/stable/2390989

Carsten Busch. If You Can’t Measure It… Maybe You Shouldn’t: Reflections on Measuring Safety, Indicators, and Goals Paperback – 30 Oct. 2019 https://amzn.to/4jw1L5r 

Merry, S.E., 2016. The seductions of quantification: Measuring human rights, gender violence, and sex trafficking. University of Chicago Press. https://amzn.to/4pnGmMZ