These two pieces originally went out in the Psychological Safety Newsletter, the second (“I was wrong about the parachutes”) a couple of weeks after the first. They have proven rather popular, so we’ve posted them online here too.

Parachutes and “Evidence based” thinking

I love this rather infamous paper by Smith and Pell on the effectiveness of parachutes. “Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials.” It makes the very effective point that nobody has ever conducted a randomised controlled study on the effectiveness of parachutes, but people in the real world still use them every single day, very effectively.

Psychological safety suffers a similar attack frequently by folks who claim to be robustly “evidence based”, when in reality they often simply dislike the idea of psychological safety for various, often ideological reasons and, as our own research suggests, from positions of relative structural advantage they may not see.

Psychological safety is the parachute. It is absurd to say that it may be better to not find out about problems, incidents, concerns or ideas for improvement. The evidentiary base for that isn’t only the academic literature — it’s the catastrophe record. Tenerife, Chernobyl, Deepwater Horizon, Challenger, the run-up to 2008 crisis: each is a case where a concern existed at the sharp end and did not travel to where it was needed, and the disaster followed. It’s a mechanism made visible by its failures, over and over, across many domains. Psychological safety’s successes are usually structurally invisible, because the averted disaster leaves no trace; which is exactly the parachute’s epistemic signature. To run the trial honestly, we would have to take a disaster we could see coming, prevent it in one arm by speaking up, and in the other simply let it happen, which sounds as sensible as jumping out of a plane without a parachute, just to see what happens.

So the demand, from a certain kind of critic, that psychological safety prove itself through a randomised controlled trial before we take it seriously isn’t rigour. It’s precisely the parachute review’s target. 

The argument has another edge though, because the catastrophe record proves the mechanism: we can’t fix a secret, we can’t utilise an idea we haven’t heard, and we can’t learn from a mistake that we don’t know about. But it does not prove the commercial models and constructs of psychological safety: the company survey, the stage model, or a metric on the dashboard. 

The parachute absolutely works, but not every parachute is made equal. Some do, in fact, fail. So whilst the RCT demand for the mechanism of psychological safety is absurd, we are right to be critical of much of the commercial apparatus around it. Nobody should be allowed to sell parachutes that don’t open.

I was wrong about the parachutes

A few weeks ago I told you that no RCT of parachutes existed. Sorry – I was wrong, there is one.

In 2018, Yeh and colleagues published one in the BMJ: the PARACHUTE trial — that’s “PArticipation in RAndomized trials Compromised by widely Held beliefs aboUt lack of Treatment Equipoise.” It was a real trial with real ethics approval, randomisation and analysis. Twenty-three participants jumped from an aircraft wearing either a parachute or an empty backpack.

The results: parachute use did not significantly reduce death or major injury (0% v 0%, P>0.9), consistent across all subgroups. In fact, “the parachute did not deploy in all 12 (100%) owing to the short duration and altitude of falls.“

So parachutes don’t work, right?

It’s worth noting that the trial struggled somewhat with enrolment. Investigators approached strangers seated near them on commercial flights, mid-flight, and asked whether they’d be willing to be randomised to jump at current altitude and velocity. Curiously, everyone declined. The authors note that owing to “difficulty in enrolling patients at several thousand meters above the ground,” the team expanded recruitment to friends, family, and themselves.

Which brings us to a key point: the 69 people who declined to participate were on commercial aircraft at a mean altitude of 9,146 metres, travelling at 800 km/h. The 23 people who opted in to the study were on stationary aircraft: a parked biplane at Martha’s Vineyard and a museum helicopter in Michigan, at a mean altitude of 0.6 metres, travelling at 0 km/h. 

The trial is a joke, of course — a sequel to Smith and Pell’s 2003 paper, published in the same BMJ Christmas issue tradition. But like its predecessor, the joke makes a serious but different point.

The 2003 paper argued that you can’t run the trial because the counterfactual is unethical, whilst the 2018 paper demonstrates something subtler about sampling. They actually did run the trial, the methodology was genuinely rigorous: nothing about the randomisation or the analysis is wrong, but it still produced an answer that is both statistically sound and completely absurd, because the conditions required to make the trial possible were precisely the conditions that removed the effect. Parachutes work as a function of altitude. Everyone at altitude declined to enrol. The trial measured the sample rather than the intervention.

“When beliefs regarding the effectiveness of an intervention exist in the community, randomized trials evaluating their effectiveness could selectively enroll individuals with a lower likelihood of benefit, thereby diminishing the applicability of trial results to routine practice.”
Yeh et al, 2018.

And it measured the sample not because the investigators were careless, but because the real version of the trial can’t be run. To test parachutes honestly we’d need to randomise people at 9,000 metres, and give half of them empty backpacks instead of parachutes just to see what happens. No ethics board would approve that, nor would participants consent to it. The only RCT that can exist is the one on the tarmac.

Instead, we could look at the observational record, and we have done (not we, I mean, but people have done). On one side (with a parachute): six deaths in 110,000 Danish sports jumps; 35 in 6.2 million French ones. On the other: falling from high altitude without a parachute is almost universally fatal. Known survivors of such events number in the dozens: survival is extraordinary.

The Dramatic Effect

Epidemiologists call this a dramatic effect: when the gap between intervention and non-intervention is this large, we don’t need an RCT. (Glasziou et al, 2007)

It’s quite clear through examining past cases of parachutes failing that the outcome leans towards bad for the subject. This isn’t an RCT though – it’s empirical data, which is exactly what the vast amount of data from disasters such as Challenger, Amagasaki, Grenfell, Chernobyl and others show about psychological safety: a lack of it results in bad things happening. 

Controlled studies on psychological safety necessarily come from lower altitudes, where failure isn’t catastrophic. The genuinely meaningful data comes instead from retrospective investigation of the disasters where voice failed. Instead of a weakness of the field, this is simply the shape of the phenomenon.

And it also speaks to a certain kind of psychological safety critic. You know the sort – the people who claim that we don’t need psychological safety – “people should just speak up“. They are, in a sense, right — but only for themselves, because they already have psychological safety. They’re standing on the tarmac, exposed to very little risk. It’s the people making the big leap out of the plane in the sky who need the parachute.

References

Ellitsgaard, N. (1987). Parachuting injuries: a study of 110,000 sports jumps. British Journal of Sports Medicine, [online] 21(1), p.13. doi:10.1136/bjsm.21.1.13.

‌Fer, C., Guiavarch, M. and Edouard, P. (2021). Epidemiology of skydiving-related deaths and injuries: A 10-years prospective study of 6.2 million jumps between 2010 and 2019 in France. Journal of Science and Medicine in Sport, [online] 24(5), pp.448–453. doi:10.1016/j.jsams.2020.11.002.

Glasziou, P., Chalmers, I., Rawlins, M. and McCulloch, P. (2007). When are randomised trials unnecessary? Picking signal from noise. BMJ, [online] 334(7589), pp.349–351. doi:10.1136/bmj.39070.527986.68.

‌Smith, G.C.S. and Pell, J.P. (2003). Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials. BMJ, 327(7429), pp.1459–1461. doi:10.1136/bmj.327.7429.1459.

‌Yeh, R.W., Valsdottir, L.R., Yeh, M.W., Shen, C., Kramer, D.B., Strom, J.B., Secemsky, E.A., Healy, J.L., Domeier, R.M., Kazi, D.S. and Nallamothu, B.K. (2018). Parachute use to prevent death and major trauma when jumping from aircraft: Randomized controlled trial. BMJ, [online] 363(363), p.k5094. doi:10.1136/bmj.k5094.