DevOps for Psychological Safety

Psychological Safety and DevOps Practices

This is a special edition article by Balázs Szakmáry

Developing software of any real complexity is a task that goes well beyond one person and one computer. The people, the machines, and the processes involved all shape each other, and thus form an interconnected socio-technical system. DevOps is at the very heart of this, because it codifies the development process in a way that both humans and machines can process. The tooling provides an interface for the humans to better understand the machines.

One of the main themes of DevOps is using various feedback loops to continuously improve the product and the process of creating it. We want to discover these, utilise them consciously, and tighten them to speed things up. Without psychological safety, these feedback loops are weak – we may not hear about issues at all, we may not pick up on weak signals, and we might not hear about a new way to gather insights or build better software. Psychological safety is foundational both to feedback loops, and to DevOps in general.

This article series explores the interactions between DevOps and psychological safety.

Part 1: CI/CD as an “Oracle of Truth”

A well-organised CI/CD system can provide excellent common ground for talking about the code, the development process, and any issues with these. Everybody (including non-developers) should be encouraged to keep an eye on the CI results.

In order to turn CI/CD into an authoritative source of information, the first thing to do is to establish trust in the feedback it gives. In many organisations, there is a well-founded, but unhealthy scepticism towards build and test results. Ironically, this is not only true for new setups that have not yet been proven to be reliable. It also happens often with large legacy codebases. Watch for people saying things like:

“Oh, that test always fails.”
“My changes have nothing to do with this build failure in my branch.”
“Can you force merge this change for me?”
“I’ll just run another build.”

There are two problems in this situation. The less important one is that this makes the number of builds go up, which clogs the queue or drives infrastructure costs up. The main issue is that the feedback is useless, so people would want to avoid using CI, but pull requests usually cannot be merged without all green builds and it becomes an annoying game of chance to get one’s changes merged.

From the perspective of psychological safety, the core issue is that this phenomenon introduces mistrust to both human-computer and person-to-person interactions. The bare minimum of successful and stress-free communication is to be sure that the participants are talking about the same thing, and that the communication is rooted in truth. This becomes especially important when opinions differ, or something has gone wrong. In a stressful situation or a crisis, wrong information can be much worse than none.

If you find yourself in this situation where the CI/CD results are not trusted, the solution is simple, but not easy: Every time when someone questions CI feedback (usually a failing build or test), an investigation has to happen, and action must be taken. There are two possible outcomes:

• There is something incorrectly configured in the CI/CD system. This needs to be immediately corrected.
• The failure is indeed a valid one (this includes flaky tests), but it did not manifest on the developer’s machine. The correctness of the result needs to be demonstrated, and optionally, the difference between the developer’s local environment and the CI environment can be pinpointed.

After a while (this can be years!), people will stop finding real issues with CI, and not much later they will trust the results as true. This reduces conflicts rooted in misunderstandings or differing assumptions, which in turn allows people to have more patience and energy for the productive conflicts about real issues.

Constantly doubting the environment is a major source of cognitive load and frustration. It is very useful to provision local developer environments using automation, ideally the same as the one that provisions the CI build environment. If this automation is in place, the gap between the environments can be closed step by step, for all developers at once.

The Oracle of Truth can be made wiser by including automated checks in the CI pipeline:

• Tests (unit, integration, smoke, etc.)
• Static analysis for style, complexity, test coverage, security, compliance, and other purposes

These are of course useful to increase the chance of the deliverables being “good”. They also provide some useful mental side effects.

Purely objective feedback with no emotions involved. This removes some unproductive conflicts, especially in code review discussions. It is completely different to make the code conform to the machine’s clearly defined expectations than to discuss the same issues with a person reviewing the code.

Common ground about the state of things. The information presented to everyone by CI is unambiguous, written, and public. This helps everybody involved in discussions to have the same mental model of the issue at hand, which reduces misunderstandings and conflicts.

Common ground about the rules. People have different perceptions about what is required and what is allowed. A great example is unit tests: stakeholders have different ideas about their usefulness (vs. shipping more features), some developers want to test everything, others think they are useless. If CI is set up to expect a certain coverage level, nobody will argue about this anymore people will argue much less. The same applies to code style, which can be a hotly contested topic without a clear standard. Formally specifying these  rules in the “definition of done” enables easy compliance, and helps transform arguments into productive discussions about what the rules should be.

Lower threshold to speaking up. When something is wrong or fishy, there is ready evidence in the logs to use. This reduces the effort to “Do you think this is weird?” Additionally, the CI system itself does some of the “speaking up” that needs to happen, by signalling about most failures.

Accepting human error as normal and hedging against it. We all make mistaks. Everybody has red builds in CI, and this is completely normal. It is the organisation that needs to be resilient to this, not individual people.

Part 2 in this series is going to examine how code hosting engines enable psychological safety.

Photo by Annie Spratt on Unsplash