Into the Woods

The cockpit of an Airbus A380. Notice the symmetry and redundancy, with two of everything (including the pilots). Wikipedia photo.

Quick now: How many hearts does an octopus have?

Answer: Three! However, two of the hearts are not backup hearts, exactly. Rather, the three-heart system is an element of octopus engineering that offloads pumping blood to the gills to two extra hearts. The two gill hearts, however, are a kind of redundancy.

Quick now: How many hearts does an earthworm have?

Answer: Five! Earthworm hearts, though, are a simpler form of heart called “aortic arches.” All five aortic arches share the load.

In us humans, hearts are a single point of failure. Maybe that’s one reason why heart failure is the leading cause of death. Some parts of our bodies are redundant, though. We have two eyes, two ears, two lungs, and two kidneys. Our redundant eyes and ears have benefits beyond redundancy, though. They provide us with stereo hearing and stereo vision. Our metabolic systems have all sorts of redundancies. As for our hearts, though they are single points of failure, they do have the ability to heal. That makes us resilient.

Quick now: How many “angle of attack” sensors were operating on the two Boeing 737 MAX planes that recently crashed?

Answer: One.

Since my post about the Boeing 737 MAX a couple of weeks ago, we’ve learned more about what went wrong, and about what Boeing intends to do about it. This piece in Vox provides some good new information. Though the airplane has two angle of attack sensors, the airplane’s control system received input from only one of them. For an extra $80,000, Boeing would include a warning light that would alert the pilots if the two sensors did not agree. The planes that crashed did not have the warning-light option. This blows my mind. Redundancy — meaning no single points of failure — was, or so I believed, an inviolable rule in aviation engineering. We can probably be pretty sure that it wasn’t Boeing engineers who decided to allow a critical crash-prevention system to have a single point of failure. Rather, it was Boeing executives, and their motive was money.

I am obsessed with redundancy. The last half of my career in newspapers (I am now retired) was in editorial systems. I was responsible for publishing systems that had to be 100 percent reliable. A failure would mean that you wouldn’t go to press. For that reason — at least back then — systems people had an understanding with the money people. The money people would say to the systems people, in essence: You’ve got to make sure that we can meet our deadlines and go to press every day. In return, the systems people would say to the money people: Well then, that’s going to cost you, because not only have you got to buy two of everything, you’ve got to build the systems in such a way that the backup system will immediately take over if the primary system fails.

In earthquake-prone San Francisco, where I worked for the last years of my career, the San Francisco Examiner and the San Francisco Chronicle had impressive levels of redundancy. There were three printing plants, geographically dispersed. At the main offices at Fifth and Mission, there was a diesel generator for backup power that was the size of a locomotive. The computer systems were redundant. If a failure was detected by “heartbeat” systems that monitor critical processes, the system would automatically “fail over” to the backup. With some systems, the failover process might take a minute or so. On some systems (such as the older Tandem mainframe computers), the failover was so fast and so smooth that you might not notice that there had been a failure. I remember one morning when a Tandem technician showed up to make repairs on the mainframe. “Really?” we asked. “What’s wrong?” The technician said that the system had failed over the night before (while the Chronicle was happily going to press, its staff of hundreds unaffected). The computer had called home to report the problem (many computers can do that), and a technician was dispatched. The computer had even told the home office what parts to bring.

An important part of my career responsibilities was risk management. I have written more “disaster recovery” plans that I care to remember. But I am still obsessed with redundancy.

Redundancy, actually, figures heavily into the plot of my first novel, Fugue in Ursa Major. In the setup and foreshadowing of the redundancy angle, Phaedrus says to Jake:

“The problem is, redundancy is not cheap…. Most people can’t afford much redundancy. I’m hard pressed for redundancy myself, these days especially. People have two cars, a spare tire, an extra toothbrush. But it’s hard to have redundancy when having just one of something you need is hard enough. But let’s don’t get ourselves depressed over dark possibilities. You’ve come to go camping on a high ridge, and smell the flowers and look at the stars. We can scare the daylights out of ourselves some other time thinking about how precarious our support systems are.”

Once upon a time (is it still true?) many systems on aircraft, such as the navigation systems, were triple redundant, like an octopus’ heart. It was very hard for me to believe that Boeing, of all companies, would allow a single sensor to bring down an airplane. Two airplanes.

According to Vox, Boeing’s fix for the 737 MAX includes monitoring two angle of attack sensors and warning the pilots if the sensors disagree. It is stunning that Boeing didn’t do things that way the first time. Boeing will pay dearly for cutting corners.

After redundancy has saved the day in Fugue in Ursa Major and as the story gets into the denouement stage, Jake teases Phaedrus, and Jake quotes his English-teacher mother. Joan is a dog:

Jake smiled up at the stars and scratched Joan’s head again.

“A single god is not redundant. If god lets you down, you have nowhere to turn to. That’s an existentially ugly place to be, as my mother might say.”