Skip to content

Redundancy and its cousin, resilience

The cockpit of an Airbus A380. Notice the symmetry and redundancy, with two of everything (including the pilots). Wikipedia photo.

Quick now: How many hearts does an octopus have?

Answer: Three! However, two of the hearts are not backup hearts, exactly. Rather, the three-heart system is an element of octopus engineering that offloads pumping blood to the gills to two extra hearts. The two gill hearts, however, are a kind of redundancy.

Quick now: How many hearts does an earthworm have?

Answer: Five! Earthworm hearts, though, are a simpler form of heart called “aortic arches.” All five aortic arches share the load.

In us humans, hearts are a single point of failure. Maybe that’s one reason why heart failure is the leading cause of death. Some parts of our bodies are redundant, though. We have two eyes, two ears, two lungs, and two kidneys. Our redundant eyes and ears have benefits beyond redundancy, though. They provide us with stereo hearing and stereo vision. Our metabolic systems have all sorts of redundancies. As for our hearts, though they are single points of failure, they do have the ability to heal. That makes us resilient.

Quick now: How many “angle of attack” sensors were operating on the two Boeing 737 MAX planes that recently crashed?

Answer: One.

Since my post about the Boeing 737 MAX a couple of weeks ago, we’ve learned more about what went wrong, and about what Boeing intends to do about it. This piece in Vox provides some good new information. Though the airplane has two angle of attack sensors, the airplane’s control system received input from only one of them. For an extra $80,000, Boeing would include a warning light that would alert the pilots if the two sensors did not agree. The planes that crashed did not have the warning-light option. This blows my mind. Redundancy — meaning no single points of failure — was, or so I believed, an inviolable rule in aviation engineering. We can probably be pretty sure that it wasn’t Boeing engineers who decided to allow a critical crash-prevention system to have a single point of failure. Rather, it was Boeing executives, and their motive was money.

I am obsessed with redundancy. The last half of my career in newspapers (I am now retired) was in editorial systems. I was responsible for publishing systems that had to be 100 percent reliable. A failure would mean that you wouldn’t go to press. For that reason — at least back then — systems people had an understanding with the money people. The money people would say to the systems people, in essence: You’ve got to make sure that we can meet our deadlines and go to press every day. In return, the systems people would say to the money people: Well then, that’s going to cost you, because not only have you got to buy two of everything, you’ve got to build the systems in such a way that the backup system will immediately take over if the primary system fails.

In earthquake-prone San Francisco, where I worked for the last years of my career, the San Francisco Examiner and the San Francisco Chronicle had impressive levels of redundancy. There were three printing plants, geographically dispersed. At the main offices at Fifth and Mission, there was a diesel generator for backup power that was the size of a locomotive. The computer systems were redundant. If a failure was detected by “heartbeat” systems that monitor critical processes, the system would automatically “fail over” to the backup. With some systems, the failover process might take a minute or so. On some systems (such as the older Tandem mainframe computers), the failover was so fast and so smooth that you might not notice that there had been a failure. I remember one morning when a Tandem technician showed up to make repairs on the mainframe. “Really?” we asked. “What’s wrong?” The technician said that the system had failed over the night before (while the Chronicle was happily going to press, its staff of hundreds unaffected). The computer had called home to report the problem (many computers can do that), and a technician was dispatched. The computer had even told the home office what parts to bring.

An important part of my career responsibilities was risk management. I have written more “disaster recovery” plans that I care to remember. But I am still obsessed with redundancy.

Redundancy, actually, figures heavily into the plot of my first novel, Fugue in Ursa Major. In the setup and foreshadowing of the redundancy angle, Phaedrus says to Jake:

“The problem is, redundancy is not cheap…. Most people can’t afford much redundancy. I’m hard pressed for redundancy myself, these days especially. People have two cars, a spare tire, an extra toothbrush. But it’s hard to have redundancy when having just one of something you need is hard enough. But let’s don’t get ourselves depressed over dark possibilities. You’ve come to go camping on a high ridge, and smell the flowers and look at the stars. We can scare the daylights out of ourselves some other time thinking about how precarious our support systems are.”

Once upon a time (is it still true?) many systems on aircraft, such as the navigation systems, were triple redundant, like an octopus’ heart. It was very hard for me to believe that Boeing, of all companies, would allow a single sensor to bring down an airplane. Two airplanes.

According to Vox, Boeing’s fix for the 737 MAX includes monitoring two angle of attack sensors and warning the pilots if the sensors disagree. It is stunning that Boeing didn’t do things that way the first time. Boeing will pay dearly for cutting corners.

After redundancy has saved the day in Fugue in Ursa Major and as the story gets into the denouement stage, Jake teases Phaedrus, and Jake quotes his English-teacher mother. Joan is a dog:

Jake smiled up at the stars and scratched Joan’s head again.

“Aha,” said Jake, “I just figured out your real objection to monotheism.”

“What’s that?”

“A single god is not redundant. If god lets you down, you have nowhere to turn to. That’s an existentially ugly place to be, as my mother might say.”


  1. frigast wrote:

    In France we need to have an alternative solution to almost everything.
    – and an alternative plan to every situation.
    It’s not redundance – only foresight 🙂

    Tandem was pretty smart (I was programmer in IBM at that time :-))
    Besides my office computer I have a little portable one – should need be 🙂
    But I only have one car – lolol
    Lost my hiking staff some time ago, didn’t have a spare, didn’t buy another one – found out life was much easier/simpler not having one to take care of 🙂
    I think my morale would be:
    If you cannot live without it, you might as well buy two of them, e.g. house keys – lolol

    Sunday, March 31, 2019 at 7:35 am | Permalink
  2. frigast wrote:

    Forgot to mention, my portable computer has four hearts – lolol
    FYI: we’re looking foreward to be hearing more about Jake & Phaedrus 🙂

    Sunday, March 31, 2019 at 7:38 am | Permalink
  3. daltoni wrote:

    Hi Frigast… Tandem made very nice hardware. It was what banks used, I believe, because it was so fault-tolerant. After the Tandems were retired, in my line of work we mostly went with redundant Sun servers running Solaris, with Oracle as the database. The Sun hardware knew how to fail over, but it was never as smooth as Tandem. 🙂

    Sunday, March 31, 2019 at 7:43 am | Permalink

Post a Comment

Your email is never published nor shared. Required fields are marked *