How survivor bias distorts reality

How survivor bias distorts reality

During World War II, researchers at the Center for Naval Analysis faced a critical problem. Many bombers were getting shot down on bombing runs over Germany. The naval researchers knew they needed hard data to solve this problem.  So they went to work. After each mission, the bullet holes and damage from each bomber were painstakingly reviewed and recorded.  On the basis of all these data, the researchers were looking for vulnerabilities. 

After a while, the data was revealing a clear pattern (see picture below): most of the damage was to the wings and body of the plane.  So the solution to avoid this damage was clear: they just needed to increase the armor on the wings and the body of the plane.  However, their analysis and the solution they implemented was completely wrong…

plane.jpg

There was one crucial item that was missing from all the data they had studied: data about every plane that had been shot down and that did not return.  Have a look at the above image again: the surviving planes rarely had damage in the cockpit, the engine and part of the tail.  Was this because of superior protection to those areas?  No.  In fact, these were the most vulnerable areas on the entire plane!  So if you think about it, the researchers had created a map of the exact places that the bomber could be shot and still survive.  So what did they finally do?  They reinforced the cockpit, the engine and parts of the tail.  And the result?  Fewer fatalities and greater success of bombing missions. 

This story is a vivid example of survivor bias. Survivor bias happens when we only look at the data of those who succeed and exclude those who fail. Survivor bias is all around us, especially in the media. You read articles about entrepreneurs who risked everything financially and who are now a success. But no one profiles the hundred other entrepreneurs who followed the same strategy and went bankrupt. 

Bottomline? We tend to gravitate towards the most successful entrepreneurs in the world when we study examples, e.g. Bill Gates, Richard Branson, Steve Jobs, Mark Zuckerberg, Elon Musk, etc.  However, we need to listen more to people who fail and mess up.  Do not just focus on people who succeeded and “survived”.  When you focus too heavily on the “survivors”, you ignore important data and learnings about the rest of the population. 

Latest articles by Filip