Abraham Wald and how to reinforce fighter planes
During World War II, the Allies studied Nazi damage to their airplanes.
Their study resulted in this figure:
The analysis of these damages resulted in the idea that their airplanes should be reinforced in the areas of the airplane showing the densest clusters of red dots. A statistician by the name Abraham Wald didn’t agree. Instead, he suggested that the airplanes should be reinforced where there were no indications of damage.
Wald was right, of course.
By studying planes that got back home somehow, the red areas indicated non-fatal damage areas. If it had been possible to study the plane that were shot down and destroyed by the Nazis, they would be likely to have found an inverse pattern.
The powerful fallacy of survivorship bias
This is a great example of a very specific fallacy — survivorship bias.
During World War I, the military forces increased the issuing of helmets for soldiers. The number of wounded soldiers skyrocketed, but this wasn’t because helmets somehow made it easier for more people to get hurt. The helmets simply rather made sure that less people died and instead showed up in the column for “wounded” soldiers.
Progress of a kind, no doubt.
No, correlation still does not equal causation
It bears repeating again and again; correlation does not equal causation.
In both popular culture and business, we’re fascinated by successful people and successful brands. We want to know how they think and how they act.
But few think to study those who tried but never became successful.
Such data sets could be many times more causative and provide much more powerful clues to your original question.
Study mistakes. Analyse failures. Learn from disasters.