Previous posts on this blog have explored learning—or failing to learn—from experience. Ultimately, survival and flourishing depend on being able to extract the right lessons from experience. Ideally, experiences would lead us to develop hypotheses, which we could test in controlled experiments. However, some questions don’t lend themselves to that kind of study. In that case, natural experiments may be the best option. Let’s look at some examples of how natural experiments inform public health.
What do we learn from experience?
When people rebuild after a flood on the same low-lying land, they may talk of resilience or courage. But if they keep doing the same thing hoping the results will be different, should that be a point of pride? Or if the utility didn’t upgrade the power grid after a catastrophic power outage, should we be surprised when it happens again?
On the other hand, if we consider a few instances and figure out what they have in common, we’re equating anecdotes with data. That may get us the right answer by pure luck. But it’s more likely to lead to a false conclusion. And once that false conclusion is out there, it takes on a life of its own. Some people will hold onto it no matter how much new information comes out later—they simply won’t change their minds.
What we take away from experience is subject to confirmation bias. That is, we interpret our observations as supporting what we already believe. It’s important to interpret experience with an open mind and be as objective as possible.
Natural experiments
The 2021 Nobel Prize in economics went to David Card, Joshua Angrist, and Guido Imbens. Despite being decades old, Card’s findings speak to some of today’s “hot button” issues. Angrist and Imbens developed a mathematical framework for determining cause and effect when it’s not possible to use strict scientific methods. Together, their work helps us extract the right lessons from natural experiments.
Card’s methodology provides more than one control group. When examining changes in public policy, it’s not enough to look at the same group before and after the change. Other changes could take place during the same period, and it may not be possible to deconvolute all their effects. For example, in his work on the effects of a minimum wage increase, he surveyed two adjacent geographic areas in New Jersey and Pennsylvania before and after the increase took effect in New Jersey. That allowed him to control for a recession that affected both areas while the experiment was going on.
Natural experiments inform public health
In the 1940s, doctors in wealthy countries became concerned about the increasing incidence of heart disease. Why was this happening? Dr. Jeremiah Morris looked for clues by studying bus drivers and conductors and postal- and telephone workers.
The idea that health and exercise were linked “wasn’t the accepted fact that we know today”, says Nick Wareham, a professor of epidemiology at Cambridge University. Some even felt that “too much physical activity was a bad thing for your health”. Navvies, miners and farm labourers did physical exercise by the spade-load. They also suffered disproportionately from various diseases, died young and featured in novels by D.H. Lawrence. It was a miserable existence.—“How London bus drivers changed the world,” The Economist, 7 September 2023
Bus conductors repeatedly climbed up and down the stairs on the double-decker buses, while drivers sat most of the time. Comparing health data from 31,000 transit workers showed dramatically better coronary health for those whose work entailed physical activity than those doing sedentary work. Similarly, mail carriers enjoyed considerably better health than more sedentary postal workers or telephone workers. At first Morris’ findings, published in 1953, met with skepticism.
Video: Physical activity is one of nine aspects of the Blue Zone lifestyle. The prototype blue zones showed how natural experiments inform public health—in this case, to promote longevity.
Cigarette smoking and lung cancer
Similarly, the incidence of lung cancer, particularly in men, increased dramatically between the 1920s and the 1940s. Sir Richard Doll and Prof. Bradford Hill considered other factors that had changed during that period, including improvements in diagnostic methods, increases in air pollution, and differences in occupation. However, their 1950 paper concluded that tobacco smoking was primarily to blame.
[T]he idea that tobacco was the cause was not very attractive, because the increase in the average consumption per person had been quite small, there being, it seemed, no obvious reason why the smoke from cigarettes, which had largely replaced cigars and pipes, should be so much more hazardous than the smoke from tobacco burnt in other ways.—“Smoking and lung cancer”, Sir Richard Doll
Sir Richard, a smoker himself when he began to work on this study, quit smoking in 1949. However, advisors to the Ministry of Health cautioned against scaring people with their conclusions until they were able to replicate their results in other cities besides London. So they held off publication until the additional results were in.
Soon after, they followed up with a longitudinal study of 40,000 British doctors. They found that the heaviest smokers had the highest mortality due to lung cancer. They were also able to connect cigarette smoking to several other diseases. By then, not surprisingly, many of the doctors had quit smoking.