Experiments usually involve a controlled environment where we can limit the number of variables. In a traditional experiment, we hold everything constant except the thing we want to study. But even in a laboratory it’s impossible to change just one thing. And the real world is less amenable to control. That’s why we need to learn from natural experiments.
In an experiment, some subjects (or specimens) serve as the control group for which conditions remain constant. Others receive one or another of the treatments the experimenter is studying. For example, some patients receive a drug while others get a placebo—a sugar pill or an injection of saline solution, say. That way they all believe they’re getting the treatment, but some are and some aren’t.
When studying policy changes, it’s less clear how to find a control group. It may not be sufficient to study the same group before and after the change, as other things can change, too. In that case it may be impossible to deconvolute the effects of the policy change from other influences. You need an otherwise comparable group that does not experience the policy change.
The 2021 Nobel Prize in economics went to three economists for their seminal work on natural experiments. David Card, Joshua Angrist, and Guido Imbens have all helped us understand how things work in the real world. Despite being decades old, Card’s findings speak to some of today’s “hot button” issues. Angrist and Imbens have developed a mathematical framework for determining cause and effect when it’s not possible to use strict scientific methods. Together, their work helps us extract the right lessons from natural experiments. That way we don’t have to keep doing the same things while expecting the results to be different.
Minimum wage and jobs
In the last few years, many states and municipalities have sought to raise their minimum wage. One objection to increasing the minimum wage is the fear that it eliminates jobs. David Card and Alan Krueger studied this question almost 30 years ago.
They analyzed jobs in fast-food restaurants in the Philadelphia area in 1992. New Jersey increased its minimum hourly wage from $4.25 to $5.05 on 1 April 1992, while Pennsylvania did not. Card and Krueger compared employment, wages, and prices at fast-food restaurants in both states before and after the increase.
In the two months before the wage increase went into effect, Card and Krueger surveyed 410 fast-food restaurants in both states. They asked about employment, starting wages, prices, and other details. A second survey 8 months after the minimum wage increase showed that the starting wage in the New Jersey restaurants had increased by 10%. However, the increased minimum wage did not carry over to workers initially earning more than the minimum wage. Perhaps surprisingly, full-time-equivalent employment increased in New Jersey relative to Pennsylvania. Prices of comparable food increased in New Jersey relative to Pennsylvania.
This study illustrates the importance of control groups in natural experiments. During the study period the recession that had already begun continued to deepen. Had they merely compared conditions in New Jersey before and after the minimum-wage increase, Card and Krueger would have had difficulty separating the effects of the minimum wage and the recession. However, the data from Pennsylvania fast-food restaurants from the same time make it clear which is which.
Immigration and the labor market
A common objection to immigration stems from the belief that by increasing the competition for jobs, it depresses wages. David Card’s study of the Miami labor market before and after the Mariel boatlift of 1980 indicates otherwise.
The Mariel boatlift followed Fidel Castro’s announcement in April 1980 that Cubans wanting to emigrate to the United States were free to leave from the port of Mariel. Over the next few months, approximately 125,000 refugees left Cuba. About half of them settled in Miami, increasing the local labor force by 7%.
Card compared wage and employment data from Miami with those of Atlanta, Los Angeles, Houston, and Tampa-St. Petersburg between 1979 and 1985. He found that the influx of workers had no real effect on wages or employment in the Miami labor market. While average wages of ethnic Cubans fell briefly, that appeared to be due to temporary high unemployment among the new arrivals.
The influx of relatively low-skilled workers was easily absorbed into Miami’s textile and apparel industries as earlier arrivals moved to better-paying jobs. However, the arrival of large numbers of workers from Mariel may have temporarily slowed the migration of low-skilled workers from other parts of the US to Miami.
Natural experiments in the pandemic
Natural experiments are a timely topic for another reason. With each state determining its own COVID response, there are many possible comparisons between low and high vaccination rates, lockdowns and business-as-usual, and masking or not. Comparisons between before and after the implementation of various measures wouldn’t be valid, as the pandemic ebbs and flows for other reasons. Also, the emergence of the delta and omicron variants complicates the analysis. But it should be possible to make valid comparisons among various jurisdictions. It’s worth doing, too, because public officials need to know, say, how effective masking is. That way they can make better informed decisions about which measures to take, and when. It might also make for more consistent policies.
It’s very likely that COVID-19 will be with us for a long time, and other viruses will undoubtedly arise. We need to learn as much as we can from these natural experiments so we’ll know better what to do next time. Careful analysis can help us avoid confirmation bias—that is, the tendency to conclude that the results support what we already believe. They can also help distinguish between science and magic.
Natural experiments aren’t new. In the 1940s they showed that smoking cigarettes causes lung cancer and other diseases. They also demonstrated the importance of physical activity to health and longevity.