In its most basic form, the scientific method is a process that allows us to understand the physical world.
- Anecdotal observations lead you to formulate a hypothesis.
- The hypothesis may be in the form of a question or a statement. Either way, it must be falsifiable. That is, certain data and/or observations could logically prove it to be false.
- Experiments are designed to test the hypothesis. In keeping with the idea of the falsifiable hypothesis, certain potential outcomes of the experiments would prove the hypothesis false. There is no such thing as proving a hypothesis true.
- As you conduct the experiments, you collect data and observations that either falsify the hypothesis or not.
- Once you have your data, you may do some graphing, number crunching, or other processing to help you understand what you’ve observed.
- Based on the preceding steps, you draw one or more conclusions.
As we’ve noted in previous blogs, the main parts of a technical report mirror the steps of the scientific method. Indeed, writing about the design and execution of an investigation helps us understand what we observed and its implications.
Understanding science
It’s natural for human beings to observe our world and infer connections. However, the results can vary widely. As we’ve discussed previously, traditional wisdom can be quite accurate. For example, indigenous fire management was considerably more effective than modern methods, and traditional architecture can inform sustainable design. But when people equate science with magic, they could fall prey to any number of fraudulent claims.
Once, when the secrets of science were the jealously guarded property of a small priesthood, the common man had no hope of mastering their arcane complexities. Years of study in musty classrooms were prerequisite to obtaining even a dim, incoherent knowledge of science. Today, all that has changed: a dim, incoherent knowledge of science is available to anyone….Indeed, today a myriad of books is available that can explain scientific facts that science itself has never dreamed of.—Tom Weller, Science Made Stupid, 1985
Weller’s clever sendup of science dates from before the internet was available to the public. But now we have far greater access to dubious information. Search engines and social media algorithms can lead us into rabbit holes. Like Alice, we may find it difficult to get out again. How can we know what’s reliable?
Using the scientific method
Even if you’re not conducting experiments, you can use the scientific method to evaluate what you read or hear about. Here are some things I’ve learned about experiment design over the years.
- The plural of “anecdote” is not “data”. If your random observations appear to show a pattern, you need to formulate a hypothesis and test it properly. Don’t just assume you know.
- Beware of confirmation bias. We all have a tendency to interpret observations in ways that reinforce beliefs we already hold. Objectivity is key.
- Make sure the sample is large enough and represents a cross section of the population under study. Medical research used to include only men and then generalize the conclusions to everyone else. Not surprisingly, women are not the same, and that extrapolation didn’t always promote women’s health.
- A good experiment needs one or more controls.
- Tests have both precision and bias. Precision refers to the variability of results when you repeat the same measurement on identical specimens. Bias refers to the tendency of a test to give artificially high or low results. If you know how a particular test is biased, you can interpret the results accordingly. For example, you may take them as upper- or lower bounds rather than actual values.
- Tests may have minimum detection limits. Sometimes the presence of certain chemicals can interfere with the detection of others. Or the preparation techniques may obscure or alter the results.
- Using multiple techniques, preferably with some overlap, can compensate for bias, blind spots, and other limitations of individual test methods.
- Another useful check on the results is comparison with other scientific publications. If other investigators have found comparable results with the same or (even better) different test methods, you can have more confidence in the conclusions.
Statistics and the scientific method
People often think of statistics as a mathematical discipline, but I believe it’s best seen as a science. It helps us distinguish between normal variation and meaningful differences. And it helps us design more rigorous and cost effective experimental programs.
We’ve discussed how to distinguish lies from statistics in a previous blog. Essentially, we need to ask five questions.
- Who says so?
- How does he know?
- What’s missing?
- Did somebody change the subject?
- Does it make sense?
There is something fascinating about science. One gets such wholesale returns of conjecture out of such a trifling investment of fact.—Mark Twain, Life on the Mississippi
Statistical design of experiments
More fundamentally, statistics can be a tremendous help in designing experiments and interpreting the results. Some investigators don’t bring statistics in until they’re analyzing their results. That’s helpful, but it misses the opportunity to improve the design of the whole test program. Of all the investigative work I’ve done in my 40-odd-year career, the two studies I’m most proud of both involved professional statisticians from the outset.
The first looked at the optimal dosage of silica fume in concrete bridge decks. It involved a range of test methods, including cracking tendency and apparent diffusion coefficients. We investigated the effects of w/cm and silica fume content for full-depth and overlay concrete mixtures. To accomplish such an ambitious study within the budget, the statistician used a factorial design. That allowed us to vary both factors simultaneously over a range of values. It was far more cost effective than testing every possible combination of the two variables over the whole range of values.
The second was a precision and bias study of ASTM C39, Standard Test Method for Compressive Strength of Cylindrical Concrete Specimens, using 4 x 8-in. specimens. The statistician designed the program in consultation with members of the relevant ASTM committee. She made sure our procedures conformed with all of ASTM’s requirements for a precision and bias study. She randomized which samples went to which laboratories so that variations among replicate specimens didn’t affect the final results. And she analyzed the results to provide the precision and bias statement ASTM needed to make the changes to ASTM C31 and C39 that we requested. Sadly, we’re still waiting for ACI 318 to accept that two 4 x 8-in. cylinders are as reliable as two 6 x 12-in. cylinders.