A friend shared this article with me. The article discusses how difficult it is to ‘know’ in the social sciences, such as economics. But it also highlights some important aspects of what science is and how it gets done.
In physics, it is fashionable to study highly idealized systems. Luckily, those systems obey relatively simple laws that we can write down so that we can use those laws to solve and inform more complicated problems. The reason this can be done in physics is that the experiments are easy. That is to say, they can be straightforwardly controlled so that the experiment isn’t interacting with its environment in unpredictable ways. There are always ways to reduce friction, air resistance, heat loss, etc. in your system until you have a very good idea of all the relevant interactions and the other interactions are chased away into very small uncertainties in a final measurement.
For most sciences, this level of control is impossible. In medicine, for example, the effectiveness of a treatment can be impossible to distinguish from effects of diet, exercise, sleep, mood, or suggestion. In economics, the effect of a tax break can be impossible to distinguish from normal fluctuations in the market. In education, the effect of a new teaching technique can be impossible to distinguish from the many interactions the student has with his or her teachers, friends, textbooks, the internet, etc. In many sciences, the system under study might be so intertwined with its environment that to separate the two is to fundamentally change the system itself.
The missing ingredient is controlled experimentation, which is what allows science positively to settle certain kinds of debates. How do we know that our physical theories concerning the wing are true? In the end, not because of equations on blackboards or compelling speeches by famous physicists but because airplanes stay up. Social scientists may make claims as fascinating and counterintuitive as the proposition that a heavy piece of machinery can fly, but these claims are frequently untested by experiment, which means that debates like the one in 2009 will never be settled. For decades to come, we will continue to be lectured by what are, in effect, Keynesian and non-Keynesian economists.
The fate of a science where controlled experiments are difficult or impossible is that they don’t mature as quickly. In a few hundred years, physics has matured to the point where physicists struggle to find ANY phenomena that doesn’t fit into a standard model. But in more difficult sciences, such as the social sciences, finding a unified model to explain the wealth of phenomena is a distant dream. This difference between the science can lead to an attitude called physics envy.
(note that many physicists who work on highly nonlinear problems, including problems where there are complicated interactions with an environment must also sometimes work without the safety net of precise, straightforward models.)
Of course, Aristotle, like other proto-scientific thinkers, relied extensively on empirical observation. The essential distinction between such observation and an experiment is control. That is, an experiment is the (always imperfect) attempt to demonstrate a cause-and-effect relationship by holding all potential causes of an outcome constant, consciously changing only the potential cause of interest, and then observing whether the outcome changes. Scientists may try to discern patterns in observational data in order to develop theories. But central to the scientific method is the stricture that such theories should ideally be tested through controlled experiments before they are accepted as reliable. Even in scientific fields in which experiments are infeasible, our knowledge of causal relationships is underwritten by traditional controlled experiments. Astrophysics, for example, relies in part on physical laws verified through terrestrial and near-Earth experiments.
The reason we know Newton’s laws in physics is because it is relatively easy to roll a wheel down an inclined plane, while controlling all of the forces affecting that wheel so that the uncertainties are low. The wheel does not interact strongly with its environment, beyond what scientists can control. But, the reason our knowledge of economics, education, and other social sciences is relatively poor is that there is no way to separate system from environment in a controlled way.
The worst part of all this is that when an experiment is not carefully controlled, scientists get fooled. You may get a certain result for all the wrong reasons. The placebo effect is the classic example of this.
Incidentally, this is exactly why it is important for Canada to handle the long census form carefully. Canada is performing a measurement, and if they don’t control that measurement carefully, everyone will get fooled.