6.3 Difficulties in Isolating Causes

Mill’s Methods are useful in discovering the causes of phenomena in the world, but their usefulness should not be overstated. Unless they are employed thoughtfully, they can lead an investigator astray. Here’s an example.

Edgy Ed applied the Method of Agreement correctly: coconut macaroons were indeed present every time. But they clearly were not the cause of his disrupted sleep. The lesson is that Mill’s Methods are useful tools for discovering causes, but their results are not always definitive. Uncritical application of the methods can lead one astray. This is especially true of the Method of Concomitant Variation. You may have heard the old saw that “correlation does not imply causation.” It’s useful to keep this corrective thought in mind when using the Method of Concomitant Variation. That two things vary concomitantly is a hint that they may be causally related, but it is not definitive proof that they are. They may be separate effects of a different, unknown cause; they may be completely causally unrelated. It is true, for example, that among children, shoe size and reading ability vary directly: children with bigger feet are better readers than those with smaller feet. Wow! So large feet cause better reading? Of course not. Larger feet and better reading ability are both effects of the same cause: getting older. Older kids wear bigger shoes than younger kids, and they also do better on reading tests. It is also true, for example, that hospital quality and death rate vary directly: that is, the higher quality the hospital (prestige of doctors, training of staff, the sophistication of equipment, etc.), on average, the higher the death rate at that hospital. That’s counterintuitive! Does that mean that high hospital quality causes high death rates? Of course not. Better hospitals have higher mortality rates because the extremely sick, most badly injured patients are taken to those hospitals, instead of to hospitals with less-experienced staff and less-sophisticated equipment. Alas, these people die more often, but not because they’re at a good hospital; it’s exactly the reverse.

Spurious correlations—those that don’t involve any causal connection at all—are easy to find in the age of “big data.” With publicly available databases archiving large amounts of data, and computers with the processing power to search them and look for correlations, it is possible to find many examples of phenomena that vary concomitantly but are obviously not causally connected. A very clever person named Tyler Vigen set about doing this and created a website where he posted his (often very amusing) discoveries. (You can visit this site at http://tylervigen.com/spurious-correlations, CC BY 4.0.) For example, he found that between 2000 and 2009, per capita cheese consumption among Americans, was very closely correlated with the number of deaths caused by people becoming entangled in their bedsheets:

Those two phenomena vary directly, but it’s hard to imagine how they could be causally related. It’s just as difficult to imagine how the following two phenomena could be causally related:

So, Mill’s Methods can’t just be applied willy-nilly; one could end up discovering causal connections where none exist. They can provide clues as to potential causal relationships, but care and critical analysis are required to confirm those results. It’s important to keep in mind that the various methods can work in concert, providing a check on each other. If edgy Ed, for example, had applied the Method of Difference—removing the coconut macaroons but keeping everything else the same—he would have discovered his error. The combination of the Methods of Agreement and Difference—the Joint Method, the controlled study—is an invaluable tool in modern scientific research. A properly conducted controlled study can provide quite convincing evidence of causal connections (or a lack thereof).

Of course, properly conducting a controlled study is not as easy as it sounds. It involves more than just the application of the Joint Method of Agreement and Difference. There are other potentially confounding factors that must be accounted for in order for such a study to yield reliable results. For example, it’s important to take great care in separating subjects into the test and control groups: there can be no systematic difference between the two groups other than the factor that we’re testing; if there is, we cannot say whether the factor we’re testing or the difference between the groups is the cause of any effects observed. Suppose we were conducting a study to determine whether or not vitamin C was effective in treating the common cold. We gather 100 subjects experiencing the onset of cold symptoms. We want one group of 50 to get vitamin C supplements, and one group of 50—the control group—not to receive them. How do we decide who gets placed into which group? We could ask for volunteers. But doing so might create a systematic difference between the two groups. People who hear “vitamin C” and think, “yeah, that’s the group for me” might be people who are more inclined to eat fruits and vegetables, for example, and might therefore be healthier on average than people who are turned off by the idea of receiving vitamin C supplements. This difference between the groups might lead to different results in how their colds progress. Instead of asking for volunteers, we might just assign the first 50 people who show up to the vitamin C group, and the last 50 to the control group. But this could lead to differences, as well. The people who show up earlier might be early-risers, who might be healthier on average than those who straggle in late. (Despite widespread belief that vitamin C is effective in treating the common cold, researchers have found very little evidence to support this claim.)

The best way to avoid systematic differences between test and control groups is to randomly assign subjects to each. We refer to studies conducted this way as randomized controlled studies. And besides randomization, other measures can be taken to improve reliability. The best kinds of controlled studies are double-blind. This means that neither the subjects nor the people conducting the study know which group is the control and which group is receiving the actual treatment. (This information is hidden from the researchers only while the study is ongoing; they are told later, of course, so they can interpret the results.) This measure is necessary because of the psychological tendency for people’s observations to be biased based on their expectations. For example, if the control group in our vitamin C experiment knew they were not getting any treatment for their colds, they might be more inclined to report that they weren’t feeling any better. Conversely, if the members of the group receiving the vitamin supplements knew that they were getting treated, they might be more inclined to report that their symptoms weren’t so bad. This is why the usual practice is to keep subjects in the dark about which group they’re in, giving a placebo to the members of the control group. It’s important to keep the people conducting the study blind for the same reasons. If they knew which group was which, they might be more inclined to observe improvement in the test group and a lack of improvement in the control group. In addition, in their interactions with the subjects, they may unknowingly give away information about which group was which via subconscious signals.

Hence, the gold standard for medical research (and other fields) is the double-blind controlled study. It’s not always possible to create those conditions—sometimes the best researchers can do is to use the Method of Agreement and merely note commonalities amongst a group of patients suffering from the same condition, for example. But the most reliable results come from double-blind controlled studies. Discovering causes is hard in many contexts. Mill’s Methods are a useful starting point, and they accurately model the underlying inference patterns involved in such research, but in practice they must be supplemented with additional measures and analytical rigor in order to yield definitive results. They can give us clues about causes, but they aren’t definitive evidence. Remember, this is inductive, not deductive, reasoning.