Introduction

Causal arguments are the heart and soul of scientific reasoning. Understanding both how to make and evaluate causal claims is essential to understanding the world. This entry will be somewhat lengthy since a lot of concepts need to be covered. You might want to make yourself a coffee…We’ll begin by looking at Mill’s methods for how to discover a causal relation then we will look at how to evaluate causal claims, including the various common errors people can make at each stage.

Part 1: How to Identify Causal Relations Using Mill’s Methods

John Stuart Mill developed four formal methods for determining causation:

• method of agreement,
• method of difference,
• joint method of agreement and difference, and
• method of concomitant variation.

These methods take what most of us already do intuitively and formalize them. The advantage of formalizing a method is that we reduce the possibility of error.

A. Method of Agreement
Suppose you go out for dinner with a group of 10 friends. Some of you order fish, some order tacos, some share a pitcher of sangria, and some order the beef fajitas. A few hours later 6 of you are really sick. You suspect food poisoning. Assuming it is food poisoning, how would you go about figuring out what caused it?

If it is a case of food poisoning, most of us might start by listing what the sick people ate. From our list we’d try to identify the common variable and that would be the most likely cause of the food poisoning.

You have just applied the method of agreement: If two or more events share only one relevant characteristic/variable then that variable is probably the cause of the event.

We can formalize the method this way:

Instance 1: Factors a, b, and c are followed by E.
Instance 2: Factors a, c, and d are followed by E.
Instance 3: Factors b and c are followed by E.
Instance 4: Factors c and d are followed by E.
Therefore, factor c is probably the cause of E.
(Lower case letter = possible causal variables, E = event/causal outcome)

Let’s apply the method to the above case to see how it works. To save ourselves a little time we can make a translation key for each variable:

f=fish, t=tacos, s=sangria, b=beef fajitas. Each instance represents a person who got sick. Suppose John, Liz, Ami, and Evan got sick. We’d formalize our reasoning this way:

John: Ate f, t, and s and got sick.
Evan: Ate f, s, and b and got sick.
Liz: Ate t and s and got sick.
Ami: Ate s and b and got sick.
Therefore, sangria is probably the cause of the food sickness.

In the method of agreement we look at all the cases where the event occurred, list all the possible causal factors, and then see which one shows up in each case where there’s an effect.

Method of Agreement in Science: Cross-Sectional Observational Studies.

Using polls, surveys, or medical records to identify the prevalence of a particular trait, disease, behavior, etc.. At the same time we can collect data on several variables that might be associated with the trait in question. When the property in question and a variable align in many cases, there is an increased likelihood of causation. 

The method of agreement is often employed by public health officials whenever a community has a higher than normal rate of some disease or condition. Officials try to determine what the common variable is. It’s often something in the water supply, food, or in the air. Once you’ve identified the source, the more difficult part can be to figure out the exact chemical in the water or air that’s causing the effect. However, figuring out which of the three is the causal variable can be tricky because people who are sick might have all three in common. To narrow it down to just one we need to use what’s called the joint method of agreement and difference. Before we learn that method we first need to learn…

B. The Method of Difference

Sometimes I have trouble sleeping. I noticed that when I eat, drink beer, and listen to podcasts before bed I can’t fall asleep. How do I figure out which of the three variables is causing my insomnia? (Ignore for the moment that it could be caused by combining all three). I’ll probably do something like this: One night I’ll eat and drink beer before bed but won’t listen to podcasts. If I don’t have insomnia then I can infer that the podcasts are causing my insomnia. If I don’t fall asleep then on another night I’ll drink beer and listen to podcasts before bed but I’ll skip eating. If I don’t have insomnia then I can infer that eating right before bed causes my insomnia.

In the example above we have applied the method of difference: If a relevant factor is present when a phenomenon occurs and absent when the phenomenon does not occur, that factor is probably the cause. To apply the the method of difference, make two lists. One list is all the probable factors present when the phenomenon occurs, the other list is all the probable factors present when the phenomenon doesn’t occur. Whatever is on the first list but not the second is probably the cause.

We can formalize the method this way:

Instance 1: Factors a, b, and c are followed by E.
Instance 2: Factors a and b are not followed by E.
Therefore, factor c is probably the cause of E.

Method of Difference in Science: Case-Control Studies

Identify the trait/behavior/disease you want to study. Select people for two groups: Those with the trait and those without the trait. Hypothesize a variety of causal factors. Look at the frequency of those causal factors in both groups. If the group with the effect (trait) is disproportionately associated with one of the hypothesized causal factors (relative to the non-trait group), we can infer probable causation.

C. Combining Methods to Discover Cause: Joint Method of Agreement and Difference

Suppose I want to know if a particular medication (X) reduces the duration of flu symptoms. First, I employ the method of agreement. I give the treatment (X) to a cohort (group of people) with the flu. Suppose their flu lasts 12 days. Does X reduce the duration of the flu? In order to answer the question I also need to employ the method of difference, or in science-talk, I need a control group. A control group is a cohort that doesn’t get the intervention. It allows me to compare a treatment/intervention group with a non-treatment group so I am able to measure effect size (if there is one). You should think of control group as part of the Relativity criterion of RRAR. We want to know, “reduces symptoms relative to what?”

Suppose my control group’s (i.e., no treatment) flu symptoms also last 12 days. There is no difference in between the control group and the treatment group; therefore, I can infer that the treatment has no effect. Had the control group’s symptoms lasted 14 days, I might infer that treatment X has some positive effect (i.e., it shortens the duration of the flu by 2 days).

Depending on the effect you’re trying to measure, your control group will vary. In most medical studies the control group is the current standard of care–not no treatment. Even if a new treatment shows some positive effects, if the effects aren’t greater than the current standard of care, there’s little reason to adopt the new treatment.

In other causal claims, the control group is what’s called the natural prevalence rate. These rates measure the natural rate of some effect or disease in a population (usually /100 000 or /million over a period of time). Anti-GMO activists will sometimes say that GMOs cause allergies. They justify their argument using the method of agreement. People who eat GMOs (i.e., the cause) have allergies (i.e., the effect). However, to establish a causal claim we’d also need to know the natural prevalence rate of allergies (among other things). That is, we’d need to know the rate at which people suffer from allergies if they don’t eat GMOs. If the prevalence rates are the same between both groups then it doesn’t make sense to say GMOs cause allergies. 

Combining the method of agreement with the method of difference is the backbone of the scientific method. Recall that with the method of agreement we identify the cause if a single factor is always present when the event occurs. Often there is more than just one common variable associated with an effect and so the method of agreement alone won’t help us identify the cause. It merely establishes correlation. For example, suppose a community has a higher than average rate of a particular cancer. If most people share a similar diet, source of food, water supply, and air how are we to determine the cause using method of agreement alone?

The method of agreement tells us that one of these variables is the cause, but if everyone in the community shares them then we’re not going to be able to distinguish causation from mere correlation (more on that in Part 2). We want to know which of those common variables is the cause. To do so we need to also employ the method of difference in conjunction with the method of agreement.

For this example, this means we need to find other community that shares all the variables except one or two and see if the same cancer rate is present in that population. Maybe we have two communities, A and B, that have a similar diet, food source, and air quality but get their water from different rivers. If the second community (B) doesn’t have the higher cancer rate we can conclude (using the method of difference) that the cause of the higher cancer rate is probably in the water supply of the first community (A).

Method of Agreement and Difference in Science: Controlled Trials

Trials that have both a treatment group and a non-treatment group are called controlled trials–among them are clinical trials. Clinical trials rely on the joint method. In a simple clinical trial we have a treatment group and a control group (placebo or current standard of care). To establish that the treatment is the cause of any observed effect, it’s given to each member of the treatment group (duh). This is applying the method of agreement. We know that the treatment group all share a common variable. 

We don’t give the control group the treatment: This is the method of difference. The treatment group has a shared variable so if there’s an effect that isn’t present in the control group, it’s probably the cause. This is the most basic set up. Let’s add a layer of complexity and show how the joint method of agreement and difference handles it.

Let’s begin by acknowledging that there are many additional variables for why a treatment group could improve apart from the treatment (e.g., similar diet, similar lifestyle, similar motivation, similar environment). 

Consider a trial for weight loss. The main problem for weight loss trials is that the people who enter them are often a self-selected group which, as you know from the lesson on polling, can lead to measurement errors. Who enters weight loss studies? People who are already motivated to lose weight. This means that in addition to the magic pill they receive in the trial, they might be cutting calories, eating more healthfully, and exercising more than they did before the trial.

Even if the magic pill has no effect, we’ll probably see weight loss (i.e., an effect) in the treatment group due to these behavioral and dietary changes. However, this weight loss will have been caused by the lifestyle changes they made around the time they entered the trial–not the pill. If we only base our conclusions off of a treatment group without a control group, we might mistakenly attribute the cause of the weight loss to the pill: Everyone took the pill and everyone lost weight therefore the pill did it! Method of agreement worked!

Not so fast…

This is why also applying randomization and the method of difference is so important. We need to control for unknown and additional causes like the lifestyle changes mentioned above. We must randomly assigned each participant to either the treatment group or the control group. This ensures that the other non-pill variables like motivation and dietary changes are approximately proportionally equal in both groups–thereby avoiding a biased sample.

We should also conceal to the participants (and observers) which group has the “real” pill and which has the sugar pill. So long as both groups engage in similar lifestyle changes, the weight loss caused by the changes will be approximately the same in both groups. Any difference between the groups will be attributable to the pill. We’ll observe a greater effect in the treatment group if the pill actually works since this is the only difference between the groups (thanks to randomization). Through the method of difference we isolated the causal factor and controlled for confounding factors. Ta da! Now you can science!

We can formalize the joint method of agreement and difference this way:

Instance 1: Factors a, b, and c are followed by E.
Instance 2: Factors a, b, and d are followed by E.
Instance 3: Factors b and c are not followed by E. (Look for counter-examples)
Instance 4: Factors b and d are not followed by E. (Look for counter examples)
Therefore, factor a is probably the cause of E.

 

D. Correlation and Causation: Method of Concomitant Variation
Suppose for a moment that you are an alien from outer space. After a hard day of making crop circles, you’re thirsty. You notice that the drinks you get in bars (while wearing your human suit) give you a headache in the morning. Using the method of agreement like in the image below, you falsely conclude that soda water causes the headache:

Sometimes when two variables consistently appear together followed by an effect, we use the method of concomitant variation to figure out which one is the causal factor and which one is merely correlated. Basically, the effect size should rise and fall in proportion to variation of the causal variable. Causal variable up=effect size goes up. Causal variable goes down=effect size goes down.

Instance 1: Factors a, b, and c are correlated with E.
Instance 2: Factors a, b and increased c are correlated with increased E.
Instance 3: Factors a, b, and decreased c are correlated with decreased E.
Therefore, factor c is causally connected with E.

In the above example, if we varied the amount of alcohol we’d figure out that it is the causal factor—not the soda water. That is, if we increase or decrease the soda water, the hangover (i.e., effect) doesn’t vary. But if we increase or decrease the amount of alcohol the intensity of the hangover varies with it.

The Method of Concomitant Variation in Science: The Dose Response Relationship

 In medical research and toxicology, concomitant variation is referred to as the dose-response relationship. If you look at a lot of alternative medicine modalities you’ll find that there is no dose response relationship with the proposed treatment. This is an indication that the treatment has no causal efficacy and that any observed effect (if any) is the subjective short-term effect of placebo, measurement errors, or a failure to adhere to double-blinding. (More on subjective measures, self-reporting, blinding, and placebos in the unit on pseudoscience).

Summary of Mill’s Method’s

Our intuitive methods of inferring causation can be formalized into four different methods. Formalizing the method allows us to apply them systematically which in turn makes our evaluations less prone to error. Mill’s methods are the backbone of the scientific method and, as we will see, important for being able to carefully evaluate causal claims.

HOMEWORK

A. (a) Determine whether the reasoning pattern is method of agreement or difference. (b) Suggest at least 1 other variables that could also explain the effect.

1. George never had trouble squatting 200 lbs before. I wonder what’s wrong. Maybe it was all those burritos he ate for lunch.

2. Sometimes I do well on my exams and sometimes I don’t. I’m not really sure what the difference is. Although, I think it might have something to do with coffee. When I have coffee before my exam, I seem to do well.

3. The price of corn has gone up again this week. I wonder if Monsantan raised its seed prices. 

4. Otis seems to bite people he doesn’t know if they reach for him while he’s on the ground. But if I hold him he’ll let them pet him.

5. I’ve always owned Honda’s and they’ve never let me down. Honda must make great cars.

6. Kids these days are violent because they watch video games. Back in my day we didn’t play video games and we were much less violent. 

7. My vitamin supplements keep me energized throughout the day. I didn’t take them yesterday and I felt tired.

B. The Joint Method of Agreement and Difference/Controlled Trial. Explain how you test hypothesis using the joint method of agreement and difference (i.e., a controlled trial). Be sure to carefully define your control group.

Example:

Some people claim that creatine helps you make more gainz. Suggest an experiment to test the claim.

First, I’d make sure all my volunteers had about the same experience level in training. This would control for some people making gainz because they’ve simply never trained before. Next, I’d randomly spit my volunteers into 2 groups. I’d give one group 25 grams of creatine/day and none to the other (i.e., the control group). I’d also assign both groups the same training program and diet to avoid confounding variables. If the creatine group made more gainz than the non-creatine group, I’d conclude that creatine causes gainz.

Questions

1. Some people claim that electro magnetic radiation (EMF) from wireless technology gives them headaches. Suggest how you would conduct an experiment with a control group to test the claim.

2. Some people claim that paying smokers to quit (for each successful month) is the most effective way to get smokers to quit. Suggest how you would conduct an experiment with a control group to test the claim.

3. Some people claim that gluten causes autism. Suggest how you could conduct an experiment with a control group to test the claim.

C. Critical Thinking in the “Real” World (Choose only 1)

1. A significant portion of the US population believes that global warming isn’t caused by greenhouse gas emissions from human activity. To explain global warming trends they instead appeal to a variety of variables including solar and volcanic activity, orbital changes, and several others.

Click on the link which graphs the effects of each variable on temperature:

In addition to greenhouse gases, select 2 other possible causal variables in the link. In light of Mill’s test of concomitant variation, suggest which has the strongest possibility of being a dominant cause of global warming. 

2. In the following article about the discovery of fluoridation and dental health, there are several examples of using the method of agreement and method of disagreement. Read to the end of the section titled “A Fruitful Collaboration”. Give one example of each method being used in the article.