An open laptop with charts on the screen.

3.3: Experiments

Learning Objectives

Upon completion of this section, you should be able to

  • Identify characteristics of an experiment
  • Identify different experiment methods
  • Identify experiments that control for the placebo effect

Experiments

You want to investigate the effectiveness of vitamin E in preventing disease. You recruit a group of subjects and ask them if they regularly take vitamin E. You notice that the subjects who take vitamin E exhibit better health on average than those who do not. Does this prove that vitamin E is effective in disease prevention? It does not. There are many differences between the two groups compared in addition to vitamin E consumption. People who take vitamin E regularly often take other steps to improve their health: exercise, diet, other vitamin supplements, choosing not to smoke. Any one of these factors could be influencing health. As described, this study does not prove that vitamin E is the key to disease prevention.

So far, we have primarily discussed observational studies – studies in which conclusions would be drawn from observations of a sample or the population. In some cases these observations might be unsolicited, such as studying the percentage of cars that turn right at a red light even when there is a “no turn on red” sign.  In other cases the observations are solicited, like in a survey or a poll.

In contrast, it is common to use experiments when exploring how subjects react to an outside influence. The purpose of an experiment is to investigate the relationship between two or more variables. When one variable causes change in another, we call the first variable the explanatory variable. The affected variable is called the response variable. In a randomized experiment, the researcher manipulates values of the explanatory variable and measures the resulting changes in the response variable. The different values of the explanatory variable are called treatments. An experimental unit is a single object or individual to be measured.

Observational Study and Experiments Terminology

An observational study is a study based on observations or measurements, but nothing new is added to the environment.

An experiment is a study in which the effects of a treatment are measured. In this case something is added to the environment (the treatments).

The explanatory variable in an experiment is what is changed or added to the environment.

The response variable in an experiment is the outcome that is being measured.

Treatments in an experiment are the different levels of the explanatory variable being added in the environment.

Experimental Unit is a single object being measured from the population in the experiment.

Example 1

Researchers want to investigate whether taking aspirin regularly reduces the risk of heart attack in men. Four hundred men between the ages of 50 and 84 are recruited as participants. The men are divided randomly into two groups: one group will take aspirin, and the other group will take a sugar pill. Each man takes one pill each day for three years, but he does not know whether he is taking aspirin or the sugar pill. At the end of the study, researchers count the number of men in each group who have had heart attacks.

Identify the following values for this study: population, sample, experimental units, explanatory variable, response variable, treatments.

Solution

The population is men aged 50 to 84.

The sample is the 400 men who participated.

The experimental units are the individual men in the study.

The explanatory variable is oral medication.

The treatments are aspirin and a sugar pill.

The response variable is whether a subject had a heart attack.

See if you can identify the difference between an observational study and an experiment in the Try It Now below.

Try it Now 1

Is each scenario describing an observational study or an experiment?

Scenario 1: The weights of 30 randomly selected people are measured.

Scenario 2: Subjects are asked to do 20 jumping jacks, and then their heart rates are measured.

Scenario 3: Twenty subjects were given one cup of coffee and twenty subjects were given one cup of tea. Subjects are then given a concentration test.

Hint 1 (click to Show/Hide)

Scenario 1: In this scenario nothing is added to the environment for the people. The weights are just measured.

Scenario 2: Exercise was added to the environment and the heart rates were than recorded.

Scenario 3: The subjects were given either a cup of coffee or tea and then took a concetration test. The coffee and tea was added to the environment.

Answer (click to Show/Hide)

Scenario 1: Observational study.

Scenario 2: Experiment; the treatment is the jumping jacks.

Scenario 3: Experiment; the treatments are coffee and tea.

Confounding

How do we know what is causing differences between two groups with different treatments? When conducting experiments, we need to isolate the treatment being tested and allow only that treatment to vary between subjects. With the vitamin E observation it was noted there were many other factors to consider about the people who were taking vitamin E (for example the life style of those individuals may be different from the groups not taking vitamin E in the observation study).

Suppose a middle school (junior high) finds that their students are not scoring well on the state’s standardized math test.  They decide to run an experiment to see if an alternate curriculum would improve scores.  To run the test, they hire a math specialist to come in and teach a class using the new curriculum.  To their delight, they see an improvement in test scores.

The difficulty with this scenario is that is not clear whether the curriculum is responsible for the improvement, or whether the improvement is due to a math specialist teaching the class.  This is called confounding – when it is not clear which factor or factors caused the observed effect.  Confounding is the downfall of many experiments, though sometimes it is hidden.

Video Confounding (3 mins 35 secs – CC)

Confounding

Confounding occurs when there are two potential variables that could have caused the outcome and it is not possible to determine which actually caused the result.

Example 2

A drug company study about a weight loss pill might report that people lost an average of 8 pounds while using their new drug.  However, in the fine print you find a statement saying that participants were encouraged to also diet and exercise.

What is a potential issue with this experiment?

Solution

It is not clear in this case whether the weight loss is due to the pill, to diet and exercise, or a combination of both.  In this case confounding has occurred.

Example 3

Researchers conduct an experiment to determine whether students will perform better on an arithmetic test if they listen to music during the test. They first give the student a test without music, then give a similar test while the student listens to music.

What is a potential issue with this experiment?

Solution

In this case, the student might perform better on the second test, regardless of the music, simply because it was the second test and they were warmed up. The order of the tests is how the confounding occured in this example.

Control Groups

Video Controlled Experiments (4 mins 52 secs – CC)

There are a number of measures that can be introduced to help reduce the likelihood of confounding. The primary measure is to use a control group.

Control group

When using a control group, the participants are divided into two or more groups, typically a control group and a treatment group. The treatment group receives the treatment being tested; the control group does not receive the treatment.

When dividing the sample into groups you would want them as similar as possible, isolating the treatment(s) as the only potential source of difference between the groups. For this reason, the method of dividing groups is important. Some researchers attempt to ensure that the groups have similar characteristics (same number of females, same number of people over 50, etc.), but it is nearly impossible to control for every characteristic. Because of this, random assignment is very commonly used.

Sometimes not giving the control group anything does not completely control for confounding variables. For example, suppose a medicine study is testing a new headache pill by giving the treatment group the pill and the control group nothing. If the treatment group showed improvement, we would not know whether it was due to the medicine in the pill, or a response to have taken any pill. This is called a placebo effect.

Placebo effect

The placebo effect is when the effectiveness of a treatment is influenced by the patient’s perception of how effective they think the treatment will be, so a result might be seen even if the treatment is ineffectual.

To control for the placebo effect, a placebo, or dummy treatment, is often given to the control group. This way, both groups are truly identical except for the specific treatment given. Look back at example 1. The sugar pill that was given was the placebo for the experiment.

Placebo and Placebo controlled experiments

A placebo is a dummy treatment given to control for the placebo effect.
An experiment that gives the control group a placebo is called a placebo controlled experiment.

Let us take a look at some examples now.

Example 4

To determine if a two day prep course would help high school students improve their scores on the SAT test, a group of students was randomly divided into two subgroups. The first group, the treatment group, was given a two day prep course. The second group, the control group, was not given the prep course. Afterwards, both groups were given the SAT.

Is this a controlled experiment?

Is it a placebo controlled? If not, is that OK?

Solution

In this case it is a controlled experiment as we have at least one group that was not given a treatment while other groups were given some sort of treatment (the prep course).

This is not a placebo controlled group as they were not giving anything. Whether or not it is ok depends on if you want to see if the prep course does better than nothing. What we don’t know is if any treatment would have had the same effect on the outcomes of the treatment group. A better approach may be to give the control group a SAT prep book and nothing else.

Example 5

A company testing a new plant food grows two crops of plants in adjacent fields, the treatment group receiving the new plant food and the control group not.  The crop yield would then be compared.

Is this a controlled experiment?

Is it a placebo controlled? If not, is that OK?

Solution

By growing them at the same time in adjacent fields, they are controlling for weather and other confounding factors.

This is not a placebo controlled experiment. This is ok as they would be getting the sample plant food as before.

Example 6

A study examined the effect of a strong painkiller against a placebo for painful dental tooth extractions. One group of patients received a dose of morphine while the other received a saltwater injection.

Is this a controlled experiment?

Is it a placebo controlled? If not, is that OK?

Solution

Yes, this is a controlled experiment as we have two groups.

It is a placebo controlled experiment. The findings were that the patients receiving a saltwater injection found as much pain relief as patients receiving a dose of morphine.

Example 7

In a study on the effect of alcohol on memory. One group is asked to drink two beers and perform a memory task. The second group is given non-alcoholic beer and the memory task. They are not told what drink they were given.

Is this a controlled experiment?

Is it a placebo controlled? If not, is that OK?

Solution

Yes, this is a controlled experiment as we have two groups.

It is a placebo controlled experiment, because the non-alcoholic beer is acting as a placebo.

Example 8

Determine if a placebo is needed.

  1. An experiment testing a new medication to prevent migraines.
  2. An experiment testing the effectivenss of a new fire retardant on fabric.
  3. An experiment testing the effect of spending time outdoors and academic performance.

Solution

  1. We would want a placebo as it is known if people take any treatment they will tend to feel better. A common approach is to use sugar pills that should have no effect treating any condition. It is also possible that the current treatment is still applied (a particular medicine) if it would be unethical to substitute it with a placebo that has no effect on the condition being measured.
  2. No placebo is needed. Here you are testing if the fire retardant works. A control group would be appropriate to measure how well the fire retardant works.
  3. No placebo is needed, but you would want to make sure the control group has all other information given to the outdoors group (exercise, diet, study tips, etc…).

In some cases, it is more appropriate to compare to a conventional treatment than a placebo. For example, in a cancer research study, it would not be ethical to deny any treatment to the control group or to give a placebo treatment. In this case, the currently acceptable medicine would be given to the second group, called a comparison group in this case. In our SAT test example, the non-treatment group would most likely be encouraged to study on their own, rather than be asked to not study at all, to provide a meaningful comparison.

When using a placebo, it would generally defeat the purpose if the participant knew they were receiving the placebo. If they know they are receiving the placebo there is a chance they would respond differently.

Blind studies

A blind study is one in which the participant does not know whether or not they are receiving the treatment or a placebo.

A double-blind study is one in which those interacting with the participants don’t know who is in the treatment group and who is in the control group.

Example 9

In a study about anti-depression medicine, you would not want the psychological evaluator to know whether the patient is in the treatment or control group either, as it might influence their evaluation.

Should this study be blind? Double blind?

Solution

The experiment should be conducted as a double-blind study as knowing a patient is receiving the treatment may influence the psychological evaluator.

Video Solution (4 mins 20 secs – CC)

Try it Now 2

A study claims that the combination of vitamin C with vitamin E is more effective for preventing the flu than vitamin C alone. In the study, a group of researchers studied 1132 subjects. The subjects were randomly assigned to out of three study groups. The first group of 310 subjects were instructed to take 600 mg daily of vitamin C. The second group of 418 subjects were instructed to take 600 mg of vitamin C and 200 IU of vitamin E daily. The third group of 404 subjects were instructed to take 600 mg of vitamin C and 400 IU of vitamin E daily. The incidence of flu over a season was recorded. All subjects knew which treatment they were receiving.

In this study, which is the control group?

  1. the third group of 404 taking 400 IU of vitamin E
  2. the first group of 310 taking vitamin C alone
  3. the second group of 418 taking 200 IU of vitamin E
  4. the 1132 subjects
  5. There is no control group
Answer (click to Show/Hide)

The control group is the first group of 310 subjects taking vitamin C alone.

This study is: Blind, Double-blind, not an experiment, not blind?

Answer (click to Show/Hide)

The study is not blind to the subjects.

In the next example we will put together what we have learned so far from this chapter.

Example 10

To assess the fitness of American college students, a researcher randomly selects 4 of the night class sections at a local community college, and asks everyone in those classed to do as many sit-ups as possible in 1 minute. On average, they could do 14 sit-ups.

  1. What is the Sample? The intended population?
  2. What is the sampling method?
  3. Is this an observational study or an experiment?
  4. Is it controlled? Placebo-controlled? Blind? Double blind?
  5. Is the data collected categorical or quantitative?
  6. Is the 14 sit-ups average a parameter or a statistic?
  7. Are there any potential sources of bias in this study?

Solution

  1. The sample is students in those four classes. The intended population was American college students (stated population of the study).
  2. Took four classes and asked everyone in those four classes. This is clustered sampling.
  3. This is an observational study as we didn’t apply a technical treatment to the groups, but rather observed how many sit-ups a person could do. One could try to argue this is an experiment and the sit-ups are the treatment applied.
  4. Not an experiment, so no to each of these. If you thought it was an experiment it would still be no to each of these as there is only a single group with no comparisons to a different treatment.
  5. Since we can do calculations to the number of sit-ups a person can do we consider this as quantitative data.
  6. The population is larger than those who participated, so we consider this number based on that sample a statistic.
  7. There is sampling bias in this study due to the selection of courses coming from night sections. There is no reason to assume that students in a night course represent a typical American college student (the intended population).

Video Solution (3 mins 47 secs – CC)

Exercises

  1. Does this describe an observational study or an experiment:  The temperature on randomly selected days throughout the year was measured.
    Answer (click to Show/Hide)

    Observational Study.

  2. Does this describe an observational study or an experiment?  A group of students are told to listen to music while taking a test and their results are compared to a group not listening to music.
    Answer (click to Show/Hide)

    Experiment.

  3. The Smell & Taste Treatment and Research Foundation conducted a study to investigate whether smell can affect learning. Subjects completed mazes multiple times while wearing masks. They completed the pencil and paper mazes three times wearing floral-scented masks, and three times with unscented masks. Participants were assigned at random to wear the floral mask during the first three trials or during the last three trials. For each trial, researchers recorded the time it took to complete the maze and the subject’s impression of the mask’s scent: positive, negative, or neutral.
    1. Describe the explanatory and response variables in this study.
    2. What are the treatments?
    3. Identify any possible confounding that could interfere with this study.
    4. Is it possible to use blinding in this study?
    Answer (click to Show/Hide)
    1. The explanatory variable is scent, and the response variable is the time it takes to complete the maze.
    2. There are two treatments: a floral-scented mask and an unscented mask.
    3. All subjects experienced both treatments. The order of treatments was randomly assigned so there were no differences between the treatment groups. Random assignment eliminates the problem of confounding.
    4. Subjects will clearly know whether they can smell flowers or not, so subjects cannot be blinded in this study. Researchers timing the mazes can be blinded, though. The researcher who is observing a subject will not know which mask is being worn.
  4. A team of researchers is testing the effectiveness of a new HPV vaccine. They randomly divide the subjects into two groups. Group 1 receives new HPV vaccine, and Group 2 receives the existing HPV vaccine. The patients in the study do not know which group they are in.
    1. Which is the treatment group?
    2. Which is the control group (if there is one)?
    3. Is this study blind, double-blind, or neither?
    4. Is this best described as an experiment, a controlled experiment, or a placebo controlled experiment?
    Answer (click to Show/Hide)
    1. Group 1 receiving new HPV vaccine
    2. Group 2 receiving existing HPV vaccine
    3. Blind. Not enough information to say Double-blind as we are not told if the clinicians are aware of which treatment a subject receives.
    4. A controlled experiment
  5. For the clinical trials of a weight loss drug containing Garcinia cambogia the subjects were randomly divided into two groups. The first received an inert pill along with an exercise and diet plan, while the second received the test medicine along with the same exercise and diet plan. The patients do not know which group they are in, nor do the fitness and nutrition advisors.
    1. Which is the treatment group?
    2. Which is the control group (if there is one)?
    3. Is this study blind, double-blind, or neither?
    4. Is this best described as an experiment, a controlled experiment, or a placebo controlled experiment?
    Answer (click to Show/Hide)
    1. Group 2 received test medicine is the treatment group
    2. Group 1 receiving inert pill is control group
    3. Double-Blind. Both the patients and the advisors do not know which treatment was given.
    4. A placebo controlled experiment.
  6. A teacher wishes to know whether the males in his/her class have more conservative attitudes than the females. A questionnaire is distributed to all students in the class assessing attitudes.
    1. Is this a sampling or a census?
    2. Is this an observational study or an experiment?
    3. Are there any possible sources of bias in this study?
    Answer (click to Show/Hide)
    1. A census
    2. Observational study
    3. Perceived lack of anonymity.
  7. A researcher wants to study the effects of birth order on personality. Explain why this study could not be conducted as a randomized experiment. What is the main problem in a study that cannot be designed as a randomized experiment?
    Answer (click to Show/Hide)

    The explanatory variable is birth order. You cannot randomly assign a person’s birth order. Random assignment eliminates the impact of lurking variables. When you cannot assign subjects to treatment groups at random, there will be differences between the groups other than the explanatory variable.

  8. You are concerned about the effects of texting on driving performance. Design a study to test the response time of drivers while texting and while driving only. How many seconds does it take for a driver to respond when a leading car hits the brakes?
    1. Describe the explanatory and response variables in the study.
    2. What are the treatments?
    3. What should you consider when selecting participants?
    4. Your research partner wants to divide participants randomly into two groups: one to drive without distraction and one to text and drive simultaneously. Is this a good idea? Why or why not?
    5. Identify any compounding that could interfere with this study.
    6. How can blinding be used in this study?
    Answer (click to Show/Hide)
    1. Explanatory: presence of distraction from texting; response: response time measured in seconds
    2. Driving without distraction and driving while texting
    3. Answers will vary. Possible responses: Do participants regularly send and receive text messages? How long has the subject been driving? What is the age of the participants? Do participants have similar texting and driving experience?
    4. If the participants were randomly assigned to one group this would be okay. It would be better to assign both treatments to each participant in random order. The second approach helps control for driver ability.
    5. Possible responses include: texting ability, driving experience, type of phone.
    6. The researchers observing the trials and recording response time could be blinded to the treatment being applied.
  9. A farmer believes that playing Barry Manilow songs to his peas will increase their yield. Describe a controlled experiment the farmer could use to test his theory.
    Answer (click to Show/Hide)

    The farmer can divide the land into a grid pattern and randomly assign half of those plots on the grid to play Barry Manilow and the other half to not play the music.

  10. A sports psychologist believes that people are more likely to be extroverted as adults if they played team sports as children. Describe two possible studies to test this theory. Design one as an observational study and the other as an experiment. Which is more practical?
    Answer (click to Show/Hide)

    An observational study could be done by taking the population and dividing it into two groups. Group 1 are those who played team sports as a child and Group 2 are those who did not play team sports as a child. Take a random sample from each group and administer a test to determine if they are extroverted as an adult (through survey questions). For the experiment we would take a random sample of children and divide them into two random groups. One group we place them on team sports and follow them into adulthood to test while the other group would be not allowed to play on team sports (ignoring the ethical problems with this). When looking at both of these it seems clear trying to divide the population of children and then following them into adulthood may be problematic in many ways. The observational study seems to be the more practical approach.

  11. Studies are often done by pharmaceutical companies to determine the effectiveness of a treatment program. Suppose that a new AIDS antibody drug is currently under study. It is given to patients once the AIDS symptoms have revealed themselves. Of interest is the average length of time in months patients live once starting the treatment. Two researchers each follow a different set of 50 AIDS patients from the start of treatment until their deaths.
    1. What is the population of this study?
    2. List two reasons why the data may differ.
    3. Can you tell if one researcher is correct and the other one is incorrect? Why?
    4. Would you expect the data to be identical? Why or why not?
    5. What sampling method is used If the first researcher collected her data by randomly selecting 10 states, then randomly selecting 5 person from each of those states.
    6. If the second researcher collected data by choosing 50 patients in the city they live in. What sampling method would that researcher have used? What concerns would you have about this data set, based upon the data collection method?
    Answer (click to Show/Hide)
    1. Population: those with AIDS whole symptoms have just revealed themselves
    2. The two researchers would very likely see different overall results due to sampling variability.
    3. You cannot tell if either researcher is correct as the true average length of time with the treatment is unknown. At best what both researchers find is an estimate to that unknown parameter.
    4. We would not expect identical data due to sampling variability.
    5. This describes quota sampling as the number of aids patients in each state is not likely to be equal and the selection would not be proportional to the overall aids patients of the country. If you said Stratified random sampling you are close, but would need to assume the there was an equal number of aids patients in each state.
    6. This is an example of convenience sampling. The concern here is if there are other environmental factors that confound the results. In addition the general population may not be as diverse as the entire population for that particular city.
  12. Find a newspaper or magazine article, or the online equivalent, describing the results of a recent study (the results of a poll are not sufficient). Give a summary of the study’s findings, then analyze whether the article provided enough information to determine the validity of the conclusions. If not, produce a list of things that are missing from the article that would help you determine the validity of the study. Look for the things discussed in the text: population, sample, randomness, blind, control, placebos, etc.

Attributions

This page contains modified content from David Lippman, “Math In Society, 2nd Edition.” Licensed under CC BY-SA 4.0.

This page contains modified content from Revision and Adaptation by Lumen Learning. Licensed under CC BY 4.0.

This page contains modified content from “OpenStax Introductory Satistics: 1.4 Experimental Design and Ethics” by Barbara Illowsky, Susan Dean. Licensed under CC BY 4.0.

This page contains content by Robert Foth, Math Faculty, Pima Community College, 2021. CC BY 4.0.

The data from Example 6 is from Levine JD, Gordon NC, Smith R, Fields HL. (1981) Analgesic responses to morphine and placebo in individuals with postoperative pain. Pain. 10:379-89.

License

Icon for the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License

Topics in Mathematics Copyright © by Robert Foth is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where otherwise noted.

Share This Book