If you’re doing research, how do you know if one thing causes another? In this lesson, we’ll look at some common threats to the internal validity of experiments, including testing effects and regression to the mean.
Lisa is a psychologist who is interested in whether teaching people puzzle-solving strategies will improve how quickly they can solve puzzles. For example, she wonders if she teaches people to systematically try different solutions while trying to solve a puzzle, will they be able to solve puzzles faster than if they weren’t taught that strategy?Lisa’s goal is to show that teaching puzzle-solving strategies (her independent variable) will result in faster puzzle-solving times (her dependent variable). But, what if someone becomes faster at solving puzzles for a reason other than her strategy-teaching session? For example, what if the person eats food or drinks caffeine, and that helps them solve the puzzle faster?Internal validity is the extent to which a researcher can say that only the independent variable caused changes in the dependent variable. For Lisa and other scientists, internal validity is the goal: she wants to be able to say that her teaching puzzle-solving strategies is what causes people to solve puzzles more quickly, not that their consumption of caffeine caused it.There are several major threats to internal validity. Let’s look closer at two common ones: testing effects and regression to the mean.
We’ve all heard the old saying that ‘practice makes perfect,’ right? Well, that’s the basic theory behind the testing effects threat to internal validity.Let’s look to Lisa for an example. When the subjects for Lisa’s study come into her lab, she gives them a puzzle to solve and times them on it. This is called a pretest.After they work on the puzzle, Lisa teaches them strategies for how to solve puzzles.Then, Lisa gives them another puzzle similar to the first one and times them on it.
She hopes to see their time become faster; this will indicate that they are better at solving puzzles, since the faster you can solve a puzzle, the better you are at it.But wait: remember that they started by taking a pretest. How does Lisa know that the lessons improved the subjects’ time and not the practice of the pretest? After all, practice makes perfect!This is testing effects: a pretest gives the subjects practice that might help them improve, regardless of whether they learned anything from the lessons or not. Lisa can control for testing effects (that is, reduce or eliminate their effect) by having a control group.For example, say that instead of teaching all the subjects the puzzle-solving strategies, Lisa taught half of them the strategies. For the other half, she just gave them a break to relax.
Then, she looked at how much more quickly both groups did on the second test than on the pretest. Because both the control group (who did not get the strategies) and the experimental group (who did get the strategies) took the pretest, Lisa can see how much effect testing has had on the results.
Imagine that you’re one of Lisa’s subjects.
You go into the experiment and take the pretest. Then you get a break to relax. During that break, you talk to your girlfriend on the phone. She says that she’s met someone new and is leaving you.
How do you think you’ll do on the second puzzle? If you’re like most people, you’ll probably not do very well. But, what if you do the puzzle task every day for a month? Chances are that eventually you’ll get a score close to your normal score.There are many things that can happen to make a person do extremely well or extremely poorly on a given day. A bad day or feeling ill can make someone do poorly on a task, while having gotten a good night’s sleep and feeling sharp on any given day can make someone do really well.But chances are that over time, you’ll end up with a score that is average for you.
This is called regression to the mean. Basically, the mean is the average score, and you are regressing (or moving back) towards it after a particularly good or a particularly bad performance.So, why is this a threat to internal validity? Imagine that some people do really, really well on the pretest. They might have just been having a good morning. But the chances are that the next time they do the puzzle, they will not do as well: they are regressing towards their mean.The opposite is true, too: some people who do really, really poorly on the pretest are likely to end up with a higher score the next time just because they are regressing towards their mean.In this case, it’s hard for Lisa to know whether the puzzle-solving strategies are helping or if the difference in the two puzzle tasks are just regression to the mean.
Of course, the chances of everyone doing really, really well or really, really poorly and then regressing towards the mean are relatively slim. The more subjects Lisa can have, the less likely it is that regression to the mean will be a factor in the final analysis of her study.
Internal validity is the extent to which a researcher can say that only the independent variable is causing changes in the dependent variable. There are several common threats to internal validity. Two major threats are testing effects, when taking a pretest automatically makes a person better at the task, and regression to the mean, when someone does really well or really poorly and then moves closer to their average on the next task.
Once you’ve completed this lesson, you’ll be able to:
- Define internal validity and explain why it is important in research
- Describe how testing effects and regression to the mean can negatively impact internal validity